Skip to content

Streaming

Set stream: true and the gateway relays server-sent events as the model generates. This page covers the parts SDKs do not handle for you: the final usage chunk, reasoning deltas, and errors that happen after HTTP 200 is already committed.

Enabling streaming

curl -N https://api.clfaigateway.dev/v1/chat/completions \
  -H "Authorization: Bearer sk-gw-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k2.6",
    "messages": [{"role": "user", "content": "Explain SSE in one sentence."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Chunks arrive as data: events and the stream ends with data: [DONE]. The final chunk always carries the authoritative token counts in usage — it is sent on every stream whether or not you ask for it. stream_options.include_usage only changes the shape of the chunks in between: with it, they carry "usage": null (exact OpenAI schema); without it, they omit the field.

JSON mode cannot stream: response_format of a JSON type together with stream: true returns 400 json_mode_no_stream.

Reasoning models

Reasoning models (all ten open models) stream their thinking as delta.reasoning_content before the answer starts in delta.content. The gateway forwards it verbatim so you can render it live. Each reasoning delta increments reasoning_tokens, reported inside completion_tokens_details of the final usage chunk.

Thinking is spent from max_tokens — budget for it

The whole completion budget covers reasoning plus the answer. If the budget runs out mid-thought you get finish_reason "length" with little or no content, and the tokens are still billed — the model really did consume them. The appetite differs by model: we measured kimi-k2.6 spending 1,500-3,000 tokens thinking on analytical prompts and up to 8,000 on a logic puzzle, at every reasoning_effort except none — so give it max_tokens of 8,000 or more for that kind of work, or send none when you do not need the thinking. GLM 5.3 is hungrier still on huge contexts — against a ~240K-token prompt we watched it spend 7,000+ tokens thinking before any visible text, so agentic work at that scale wants 8,000+ of headroom; several agent tools default lower and starve the answer. If you see an empty answer, check finish_reason before blaming the model: "length" means the budget died, not the model.

Nothing on screen yet? Prefill and thinking come first

On very large prompts the first token is late by design: prefill runs at roughly 72-77 ms per 1,000 input tokens on the 1M-window models, so a 250K-token prompt spends ~20 seconds before anything streams — and a reasoning model may then think for minutes before delta.content starts. Render delta.reasoning_content as it arrives so users can see progress, and keep client read timeouts at 60 s or more for large prompts. In our logs the most common "hang" is actually a client giving up mid-think: it lands in your usage log as a success flagged client-disconnected, billed only for the tokens delivered.

reasoning_tokens is informational

Reasoning tokens are a subset of completion_tokens and are never billed as a separate dimension — the count exists so you can see why an answer was longer than its visible text. On non-streaming requests the split cannot be observed, so reasoning_tokens is 0 there.

reasoning_effort by model

Each model accepts its own set of reasoning_effort levels, and the same name does not mean the same depth everywhere. We ran every accepted level on 25 Sep 2026 — a short logic puzzle, 3-8 runs per level — and report the median completion tokens (thinking plus answer). Thinking length swings a lot from run to run, so read these as tendencies, not guarantees.

ModelAccepted levelsWhat the level actually changes
deepseek-v4-flashlow, medium, high, xhigh, maxBarely steers it — low even thought longer (~2,600 tokens) than high (~1,100). Thinking cannot be switched off.
deepseek-v4-prolow, medium, high, xhigh, maxReal but uneven: low ~880 → high ~1,300 → xhigh ~4,100 tokens; max landed in between (~1,600). Thinking cannot be switched off.
deepseek-v4.1-flashnone, low, medium, high, xhigh, maxnone really switches thinking off (no reasoning_content) — on hard problems the model then works step by step in the visible answer (~930 tokens), so it is not cheaper there. low ~730 and high ~840 both solved the puzzle every time; max thinks longest (~1,040). medium and xhigh are upstream aliases of high and max. Measured 29 Oct 2026, 6 runs per level.
glm-4.7-flashlow, medium, highNo consistent difference — it thinks long (4,000-7,500 tokens) at every level.
glm-5.2low, medium, high, xhigh, maxTwo tiers: low, medium and high run alike (~1,800-2,900 tokens); xhigh and max think longer (~3,100-4,500). Thinking cannot be switched off.
glm-5.3low, medium, high, maxTwo tiers: low and high think briefly (~460 tokens); medium and max think fully (~1,300). Thinking cannot be switched off.
glm-5.3-flashlow, medium, high, xhigh, maxThree tiers: low barely thinks (~95 tokens — fastest, but it missed our puzzle 2 times out of 3); high thinks briefly (~415); medium, xhigh and max think fully (~630-1,200).
kimi-k2.6none, low, medium, highnone switches thinking off (~880 tokens, less accurate on multi-step problems); low, medium and high all think at full depth (~6,600-7,100).
kimi-k2.7-codelow, medium, highAlways thinks at full depth — the level makes no measurable difference.
qwen3.8-27blow, medium, xhighA real, gentle ramp: low ~650 → medium ~740 → xhigh ~1,000 tokens.

When you do not send reasoning_effort, the gateway sends low — except on deepseek-v4-flash and glm-5.3-flash, where it sends high: on DeepSeek V4 Flash low actually thinks longer, so it costs more, and on GLM 5.3 Flash low barely thinks. Send max to GLM 5.3 Flash for the hardest multi-step work. A level the model does not accept returns 400 reasoning_effort_not_supported listing the valid ones. That includes none on the models that cannot stop thinking: upstream would quietly run it as a full-depth level and bill the thinking, so the gateway refuses it instead.

Tools written for other providers sometimes send the level in their own shape. reasoning: { "effort": "high" } is read exactly like reasoning_effort when reasoning_effort itself is absent (it wins when both are sent). thinking and enable_thinking are accepted and ignored — they never switch thinking on or off here; use reasoning_effort for that.

Errors after the stream has started

Once the first byte is sent, the HTTP status is already 200 and cannot change. If the upstream fails mid-generation, the gateway emits one final event with finish_reason: "error" and an error object, then [DONE]:

the last event before [DONE]
data: {"id":"req_01j9zxg0aabbccddeeff00112233","object":"chat.completion.chunk","created":1774694600,"model":"kimi-k2.6","choices":[{"index":0,"delta":{},"finish_reason":"error"}],"error":{"message":"Upstream provider error while streaming. You were charged only for tokens already delivered.","type":"api_error","code":"upstream_error","param":null}}

data: [DONE]

SDKs will not raise this for you

An SDK that only checks the HTTP status sees a successful stream that simply ended. Check finish_reason on every chunk — the Python and JavaScript samples above do — and treat "error" as a failed request. You are charged only for the tokens delivered before the failure.

If you disconnect first

Disconnecting does not cancel what was already generated: you are charged for the tokens delivered up to the disconnect, counted exactly from per-chunk usage — never estimated. Cap your worst case with max_tokens. The full money-side policy, including zero-completion insurance, is on Billing.

How a stream settles: the status field

Every request appears in GET /v1/generation?id={x-request-id} after it settles, with a status:

statusMeaningWhat you pay
successThe stream finished — including the case where you disconnected early.All delivered tokens.
partialThe upstream failed mid-stream; you received finish_reason: "error".Only the tokens delivered before the failure.
errorThe request failed before any output.$0 when zero-completion insurance applies — see Billing.

Stream costs are read after the fact

Streams do not carry the x-gw-cost-nano response header — headers are sent before the cost is known. Read the settled cost from GET /v1/generation using the x-request-id header (also the id field of every chunk).

One more stream-specific rule: Idempotency-Key is not supported on streaming requests — Idempotency explains why.

If your stream arrives all at once

The gateway flushes every chunk immediately and never compresses text/event-stream. If chunks still arrive in one burst at the end, something between us and your code is buffering — almost always a corporate proxy, or your own reverse proxy (nginx buffers responses by default: set proxy_buffering off; for SSE routes, or honor the X-Accel-Buffering: no convention). Serverless platforms that buffer function responses have the same effect.

A quick differential: run the same request with curl -N from the affected network. If curl streams smoothly, the buffering lives in your stack, not on the wire.

Behind a corporate proxy, configure the SDK explicitly (httpx ≥ 0.28 renamed the option to singular proxy): Python OpenAI(http_client=httpx.Client(proxy="http://proxy:8080")), Node via fetchOptions. If you get APIConnectionError, the request never reached us — check proxy/DNS/TLS on your side; APIStatusError means it did.

Client timeouts and reasoning models

The SDK defaults (10-minute overall timeout, Python applies it per-read while streaming) are safe for reasoning models. The common mistake is lowering it globally — timeout=30 — because reasoning models can think for 30–70 s before the first visible token on long prompts (we stream reasoning_content early precisely so the connection is never silent for long). A too-low timeout raises APITimeoutError mid-generation, and the SDK then retries the whole request — on a non-streaming call without an Idempotency-Key, that can bill twice.

granular timeout
# Python — tight connect, generous read (instead of one small global timeout)
import httpx
client = OpenAI(
    base_url="https://api.clfaigateway.dev/v1",
    api_key=os.environ["CLF_API_KEY"],
    timeout=httpx.Timeout(600.0, connect=5.0),
)