Streaming
Set stream: true and the gateway relays server-sent events as the model generates. This page covers the parts SDKs do not handle for you: the final usage chunk, reasoning deltas, and errors that happen after HTTP 200 is already committed.
Enabling streaming
curl -N https://api.clfaigateway.dev/v1/chat/completions \
-H "Authorization: Bearer sk-gw-..." \
-H "Content-Type: application/json" \
-d '{
"model": "kimi-k2.6",
"messages": [{"role": "user", "content": "Explain SSE in one sentence."}],
"stream": true,
"stream_options": {"include_usage": true}
}'Chunks arrive as data: events and the stream ends with data: [DONE]. The final chunk always carries the authoritative token counts in usage — it is sent on every stream whether or not you ask for it. stream_options.include_usage only changes the shape of the chunks in between: with it, they carry "usage": null (exact OpenAI schema); without it, they omit the field.
JSON mode cannot stream: response_format of a JSON type together with stream: true returns 400 json_mode_no_stream.
Reasoning models
Reasoning models (all ten open models) stream their thinking as delta.reasoning_content before the answer starts in delta.content. The gateway forwards it verbatim so you can render it live. Each reasoning delta increments reasoning_tokens, reported inside completion_tokens_details of the final usage chunk.
Thinking is spent from max_tokens — budget for it
The whole completion budget covers reasoning plus the answer. If the budget runs out mid-thought you get finish_reason "length" with little or no content, and the tokens are still billed — the model really did consume them. The appetite differs by model: we measured kimi-k2.6 spending 1,500-3,000 tokens thinking on analytical prompts and up to 8,000 on a logic puzzle, at every reasoning_effort except none — so give it max_tokens of 8,000 or more for that kind of work, or send none when you do not need the thinking. GLM 5.3 is hungrier still on huge contexts — against a ~240K-token prompt we watched it spend 7,000+ tokens thinking before any visible text, so agentic work at that scale wants 8,000+ of headroom; several agent tools default lower and starve the answer. If you see an empty answer, check finish_reason before blaming the model: "length" means the budget died, not the model.
Nothing on screen yet? Prefill and thinking come first
On very large prompts the first token is late by design: prefill runs at roughly 72-77 ms per 1,000 input tokens on the 1M-window models, so a 250K-token prompt spends ~20 seconds before anything streams — and a reasoning model may then think for minutes before delta.content starts. Render delta.reasoning_content as it arrives so users can see progress, and keep client read timeouts at 60 s or more for large prompts. In our logs the most common "hang" is actually a client giving up mid-think: it lands in your usage log as a success flagged client-disconnected, billed only for the tokens delivered.
reasoning_tokens is informational
Reasoning tokens are a subset of completion_tokens and are never billed as a separate dimension — the count exists so you can see why an answer was longer than its visible text. On non-streaming requests the split cannot be observed, so reasoning_tokens is 0 there.
reasoning_effort by model
Each model accepts its own set of reasoning_effort levels, and the same name does not mean the same depth everywhere. We ran every accepted level on 25 Sep 2026 — a short logic puzzle, 3-8 runs per level — and report the median completion tokens (thinking plus answer). Thinking length swings a lot from run to run, so read these as tendencies, not guarantees.
| Model | Accepted levels | What the level actually changes |
|---|---|---|
deepseek-v4-flash | low, medium, high, xhigh, max | Barely steers it — low even thought longer (~2,600 tokens) than high (~1,100). Thinking cannot be switched off. |
deepseek-v4-pro | low, medium, high, xhigh, max | Real but uneven: low ~880 → high ~1,300 → xhigh ~4,100 tokens; max landed in between (~1,600). Thinking cannot be switched off. |
deepseek-v4.1-flash | none, low, medium, high, xhigh, max | none really switches thinking off (no reasoning_content) — on hard problems the model then works step by step in the visible answer (~930 tokens), so it is not cheaper there. low ~730 and high ~840 both solved the puzzle every time; max thinks longest (~1,040). medium and xhigh are upstream aliases of high and max. Measured 29 Oct 2026, 6 runs per level. |
glm-4.7-flash | low, medium, high | No consistent difference — it thinks long (4,000-7,500 tokens) at every level. |
glm-5.2 | low, medium, high, xhigh, max | Two tiers: low, medium and high run alike (~1,800-2,900 tokens); xhigh and max think longer (~3,100-4,500). Thinking cannot be switched off. |
glm-5.3 | low, medium, high, max | Two tiers: low and high think briefly (~460 tokens); medium and max think fully (~1,300). Thinking cannot be switched off. |
glm-5.3-flash | low, medium, high, xhigh, max | Three tiers: low barely thinks (~95 tokens — fastest, but it missed our puzzle 2 times out of 3); high thinks briefly (~415); medium, xhigh and max think fully (~630-1,200). |
kimi-k2.6 | none, low, medium, high | none switches thinking off (~880 tokens, less accurate on multi-step problems); low, medium and high all think at full depth (~6,600-7,100). |
kimi-k2.7-code | low, medium, high | Always thinks at full depth — the level makes no measurable difference. |
qwen3.8-27b | low, medium, xhigh | A real, gentle ramp: low ~650 → medium ~740 → xhigh ~1,000 tokens. |
When you do not send reasoning_effort, the gateway sends low — except on deepseek-v4-flash and glm-5.3-flash, where it sends high: on DeepSeek V4 Flash low actually thinks longer, so it costs more, and on GLM 5.3 Flash low barely thinks. Send max to GLM 5.3 Flash for the hardest multi-step work. A level the model does not accept returns 400 reasoning_effort_not_supported listing the valid ones. That includes none on the models that cannot stop thinking: upstream would quietly run it as a full-depth level and bill the thinking, so the gateway refuses it instead.
Tools written for other providers sometimes send the level in their own shape. reasoning: { "effort": "high" } is read exactly like reasoning_effort when reasoning_effort itself is absent (it wins when both are sent). thinking and enable_thinking are accepted and ignored — they never switch thinking on or off here; use reasoning_effort for that.
Errors after the stream has started
Once the first byte is sent, the HTTP status is already 200 and cannot change. If the upstream fails mid-generation, the gateway emits one final event with finish_reason: "error" and an error object, then [DONE]:
data: {"id":"req_01j9zxg0aabbccddeeff00112233","object":"chat.completion.chunk","created":1774694600,"model":"kimi-k2.6","choices":[{"index":0,"delta":{},"finish_reason":"error"}],"error":{"message":"Upstream provider error while streaming. You were charged only for tokens already delivered.","type":"api_error","code":"upstream_error","param":null}}
data: [DONE]SDKs will not raise this for you
An SDK that only checks the HTTP status sees a successful stream that simply ended. Check finish_reason on every chunk — the Python and JavaScript samples above do — and treat "error" as a failed request. You are charged only for the tokens delivered before the failure.
If you disconnect first
Disconnecting does not cancel what was already generated: you are charged for the tokens delivered up to the disconnect, counted exactly from per-chunk usage — never estimated. Cap your worst case with max_tokens. The full money-side policy, including zero-completion insurance, is on Billing.
How a stream settles: the status field
Every request appears in GET /v1/generation?id={x-request-id} after it settles, with a status:
| status | Meaning | What you pay |
|---|---|---|
success | The stream finished — including the case where you disconnected early. | All delivered tokens. |
partial | The upstream failed mid-stream; you received finish_reason: "error". | Only the tokens delivered before the failure. |
error | The request failed before any output. | $0 when zero-completion insurance applies — see Billing. |
Stream costs are read after the fact
Streams do not carry the x-gw-cost-nano response header — headers are sent before the cost is known. Read the settled cost from GET /v1/generation using the x-request-id header (also the id field of every chunk).
One more stream-specific rule: Idempotency-Key is not supported on streaming requests — Idempotency explains why.
If your stream arrives all at once
The gateway flushes every chunk immediately and never compresses text/event-stream. If chunks still arrive in one burst at the end, something between us and your code is buffering — almost always a corporate proxy, or your own reverse proxy (nginx buffers responses by default: set proxy_buffering off; for SSE routes, or honor the X-Accel-Buffering: no convention). Serverless platforms that buffer function responses have the same effect.
A quick differential: run the same request with curl -N from the affected network. If curl streams smoothly, the buffering lives in your stack, not on the wire.
Behind a corporate proxy, configure the SDK explicitly (httpx ≥ 0.28 renamed the option to singular proxy): Python OpenAI(http_client=httpx.Client(proxy="http://proxy:8080")), Node via fetchOptions. If you get APIConnectionError, the request never reached us — check proxy/DNS/TLS on your side; APIStatusError means it did.
Client timeouts and reasoning models
The SDK defaults (10-minute overall timeout, Python applies it per-read while streaming) are safe for reasoning models. The common mistake is lowering it globally — timeout=30 — because reasoning models can think for 30–70 s before the first visible token on long prompts (we stream reasoning_content early precisely so the connection is never silent for long). A too-low timeout raises APITimeoutError mid-generation, and the SDK then retries the whole request — on a non-streaming call without an Idempotency-Key, that can bill twice.
# Python — tight connect, generous read (instead of one small global timeout)
import httpx
client = OpenAI(
base_url="https://api.clfaigateway.dev/v1",
api_key=os.environ["CLF_API_KEY"],
timeout=httpx.Timeout(600.0, connect=5.0),
)