Streaming

Set stream: true and the gateway relays server-sent events as the model generates. This page covers the parts SDKs do not handle for you: the final usage chunk, reasoning deltas, and errors that happen after HTTP 200 is already committed.

Enabling streaming

curl -N https://api.clfaigateway.dev/v1/chat/completions \
  -H "Authorization: Bearer sk-gw-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k2.6",
    "messages": [{"role": "user", "content": "Explain SSE in one sentence."}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

Chunks arrive as data: events and the stream ends with data: [DONE]. The final chunk always carries the authoritative token counts in usage — it is sent on every stream whether or not you ask for it. stream_options.include_usage only changes the shape of the chunks in between: with it, they carry "usage": null (exact OpenAI schema); without it, they omit the field.

JSON mode cannot stream: response_format of a JSON type together with stream: true returns 400 json_mode_no_stream.

Reasoning models

Reasoning models (all six open models) stream their thinking as delta.reasoning_content before the answer starts in delta.content. The gateway forwards it verbatim so you can render it live. Each reasoning delta increments reasoning_tokens, reported inside completion_tokens_details of the final usage chunk.

reasoning_tokens is informational

Reasoning tokens are a subset of completion_tokens and are never billed as a separate dimension — the count exists so you can see why an answer was longer than its visible text. On non-streaming requests the split cannot be observed, so reasoning_tokens is 0 there.

Errors after the stream has started

Once the first byte is sent, the HTTP status is already 200 and cannot change. If the upstream fails mid-generation, the gateway emits one final event with finish_reason: "error" and an error object, then [DONE]:

the last event before [DONE]

data: {"id":"req_01j9zxg0aabbccddeeff00112233","object":"chat.completion.chunk","created":1774694600,"model":"kimi-k2.6","choices":[{"index":0,"delta":{},"finish_reason":"error"}],"error":{"message":"Upstream provider error while streaming. You were charged only for tokens already delivered.","type":"api_error","code":"upstream_error","param":null}}

data: [DONE]

SDKs will not raise this for you

An SDK that only checks the HTTP status sees a successful stream that simply ended. Check finish_reason on every chunk — the Python and JavaScript samples above do — and treat "error" as a failed request. You are charged only for the tokens delivered before the failure.

If you disconnect first

Disconnecting does not cancel what was already generated: you are charged for the tokens delivered up to the disconnect, counted exactly from per-chunk usage — never estimated. Cap your worst case with max_tokens. The full money-side policy, including zero-completion insurance, is on Billing.

How a stream settles: the status field

Every request appears in GET /v1/generation?id={x-request-id} after it settles, with a status:

status	Meaning	What you pay
`success`	The stream finished — including the case where you disconnected early.	All delivered tokens.
`partial`	The upstream failed mid-stream; you received `finish_reason: "error"`.	Only the tokens delivered before the failure.
`error`	The request failed before any output.	$0 when zero-completion insurance applies — see Billing.

Stream costs are read after the fact

Streams do not carry the x-gw-cost-nano response header — headers are sent before the cost is known. Read the settled cost from GET /v1/generation using the x-request-id header (also the id field of every chunk).

One more stream-specific rule: Idempotency-Key is not supported on streaming requests — Idempotency explains why.

If your stream arrives all at once

The gateway flushes every chunk immediately and never compresses text/event-stream. If chunks still arrive in one burst at the end, something between us and your code is buffering — almost always a corporate proxy, or your own reverse proxy (nginx buffers responses by default: set proxy_buffering off; for SSE routes, or honor the X-Accel-Buffering: no convention). Serverless platforms that buffer function responses have the same effect.

A quick differential: run the same request with curl -N from the affected network. If curl streams smoothly, the buffering lives in your stack, not on the wire.

Behind a corporate proxy, configure the SDK explicitly (httpx ≥ 0.28 renamed the option to singular proxy): Python OpenAI(http_client=httpx.Client(proxy="http://proxy:8080")), Node via fetchOptions. If you get APIConnectionError, the request never reached us — check proxy/DNS/TLS on your side; APIStatusError means it did.

Client timeouts and reasoning models

The SDK defaults (10-minute overall timeout, Python applies it per-read while streaming) are safe for reasoning models. The common mistake is lowering it globally — timeout=30 — because reasoning models can think for 30–70 s before the first visible token on long prompts (we stream reasoning_content early precisely so the connection is never silent for long). A too-low timeout raises APITimeoutError mid-generation, and the SDK then retries the whole request — on a non-streaming call without an Idempotency-Key, that can bill twice.

granular timeout

# Python — tight connect, generous read (instead of one small global timeout)
import httpx
client = OpenAI(
    base_url="https://api.clfaigateway.dev/v1",
    api_key=os.environ["CLF_API_KEY"],
    timeout=httpx.Timeout(600.0, connect=5.0),
)

PreviousAuthentication NextBilling