Context & caching

Long-context and agent workloads live or die on three things: knowing the limit, knowing where you stand against it, and getting cache pricing without doing anything special. This page covers all three — plus one promise: we never rewrite or truncate your messages.

Context windows

Every model has a fixed context window covering input tokens plus your max_tokens budget — the provider counts both when deciding whether a request fits, so a prompt that is technically under the window can still be rejected if max_tokens pushes the total over.

The live list is on the models page and in GET /v1/models. Output length is separately capped by max_tokens (16,384 per request by default for an organization).

When a request is too large

Oversized requests fail fast with 400 context_length_exceeded — rejected before any generation starts, so nothing is billed. The message carries the estimated total and the limit:

# 400 context_length_exceeded
{
  "error": {
    "message": "Estimated 332876 tokens (input + max_tokens) exceeds the model context window of 262144 tokens. Reduce message length or max_tokens.",
    "type": "invalid_request_error",
    "code": "context_length_exceeded",
    "param": null
  }
}

• Do not retry as-is. The same payload gets the same rejection on every attempt. Shorten the conversation or lower max_tokens, then retry.
• The gate is an estimate. The rejection threshold is computed from an approximation, not an exact tokenizer pass — treat the numbers as a firm boundary with a little fuzz, and keep headroom instead of engineering to the exact token.

Counting tokens before you send

POST /v1/count_tokens returns an estimate for your messages without touching the model — free, no rate-limit charge, nothing billed:

curl https://api.clfaigateway.dev/v1/count_tokens \
  -H "Authorization: Bearer sk-gw-..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kimi-k2.6",
    "messages": [{"role": "user", "content": "Summarize this repository..."}]
  }'

# 200
{
  "object": "token_count",
  "model": "kimi-k2.6",
  "estimated": true,
  "method": "chars-per-model",
  "estimated_input_tokens": 1210,
  "context_window": 262144
}

The estimate is ceil(total content characters / 4) — the same family of approximation the provider uses at its gate, which is why we mirror it instead of shipping a tokenizer that would disagree with the gatekeeper. It is labeled "estimated": true and the honest error bars, measured against real usage, are:

Context headers on every response

Automatic prefix caching

Prompt caching is automatic. There is no field to set and no header to send — repeated prompt prefixes are served from the provider cache and billed at the cheaper cached_input rate (see live pricing) wherever your requests land.

• Cache granularity is 64-token blocks — very short prompts do not cache; the win starts at a few hundred tokens of stable prefix.
• Cached prefixes stay warm for at least an hour of reuse, which covers chat sessions and agent loops comfortably.
• Keep your prefix byte-stable to earn it: fixed system prompt first, append-only history, and no timestamps, request ids, or random values near the top of the conversation. One changed byte early in the prompt invalidates every block after it.
• Tool definitions are serialized deterministically by the gateway (stable key order), so identical tool sets never break your cache by accident.

We never modify your messages

The gateway forwards messages exactly as you sent them — no truncation, no summarization, no middle-out compression, ever, not even as an opt-in. A request that does not fit fails loudly with the error above instead of being silently trimmed into a different question. Context management strategies (sliding windows, summarizing old turns) belong in your application, where you know what matters; this API guarantees the model sees exactly what you sent.