Streaming

Streaming works the way each vendor designed it: server-sent events in the vendor's own format, passed to you byte for byte. The gateway reads along only to count usage and to shield you from failing channels. Your existing stream-parsing code needs no changes.

Enable streaming#

SDK code for each is in the API guides: Anthropic, OpenAI, Gemini.

The first token is held, failures before it are retried#

Sometimes a channel accepts a request and then gives up before it has produced anything, for example by sending message_start and then an overload error. To hide this, the gateway holds the beginning of every stream until the first piece of real content arrives (text, thinking or a tool call). A channel that fails before that point has sent you nothing, so the gateway quietly re-sends the request to another channel and you receive one clean stream from the start.

Slow starters are treated the same way. For Claude models without extended thinking, a channel that has produced no token after 10 seconds (plus 1 second per 10K prompt tokens) is abandoned for another, unless it is the last one left. With thinking enabled, the model may think as long as it needs. Non-streaming calls follow the same rule.

For your client this means:

• The status line and headers arrive with the first token: time to first byte is time to first token.
• If every channel fails before the first token, you receive a normal HTTP error in your API's format, not a broken stream.
• The hold lasts at most 15 seconds. A model that thinks silently longer than that starts streaming then, and the request is committed to its channel from that moment.
• Give the response at least 30 seconds to start. A request may also queue a few seconds for capacity first; see Rate limits & failover.

Sample streams#

event: message_start
data: {"type":"message_start","message":{"id":"msg_01Qe4n","type":"message","role":"assistant","model":"claude-sonnet-5-5","content":[],"stop_reason":null,"usage":{"input_tokens":12,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Red, green, blue."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":7}}

event: message_stop
data: {"type":"message_stop"}

When a running stream breaks#

After the first token, the request belongs to its channel and can no longer be moved. Two things can still cut it short.

Silence. If the channel sends nothing for 3 minutes, the gateway closes the stream with an error event in your API's format:

event: error
data: {"type":"error","error":{"type":"api_error","message":"Upstream stopped responding mid-stream"}}

An error raised by the channel, such as an overloaded model. You receive a neutral error event in your format rather than the channel's own wording: overloaded_error (OpenAI rate_limit_exceeded, Gemini RESOURCE_EXHAUSTED) if the channel is busy, otherwise api_error (Gemini UNAVAILABLE) with the message The model provider returned an error. Please retry.

Either way the stream ends without its usual closing event (message_stop, data: [DONE] or a finishReason). Treat any stream that lacks it as incomplete: keep or drop the partial text, and repeat the whole request if you need a full answer. The Anthropic and OpenAI SDKs raise an exception when they see an error event.

Stop a stream early#

Close the connection to stop generating: break out of the loop, or call stream.close() / controller.abort(). The gateway stops the model and bills, at half the list price, only what was produced so far:

• Input: the count the stream already reported (Anthropic sends it at the start), or about one token per four characters of your request if nothing was reported.
• Output: the larger of the count reported so far and about one token per four characters of text already sent to you. Streams that end early rarely report a final output count, so the estimate is what usually applies.

A stream closed after a stall is billed the same way. Every cancelled or stalled stream appears in Logs with its charge. See also Models & pricing.