Rate limits & failover

Cheaper tokens are only useful if requests get through. Between your request and the model sit three things: limits you choose for your key, the capacity of the channels behind the model, and how healthy those channels are. The gateway deals with most capacity and health trouble on its own, by waiting briefly and by retrying elsewhere. When it can't, you see a short, retryable 429 or 502.

Automatic failover#

When a channel fails, the request is retried on another channel of the same channel group, up to three attempts in all. A retry happens when a channel:

• times out, can't be reached, or answers 408, 409, 425, 429 or any 5xx;
• refuses the request for a reason on its side, not yours;
• fails mid-stream before the first token (see Streaming).

New conversations are spread across the channels that serve a model. A channel that throws more errors than usual, or runs much slower than its norm, receives fewer new conversations until it recovers. One that keeps failing is taken out of rotation for a while, and conversations pinned to it move to a healthy channel.

Two situations are never retried:

• Your request is invalid. If the model rejects it with 400, 404, 413 or 422, another channel would reject it too. You get the status and the reason.
• Output has started. After the first token, the request is tied to its channel. A later failure arrives as an error event inside the stream.

Waiting for capacity#

Each channel handles a limited number of requests at once. If the channel a request would use is full, the gateway tries the other channels serving the model. If all of them are full, the request queues for whichever channel should free up first.

Total waiting time is capped at 10 seconds. An admitted request reports its wait in x-halfoff-queued-ms. One that is not admitted fails with 429, the message All upstream capacity for this model is busy. Retry shortly., and a retry-after header.

Waiting costs nothing; billing only starts once a request runs, and a request rejected with 429 is never billed.

Limits on your own keys#

You can give each key a Requests per minute cap and a Max concurrent requests cap under Limits & restrictions, when creating or editing it in API keys. Neither is set by default.

A request over a limit waits up to 2 seconds for a slot. If none opens, it fails with 429, the message This API key has reached its rate or concurrency limit., and retry-after in seconds. Giving each workload its own key with its own limits stops one job from starving another. Other key settings are in API keys.

Status codes#

The error type matches your API: rate_limit_error (Anthropic), rate_limit_exceeded (OpenAI) or RESOURCE_EXHAUSTED (Gemini) for 429, and api_error (UNAVAILABLE for Gemini) for 502. All codes are listed in Errors.

Retrying from your code#

By the time you see a 429 or 502, the gateway has already tried several channels. Retry a few more times with exponential backoff and jitter, and respect retry-after when it is sent.

The Anthropic and OpenAI SDKs already retry 408, 409, 429 and 5xx twice by default. For long batch runs, allow more:

import os
from openai import OpenAI

client = OpenAI(base_url="https://halfoff.dev/v1", api_key=os.environ["HALFOFF_API_KEY"], max_retries=6)

Over plain HTTP, apply the same policy yourself:

# --retry handles 429 and 5xx and respects retry-after
curl --retry 6 --retry-max-time 180 https://halfoff.dev/v1/chat/completions \
  -H "Authorization: Bearer $HALFOFF_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "gpt-5.6-luna", "messages": [{"role": "user", "content": "Classify: \"the app crashes on login\""}]}'

With streams, retry only if the error arrives before any content. A stream that breaks halfway has already been billed for its partial output, so decide whether to keep that output or ask again for the whole answer.

Balance and spend limits#

Before it runs, every request checks your balance and the key's spend limit. If the balance is at or below zero, or the key has used up its limit, you get 402 with Insufficient balance. Top up at the dashboard. or This API key has reached its spend limit. The error type is insufficient_quota (OpenAI), billing_error (Anthropic) or RESOURCE_EXHAUSTED (Gemini).

The check runs at the start of a request and the charge settles a few seconds after it finishes. A request that starts with money in the account always runs to the end, so the balance can dip slightly below zero; your next top-up covers the difference. See Billing.