---
title: Rate limits & failover
description: Per-key request and concurrency limits, short capacity queues and automatic failover across channels. Status codes to expect and how to retry safely.
---

Cheaper tokens are only useful if requests get through. Between your request and the model sit three things: limits you choose for your key, the capacity of the channels behind the model, and how healthy those channels are. The gateway deals with most capacity and health trouble on its own, by waiting briefly and by retrying elsewhere. When it can't, you see a short, retryable `429` or `502`.

## Automatic failover {#failover}

When a channel fails, the request is retried on another channel of the same [channel group](/docs/models/channel-groups#failover), up to three attempts in all. A retry happens when a channel:

- times out, can't be reached, or answers `408`, `409`, `425`, `429` or any `5xx`;
- refuses the request for a reason on its side, not yours;
- fails mid-stream before the first token (see [Streaming](/docs/models/streaming#first-token-hold)).

New conversations are spread across the channels that serve a model. A channel that throws more errors than usual, or runs much slower than its norm, receives fewer new conversations until it recovers. One that keeps failing is taken out of rotation for a while, and conversations pinned to it move to a healthy channel.

Two situations are never retried:

- **Your request is invalid.** If the model rejects it with `400`, `404`, `413` or `422`, another channel would reject it too. You get the status and the reason.
- **Output has started.** After the first token, the request is tied to its channel. A later failure arrives as an error event inside the stream.

## Waiting for capacity {#queueing}

Each channel handles a limited number of requests at once. If the channel a request would use is full, the gateway tries the other channels serving the model. If all of them are full, the request queues for whichever channel should free up first.

Total waiting time is capped at **10 seconds**. An admitted request reports its wait in `x-halfoff-queued-ms`. One that is not admitted fails with `429`, the message `All upstream capacity for this model is busy. Retry shortly.`, and a `retry-after` header.

Waiting costs nothing; billing only starts once a request runs, and a request rejected with `429` is never billed.

## Limits on your own keys {#key-limits}

You can give each key a **Requests per minute** cap and a **Max concurrent requests** cap under **Limits & restrictions**, when creating or editing it in [API keys](/app/keys). Neither is set by default.

| Limit | Counts |
| --- | --- |
| Requests per minute | Requests started in the past 60 seconds, on a rolling window. |
| Max concurrent requests | Requests in flight, open streams included until they finish. |

A request over a limit waits up to 2 seconds for a slot. If none opens, it fails with `429`, the message `This API key has reached its rate or concurrency limit.`, and `retry-after` in seconds. Giving each workload its own key with its own limits stops one job from starving another. Other key settings are in [API keys](/docs/account/api-keys).

## Status codes {#status-codes}

| Status | Cause | What to do |
| --- | --- | --- |
| `429` | The key's RPM or concurrency cap. Sends `retry-after`. | Wait `retry-after` seconds, or raise the cap. |
| `429` | All capacity for the model stayed busy for 10 seconds. Sends `retry-after`. | Wait `retry-after` seconds, then retry. |
| `429` | Every channel tried returned a rate limit. Message `Upstream capacity for this model is busy. Retry shortly.` | Retry with backoff. |
| `502` | Every channel tried failed. Message `The model provider returned an error. Please retry.` | Retry with backoff. |
| `500` | Unexpected gateway error. | Retry once or twice; if it persists, contact support with the `x-halfoff-request-id`. |
| `402` | Balance at or below zero, or the key's spend limit reached. | Top up or raise the limit. Don't retry. |
| `400`, `404`, `413`, `422` | Invalid or oversized request, or unknown model id. | Fix the request. Don't retry. |
| `401`, `403` | Key missing, revoked or expired, or not allowed this model. | Fix the key. Don't retry. |

The error type matches your API: `rate_limit_error` (Anthropic), `rate_limit_exceeded` (OpenAI) or `RESOURCE_EXHAUSTED` (Gemini) for `429`, and `api_error` (`UNAVAILABLE` for Gemini) for `502`. All codes are listed in [Errors](/docs/api/errors).

## Retrying from your code {#client-retries}

By the time you see a `429` or `502`, the gateway has already tried several channels. Retry a few more times with exponential backoff and jitter, and respect `retry-after` when it is sent.

The Anthropic and OpenAI SDKs already retry `408`, `409`, `429` and `5xx` twice by default. For long batch runs, allow more:

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://halfoff.dev/v1", api_key=os.environ["HALFOFF_API_KEY"], max_retries=6)
```

Over plain HTTP, apply the same policy yourself:

```bash title="cURL"
# --retry handles 429 and 5xx and respects retry-after
curl --retry 6 --retry-max-time 180 https://halfoff.dev/v1/chat/completions \
  -H "Authorization: Bearer $HALFOFF_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model": "gpt-5.6-luna", "messages": [{"role": "user", "content": "Classify: \"the app crashes on login\""}]}'
```

```python title="Python"
import os
import random
import time

import requests

RETRY_ON = {429, 500, 502, 503, 504}

def post_chat(payload, max_attempts=6):
    for attempt in range(max_attempts):
        r = requests.post(
            "https://halfoff.dev/v1/chat/completions",
            headers={"Authorization": f"Bearer {os.environ['HALFOFF_API_KEY']}"},
            json=payload,
            timeout=600,
        )
        if r.status_code not in RETRY_ON or attempt == max_attempts - 1:
            r.raise_for_status()
            return r.json()
        wait = r.headers.get("retry-after")
        delay = float(wait) if wait else min(2 ** attempt, 30)
        time.sleep(delay + random.uniform(0, 1))
```

```js title="Node.js"
const RETRY_ON = new Set([429, 500, 502, 503, 504]);

async function postChat(payload, maxAttempts = 6) {
  for (let attempt = 0; ; attempt++) {
    const res = await fetch('https://halfoff.dev/v1/chat/completions', {
      method: 'POST',
      headers: {
        authorization: `Bearer ${process.env.HALFOFF_API_KEY}`,
        'content-type': 'application/json',
      },
      body: JSON.stringify(payload),
    });
    if (res.ok) return res.json();
    if (!RETRY_ON.has(res.status) || attempt === maxAttempts - 1) {
      throw new Error(`${res.status}: ${await res.text()}`);
    }
    const delay = Number(res.headers.get('retry-after')) || Math.min(2 ** attempt, 30);
    await new Promise((r) => setTimeout(r, (delay + Math.random()) * 1000));
  }
}
```

With streams, retry only if the error arrives before any content. A stream that breaks halfway has already been billed for its partial output, so decide whether to keep that output or ask again for the whole answer.

## Balance and spend limits {#balance}

Before it runs, every request checks your balance and the key's spend limit. If the balance is at or below zero, or the key has used up its limit, you get `402` with `Insufficient balance. Top up at the dashboard.` or `This API key has reached its spend limit.` The error type is `insufficient_quota` (OpenAI), `billing_error` (Anthropic) or `RESOURCE_EXHAUSTED` (Gemini).

The check runs at the start of a request and the charge settles a few seconds after it finishes. A request that starts with money in the account always runs to the end, so the balance can dip slightly below zero; your next top-up covers the difference. See [Billing](/docs/account/billing).