Prompt caching

Half price is the starting point. Prompt caching takes input costs down much further for workloads that resend the same prefix: long chats, coding assistants, retrieval with a fixed set of documents. A model that has recently processed a prefix can reuse that work, and reused tokens are billed at the cache-read rate. On claude-sonnet-5.5, that is $0.05 per million tokens, against $1.00 for uncached input.

Caching through Half Off Dev rests on two pieces:

1. The vendor's own caching, used exactly as each vendor documents it. Fields like cache_control and prompt_cache_key are forwarded unchanged.
2. Conversation affinity. A cache exists only on the channel that wrote it. The gateway recognizes successive turns of the same conversation and sends them back to that channel, so the cache is still there for the next turn.

Claude: mark what to cache#

Claude caches only the parts you mark. Add cache_control to the last block of the stable prefix (tool definitions, system prompt, earlier turns). Up to four breakpoints are allowed, or a single top-level cache_control caches everything up to the last block.

response = client.messages.create(
    model="claude-sonnet-5.5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": PRODUCT_MANUAL,
        "cache_control": {"type": "ephemeral"},
    }],
    messages=chat_history,
)
print(response.usage)

usage reports cache_creation_input_tokens (written, billed at the cache-write rate) and cache_read_input_tokens (read, billed at the cache-read rate). Prefixes shorter than the model's minimum are not cached. Anthropic's caching guide covers where to place breakpoints and what invalidates a cache.

GPT: automatic, with an optional key#

GPT models cache long prefixes by themselves, so there is nothing to mark. Put the stable content first and the changing content last. A prompt_cache_key that stays the same for a conversation or a user improves the hit rate, and the gateway uses the same value to keep that conversation on one channel.

response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=chat_history,
    prompt_cache_key="thread_51c7d0",
)
print(response.usage.prompt_tokens_details.cached_tokens)

usage.prompt_tokens_details.cached_tokens is billed at the cache-read rate; the remainder of prompt_tokens at the input rate. There is no separate cache-write charge for GPT models. Details in OpenAI's caching guide.

Gemini: implicit caching#

Gemini models cache repeated prefixes automatically. Cached tokens appear in usageMetadata.cachedContentTokenCount and are billed at the cache-read rate, with no cache-write charge. Explicit context caching via the cachedContents API is not offered.

How turns are matched to a conversation#

Each conversation gets a fingerprint, taken from the first of these sources that is present:

The last row works without any effort on your side. A conversation grows by appending messages, so its system prompt and opening user message stay put, and every turn produces the same fingerprint. Only the first 4,000 characters of each are compared.

A fingerprint belongs to one API key, one model and one channel group. Send the same conversation with another key or to another model and it starts a fresh pin.

How long a pin lasts#

The first successful call in a conversation pins it to the channel that served it, and later turns try that channel first.

• Lifetime. A pin lives for up to an hour and is refreshed while the conversation stays active. After 30 to 60 minutes of silence it lapses, and the next turn is placed as if the conversation were new.
• Cache already gone. In Anthropic Messages format a pin also ends once the conversation has been idle longer than its cache survives: 5 minutes, or 1 hour if your cache_control breakpoints use ttl: "1h". There is nothing left to protect, so the next turn is placed freshly rather than waiting on its old channel.
• Channel busy. When the pinned channel is at capacity, the turn waits a moment for it (about a second for short prompts, up to ten seconds for long ones) instead of moving and paying for a new cache write. If it is still full, the turn moves.
• Channel failing. When the pinned channel is unhealthy, the turn moves to a healthy one and the pin moves with it. That turn writes a new cache, and the turns after it read from it.

Give conversations an explicit id#

The automatic fingerprint breaks if the first user message changes from turn to turn, for example when your app condenses or rewrites old history. Then send one stable id per conversation. A header works in every format:

curl https://halfoff.dev/v1/messages \
  -H "x-api-key: $HALFOFF_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-session-id: thread_51c7d0" \
  -H "content-type: application/json" \
  -d '{"model": "claude-sonnet-5.5", "max_tokens": 1024,
       "messages": [{"role": "user", "content": "Pick up where we left off."}]}'

Body fields do the same job: metadata.user_id in Anthropic format, prompt_cache_key in OpenAI format. Keep it to one id per conversation; sharing one id across many unrelated conversations crowds them all onto one channel.

Cache use in logs and billing#

For each call, Logs show:

The dashboard overview shows your cache hit rate across all calls. Every token kind has its own half-price rate; see the formula.