---
title: Prompt caching
description: Cut input costs further on top of half price: use each vendor's prompt caching, and let Half Off Dev route every conversation back to the channel holding its cache.
---

Half price is the starting point. Prompt caching takes input costs down much further for workloads that resend the same prefix: long chats, coding assistants, retrieval with a fixed set of documents. A model that has recently processed a prefix can reuse that work, and reused tokens are billed at the cache-read rate. On `claude-sonnet-5.5`, that is $0.05 per million tokens, against $1.00 for uncached input.

Caching through Half Off Dev rests on two pieces:

1. **The vendor's own caching**, used exactly as each vendor documents it. Fields like `cache_control` and `prompt_cache_key` are forwarded unchanged.
2. **Conversation affinity.** A cache exists only on the channel that wrote it. The gateway recognizes successive turns of the same conversation and sends them back to that channel, so the cache is still there for the next turn.

## Claude: mark what to cache {#anthropic}

Claude caches only the parts you mark. Add `cache_control` to the last block of the stable prefix (tool definitions, system prompt, earlier turns). Up to four breakpoints are allowed, or a single top-level `cache_control` caches everything up to the last block.

```python title="Python"
response = client.messages.create(
    model="claude-sonnet-5.5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": PRODUCT_MANUAL,
        "cache_control": {"type": "ephemeral"},
    }],
    messages=chat_history,
)
print(response.usage)
```

```js title="Node.js"
const response = await client.messages.create({
  model: 'claude-sonnet-5.5',
  max_tokens: 1024,
  system: [{
    type: 'text',
    text: PRODUCT_MANUAL,
    cache_control: { type: 'ephemeral' },
  }],
  messages: chatHistory,
});
console.log(response.usage);
```

`usage` reports `cache_creation_input_tokens` (written, billed at the cache-write rate) and `cache_read_input_tokens` (read, billed at the cache-read rate). Prefixes shorter than the model's minimum are not cached. [Anthropic's caching guide](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) covers where to place breakpoints and what invalidates a cache.

## GPT: automatic, with an optional key {#openai}

GPT models cache long prefixes by themselves, so there is nothing to mark. Put the stable content first and the changing content last. A `prompt_cache_key` that stays the same for a conversation or a user improves the hit rate, and the gateway uses the same value to keep that conversation on one channel.

```python title="Python"
response = client.chat.completions.create(
    model="gpt-6.1-sol",
    messages=chat_history,
    prompt_cache_key="thread_51c7d0",
)
print(response.usage.prompt_tokens_details.cached_tokens)
```

```js title="Node.js"
const response = await client.chat.completions.create({
  model: 'gpt-6.1-sol',
  messages: chatHistory,
  prompt_cache_key: 'thread_51c7d0',
});
console.log(response.usage.prompt_tokens_details?.cached_tokens);
```

`usage.prompt_tokens_details.cached_tokens` is billed at the cache-read rate; the remainder of `prompt_tokens` at the input rate. There is no separate cache-write charge for GPT models. Details in [OpenAI's caching guide](https://platform.openai.com/docs/guides/prompt-caching).

## Gemini: implicit caching {#gemini}

Gemini models cache repeated prefixes automatically. Cached tokens appear in `usageMetadata.cachedContentTokenCount` and are billed at the cache-read rate, with no cache-write charge. Explicit context caching via the `cachedContents` API is not offered.

## How turns are matched to a conversation {#conversation-matching}

Each conversation gets a fingerprint, taken from the first of these sources that is present:

| Source | API format |
| --- | --- |
| `x-session-id` header | All |
| `x-halfoff-session` header | All |
| `x-claude-code-session-id` header (Claude Code sends it) | All |
| `metadata.user_id` in the body | Anthropic |
| `prompt_cache_key` in the body | OpenAI |
| `user` in the body | OpenAI |
| System prompt plus first user message | All |

The last row works without any effort on your side. A conversation grows by appending messages, so its system prompt and opening user message stay put, and every turn produces the same fingerprint. Only the first 4,000 characters of each are compared.

A fingerprint belongs to one API key, one model and one channel group. Send the same conversation with another key or to another model and it starts a fresh pin.

## How long a pin lasts {#pin-duration}

The first successful call in a conversation pins it to the channel that served it, and later turns try that channel first.

- **Lifetime.** A pin lives for up to an hour and is refreshed while the conversation stays active. After 30 to 60 minutes of silence it lapses, and the next turn is placed as if the conversation were new.
- **Cache already gone.** In Anthropic Messages format a pin also ends once the conversation has been idle longer than its cache survives: 5 minutes, or 1 hour if your `cache_control` breakpoints use `ttl: "1h"`. There is nothing left to protect, so the next turn is placed freshly rather than waiting on its old channel.
- **Channel busy.** When the pinned channel is at capacity, the turn waits a moment for it (about a second for short prompts, up to ten seconds for long ones) instead of moving and paying for a new cache write. If it is still full, the turn moves.
- **Channel failing.** When the pinned channel is unhealthy, the turn moves to a healthy one and the pin moves with it. That turn writes a new cache, and the turns after it read from it.

## Give conversations an explicit id {#explicit-session-id}

The automatic fingerprint breaks if the first user message changes from turn to turn, for example when your app condenses or rewrites old history. Then send one stable id per conversation. A header works in every format:

```bash title="cURL"
curl https://halfoff.dev/v1/messages \
  -H "x-api-key: $HALFOFF_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-session-id: thread_51c7d0" \
  -H "content-type: application/json" \
  -d '{"model": "claude-sonnet-5.5", "max_tokens": 1024,
       "messages": [{"role": "user", "content": "Pick up where we left off."}]}'
```

```python title="Python"
response = client.messages.create(
    model="claude-sonnet-5.5",
    max_tokens=1024,
    messages=chat_history,
    extra_headers={"x-session-id": "thread_51c7d0"},
)
```

```js title="Node.js"
const response = await client.messages.create(
  { model: 'claude-sonnet-5.5', max_tokens: 1024, messages: chatHistory },
  { headers: { 'x-session-id': 'thread_51c7d0' } },
);
```

Body fields do the same job: `metadata.user_id` in Anthropic format, `prompt_cache_key` in OpenAI format. Keep it to one id per conversation; sharing one id across many unrelated conversations crowds them all onto one channel.

## Cache use in logs and billing {#cache-in-logs}

For each call, [Logs](/app/logs) show:

| Field | Meaning |
| --- | --- |
| **In** | Uncached input plus cache writes. |
| **Cached** | Input tokens read from the cache. |
| **Cache write** (details) | Input tokens written to the cache. |
| **Cache hit** (details) | Cached tokens as a share of all input. |
| **Affinity** (details) | `explicit`: you supplied a session id. `sticky`: the conversation was pinned and returned to its channel. `new`: first call, or the pin had expired or its channel was unavailable. `none`: no fingerprint could be built. |

The dashboard overview shows your cache hit rate across all calls. Every token kind has its own half-price rate; see [the formula](/docs/models/pricing#charge-formula).

> [!TIP] Low hit rate?
> If every turn of a conversation shows `new`, something in its system prompt or first user message changes between turns. The usual culprit is a timestamp in the system prompt, which also defeats the vendor's cache: move it to the latest user message, or send an explicit session id. If turns show `sticky` but **Cached** stays empty, check that the prefix meets the model's minimum length and, for Claude, that it carries `cache_control`.