How We Keep the Prompt Cache Warm While Rewriting Every Request

Every request one of our coding agents sends to a model passes through a gateway first. The gateway rarely forwards the request unchanged. It adds the project’s working rules, a directory of the user’s applications and repositories, credential handling, and notes from reviews. On a typical day it makes an injection decision on almost every request.

That raises two questions. Why rewrite requests at all, when the agent could discover the same things itself? And how do you rewrite every request without wrecking the prompt cache, the provider feature that makes long agent sessions affordable?

An agent without context starts every session lost

A coding agent arrives knowing a lot about programming and nothing about you. It does not know your repositories, which one holds the app you mean, where your app is deployed, which branch is the integration branch, or which commands are safe to run. Ask it to “update the media app” and the first stretch of the session goes to searching the disk, guessing, and sometimes editing the wrong copy of the code.

> update the media app to hide watched titles

Searching for "media"... found 14 matches.
ui/app/media/ looks related; it is the
platform's generic media viewer.
Editing ui/app/media/Grid.vue ...

(wrong repository: the app lives elsewhere)

> update the media app to hide watched titles

The Media app lives in home/media (branch main);
app.toml owns its views, pipelines/ owns history.
Reading DISCOVERY.md for watch-history rules...
Planning the change in pipelines/history.toml.

So the gateway adds what a new teammate would want on day one. Each layer answers a question the agent would otherwise spend tokens and time rediscovering:

• Working rules. How this team branches, tests, reviews and deploys, and what is off limits, such as never resuming a paused GPU autoscaler without being told to.
• An app and repository directory. Each application the user owns, its aliases, where its code lives, and which docs to read first.
• Credential handling. The agent asks for a secret by name. The gateway swaps in the real value at the edge, so a key never sits in the conversation, the logs or the cache.
• Memories and review notes. Durable decisions from earlier sessions, and findings from a reviewer that is watching the session.

The catch: the cache only matches an exact prefix

Providers cache the processed form of a prompt and charge a small fraction of the normal price to reuse it, but only for an exact, byte-for-byte prefix. Change one character near the top of a 100k-token conversation and everything after it is billed at full price again. Worse, the cache write itself costs more than a plain read, so a request that breaks the prefix pays a premium to rebuild what it just threw away.

Context injection is a machine for changing the start of prompts. Done naively, it guarantees a miss on every turn. The fix is to treat request order as a design constraint: put what never changes first, what changes per session next, and what changes per turn last.

Rules that keep the prefix stable

• Freeze injected context for the life of a context window. The directory and rules are rendered once per conversation and stored. If the user edits their directory mid-conversation, the old snapshot keeps being sent; the edit appears in new conversations, or when a long one is compacted and its prefix is rebuilt anyway.
• Append, never edit. New information, such as a review note, is added as a new message at the end. Earlier messages are never rewritten, even to fix a typo in something the gateway added.
• Deterministic rendering. Anything rendered into a prompt is sorted and serialized the same way every time. A map that iterates in a different order is enough to miss the cache.
• Swap secrets after the cache key, not before. Credential replacement happens on headers and outbound values, never by editing cached message text.
• Keep cache lifetimes ordered. A breakpoint with a short lifetime may not sit in front of one with a longer lifetime. The gateway raises the earlier one instead of letting the provider reject or ignore it.

Guards for when someone breaks the rules anyway

Rules in a document do not stop a bug. Every injection path is new code, and a single one that rewrites history turns every cached read in the sessions it touches into a full-price read. So the gateway checks its own work.

For each conversation it remembers what the previous request’s prefix looked like and how much of it the provider reported as cached. If a new request should have matched a warm cache and the provider reports a large uncached read, the gateway records it as a warm miss. Repeated warm misses for the same scope trip a breaker, which refuses further requests that would keep paying full price. The breaker stays latched until an operator has looked at the cause and resets it with a written reason, which is kept with the scope for audit.

Most decisions are plain admissions. The second-largest group waits: the guard holds a request until the cache it should reuse is ready, instead of letting it pay to write the same prefix again.

Measure it, or assume it is broken

Providers report cached, newly written and uncached input tokens on every response. The gateway records all three per backend and per agent slot, and the hit rate becomes a dashboard number like any other service metric. When a deploy moves it, someone notices that afternoon instead of at the end of the month.

The chart is the argument for measuring. Most weeks sit above 90%, and the bad weeks are severe: below 20% means nearly every input token that week was billed at full price. Codex has stayed between 91% and 98% every week since history edits were rejected on that route. Claude, which that rule did not cover, still had two bad weeks afterwards. Since the guard shipped, Claude weeks have been between 87% and 96%.

The economics are lopsided. A cached read costs about a tenth of an uncached one, so at a 90% hit rate the input bill is about a fifth of what it would be with no caching at all, and at 15% it is nearly the full price. The roughly 10,000 tokens of context we inject cost full price once per conversation and a tenth of that on every turn after.

What we would tell someone starting out

• Decide what the agent should know before it reads a single file, and put it in front of the conversation once.
• Order every request from most stable to least stable, and never edit what has already been sent.
• Record cached, written and uncached tokens for every call. A hit rate you do not measure is a hit rate you do not have.
• Add a breaker. The bug that breaks caching will be in code nobody thinks of as cache code.