Tracking Down Agent Spending Anomalies With One Usage Ledger

All of our model traffic goes through one gateway: coding agents, chat in the UI, background summarizers, app features. For every call, the gateway writes one row to a usage_event table in Postgres. Over the last 30 days that table gained about 1.16 million rows, 114 billion tokens, and roughly $3,080 of estimated spend.

A provider dashboard can tell you that spend went up. It cannot tell you which agent, task or feature caused it, because it only sees an API key. The ledger exists to answer that second question. Each row carries the tokens, plus the keys we found we needed when something looked wrong:

• Who: user, API key, tenant and the agent role (a coding agent, the chat UI, a summarizer).
• What work: session, conversation, ticket, workstream and pull request.
• Where: model, backend, machine, pod and source address.
• Tokens, split the way the provider reported them: uncached input, cache reads, cache writes, output and reasoning.

A command-line tool ranks spend by any of these keys over any window (top --by role, --by day, --by session and so on). It reads Postgres directly, so there is no separate analytics pipeline that can fall behind. The four cases below are all from the last month.

1. Spend more than doubled in two days

On 3 and 4 October, estimated spend jumped to $221 and $245 a day, against a 28-day average of about $100. The first instinct was a runaway agent stuck in a loop. Ranking those two days by session showed no single outlier: the top sessions each used about 3% to 7% of the total, and their average prompt size matched every other day.

Counting distinct agent sessions per day explained it. The fleet grew from 27 coding-agent sessions on 2 October to 45 and then 51, and each session made somewhat more calls. Spend per active session moved much less than total spend did, and those were also the two busiest merge days of the month, with 197 and 182 pull requests merged. The fleet was doing more work, not repeating itself.

Lesson: before looking for a bug, divide by the number of things running. A spike that disappears per session is a capacity decision, not an incident. It may still be the wrong decision, but it needs a different conversation.

2. Prompts appeared to shrink by 75%

In mid-October, average input per request fell from about 125,000 tokens to as low as 20,000 on some days. That looked like good news: perhaps context trimming had started working. It was wrong.

The two providers report input differently. OpenAI counts cached tokens inside its input total and reports the cached part as a subset. Anthropic reports uncached input, cache reads and cache writes as three separate numbers. A Claude call with a 140,000-token conversation typically records a few hundred tokens of input and the rest as cache reads. When agents shifted part of their work to Claude in mid-October, the naive average fell, even though prompts had not changed.

With input counted once, the average rose slightly over the month, from about 125,000 to about 140,000 tokens. Our reporting now stores a normalized total next to the raw provider fields, and every cache hit rate is computed against that total.

Lesson: never average a token field across providers without checking what it includes. Keep the raw fields, because they map to how each provider bills, and add one normalized total for comparisons.

3. Calls nobody owned

About 4.4% of calls and $157 of the month’s spend had no agent role. The ledger has a view for exactly this slice, broken down by model and session. Broken down by model, half of it was calls to our main agent model that arrived without a role header, and another 30% was two smaller models that are not used through the normal agent launcher. The breakdown by session points to the specific clients to fix.

Unowned spend matters more than its share suggests. A budget that is enforced per role cannot stop a client that has no role. A growing unattributed slice is best treated as a bug in whichever client stopped sending headers.

Lesson: attribution has to be stamped by the client making the call, or by the gateway from the client’s credential. Anything that relies on someone joining logs later will drift.

4. The busiest session was not the expensive one

Ranking session-hours by call count, the top entry was one agent making 894 calls in a single hour, about one every four seconds. It looked like a loop. Its uncached input for that hour was close to zero, and its output was small: it was reading and editing many small files, with nearly all of its context served from cache.

The sessions that did cost money looked ordinary by call count. Their cost came from cache misses on large contexts: a session that switches model, or has its history rewritten, pays full price for 100,000 or more tokens on the next call. One older model on our gateway ran at a 27% cache hit rate for the month, against 91% to 97% for the models agents use day to day on the few clients still using it, so most of its input was billed at full price.

Lesson: rank by estimated cost, not by request count. Then check the cache hit rate of the most expensive sessions before anything else.

The order we check things in now

1. Spend per day, then per active session. If the per-session line is flat, the cause is concurrency.
2. Normalized input per call and cache hit rate, by model. A falling hit rate on the same work means something changed the prompt prefix.
3. The unattributed slice. If it grew, find the client that stopped sending headers.
4. Top sessions by estimated cost for the window, and what ticket or pull request each was working on.

None of this needs special infrastructure: one wide table, written once per call by the component that already sees every request. The hard part was deciding up front which keys to stamp, and adding the ones we missed each time an anomaly could not be explained.