Spreading Agent Traffic Across Your Own Accounts Without Breaking the Cache

Every LLM account has limits: a rolling five-hour usage window, a weekly cap, and per-minute rate limits. One busy coding agent can reach them. Several agents working for one person reach them every day, and the usual answer is to add a second account of your own and switch by hand when the first one runs out.

Our gateway does that switching for you. You bring your own accounts and keys, and the gateway decides which one each new conversation uses. Doing that well comes down to two things: knowing how much room each account has left, and never moving a conversation away from the account that holds its prompt cache.

Learning each account’s limits for free

Anthropic’s responses carry headers with the account’s usage in each window, whether it is close to a limit, and when each window resets. The gateway reads them from every response, so it learns each account’s state as a side effect of normal traffic. Readings older than ten minutes count for less, and every gateway replica shares what it learns.

OpenAI’s responses carry no equivalent for subscription usage, so for those accounts a background poller reads weekly usage instead.

Picking an account for a new conversation

A weighted score decides. It favours:

• Budget about to expire. Weekly allowance that will be lost at the next reset if nobody uses it.
• Accounts behind pace. Ones using less than a steady pace that keeps 15% of the week in reserve.
• Room in the short window. Accounts below 80% of their five-hour window.
• The right model. Requests for the most expensive model are steered away from accounts that still have room for the default one.

An account at 97% of any window is all but excluded. An account the gateway has never seen gets three trial requests before it is trusted. For OpenAI accounts, a new conversation goes to the account whose weekly window resets soonest.

Never move a conversation

Prompt caches belong to one account. Move a long conversation to another account and its next request pays full price for the entire history. So each conversation is pinned to the account it started on, and the pin expires after an hour of inactivity, the same as the cache lifetime.

Pins are stored in Postgres, so restarting the gateway does not scatter conversations across accounts. If that store is unavailable, the gateway refuses the request rather than guess. In the window we measured, 1,184 conversations kept their account, 5 moved and 6 were newly assigned.

OpenAI raises the stakes further. A conversation’s encrypted reasoning state only works on the account that produced it, so a Codex conversation never moves at all.

When an account says no

• A 429 with rate-limit headers puts that account on cooldown until its window resets, capped at seven days.
• A 429 without those headers is treated as a network-level rejection and gets a five-second pause. That rule came from an outage: a VPN failure produced bare 429s, and the gateway put every account on cooldown while all of them were fine.
• A weekly-cap rejection blocks only that model on that account. A five-hour rejection blocks the whole account.
• No retry on the same account, and no automatic retry of server errors.
• Background work yields first. Low-priority traffic such as summaries and background reviews is turned away when headroom drops below 15%, so interactive work keeps its share.

What went wrong along the way

• The VPN outage above, which cooled down every healthy account at once.
• An OpenAI account at 100% of its weekly limit kept receiving new conversations until the poller existed.
• A cooldown stayed set for six days after a plan’s limits were reset early, because the gateway trusted the old reset time.
• Pins kept only in memory were lost on every restart, and conversations lost their cache with them.

If you build this yourself

• Read the limits the provider already sends you before building any polling.
• Pin conversations to accounts, persist the pins, and expire them on the cache lifetime.
• Treat 429s without rate-limit headers as a network problem, not an exhausted account.
• Scope rejections narrowly: a model, then an account, never the whole pool.