Building a Large Rust Monorepo for Twenty Agents Without Compiling Locally
When a fleet of coding agents all ran cargo on one development machine, the machine stopped being usable. We moved every build to shared remote builders. Here is how the system works, what it costs, and what broke.
A coding agent compiles constantly. It checks a change, runs the tests for one crate, builds a binary to try it, and does it again. In a large Rust workspace each of those can take minutes of CPU and gigabytes of memory. One agent doing that is fine. Twenty doing it on the same machine is not.
On July 16 our development server hit a load average of about 70 because agents were running cargo locally, and a watchdog was killing their builds within seconds. The same day the rule became: no local cargo. This post describes what replaced it.
- July 16: the development server is overloaded; local cargo is banned.
- August 6: the first builder service and remote cargo client.
- August 9: remote builds become the default, and a failed remote build stops instead of compiling locally.
- August 16: builders share one compile cache through S3, and burst builders can scale.
- August 26: a scheduling deadlock that stalled one builder for 11 hours is fixed.
- October 18: builds whose builder scaled away mid-request are re-admitted elsewhere.
The idea: agents need compile results, not compilers
The design note that started this has one sentence worth repeating: an agent never needs to be on a compile machine, it needs compile results. So an agent’s cargo build, check or test becomes a request to a builder service. The agent sends its source, the builder compiles, and output, test results and any requested binaries stream back.
Agents keep their editing environment, which is cheap. Compilation moves to a few large machines with warm caches, which is expensive to duplicate in every agent’s workspace.
Sending source without sending the repository
The workspace is tens of thousands of files, and builders are on another continent from the development machine. Sending everything on every build would be slow, so the client sends almost nothing.
- By default it sends the commit the builder can already fetch from the forge, plus a diff of the agent’s uncommitted edits, capped at 32 MiB.
- For trees the forge does not have, it sends a manifest of every file’s BLAKE3 hash. The builder answers with the hashes it is missing, and only those files are uploaded. A typical build of a 26,000-file checkout uploads a dozen files.
Choosing a builder
Build caches are much more effective when the same checkout keeps landing on the same machine. Each checkout gets a stable key, from its host and path or, for CI, its branch, and that key hashes to a preferred builder.
Affinity is a preference, not a rule. A builder that already has a warm slot for the key wins if its load is within 8 builds of the least-loaded builder. The hash-preferred builder wins if it is within 2. Otherwise the request goes to the least-busy machine, because a warm cache is not worth waiting in a long queue.
Keeping builders from falling over
- Two levels of limits. Each builder accepts a fixed number of builds at once, and a node-wide jobserver caps how many compiler processes those builds run in total, so one large build cannot starve the rest.
- Narrow under memory pressure. When memory runs low, the builder lowers compile parallelism instead of letting the kernel kill compilers halfway through.
- Deadlines everywhere. Admission and queueing have a 5-minute budget by default, and a build has 15 minutes. If a builder disappears mid-build, the request is re-admitted elsewhere up to twice within the original deadline.
- Burst capacity. Three extra builders scale from zero to one each when a request has waited 30 seconds, and are released after 20 idle minutes.
Show the data
| Build slots | Compiler job tokens | |
|---|---|---|
| builder-0 (12 CPU) | 6 | 6 |
| builder-1 (6 CPU) | 2 | 4 |
| each burst builder (14 CPU) | 8 | 4 |
One shared compile cache
Builders use a rustc wrapper cache, shared through S3 since mid-August, so a crate compiled on one builder is a cache hit on all of them. The cache reports a hit rate weighted by compile time, which is the number that matters: missing a two-second crate costs little, missing the largest crate costs minutes.
Results vary widely with what changed. In a small sample of 51 recent builds, the median weighted hit rate was 38%, or 60% among builds that hit the cache at all, and the median build saved 50 seconds. The best single build in the sample saved almost 22 minutes. Edits deep in the dependency graph invalidate a lot; edits in a leaf crate invalidate almost nothing.
What broke
- A jobserver deadlock parked one builder for 11 hours: builds held slots while waiting for compiler tokens that other waiting builds held.
- Disk. Build directories fill disks quickly. Cleanup had to learn not to delete a target directory while a build was still writing to it.
- Builders that scaled away. A burst builder could be released after accepting a request but before finishing it. The client now detects that and re-admits the build elsewhere.
- Cost. The burst cap moved from one builder to six and back to three as we traded queue time against server cost.
Fail loudly instead of falling back
The first version fell back to compiling locally whenever the builders were unavailable. That brought back exactly the overload it was built to prevent, at the moment the system was already struggling. Now a remote build that cannot be placed fails with an error, and local compilation is blocked by the tool itself. An agent that hits the block reports a builder problem; it does not route around it.
That last rule generalises. When an agent fleet depends on shared infrastructure, any automatic fallback becomes the path every agent takes the first time something is slow.