Cutting What a Schema Edit Rebuilds in a 1,000-Crate Rust Workspace

Our Rust monorepo has about a thousand workspace crates. Around 139 of them, 148 today, hold shared schema types: the serde structs we store, send between services and hash. One more crate, the aggregator, depended on most of them and re-exported their types, and most consumers depended on the aggregator instead of on the schema crate they used.

Cargo recompiles every crate downstream of a crate whose source changes. In mid-September, 58 of the 139 schema crates sat underneath the aggregator, so an edit to any of them rebuilt at least 440 of the 980 workspace crates. In the last week of September a short series of changes removed the aggregator from every production crate, and two other changes went after compile time inside the crates. This post covers what was measured, what helped and what made no difference.

Counting the rebuild set

We call the workspace crates that depend on a crate, directly or through other crates, its rebuild set. It can be computed from the Cargo manifests alone, so it is cheap to track over months of history, and it shows where an edit fans out before anyone measures a compile.

In mid-September the average schema crate had 207 crates in its rebuild set. The distribution had two humps: 59 schema crates rebuilt 400 or more crates, and 72 rebuilt 50 or fewer. The large ones were almost exactly the crates underneath the aggregator. Today the average is 78 of 995, 5 schema crates rebuild 400 or more, and 101 rebuild 50 or fewer. The aggregator itself has one remaining dependent: a fixture that compile-checks model-generated code written against the old import paths.

Moving consumers to the crates that own the types

• Twenty crates first (26 September). The first change replaced twenty direct aggregator dependencies with the micro-crates that own the types. Nine of them were schema crates whose transaction builders had depended on the aggregator. Thirteen of the twenty lost their path to the aggregator entirely, and their own dependency closures shrank; one schema crate went from 68 workspace dependencies to 23, and the transform runtime from 68 to 15.
• Moving real code out of the aggregator (27 September). The aggregator held more than re-exports. Its CBOR encoding traits moved to the base schema crate, and data-folder file access moved to a new crate, each with a compatibility re-export at the old path so existing imports still compiled. Twelve more consumers moved. The audit recorded the aggregator’s transitive dependents falling from 418 to 230.
• The rest in one change (28 September). The last change touched 1,147 files. It moved the remaining implementations, among them the market-data record and the node configuration, into owning crates and migrated every production consumer, including the storage engine and the data engine service. All 160 affected crates passed cargo check --tests, and 827 schema and node tests passed. Getting through CI took several rounds: example binaries that imported their own library with crate:: paths, a run that hit the two-hour wall limit, and a rebase conflict with concurrent work.
• A test that keeps it that way. An inline test in the aggregator crate reads every workspace manifest, including target-specific tables and renamed dependencies, and fails if any crate other than the generated-code fixture depends on the aggregator. A second test applies the same rule to an old facade crate for schema macros.

The effect is uneven. The agent workflow schema crate’s rebuild set fell by three quarters, because most of its old dependents reached it only through the aggregator. The base and error crates barely moved: most of the workspace uses them directly, and the error crate’s set grew from 760 to 786 as crates were added. The average has also crept up, from 70 at the end of September to 78 now. We read that as new crates taking on dependencies, without having checked each one.

Sharing serde codec instances between crates

Our schema types are encoded as CBOR through serde. serde’s deserializers are generic, so every crate that decodes a type has rustc monomorphize the full deserializer for that type and generate its code again. Before the change, the transaction type was decoded in 33 crates and the operation event type in 28.

The fix routes every encode and decode through two plain generic functions and adds a macro that a schema crate invokes with a list of its types. The macro stores pointers to those two function instances in a static marked #[used], which makes rustc generate them in the schema crate. At optimization levels 0 and 1, the levels our development, CI and release profiles use, rustc shares generic instances between crates by default, so downstream crates link the schema crate’s copy instead of generating their own. The bytes on the wire are unchanged.

pub const fn of<T: Serialize + DeserializeOwned>() -> Self {
    CodecRoot(
        to_cbor::<T> as fn(&T) -> Result<Vec<u8>, serde_cbor::Error> as *const (),
        from_cbor::<T> as fn(&[u8]) -> Result<T, serde_cbor::Error> as *const (),
    )
}

macro_rules! share_cbor_codecs {
    ($($ty:ty),* $(,)?) => {
        const _: () = {
            #[used]
            static CBOR_CODEC_ROOTS: &[$crate::CodecRoot] =
                &[$($crate::CodecRoot::of::<$ty>()),*];
        };
    };
}

The pilot listed 39 agent workflow types. Measured with rustc’s own pass timings, three runs each:

• The workflow record crate went from 5,582 monomorphized items to 2,390, and from 1.8 to 1.1 seconds of rustc time.
• The subscriber supervisor went from 13.4 to 12.3 seconds. The workflow runner went from 11.2 to 9.5 seconds, but its runs ranged from 9.2 to 13.4 seconds, so that result is within noise.
• The schema crate pays for it. It went from 5.2 to 7.5 seconds, about 60 milliseconds per listed type, on every schema edit. So the rule is to list only types that two or more crates decode.

Extending it to seven widely decoded types brought smaller gains on the crates measured: 1.44 to 1.24 seconds for the workflow record crate and 0.20 to 0.17 seconds for a row-writer crate.

The data engine service and its route tree

Our heaviest service crate is the data engine’s HTTP service, about 48,000 lines in one crate, at roughly two minutes of rustc time in these measurements. Codecs were a small part of it: serde_cbor accounted for 9% of its monomorphized size. The profile put about 73 seconds in the monomorphization collector’s graph walk and about 8 seconds in LLVM. The service uses warp, where routes combine with or into nested generic filter types, and it has about 70 routes.

• Normalizing replies first. Converting each route’s reply to a plain response before joining route groups gives every or join the same response and filter type. Median rustc time went from 120.79 to 103.56 seconds, about 14%, and the type-checking phase from 21.22 to 11.97 seconds. Route order, rejection recovery and CORS behavior stayed the same and were tested against the original typed trees. This shipped.
• Joining groups statically. 120.21 seconds, no measurable change.
• Boxing each group. 113.85 seconds, a smaller gain than normalizing individual routes.

The September plans leave splitting the service into smaller crates as separate work, on the reasoning that most of its remaining time is spent walking one very large crate.

Trying Cranelift

Cranelift is an alternative code generator for rustc that produces unoptimized code faster than LLVM. Our builder toolchains did not include it at first, so rustc failed to find the backend. The September rollout added the preview component to the CI and builder images without changing the default.

• Workflow runner: 10.27 to 8.94 seconds, about 13%.
• Agent workflow schema crate: 11.57 to 9.17 seconds, about 21%.
• Data engine service: 100.29 to 100.23 seconds, no gain. Its time goes to the compiler stages before code generation.

LLVM is still the default, and the line that would enable the backend feature in the root manifest is still commented out. The handoff notes expected a one-time miss on every build cache when the backend changes.

Where external dependencies fit

An August study of compile CPU looked at external crates first. Removing an unused gRPC dependency from a crate underneath the aggregator was estimated at 1.4% of weekly compile CPU, and duplicate versions of external crates at about 2%. The largest share then came from compiling the same crates repeatedly in separate CI target directories, which the move to remote builders addressed. The schema work targets a different cost: how much has to be recompiled after an edit.

That study also recorded a counting trap that applies to this post. Summing the compile cost of every crate in a closure counts shared crates once per dependent and overstated savings by about ten times. Scaling cost by the number of users removed matched reality. The rebuild-set counts here are graph sizes, and we do not convert them into seconds.

A CI gate that depends on the smaller graph

In October, CI started compiling every reverse dependency of a changed schema crate, library and binary targets only, without their tests. Before that it compiled at most three consumers, ranked by deployment. That missed library crates that build schema structs with full struct literals: adding a field, even an optional one, breaks those literals with error E0063, and one such break failed a publication. The gate fails closed when it cannot resolve the dependency graph.

Our inference, without a before-and-after measurement of gate time: compiling every consumer is affordable partly because the rebuild sets are smaller. Under the mid-September graph, an edit to any of the 58 schema crates under the aggregator would have pulled at least 440 crates into the gate.

What we would do again

Our inference from this work, rather than measured results:

• Compute rebuild sets from the manifests before timing anything. It is cheap: for this post, a short script over the manifests covered four months of history, and the counts point straight at the aggregator.
• Leave compatibility re-exports at the old paths during the move, and add the regression test in the same change that finishes the migration, so the graph cannot drift back.
• Profile the slowest crate before applying a fix that worked elsewhere. Codec sharing helped codec-heavy crates and did little for the data engine service, whose time went to its route tree.
• Trial a codegen backend per crate. Cranelift helped crates that spend their time in code generation and did nothing for the slowest one.

Limits of this account: the graph counts exclude dev-only dependencies, so test builds recompile more than shown. The average weights every schema crate equally, while real edits cluster on a few of them. The compile timings come from three samples on one builder with incremental compilation off, and baselines differ between measurement days; the data engine service measured 78 seconds of rustc time in one run and 100 to 121 seconds in later series. We have no fleet-wide before-and-after measurement of rebuild time.