Keeping Concurrent Deploy Renders From Overwriting Each Other

Our Kubernetes manifests are generated by Rust code in the monorepo’s CI pipeline crate. After a pull request merges, a publish job builds whatever images changed, runs the generators and commits the rendered YAML into a deploy directory on the same integration branch. ArgoCD watches that directory and syncs the cluster to it. Nobody edits the rendered files by hand in the normal path.

In 26 September to 25 October 2026 (UTC), the branch took 3,159 merged pull requests, 1,494 manifest-only render commits and 177 render commits that re-pinned an image. Several publish jobs run at the same time, each rendering from the commit it started on. Any of them can find that the branch has moved by the time it pushes. This post covers the rules that decide what happens then.

A render commit should not start another pipeline

A render touches only the deploy directory, which builds no crate and produces no image. Without care, each render push would still start a full CI run that detects no changes, plus the workflows chained to its completion. Render commits carry a skip-CI marker, and each render is pushed on its own, so the marker is on the push’s head commit, which is where the forge checks for it. A merge is a separate push whose head commit has no marker, so real changes are never skipped.

Label the commit by what it changed

The first render commits always read “update images to” a tag. When three renders in a row rewrote other manifest fields without rebuilding anything, the log showed three commits pinning the identical tag, and searching the history for a tag returned manifest churn as if it were duplicate deploys.

The publisher now scans the staged diff for added or removed image references. Only a render that changes one gets the image subject; every other render says “update deploy manifests (no image change)”. In the window above, 89% of render commits were manifest-only. Every manifest-only commit also records the source commit it was rendered from in a trailer line. Most of the rules below depend on that trailer.

ci(dev): update deploy manifests (no image change) [skip ci]

Deploy-Source-SHA: <source commit>

Losing a race can be a success

A render clones the branch shallowly, generates, commits and pushes. If a merge or another render lands in between, the push is rejected. The publisher then rebases the render onto the new tip and retries. On 11 July it also learned to drop a render that the rebase showed was already present, such as one identical to a concurrent job’s. If the rebase conflicted, it opened a fallback pull request.

That broke when a newer job had already rendered different pins. The older render conflicted with it, opened a conflicted fallback pull request and left a branch behind. By 12 July, 36 orphaned render branches had accumulated. Pushing such a render would also have moved every image pin back to an older tag. The fix checks whether the branch already carries a render for a descendant of this job’s source commit. If it does, the job drops its render and reports success, because a newer render already covers it.

Decide staleness by what the render writes

The harder question is when an older render is still safe to push. The answer changed three times.

• Gate on render inputs only (August). The first staleness check reused the list of paths that trigger post-deploy work. That list includes high-churn image sources such as the agent CLI, whose commits change no rendered byte. A CLI-only commit landing mid-job refused an otherwise valid render. The gate now covers only generator code and the files generators embed, and a test asserts that every generator module and embedded file sits under one of those roots.
• Compare Kubernetes resources, not files (October 3). A render scoped to one service may share a YAML file with unrelated templates. Dropping a scoped render now requires every Kubernetes resource it changed to be already present on the branch, or last changed by a render from a strictly newer source. A same-source render, or a newer render of different resources, proves nothing. A failed read or ambiguous history never counts as delivery.
• Push older renders that overwrite nothing newer (October 11). Refusing every render whose source was behind the tip could never keep up with the merge rate. An older render is now pushed unless a render from a newer source, or an edit with no recorded source, has already changed one of the files it writes. Unrecorded edits count as newer, so a stale render cannot revert a manual fix.

The coverage and overwrite checks read first-parent history, so a merge resolution without a source trailer cannot borrow coverage from a trailer on the merged side branch.

A failed generator should not delete other manifests

A full render clears the deploy directory and regenerates it, which also removes manifests for services that no longer exist. On 26 July four generators failed partway through, and the render staged 146 deletions. None were files those generators write: they included documentation and compose manifests and about 120 rendered pipeline definitions.

The fix restores every deleted path from the checked-out branch when any generator fails. Paths that a working generator re-rendered keep their new content, because they are modifications rather than deletions. Cleanup of stale paths still runs on every clean render and is suspended only for a run with a failure.

Prove the generator matches its source

The push rules above trust the source commit the render claims. In October we noticed that nothing stopped a fetched, older build of the generator from running in a newer checkout and stamping the newer commit on its output. The pipeline’s build script now embeds a fingerprint of its inputs: paths, file modes and object hashes for the workspace code and every embedded render input. The real render path checks that fingerprint against the checkout before rendering and again before publishing. Dirty or untracked inputs fail the check, and an older driver that lacks the proof command fails closed.

What we would reuse

Our inference from this history, rather than measured results:

• Record the source commit in every generated commit. Each ordering decision above needs to know which merge a render came from.
• Decide staleness by what a render writes, at the granularity of the deployed resource. Whether the branch moved since the job started is the wrong question at a high merge rate.
• Make “a newer render covers this one” an explicit success outcome. Treating it as a conflict produced fallback pull requests and orphaned branches.
• Scope deletions to what the failing step owned. A cleanup that is correct on a clean run can delete unrelated files when one generator fails.
• Bind the generator binary to its source before trusting the provenance it stamps.

Limits of this account: it is reconstructed from code, comments, plans and commit history, plus the render-commit counts in the chart. We have no count of how many push races occurred, how many pins were downgraded before the July fix, or how often each later rule fired.