Replacing Long-Lived Agent Pods Without Killing Their Work

Each coding agent in our fleet runs in a long-lived Kubernetes pod with a persistent state volume. Some pods take dispatched work: CI, review and fix jobs handed out by a supervisor. Others host one interactive agent conversation. Either kind can be partway through a multi-hour task at any moment.

The pods are Deployments with the Recreate strategy, and a reconciler re-applies every Deployment on each sweep. Any change to the pod template, whether an environment variable, a mount or an image, deletes the running pod and starts a new one. For an idle pod that is harmless. For a pod running a CI job, the job dies and its pull request is left without a result.

Until July the safe path was a manual drain-and-cycle command, and in practice it went unused. One dispatch pod ran for a day without a new database URL variable after the change that added it had merged. This post describes the automatic rotation that replaced the manual step, and the incidents that set each of its limits.

Detect drift with a hash, then drain before applying

The reconciler content-hashes the desired pod template and stamps the hash into a template annotation. The hash excludes the annotation itself, so stamping does not change it. Before each re-apply, the reconciler compares the desired hash with the one on the live Deployment. Equal hashes mean there is nothing to do.

On drift, for a pod with work on it, the reconciler runs the same drain the manual command used. It writes a DRAIN marker file on the pod’s persistent volume and sets a drained flag in the pod allocator, so new work goes to the group’s other pods. It then skips the apply on every sweep until live work sessions and unfinished execution claims reach zero. Only then does the apply, and the Recreate it implies, go through. Once the replacement pod is ready, the drain is cleared.

Rotation state lives in the cluster rather than in the reconciler process. The marker carries a tag and a start timestamp, so a restarted reconciler can resume or unwind a rotation by probing the pod. An empty marker means an operator drained the pod by hand, and the reconciler never clears that one.

Each limit came from an incident

The first version, merged on 22 July, waited for work to finish and forced the apply after a one-hour ceiling. Over the next three months the module grew to 2,099 lines and 22 tests across 33 commits. Most of those changes added a bound or narrowed one after a specific failure.

• Every pod drained at once (28 July). A reconciler rollout changed the template for all five dispatch pods in one sweep. Each held a warm prelaunch session, which never exits on its own, so all five drained together and stayed drained. For about an hour, until the ceiling forced the applies, every dispatch attempt failed because no pod was accepting work. Fix: at most one dispatch pod rotates at a time, so the rest of the group keeps serving.
• The ceiling killed running jobs (30 July). Another template change rolled the five dispatch pods one after another, and at each pod the forced apply killed whichever CI, fix or review job was running. Fix: past the ceiling, the apply goes through only while no work sessions are live. Idle warm sessions no longer hold a rotation open, and live work still does.
• A broken pod was about to be parked (20 August). A dispatch pod had been running an 11-day-old spec that lacked a credential mount added since. Its controller got authorization errors on every keepalive write, so the reaper treated it as dead and would have parked it. Nothing re-attaches a parked environment without an operator, so parking would also have discarded the spec that fixed it. Fix: when a pod with an expired keepalive is also drifted and has passed the empty-occupancy probe, the reconciler re-applies it instead of parking it.
• Probe failures deferred forever (21 August). When the reconciler cannot exec into a pod to read its marker, it defers the apply, because applying blind could recreate a busy pod. With no cumulative limit, a pod whose exec kept failing deferred on every sweep. Fix: after four hours of failed probes, the apply is forced. This clock is process-local; a reconciler restart resets it, which can only lengthen the wait.

Conversation pods need the agent to cooperate

In October rotation was extended to conversation pods. Counting sessions from outside is unreliable there, because only the agent process knows whether it is partway through a turn. So the reconciler asks it. It sends an authenticated relocation request to the runtime inside the pod. The runtime takes a fence that refuses new work, writes a checkpoint and reports back.

Only a specific report authorizes the Recreate: matching environment and pod, Running status, Idle activity, a heartbeat less than 30 seconds old and a checkpoint tagged as belonging to the rotation. Transcripts and session records are stored outside the pod, so a managed conversation resumes on its next input on the replacement pod.

The handshake brought its own failure modes, and most of the module’s commits from 6 to 18 October address them. That period includes the chart’s peak of 11 commits in one week.

• Failed checkpoints held the rotation slot. An error now keeps the protective drain but releases the reservation. A retry must acquire it again, and a marker alone never authorizes a Recreate.
• There was no deadline. A runtime gets 20 minutes to finish its checkpoint, then yields the slot for an hour before trying again. A refused handshake runs on the same deadline. The first version kept that clock in memory, and the development environment redeploys the reconciler every 30 to 60 minutes, so the 20-minute deadline kept restarting and long-idle runtimes never reached their fallback. The marker now records when the current attempt began, and a restarted reconciler continues that attempt.
• One slow runtime blocked the rest. Conversation pods shared the dispatch cap of one, so a runtime that never cooperated held the only slot for its full deadline while a dozen idle runtimes waited behind it for hours. Conversation rotations now have their own pool with a cap of four. Each conversation pod serves one conversation, so rotating it removes no capacity another depends on.
• The idle fallback could never fire. For a runtime whose handshake keeps failing, such as a terminal session that stopped answering, the fallback proof is a fresh Idle report with 30 minutes of no meaningful activity. The first version read that report while the rotation’s own fence was still held, and a fenced runtime never reports Idle. One pod stayed fenced for three days with only idle sessions on it. The check now runs after the fence is released.
• A fence could land on a returning conversation. The runtime’s activity report lags behind a new message. The data engine stamps the time when a conversation claims its runtime, and input from the last ten minutes now keeps the pod out of rotation.

A new image alone no longer restarts a pod

The change on 24 October removed the most frequent trigger. Every agent image publish moves the image digest, and applying it under Recreate would interrupt live conversations and jobs. The controller in each pod already replaces the CLI and other binaries in place from the development release channel, so a restart for the image alone gains little.

When the image is the only difference, the reconciler copies the live image fields into the desired template before hashing, and no drift is recorded. The new image arrives with the next non-image change or the next fresh attach, and an environment variable can restore the old roll-on-image behavior. A fix the next day handled a false drift: the API server fills in a default image pull policy for init containers that our template builder omits, and copying that default produced a hash that never matched the live stamp.

What we would carry to another system

Our inference from this history, rather than measured results:

• Keep rotation state where the next reconciler can read it. A marker on the persistent volume and a flag in the allocator survive restarts. Clocks that must stay in process memory should reset in the direction that waits longer.
• Give every wait a ceiling, and state what the ceiling may destroy. Two of the incidents above came from a missing ceiling and one from a ceiling that ignored live work.
• Cap concurrent drains per capacity pool. Pools that share no capacity should have separate caps, so one cannot starve the other.
• Ask a stateful agent process before replacing it. Session counts from outside cannot tell an agent partway through a turn from an idle one.
• Separate a binary update from a pod spec change when the process can update itself in place.

Limits of this account: it is reconstructed from the module’s code, comments, tests and commit history. We do not have a count of rotations performed, or of work interrupted before and after each fix. The incidents listed are the ones recorded in comments when their fixes landed.