Bringing Up a Firecracker MicroVM Sandbox Inside Kubernetes: 14 Bugs in 16 Hours

Our data engine runs dataflows, and some stages are user-defined functions: code that a user or an agent wrote and that we do not trust. Trusted operators run in-process. Untrusted functions run in a Firecracker microVM, one per invocation, so the function gets its own kernel, has no network, and reads its code and input from block devices. If no sandbox backend is registered, the executor refuses the stage.

By June every piece existed: the launcher crate, the backend, a guest root filesystem, a DaemonSet that stages kernel and rootfs onto each node, and tests for the engine-side contract. Yet no deployed executor had ever registered the backend, so every user-function stage in the cluster was refused. This post follows the work of closing that gap.

Between 12 June 2026, 21:31 UTC to 13 June 2026, 13:37 UTC, 10 fix commits landed before the first green end-to-end run. They fixed 14 separate bugs. All of the commits came from one coding agent working against the live development cluster, and the running log it kept in a plan file is the main source for this post.

Where the microVM can run

The original test job asked for the executor variant that runs under the Kata Containers Firecracker runtime class. That could never work. A Kata Firecracker pod is itself a Firecracker guest, and it has no /dev/kvm to start another VM. The backend can only register in an ordinary runc pod on a node with KVM.

So the change that enabled the sandbox added a fourth executor variant: plain runc, privileged for /dev/kvm and the jailer’s chroot and device setup, read-only host mounts for the Firecracker binaries and the staged guest images, and an emptyDir for the jailer’s chroot. That pod is trusted host-side machinery; the isolation boundary for user code is the per-invocation microVM. The same change made the stager’s symlinks relative, because its absolute targets embedded the stager pod’s own mount prefix and dangled from any other mount.

Scheduling and the control plane

• Nodes without the prerequisites. A check of the three dedicated nodes found only one fully ready. One could not reach object storage, so the stager never downloaded the images. The other had no Firecracker binaries. A node selector on the dedicated pool could land the pod on either, where its host-path mounts would fail and the pod would hang in ContainerCreating. The variant now selects a readiness label that is applied to a node by hand after its prerequisites are checked.
• Registration rejected by a new auth gate. A service-token requirement on the executor registry, which took effect in the development environment the day before, meant every executor variant, not only the new one, failed to register with “authorization metadata required”. Requests could no longer be routed to an executor by runtime header. The fix mounted the service token into all variants.
• A dead pod kept receiving work. The registry had no eviction. After the executor pod was replaced, its old registration stayed first in iteration order, and the first-match picker sent every task to the dead address. Registration snapshots now drop peers whose heartbeats have lapsed for three intervals, with a 30-second floor, and order the rest freshest first.

File ownership under the jailer

Firecracker’s jailer starts as root, builds a chroot, then drops to an unprivileged user before starting the VMM. Three permission failures came from that handoff.

• The jailer could not be spawned. The executor image runs as a non-root user, and the host’s jailer binary was root-owned with mode 0744. The jailer has to start as root anyway, so the variant now runs as root.
• The VMM could not be executed. The jailer copies the Firecracker binary into the chroot and keeps its original mode, so the unprivileged user could not execute the copy. An init container now stages 0755 copies of both binaries into a shared volume.
• The output drive could not be opened. Input and code drives were staged root-owned at 0644, which the unprivileged user can read. The output drive is read-write, so Firecracker failed with “Permission denied” and the guest never booted. The launcher now chowns every staged file to the jailer’s user and group.

A fourth launch bug had the same symptom. The jailer rejects instance IDs longer than 64 characters, and our ID was a four-character prefix plus a 64-character hash. Both this bug and the 0744 binary first appeared as “api.socket did not become ready within 10s”, because the launcher’s timeout path dropped the child’s stderr. The launcher now kills the child and includes its drained stderr in the error.

jailer/firecracker stderr: Invalid instance ID: Invalid len (68)

The guest’s init

With the jail working, the guest kernel started and immediately panicked. The guest bugs were found by booting the real root filesystem on the node and reading the serial console.

• The kernel never ran /init. When booting from a raw disk without an initramfs, the kernel tries /sbin/init, /etc/init, /bin/init and /bin/sh, but never /init. The rootfs now links /sbin/init to /init.
• Installed packages were missing. As PID 1, the init script started Python through env, and CPython then resolved its prefix to /, which left the dist-packages directory holding pyarrow off the module path. The script now calls the interpreter by absolute path.
• An extra argument. The host passes an exec-style argument vector whose first token is the program path. Forwarded to the runner script, that token became a stray positional argument. The init script now drops it.

Firecracker behavior we had assumed

The last three bugs came from assumptions about Firecracker itself. Two of them only appeared once the real executor path ran end to end.

• Drives are rounded down to whole sectors. Firecracker sizes a file-backed block device in 512-byte sectors and drops the remainder. A user function of about 250 bytes became an empty code drive. The backend now zero-pads every drive to a sector boundary, and the guest runner strips the trailing zero bytes before hashing the code.
• The API connection stays open. The launcher sent each API request with “Connection: close” and read the response until end of file. In our test, Firecracker v1.8.0 answered with keep-alive and a Content-Length header and held the socket open, so the first configuration request blocked forever. The VMM process was up, but the instance was never started. Earlier manual proofs had driven the API with curl, which frames responses correctly, so the launcher’s own client had not run against a real VMM before. The client now reads exactly Content-Length bytes, with a 10-second read timeout.
• Power-off does not stop the VMM. The guest ran the function, wrote a valid Arrow stream to its output drive and powered off. Firecracker kept running with a halted vCPU, and the executor’s wait never returned. Firecracker exits when the guest resets. Rebooting the guest stopped the VMM in about two seconds. The init script now reboots, with a SysRq reset and a kernel panic configured to reboot as fallbacks.

# Firecracker exits the VMM when the guest RESETS (reboot), not when it
# powers off ...
reboot -f 2>/dev/null
# Fallbacks if reboot -f didn't take: sysrq 'b' is an immediate reset;
# failing that, init exiting trips panic=1 -> reboot=k -> VMM exit.
echo b > /proc/sysrq-trigger 2>/dev/null

The fix that did not ship

The ownership fix was merged, but the deployed executor did not pick it up. The bring-up log first attributed the delay to a broken image push in CI. The actual cause: the executor image’s trigger paths did not list the launcher crate or two other statically linked crates, so CI never selected the image for a rebuild, and the cluster kept running the image from before the fix. Adding the three crates to the trigger paths shipped it. The same commit made the jailer ID unique per launch, because a fixed ID per root filesystem would collide in the chroot directory when two invocations ran at once.

After the reboot fix, the end-to-end job passed: five rows went through the data engine, executor, jailer, microVM, the Python runner and back, each with the expected output, in about three seconds. A post-deploy smoke hook was added later that day. It is report-only, so a failure does not block a sync.

What we would do again

Our inference from this bring-up, rather than measured results:

• Run the production code path before trusting a manual proof. Booting with curl verified the guest and skipped the launcher’s HTTP client, which held one of the last bugs.
• Return the child process’s stderr in launch errors. Two different causes looked identical as a socket timeout.
• Check a fix is in the running artifact before debugging further. One investigation chased a push failure while the image had never been rebuilt.
• List every user ID that touches a file when a privileged parent hands work to an unprivileged child. Three of the 14 bugs were file ownership.
• Give a report-only check an owner. When we checked on 27 October, the smoke hook’s latest run had failed at the data engine’s authorization gate, because the hook sends no service token. It never reached the sandbox. Being report-only, it blocked nothing. We have no record of when it started failing, and a fix is tracked separately.

Limits of this account: it is reconstructed from commit history, the bring-up log and the current code. We know the order of the fixes and their commit times, but not how many boot attempts or how much investigation each one took. The layer grouping is ours. We have not run a concurrency or performance test of the sandbox for this post.