fd31fa9e

feat(agent): reconcile each VM in its own worker

a73x   2026-07-26 08:24

feat(agent): reconcile each VM in its own worker

The agent drove every VM on a host in one synchronous pass, so a slow
operation on one VM — an image fetch, a disk copy, a boot — delayed every
other VM and, critically, the host's ActualStateReport. That report is the
heartbeat: the control plane marks a host offline after 30s of silence, so
one slow VM could flap an entire healthy host offline.

Each VM now has its own long-lived worker goroutine. Step keeps its
signature and becomes a router:

  fence stale snapshots
    -> dispatch one assignment per VM to that VM's worker (spawn/reap)
    -> aggregate each worker's last-published result, never waiting

Serialization of a single VM's operations is structural — one goroutine per
VM, no per-VM lock. Everything shared between VMs is guarded where it lives:
the compute ledger under Engine.mu, the DHCP reservation table under the dhcp
server's own lock, and the state store by one file per VM written via rename.

Two deliberate consequences. The report is last-known rather than post-tick,
so a busy VM contributes its previous row — that is precisely what makes the
heartbeat unblockable. And compute freed by quarantining one VM reaches a
sibling on a later tick rather than the same one; a quota refusal is
non-terminal and the loop is level-triggered, so it self-heals.

The watchdog moves from bounding a whole host step to bounding one VM's pass
(StepTimeout -> VMTimeout, -step-timeout -> -vm-timeout), which also stops a
wedged VM from starving a sibling of its turn.

Concurrent creates are bounded so they cannot saturate the disk the heartbeat
path also reads: MaxConcurrentCreates (-max-concurrent-creates, default 4)
gates the I/O-heavy region of create, and imagecache collapses concurrent
fetches of the same image into a single download via singleflight. Abandoned
download and convert temporaries are swept at agent start.

Two properties the worker model leans on:

  - state.Get reports not-found only for IsNotExist: an unreadable record is
    not an absent one, so a pass that cannot observe its own record changes
    nothing and keeps its last-known row. Anything else would route a live VM
    back through create(), rebuilding its disk under a running guest and
    orphaning its hypervisor process.

  - A published vmResult is aliased by every report taken between two
    publishes, so aggregate clones each proto row into the report rather than
    depending on how many goroutines might marshal it.

syncclient is unchanged.