a73x

45ac02ec

feat(agent): the host says whether it can run a guest, before create spends anything

a73x   2026-08-06 09:12

Commit message
feat(agent): the host says whether it can run a guest, before create spends anything

Reconcile asks the backend one question at the top of every create: can this
host run a guest at all. A host that can never boot one now refuses in its own
words, ahead of both the image fetch and the create throttle — before, a VM
misplaced onto a host with no VM runtime paid for a multi-gigabyte download and
then failed with an error naming the image toolchain rather than the host.

The refusal is permanent, so it is terminal in one attempt rather than three
identical ones, and it is asked per create rather than once at start, because a
host's answer can change while the agent runs.
docs/assumptions.md
Old New
@@ -124,6 +124,19 @@ that hosts stay on that LAN.
124 **Unverified**: one fleet host is a laptop that leaves the network, and a 124 **Unverified**: one fleet host is a laptop that leaves the network, and a
125 public hostname resolving to a public address would route the long way round. 125 public hostname resolving to a public address would route the long way round.
126 126
127 ### A backend that refuses preflight refuses everything after it
128
129 Reconcile asks the backend once, before the image fetch, whether this host can
130 run a guest at all, and treats a permanent refusal as the VM's verdict.
131 Underpins spending nothing—no download, no create slot—on a VM the host can
132 never boot, and reporting the host's own reason rather than the first expensive
133 step's symptom.
134 **Partly proven**: pinned in reconcile's tests, and `inert` is the only backend
135 that refuses today. It assumes refusal is a static property of the host; a
136 backend that could run guests only sometimes (a Mac whose VM entitlement comes
137 and goes) would need preflight consulted per attempt to stay honest, which is
138 why it is asked per create rather than once at start.
139
127 ### A gzipped image wraps a raw one 140 ### A gzipped image wraps a raw one
128 141
129 Gzip is transport, not a disk format, so a `.gz` download is decompressed and 142 Gzip is transport, not a disk format, so a `.gz` download is decompressed and
internal/agent/cloudhv/cloudhv.go
Old New
@@ -154,6 +154,13 @@ func permanentf(format string, args ...any) error {
154 return permanentError{err: fmt.Errorf(format, args...)} 154 return permanentError{err: fmt.Errorf(format, args...)}
155 } 155 }
156 156
157 // Preflight passes unconditionally: this backend exists only on a host that
158 // runs guests, and the one thing it needs beyond itself — the
159 // cloud-hypervisor binary — is installed by the agent's bootstrap step at
160 // start, long before any create. Re-checking it per create would trade a
161 // clear startup failure for a per-VM one.
162 func (p *Provisioner) Preflight(_ context.Context) error { return nil }
163
157 // PrepareRootDisk creates the VM's root disk by making a reflink copy of 164 // PrepareRootDisk creates the VM's root disk by making a reflink copy of
158 // basePath (instant on XFS/btrfs; silent full-copy fallback on ext4) and then 165 // basePath (instant on XFS/btrfs; silent full-copy fallback on ext4) and then
159 // truncating it to spec.DiskGB gigabytes. 166 // truncating it to spec.DiskGB gigabytes.
internal/agent/inert/inert.go
Old New
@@ -52,6 +52,12 @@ func refuse(cause error) error {
52 // refuses, and nothing is ever running. 52 // refuses, and nothing is ever running.
53 type Provisioner struct{} 53 type Provisioner struct{}
54 54
55 // Preflight is where a VM misplaced onto this host should now fail: reconcile
56 // asks it before the image fetch, so the answer an operator reads is this
57 // host's verdict rather than whatever the create's first expensive step
58 // happened to trip over.
59 func (Provisioner) Preflight(_ context.Context) error { return refuse(ErrNoRuntime) }
60
55 func (Provisioner) PrepareRootDisk(_ context.Context, _ state.VMSpec, _ string) error { 61 func (Provisioner) PrepareRootDisk(_ context.Context, _ state.VMSpec, _ string) error {
56 return refuse(ErrNoRuntime) 62 return refuse(ErrNoRuntime)
57 } 63 }
internal/agent/inert/inert_test.go
Old New
@@ -36,6 +36,7 @@ func TestProvisionerRefusesEveryLifecycleCallPermanently(t *testing.T) {
36 ctx := context.Background() 36 ctx := context.Background()
37 p := Provisioner{} 37 p := Provisioner{}
38 calls := map[string]error{ 38 calls := map[string]error{
39 "Preflight": p.Preflight(ctx),
39 "PrepareRootDisk": p.PrepareRootDisk(ctx, state.VMSpec{}, "/base.img"), 40 "PrepareRootDisk": p.PrepareRootDisk(ctx, state.VMSpec{}, "/base.img"),
40 "Boot": p.Boot(ctx, "vm-1", state.VMSpec{}), 41 "Boot": p.Boot(ctx, "vm-1", state.VMSpec{}),
41 "Shutdown": p.Shutdown(ctx, "vm-1"), 42 "Shutdown": p.Shutdown(ctx, "vm-1"),
internal/agent/reconcile/reconcile.go
Old New
@@ -56,6 +56,17 @@ import (
56 // mechanism and do not generalize. What generalizes is the data: the 56 // mechanism and do not generalize. What generalizes is the data: the
57 // deterministic MAC, the host's subnet, and a VM's current address. 57 // deterministic MAC, the host's subnet, and a VM's current address.
58 type Provisioner interface { 58 type Provisioner interface {
59 // Preflight reports whether this backend can run a guest on this host at
60 // all. It is asked once per create attempt, before any expensive work, so
61 // that a host which can never boot a guest refuses in its own words instead
62 // of failing later at whatever step happens to notice first — on a host with
63 // no VM runtime, that was a multi-gigabyte image download followed by an
64 // error naming the image toolchain rather than the host.
65 //
66 // nil means the backend is willing to try; a Permanent error terminal-fails
67 // the VM in one attempt, exactly as a refusal from the lifecycle verbs does.
68 Preflight(ctx context.Context) error
69
59 // PrepareRootDisk materialises the VM's root disk from a base image 70 // PrepareRootDisk materialises the VM's root disk from a base image
60 // (clone + grow). It is root-disk-only by contract: reconcile's rebuild 71 // (clone + grow). It is root-disk-only by contract: reconcile's rebuild
61 // path calls create again, so routing a user volume through it would 72 // path calls create again, so routing a user volume through it would
@@ -648,6 +659,15 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
648 return 659 return
649 } 660 }
650 661
662 // Ask the backend whether this host can run a guest before spending anything
663 // on finding out. Ahead of the throttle as well as the fetch: a create that
664 // can only be refused should not queue for a slot that a create which might
665 // succeed could be using.
666 if err := e.Prov.Preflight(ctx); err != nil {
667 e.failCreate(ctx, rec, err, res)
668 return
669 }
670
651 // Throttle: cap how many VMs on this host may be inside the I/O-heavy part 671 // Throttle: cap how many VMs on this host may be inside the I/O-heavy part
652 // of create at once. Per-VM workers made these concurrent — N simultaneous 672 // of create at once. Per-VM workers made these concurrent — N simultaneous
653 // creates mean N image downloads, N image decodes and N multi-GB disk 673 // creates mean N image downloads, N image decodes and N multi-GB disk
internal/agent/reconcile/reconcile_test.go
Old New
@@ -36,9 +36,10 @@ type fakeProv struct {
36 booted []string 36 booted []string
37 shutdown []string 37 shutdown []string
38 destroyed []string 38 destroyed []string
39 prepErr error 39 prepErr error
40 bootErr error // one-shot: consumed and cleared on first Boot call 40 bootErr error // one-shot: consumed and cleared on first Boot call
41 destroyErr error // sticky: every Destroy fails until it is cleared 41 destroyErr error // sticky: every Destroy fails until it is cleared
42 preflightErr error // sticky: the backend refuses this host outright
42 43
43 cidr string 44 cidr string
44 addrs map[string]string // vmID -> ip (sticky, mirrors the DHCP table) 45 addrs map[string]string // vmID -> ip (sticky, mirrors the DHCP table)
@@ -57,6 +58,12 @@ func newFakeProv() *fakeProv {
57 } 58 }
58 } 59 }
59 60
61 func (f *fakeProv) Preflight(_ context.Context) error {
62 f.mu.Lock()
63 defer f.mu.Unlock()
64 return f.preflightErr
65 }
66
60 func (f *fakeProv) PrepareRootDisk(_ context.Context, s state.VMSpec, _ string) error { 67 func (f *fakeProv) PrepareRootDisk(_ context.Context, s state.VMSpec, _ string) error {
61 f.mu.Lock() 68 f.mu.Lock()
62 defer f.mu.Unlock() 69 defer f.mu.Unlock()
@@ -637,6 +644,41 @@ func TestPermanentCreateErrorFailsTerminallyInOneAttempt(t *testing.T) {
637 assert.Equal(t, 1, f.prov.prepCalls, "no further provisioner attempts after a permanent failure") 644 assert.Equal(t, 1, f.prov.prepCalls, "no further provisioner attempts after a permanent failure")
638 } 645 }
639 646
647 // TestPreflightRefusalFailsBeforeTheImageFetch is the ordering this seam exists
648 // for. A host that cannot run guests must say so before create spends anything:
649 // the image fetch is a multi-gigabyte download, and letting it run first means
650 // the operator reads whatever it tripped over — an image error, say — instead
651 // of the host's own verdict.
652 func TestPreflightRefusalFailsBeforeTheImageFetch(t *testing.T) {
653 f := setup(t)
654 var fetches int
655 f.eng.Images = func(context.Context, string, string) (string, error) {
656 fetches++
657 return "/cache/x.raw", nil
658 }
659 f.prov.preflightErr = permErr{"no VM runtime on this host (darwin/arm64)"}
660
661 rep := f.step(snap(1, vm("vm1")))
662 row := findVM(rep, "vm1")
663 require.NotNil(t, row)
664 assert.Equal(t, "failed", row.Phase, "a permanent refusal is terminal on attempt 1")
665 assert.Contains(t, row.LastError, "no VM runtime on this host",
666 "the reported error must be the host's verdict, not a later step's symptom")
667 assert.Zero(t, fetches, "must not download an image for a VM this host can never boot")
668 assert.Zero(t, f.prov.prepCalls, "must not reach the disk either")
669 }
670
671 // TestPreflightPassingLeavesCreateUnchanged guards the other direction: the new
672 // gate must be invisible on a host that can run guests.
673 func TestPreflightPassingLeavesCreateUnchanged(t *testing.T) {
674 f := setup(t)
675 rep := f.step(snap(1, vm("vm1")))
676 row := findVM(rep, "vm1")
677 require.NotNil(t, row)
678 assert.Equal(t, "ready", row.Phase)
679 assert.Equal(t, []string{"vm1"}, f.prov.booted)
680 }
681
640 // TestTwoVMsCreatedInOneStepGetDistinctIPs pins that when a single Step creates 682 // TestTwoVMsCreatedInOneStepGetDistinctIPs pins that when a single Step creates
641 // two VMs, each gets a distinct address. The backend owns the used-address set 683 // two VMs, each gets a distinct address. The backend owns the used-address set
642 // under its own lock (netenv's DHCP table does; the fake mirrors it), so two 684 // under its own lock (netenv's DHCP table does; the fake mirrors it), so two
internal/agent/syncclient/client_test.go
Old New
@@ -30,6 +30,7 @@ import (
30 // can drive a real Client.Run against a real syncsvc server over QUIC loopback. 30 // can drive a real Client.Run against a real syncsvc server over QUIC loopback.
31 type noopProv struct{} 31 type noopProv struct{}
32 32
33 func (noopProv) Preflight(context.Context) error { return nil }
33 func (noopProv) PrepareRootDisk(context.Context, state.VMSpec, string) error { return nil } 34 func (noopProv) PrepareRootDisk(context.Context, state.VMSpec, string) error { return nil }
34 func (noopProv) Boot(context.Context, string, state.VMSpec) error { return nil } 35 func (noopProv) Boot(context.Context, string, state.VMSpec) error { return nil }
35 func (noopProv) Shutdown(context.Context, string) error { return nil } 36 func (noopProv) Shutdown(context.Context, string) error { return nil }