a73x

b5e82f75

comments: a claim a test already fails on is not one

a73x   2026-08-23 09:27

Commit message
comments: a claim a test already fails on is not one

Every comment in eight packages was asked one question: mutate what it claims
and see what breaks. A test that fails is the alarm, and only one of the two
can rot — so the prose goes and the test keeps the claim, which is why the
test names here read as sentences. A compile error is louder still but cannot
say why, so those comments stay and now say so. ST1020 does the same for an
exported symbol's summary line. What no alarm reaches is either a shape the
code should carry or a contract a reader outside the code cannot run: what a
wire field means, what an old agent does on absence, what macOS will not let
a Linux test observe.

546 lines go. The number is not the point — where they go from is. Deletions
land on the tested half: guestAddr and resolveGuestCIDR sit near full
coverage and give up nearly every line, while RunCLI and newPlatform sit at
zero and give up none. capacity.go and networks.go give up nothing at all,
having been written this way from the start.

Lifecycle claimed the control loops never read it and nothing enforced that;
an arch test now walks agent, store, syncsvc and hub and names the axes to
branch on instead. RevokedCert.Serial keeps its comment for the opposite
reason: widening it is a compile error at call sites two files away.
internal/agent/netenv/named.go
Old New
@@ -13,8 +13,6 @@ import (
13 "github.com/a73x/eitri/internal/agent/state" 13 "github.com/a73x/eitri/internal/agent/state"
14 ) 14 )
15 15
16 // sysfsIsBridge reports whether name is a bridge device:
17 // /sys/class/net/<name>/bridge is a directory that exists for exactly those.
18 // Bridge-ness is checked rather than assumed because attaching a tap to a link 16 // Bridge-ness is checked rather than assumed because attaching a tap to a link
19 // that is not a bridge fails later with a raw RTNETLINK error naming neither 17 // that is not a bridge fails later with a raw RTNETLINK error naming neither
20 // the network nor the flag that asked for it. 18 // the network nor the flag that asked for it.
@@ -26,12 +24,10 @@ func sysfsIsBridgeAt(root, name string) bool {
26 return err == nil && fi.IsDir() 24 return err == nil && fi.IsDir()
27 } 25 }
28 26
29 // VerifyNetworks confirms every configured network's bridge exists and is a 27 // VerifyNetworks refuses to start when a configured network's bridge is absent.
30 // bridge, before any VM can be asked for. eitri never creates these — a missing 28 // eitri never creates operator bridges — a missing one is a netplan/networkd
31 // one is the operator's netplan/networkd change not yet made, and the only 29 // change not yet made. Refusing at startup beats coming up and advertising a
32 // honest response is a startup failure that names it. Coming up anyway would 30 // network this host cannot honor, which fails every VM placed on it instead.
33 // advertise a network this host cannot honor, and every VM placed on it would
34 // fail one at a time instead.
35 func (n *Net) VerifyNetworks() error { 31 func (n *Net) VerifyNetworks() error {
36 for name, br := range n.networks { 32 for name, br := range n.networks {
37 if !n.isBridge(br) { 33 if !n.isBridge(br) {
@@ -43,58 +39,35 @@ func (n *Net) VerifyNetworks() error {
43 return nil 39 return nil
44 } 40 }
45 41
46 // blockGuestDHCP installs the one rule a host that lends out an operator bridge 42 // blockGuestDHCP drops DHCPv4 server talk entering an operator bridge from a
47 // owes it: a frame entering that bridge from an eitri guest tap with UDP source 43 // guest tap. One rule serves any number of named networks: it keys on the
48 // port 67 — DHCPv4 server talk — is dropped. Called from EnsureBridge, so it is 44 // eil- tap prefix, not on the bridge.
49 // reprogrammed on every agent start; a host with no configured network runs
50 // none of it and its ruleset stays exactly what it was.
51 // 45 //
52 // One rule covers a host serving any number of named networks, because it keys 46 // It buys two things. The first is the half of the address snoop no socket can
53 // on the tap-name prefix and not on the bridge: every named tap this agent 47 // settle: netsnoop accepts only frames the kernel marks outgoing, which proves
54 // creates is eil-something, whichever operator bridge it was enslaved to. 48 // a guest cannot mint its own address — but an ACK one guest unicasts at a
49 // neighbour's MAC is forwarded by the bridge and leaves the neighbour tap
50 // outgoing too, indistinguishable there from the site answering. Prerouting is
51 // upstream of the bridge's port choice, so the frame never reaches a sibling
52 // guest or the physical uplink and the question cannot be asked. The second is
53 // plain LAN protection: a guest handing out leases would address the operator's
54 // real machines, and eitri is what put it on their network.
55 // 55 //
56 // It buys two separate things. The first is the half of the address snoop that 56 // DHCPv6 and IPv6 RAs are deliberately unfiltered: a bridged guest is meant to
57 // no socket can settle by itself: internal/agent/netsnoop accepts only frames 57 // be a full peer, an operator may run a router guest on purpose, and the snoop
58 // the kernel marks outgoing, which proves a guest cannot mint its own address, 58 // reads IPv4 ACKs only.
59 // but an ACK one guest unicasts at a neighbour's MAC is forwarded by the bridge
60 // and leaves the neighbour tap outgoing too, indistinguishable there from the
61 // site answering. Dropping it at ingress settles it before the question can be
62 // asked — the frame never reaches the bridge, so there is nothing to forward.
63 // The second is plain protection for the LAN: a guest handing out leases would
64 // otherwise address the operator's real machines, and eitri is what put it on
65 // their network.
66 //
67 // Ingress at the tap is the hook point precisely because it is upstream of
68 // every path the frame could take. A drop at prerouting happens before the
69 // bridge picks a port, so the forged lease reaches neither a sibling guest nor
70 // the physical uplink and the switch beyond it.
71 //
72 // DHCPv6 (source port 547) and IPv6 router advertisements are deliberately not
73 // filtered, and the omission is the design rather than a gap in it: a bridged
74 // guest is meant to be a full peer on the operator's LAN, an operator may run a
75 // router guest on purpose, and neither v6 path can poison anything eitri
76 // records, since the snoop reads IPv4 ACKs only.
77 func (n *Net) blockGuestDHCP(ctx context.Context) error { 59 func (n *Net) blockGuestDHCP(ctx context.Context) error {
78 if len(n.networks) == 0 { 60 if len(n.networks) == 0 {
79 return nil 61 return nil
80 } 62 }
81 // Same idempotency as the NAT chain: add table/add chain are no-ops when the 63 // Two invocations are two transactions, so a restart on a host with live
82 // object is already there, and the flush before the rule keeps restarts from 64 // taps leaves the chain empty between them. The fix is one nft -f
83 // stacking copies of it. Two invocations are two transactions, though, so a 65 // transaction, which the NAT chain needs identically.
84 // restart on a host with live taps leaves the chain empty for the moment
85 // between them; the honest fix is one nft -f transaction, which the NAT chain
86 // needs identically and which belongs to its own change rather than this one.
87 steps := [][]string{ 66 steps := [][]string{
88 {"nft", "add", "table", "bridge", "eitri"}, 67 {"nft", "add", "table", "bridge", "eitri"},
89 // The priority is the number and not the name filter, which is -200 in the
90 // bridge family but 0 in the ip family the NAT chain above lives in: the
91 // number reads the same whichever table a reader has in mind.
92 {"nft", "add", "chain", "bridge", "eitri", "prerouting", 68 {"nft", "add", "chain", "bridge", "eitri", "prerouting",
93 "{ type filter hook prerouting priority -200 ; }"}, 69 "{ type filter hook prerouting priority -200 ; }"},
94 {"nft", "flush", "chain", "bridge", "eitri", "prerouting"}, 70 {"nft", "flush", "chain", "bridge", "eitri", "prerouting"},
95 // The quotes around the pattern are nft syntax, not shell quoting that a
96 // runner without a shell would have eaten: nft re-lexes its arguments
97 // joined, and an unquoted trailing star is a parse error there.
98 {"nft", "add", "rule", "bridge", "eitri", "prerouting", 71 {"nft", "add", "rule", "bridge", "eitri", "prerouting",
99 "iifname", fmt.Sprintf("%q", netTapPrefix+"*"), "udp", "sport", "67", "drop"}, 72 "iifname", fmt.Sprintf("%q", netTapPrefix+"*"), "udp", "sport", "67", "drop"},
100 } 73 }
@@ -107,13 +80,8 @@ func (n *Net) blockGuestDHCP(ctx context.Context) error {
107 } 80 }
108 81
109 // createNamedTap is CreateTap's second half for a guest that asked for a named 82 // createNamedTap is CreateTap's second half for a guest that asked for a named
110 // network: a SECOND tap -> enslave to the operator's bridge -> up -> start the 83 // network. No reservation, no masquerade, no forced DNS — the site's DHCP
111 // address snoop. No reservation, no masquerade, no forced DNS — the site's DHCP 84 // server owns all three on this NIC.
112 // server owns all three on this NIC. The guest's first tap, on eitri0, was
113 // created before this ran and is untouched by it.
114 //
115 // The network is known to be configured: CreateTap refuses an unknown one
116 // before it creates anything at all.
117 func (n *Net) createNamedTap(ctx context.Context, vmID, network string) error { 85 func (n *Net) createNamedTap(ctx context.Context, vmID, network string) error {
118 bridge := n.networks[network] 86 bridge := n.networks[network]
119 tap := n.NetTapName(vmID) 87 tap := n.NetTapName(vmID)
@@ -133,24 +101,17 @@ func (n *Net) createNamedTap(ctx context.Context, vmID, network string) error {
133 n.mu.Lock() 101 n.mu.Lock()
134 n.attachLocked(vmID) 102 n.attachLocked(vmID)
135 n.mu.Unlock() 103 n.mu.Unlock()
136 // A snoop that will not start fails the create. The named NIC's address 104 // The named NIC's address arrives only via the snoop, so a guest booted
137 // arrives only this way, so a guest booted without one is a guest whose LAN 105 // without one is a guest whose LAN address nothing above the host can ever
138 // address nothing above the host can ever learn — visibly failing the 106 // learn. Failing the create beats reporting a silently half-reported VM.
139 // create beats reporting a networked guest that is silently half-reported.
140 // The one refusal that is not a failure is the VM being deleted while this
141 // ran: that takes the attachment with it, and startSnoop leaves a guest on
142 // its way out unwatched rather than arming a goroutine for a dead tap.
143 //nolint:contextcheck // the snoop's lifetime is the tap's, not this call's: inheriting ctx would end discovery the moment CreateTap returned 107 //nolint:contextcheck // the snoop's lifetime is the tap's, not this call's: inheriting ctx would end discovery the moment CreateTap returned
144 return n.startSnoop(vmID, tap) 108 return n.startSnoop(vmID, tap)
145 } 109 }
146 110
147 // attachLocked returns vmID's attachment, creating it if the VM has none. 111 // attachLocked returns vmID's attachment, creating it if the VM has none.
148 // Called with n.mu held. 112 // Called with n.mu held. It keeps any existing entry because a Boot retry
149 // 113 // re-runs createNamedTap on a guest whose snoop is live, and neither that
150 // It keeps any existing entry rather than replacing it, because both callers 114 // snoop nor the address it found may be dropped by the re-attach.
151 // can arrive at a VM that is already attached: a Boot retry re-runs
152 // createNamedTap on a guest whose snoop is live, and neither that snoop nor the
153 // address it has already found may be dropped on the floor by the re-attach.
154 func (n *Net) attachLocked(vmID string) *attachment { 115 func (n *Net) attachLocked(vmID string) *attachment {
155 a, live := n.attached[vmID] 116 a, live := n.attached[vmID]
156 if !live { 117 if !live {
@@ -160,13 +121,10 @@ func (n *Net) attachLocked(vmID string) *attachment {
160 return a 121 return a
161 } 122 }
162 123
163 // startSnoop begins DHCP-ACK discovery for vmID on tap, the VM's named-network 124 // startSnoop begins DHCP-ACK discovery for vmID's named-network tap. It keys on
164 // one. It keys on state.NetMAC — the second NIC's address — so the guest's NAT 125 // state.NetMAC — the second NIC's address — so the VM's NAT lease, which this
165 // lease, which crosses a different tap and this host granted itself, can never 126 // host granted itself over a different tap, can never be mistaken for what the
166 // be mistaken for what the site's server said. Idempotent per VM: a second call 127 // site's server said.
167 // while a snoop is live is a no-op, which is what makes it safe on a Boot retry
168 // and on the restart replay. A VM with no attachment has no NIC to watch, and
169 // gets no listener.
170 func (n *Net) startSnoop(vmID, tap string) error { 128 func (n *Net) startSnoop(vmID, tap string) error {
171 n.mu.Lock() 129 n.mu.Lock()
172 a, live := n.attached[vmID] 130 a, live := n.attached[vmID]
@@ -184,11 +142,10 @@ func (n *Net) startSnoop(vmID, tap string) error {
184 n.mu.Unlock() 142 n.mu.Unlock()
185 143
186 if err := n.listen(ctx, tap, mac, n.noteDiscovered(vmID)); err != nil { 144 if err := n.listen(ctx, tap, mac, n.noteDiscovered(vmID)); err != nil {
187 // Leave nothing armed behind a failure: the cancel above would make the 145 // Leave nothing armed: the cancel above would make a retry believe a
188 // retry believe a snoop is already running and skip starting one. Only 146 // snoop is running. Only this attachment is disarmed — a VM deleted and
189 // this attachment is disarmed — a VM deleted and re-created while the 147 // re-created while the listener was refusing has a new entry, and that
190 // listener was refusing has a new entry, and that one's snoop is not 148 // one's snoop is not this failure's to un-arm.
191 // this failure's to un-arm.
192 n.mu.Lock() 149 n.mu.Lock()
193 if cur, live := n.attached[vmID]; live && cur == a { 150 if cur, live := n.attached[vmID]; live && cur == a {
194 cur.stopSnoop = nil 151 cur.stopSnoop = nil
@@ -201,10 +158,9 @@ func (n *Net) startSnoop(vmID, tap string) error {
201 } 158 }
202 159
203 // NetworkAddress returns the address the site's DHCP server granted this VM's 160 // NetworkAddress returns the address the site's DHCP server granted this VM's
204 // named-network NIC, or "" — for a VM that has no such NIC, and for a VM whose 161 // named-network NIC, or "". Empty is "not yet known", never "unreachable": the
205 // guest has not yet completed a DHCP exchange on it. Empty is "not yet known", 162 // VM's NAT address (Address) is what eitri reaches it by, and that exists from
206 // never "unreachable": the VM's NAT address (Address) is what eitri itself 163 // boot.
207 // reaches it by, and that one exists from boot.
208 func (n *Net) NetworkAddress(vmID string) string { 164 func (n *Net) NetworkAddress(vmID string) string {
209 n.mu.Lock() 165 n.mu.Lock()
210 defer n.mu.Unlock() 166 defer n.mu.Unlock()
@@ -214,15 +170,12 @@ func (n *Net) NetworkAddress(vmID string) string {
214 return "" 170 return ""
215 } 171 }
216 172
217 // noteDiscovered builds vmID's snoop callback. It records what the ACK granted, 173 // noteDiscovered records what an ACK granted, but only while the VM is still
218 // but only while the VM is still attached: the listener checks its context 174 // attached: the listener checks its context between reads, so a frame from the
219 // between reads, so a frame that arrived in the last poll window (~1s) can call 175 // last poll window (~1s) can call back after DeleteTap cancelled it, and a
220 // back after DeleteTap already cancelled it, and a deleted VM must not come 176 // deleted VM must not come back holding an address. It looks the attachment up
221 // back holding an address. 177 // on every ACK rather than closing over the entry — an entry DeleteTap dropped
222 // 178 // is a place nothing may still be writing to.
223 // The callback closes over the id and looks the attachment up on every ACK,
224 // never over the entry itself: an entry DeleteTap dropped is a place nothing
225 // may still be writing to, and only the map can say whether it is still there.
226 func (n *Net) noteDiscovered(vmID string) func(ip string) { 179 func (n *Net) noteDiscovered(vmID string) func(ip string) {
227 return func(ip string) { 180 return func(ip string) {
228 n.mu.Lock() 181 n.mu.Lock()
@@ -233,24 +186,16 @@ func (n *Net) noteDiscovered(vmID string) func(ip string) {
233 } 186 }
234 } 187 }
235 188
236 // AdoptNetwork rebuilds a networked VM's in-memory discovery state after an 189 // AdoptNetwork rebuilds a networked VM's discovery state after an agent
237 // agent restart: the record's last known named-NIC address (so the fleet keeps 190 // restart. It says nothing about the VM's NAT attachment — that half is
238 // reading it before the next DHCP renewal is snooped) and, when the tap is 191 // AddReservation's, and the replay calls both for the same guest.
239 // still up, a fresh snoop. It says nothing about the VM's NAT attachment —
240 // that half is AddReservation's, and the replay calls both for the same guest.
241 //
242 // It adopts the VM whether or not this agent still serves the network named in
243 // the record, and that is the kind answer rather than an oversight: an agent
244 // restarted without one of its --host-network flags is looking at a guest that
245 // is still running, still holding the tap the previous agent enslaved, still
246 // answering on the address the site gave it. Forgetting the attachment would
247 // take that address off the VM's report and disarm the snoop that keeps it
248 // current, punishing the guest for a change made on the host.
249 // 192 //
250 // What it will not do is stay quiet about it. The VM survives on borrowed 193 // It adopts the VM even when this agent no longer serves the named network.
251 // configuration — CreateTap refuses the network permanently now, so the guest 194 // That guest is still running, still holding the tap, still answering on the
252 // fails the next time it is booted — and the only moment an operator can be 195 // site's address; forgetting it would take that address off the VM's report and
253 // told before that happens is this one, so the drift is warned here. 196 // disarm the snoop, punishing the guest for a change made on the host. It
197 // survives on borrowed configuration — CreateTap refuses the network now, so
198 // the next boot fails — and this is the only moment to say so.
254 func (n *Net) AdoptNetwork(vmID, network, ip string) { 199 func (n *Net) AdoptNetwork(vmID, network, ip string) {
255 n.mu.Lock() 200 n.mu.Lock()
256 a := n.attachLocked(vmID) 201 a := n.attachLocked(vmID)
@@ -266,17 +211,11 @@ func (n *Net) AdoptNetwork(vmID, network, ip string) {
266 } 211 }
267 tap := n.NetTapName(vmID) 212 tap := n.NetTapName(vmID)
268 if n.isTap(tap) { 213 if n.isTap(tap) {
269 // No tap means the VM is not running; Boot creates both it and the 214 // A snoop that refuses to start is not worth failing agent startup
270 // snoop. A snoop that refuses to start here is not worth failing the 215 // over: the address above is already reported, and the VM's own
271 // agent's startup over: the address above is already reported, and the 216 // converge is where a broken tap becomes a story. The cost is real
272 // VM's own converge is where a broken tap becomes a story. 217 // though — an already-running VM keeps no re-arm path until it reboots,
273 // 218 // and reports the replayed address until then.
274 // The cost is bounded but real: startSnoop leaves nothing armed after a
275 // failure, so a VM that is later booted re-arms — but a VM already
276 // running keeps no re-arm path until it reboots or the agent restarts,
277 // and reports the replayed address until then. That address is almost
278 // certainly still the guest's: the MAC is derived from the VM id and
279 // the LAN's lease table hands the same one back.
280 _ = n.startSnoop(vmID, tap) 219 _ = n.startSnoop(vmID, tap)
281 } 220 }
282 } 221 }
internal/agent/reconcile/reconcile.go
Old New
@@ -2,11 +2,8 @@
2 // 2 //
3 // Definitions (normative, from the spec): 3 // Definitions (normative, from the spec):
4 // 4 //
5 // exists = state-dir record present AND create completed (rec.BootID != "")
6 // lost = boot ID changed OR process died without a recorded stop request; 5 // lost = boot ID changed OR process died without a recorded stop request;
7 // a deliberately stopped VM is stopped, NOT lost 6 // a deliberately stopped VM is stopped, NOT lost
8 // destroyed[] ack = level-triggered: every tombstoned vm_id with no local
9 // record, repeated until the server hard-deletes it
10 // 7 //
11 // Shape: Step is a router, not a worker. It fences stale snapshots, hands each 8 // Shape: Step is a router, not a worker. It fences stale snapshots, hands each
12 // VM its slice of desired state to that VM's own long-lived goroutine, and 9 // VM its slice of desired state to that VM's own long-lived goroutine, and
@@ -15,15 +12,7 @@
15 // goroutine per VM is also the serialization primitive: a single VM's 12 // goroutine per VM is also the serialization primitive: a single VM's
16 // operations are serial by construction. See worker.go. 13 // operations are serial by construction. See worker.go.
17 // 14 //
18 // The "exists" definition deserves a comment: 15 // Belt-and-suspenders: PrepareRootDisk and seed.Build both
19 // rec.BootID is the sole completion witness. create() sets it to the current
20 // host boot ID only after every side effect (image, disk, seed, boot) has
21 // succeeded, so rec.BootID != "" means — and only means — a create finished.
22 // Disk presence is deliberately NOT consulted: create() writes disk.raw in the
23 // middle of the sequence, so a create that fails after the disk is written but
24 // before boot leaves a disk on a still-empty BootID; treating that as "exists"
25 // would divert the retry to converge() (which rebuilds neither disk nor seed)
26 // and strand the VM. Belt-and-suspenders: PrepareRootDisk and seed.Build both
27 // write via a temp file + rename, so a killed create can never leave a torn artifact. 16 // write via a temp file + rename, so a killed create can never leave a torn artifact.
28 package reconcile 17 package reconcile
29 18
@@ -57,14 +46,7 @@ import (
57 // deterministic MAC, the host's subnet, and a VM's current address. 46 // deterministic MAC, the host's subnet, and a VM's current address.
58 type Provisioner interface { 47 type Provisioner interface {
59 // Preflight reports whether this backend can run a guest on this host at 48 // Preflight reports whether this backend can run a guest on this host at
60 // all. It is asked once per create attempt, before any expensive work, so 49 // all.
61 // that a host which can never boot a guest refuses in its own words instead
62 // of failing later at whatever step happens to notice first — on a host with
63 // no VM runtime, that was a multi-gigabyte image download followed by an
64 // error naming the image toolchain rather than the host.
65 //
66 // nil means the backend is willing to try; a Permanent error terminal-fails
67 // the VM in one attempt, exactly as a refusal from the lifecycle verbs does.
68 Preflight(ctx context.Context) error 50 Preflight(ctx context.Context) error
69 51
70 // PrepareRootDisk materialises the VM's root disk from a base image 52 // PrepareRootDisk materialises the VM's root disk from a base image
@@ -84,10 +66,6 @@ type Provisioner interface {
84 66
85 // Destroy stops the VM and releases every host resource it holds — its 67 // Destroy stops the VM and releases every host resource it holds — its
86 // process and its network attachment, address reservation included. 68 // process and its network attachment, address reservation included.
87 // Idempotent: it is called on every tick past a VM's grace until it
88 // returns nil. A non-nil error KEEPS the VM's record so a later tick
89 // retries; returning nil is the backend's promise that nothing is left to
90 // reap, and reconcile deletes the record on the strength of it.
91 Destroy(ctx context.Context, vmID string) error 69 Destroy(ctx context.Context, vmID string) error
92 70
93 Running(vmID string) bool 71 Running(vmID string) bool
@@ -97,17 +75,13 @@ type Provisioner interface {
97 // should be running is not, to give that report a cause: the process table 75 // should be running is not, to give that report a cause: the process table
98 // can say a guest is gone but never why, and every backend already keeps its 76 // can say a guest is gone but never why, and every backend already keeps its
99 // hypervisor's own output on disk. 77 // hypervisor's own output on disk.
100 //
101 // Empty means "nothing to add", NOT "nothing went wrong" — a guest killed
102 // by a host reboot leaves no complaint behind and is still lost.
103 FailureReason(vmID string) string 78 FailureReason(vmID string) string
104 79
105 // Address returns the VM's current guest address, or "" when the backend 80 // Address returns the VM's current guest address, or "" when the backend
106 // does not know one (never booted, or gone). It is POLLED rather than 81 // does not know one (never booted, or gone). It is POLLED rather than
107 // returned by Boot: where the host OS's own DHCP server hands out the 82 // returned by Boot: where the host OS's own DHCP server hands out the
108 // address, it does not exist until the guest has booted and asked for one, 83 // address, it does not exist until the guest has booted and asked for one,
109 // and Boot must not block inside the create slot waiting for that. Empty 84 // and Boot must not block inside the create slot waiting for that.
110 // means "no answer", never "no address" — see noteAddress.
111 Address(vmID string) string 85 Address(vmID string) string
112 86
113 // NetworkAddress returns the address the VM's SECOND NIC — the one on the 87 // NetworkAddress returns the address the VM's SECOND NIC — the one on the
@@ -119,7 +93,7 @@ type Provisioner interface {
119 // Separate from Address rather than replacing it because the two are 93 // Separate from Address rather than replacing it because the two are
120 // different facts with different timings: Address is allocated by the host 94 // different facts with different timings: Address is allocated by the host
121 // and known at boot, this one is granted by someone else's DHCP server and 95 // and known at boot, this one is granted by someone else's DHCP server and
122 // discovered afterwards. Empty means "not yet known", never "unreachable". 96 // discovered afterwards.
123 NetworkAddress(vmID string) string 97 NetworkAddress(vmID string) string
124 } 98 }
125 99
@@ -184,7 +158,7 @@ type Engine struct {
184 158
185 // MaxConcurrentCreates caps how many VMs on this host may be inside the 159 // MaxConcurrentCreates caps how many VMs on this host may be inside the
186 // I/O-heavy part of create at once (image fetch, disk materialisation). 160 // I/O-heavy part of create at once (image fetch, disk materialisation).
187 // Zero means unlimited. Per-VM workers made creates concurrent; this bounds 161 // Per-VM workers made creates concurrent; this bounds
188 // how much of that concurrency reaches the disk. See acquireCreateSlot. 162 // how much of that concurrency reaches the disk. See acquireCreateSlot.
189 MaxConcurrentCreates int 163 MaxConcurrentCreates int
190 164
@@ -194,19 +168,14 @@ type Engine struct {
194 createSlots chan struct{} 168 createSlots chan struct{}
195 169
196 // MaxVCPUs, MaxMemMB, MaxDiskGB cap the host resources this agent will 170 // MaxVCPUs, MaxMemMB, MaxDiskGB cap the host resources this agent will
197 // commit to live VMs (0 = unlimited). A create whose resources would push 171 // commit to live VMs (0 = unlimited). This is the
198 // the running total past a cap is refused — see admit. This is the
199 // enforced half of agent-side quotas; syncclient advertises the same caps. 172 // enforced half of agent-side quotas; syncclient advertises the same caps.
200 MaxVCPUs int64 173 MaxVCPUs int64
201 MaxMemMB int64 174 MaxMemMB int64
202 MaxDiskGB int64 175 MaxDiskGB int64
203 176
204 // VMTimeout bounds ONE VM's reconcile pass — every operation for that VM in 177 // VMTimeout bounds ONE VM's reconcile pass — every operation for that VM in
205 // that pass, not each operation and not the whole host tick. Zero disables 178 // that pass, not each operation and not the whole host tick. Convergence across ticks is guaranteed
206 // the watchdog. A wedged operation (disk prep, seed build, image fetch) then
207 // fails with the ctx error instead of freezing that VM's reconcile forever;
208 // the next tick retries, and ctx-expiry failures are refunded from the create
209 // retry budget (see failCreate). Convergence across ticks is guaranteed
210 // because completed image downloads are durably cached per-sha. Set 179 // because completed image downloads are durably cached per-sha. Set
211 // comfortably above the longest legitimate operation (imagecache's HTTP 180 // comfortably above the longest legitimate operation (imagecache's HTTP
212 // client allows 10m for a first-time image download). 181 // client allows 10m for a first-time image download).
@@ -347,7 +316,6 @@ func (e *Engine) fenceReport(currentEpoch uint64) *pb.Report {
347 for _, rec := range recs { 316 for _, rec := range recs {
348 var res vmResult 317 var res vmResult
349 res.hostPubKey = rec.HostPubKey 318 res.hostPubKey = rec.HostPubKey
350 // Quarantined VMs belong in Quarantined[], not Vms[].
351 if rec.QuarantinedAt != nil { 319 if rec.QuarantinedAt != nil {
352 res.quarantined = quarantinedEntry(rec, e.graceFor(rec)) 320 res.quarantined = quarantinedEntry(rec, e.graceFor(rec))
353 res.merge(rep) 321 res.merge(rep)
@@ -386,13 +354,6 @@ type publisher func(vmResult)
386 // subsystems it does NOT own — the image cache (Images), and the host backend 354 // subsystems it does NOT own — the image cache (Images), and the host backend
387 // (Prov.Boot/Destroy), which owns both the guest's network attachment and its 355 // (Prov.Boot/Destroy), which owns both the guest's network attachment and its
388 // serial pump — each of which carries its own locking. 356 // serial pump — each of which carries its own locking.
389 //
390 // The bool reports whether the pass RAN. A pass that cannot read this VM's own
391 // record cannot tell an absent VM from a live one, so it changes nothing and
392 // publishes nothing: the loop is level-triggered, the next tick retries once the
393 // store recovers, and the VM keeps its last-known row in the report meanwhile
394 // (see worker.run). Acting on the unreadable record instead would route a
395 // running VM into create().
396 func (e *Engine) reconcileOne(ctx context.Context, id string, a assignment, pub publisher) (vmResult, bool) { 357 func (e *Engine) reconcileOne(ctx context.Context, id string, a assignment, pub publisher) (vmResult, bool) {
397 if e.VMTimeout > 0 { 358 if e.VMTimeout > 0 {
398 var cancel context.CancelFunc 359 var cancel context.CancelFunc
@@ -418,10 +379,6 @@ func (e *Engine) reconcileOne(ctx context.Context, id string, a assignment, pub
418 res.hostPubKey = rec.HostPubKey 379 res.hostPubKey = rec.HostPubKey
419 380
420 if a.desired != nil && !a.tombstoned { 381 if a.desired != nil && !a.tombstoned {
421 // Un-delete path: the VM re-appears in desired while still carrying a
422 // stale QuarantinedAt from a previous tombstone. Clear it so the NEXT
423 // delete starts a fresh grace window (not an instant kill from a stale
424 // timestamp).
425 if ok && rec.QuarantinedAt != nil { 382 if ok && rec.QuarantinedAt != nil {
426 rec.QuarantinedAt = nil 383 rec.QuarantinedAt = nil
427 rec.QuarantineTombstoned = false 384 rec.QuarantineTombstoned = false
@@ -439,11 +396,6 @@ func (e *Engine) reconcileOne(ctx context.Context, id string, a assignment, pub
439 return res, true 396 return res, true
440 } 397 }
441 398
442 // ackDestroyed appends every tombstoned VM that recs — the caller's record view
443 // — no longer holds a record for. Level-triggered off the most recently accepted
444 // snapshot's delete set, and repeated in every report until the control plane
445 // hard-deletes the VM.
446 //
447 // Step passes the view it dispatched with, read before this tick's destroying 399 // Step passes the view it dispatched with, read before this tick's destroying
448 // passes ran, so the ack lands one or more ticks AFTER the pass that removed the 400 // passes ran, so the ack lands one or more ticks AFTER the pass that removed the
449 // record, not in the same tick. Being level-triggered is what makes that fine. 401 // record, not in the same tick. Being level-triggered is what makes that fine.
@@ -462,9 +414,6 @@ func (e *Engine) ackDestroyed(rep *pb.Report, tombstoned map[string]bool, recs m
462 // This is the teardown branch of a VM's reconcile pass (see reconcileOne), 414 // This is the teardown branch of a VM's reconcile pass (see reconcileOne),
463 // split out so a single VM's teardown is expressible on its own. 415 // split out so a single VM's teardown is expressible on its own.
464 func (e *Engine) reapVM(ctx context.Context, id string, rec state.Record, isTombstoned bool, res *vmResult) { 416 func (e *Engine) reapVM(ctx context.Context, id string, rec state.Record, isTombstoned bool, res *vmResult) {
465 // Being reaped (absent from desired or tombstoned): free its compute
466 // from the admission ledger so a new VM can use it. Idempotent; the
467 // address is released by the backend's Destroy.
468 e.releaseCompute(id) 417 e.releaseCompute(id)
469 418
470 now := e.Now() 419 now := e.Now()
@@ -490,11 +439,6 @@ func (e *Engine) reapVM(ctx context.Context, id string, rec state.Record, isTomb
490 grace := e.graceFor(rec) 439 grace := e.graceFor(rec)
491 440
492 if now.Sub(*rec.QuarantinedAt) >= grace { 441 if now.Sub(*rec.QuarantinedAt) >= grace {
493 // Grace expired: destroy the VM. Destroy stops the guest AND releases
494 // its host network resources; on failure the record is KEPT so a later
495 // tick retries, because deleting it would orphan whatever the backend
496 // still holds with nothing left to reap it by.
497 //
498 // NOTE: this may run with an expired pass ctx (watchdog). Safe today 442 // NOTE: this may run with an expired pass ctx (watchdog). Safe today
499 // because the cloud-hypervisor backend's kill ignores ctx (SIGKILL via 443 // because the cloud-hypervisor backend's kill ignores ctx (SIGKILL via
500 // pidfile) — a backend that honors ctx throughout would skip the 444 // pidfile) — a backend that honors ctx throughout would skip the
@@ -507,7 +451,6 @@ func (e *Engine) reapVM(ctx context.Context, id string, rec state.Record, isTomb
507 // Do NOT record a quarantined entry — VM is gone. 451 // Do NOT record a quarantined entry — VM is gone.
508 return 452 return
509 } 453 }
510 // Still in grace: report as quarantined.
511 res.quarantined = quarantinedEntry(rec, grace) 454 res.quarantined = quarantinedEntry(rec, grace)
512 } 455 }
513 456
@@ -530,8 +473,6 @@ func (e *Engine) reconcileVM(ctx context.Context, d *pb.VMSpec, rec state.Record
530 e.converge(ctx, d, rec, res) 473 e.converge(ctx, d, rec, res)
531 } 474 }
532 475
533 // graceFor returns the quarantine grace period for rec: the shorter
534 // TombstoneGrace for a tombstoned VM, else VanishGrace.
535 func (e *Engine) graceFor(rec state.Record) time.Duration { 476 func (e *Engine) graceFor(rec state.Record) time.Duration {
536 if rec.QuarantineTombstoned { 477 if rec.QuarantineTombstoned {
537 return e.TombstoneGrace 478 return e.TombstoneGrace
@@ -539,12 +480,6 @@ func (e *Engine) graceFor(rec state.Record) time.Duration {
539 return e.VanishGrace 480 return e.VanishGrace
540 } 481 }
541 482
542 // admit is the single serialized admission gate: it checks compute quota and,
543 // if admitted, commits the VM's compute to the ledger. It returns a non-empty
544 // reason for a NON-TERMINAL quota refusal, in which case nothing is committed.
545 //
546 // Committing here — before any of the create's side effects run — is what makes
547 // a same-tick sibling count a VM whose create later fails.
548 func (e *Engine) admit(vmID string, spec state.VMSpec) string { 483 func (e *Engine) admit(vmID string, spec state.VMSpec) string {
549 e.mu.Lock() 484 e.mu.Lock()
550 defer e.mu.Unlock() 485 defer e.mu.Unlock()
@@ -560,8 +495,7 @@ func (e *Engine) admit(vmID string, spec state.VMSpec) string {
560 495
561 // acquireCreateSlot takes one of the host's create slots, returning the release 496 // acquireCreateSlot takes one of the host's create slots, returning the release
562 // func. It blocks until a slot frees or ctx expires — never longer, so a queued 497 // func. It blocks until a slot frees or ctx expires — never longer, so a queued
563 // VM cannot outlive its own pass. MaxConcurrentCreates == 0 disables the 498 // VM cannot outlive its own pass.
564 // throttle and returns a no-op release.
565 // 499 //
566 // This is deliberately NOT the admission gate: quota decides whether a VM may 500 // This is deliberately NOT the admission gate: quota decides whether a VM may
567 // exist on this host at all and is recorded durably, whereas a create slot is 501 // exist on this host at all and is recorded durably, whereas a create slot is
@@ -593,8 +527,6 @@ func (e *Engine) note(vmID string, spec state.VMSpec) {
593 e.committed[vmID] = spec 527 e.committed[vmID] = spec
594 } 528 }
595 529
596 // releaseCompute frees a VM's compute from the ledger (at quarantine). The VM's
597 // address is released separately, on destroy, by the backend.
598 func (e *Engine) releaseCompute(vmID string) { 530 func (e *Engine) releaseCompute(vmID string) {
599 e.mu.Lock() 531 e.mu.Lock()
600 defer e.mu.Unlock() 532 defer e.mu.Unlock()
@@ -602,12 +534,9 @@ func (e *Engine) releaseCompute(vmID string) {
602 } 534 }
603 535
604 // noteAddress folds the backend's current answers for this VM's addresses into 536 // noteAddress folds the backend's current answers for this VM's addresses into
605 // rec, reporting whether either changed. An empty answer never clears a known 537 // rec, reporting whether either changed.
606 // address: "I don't know yet" is not "it has none", and blanking rec.IP would
607 // break the guest SSH tunnel (syncclient refuses an empty address) for a VM
608 // that is perfectly reachable.
609 // 538 //
610 // Both NICs are asked on every poll, and the same rule covers both. The named 539 // The named
611 // NIC's address is the one that genuinely arrives late — the host discovers it 540 // NIC's address is the one that genuinely arrives late — the host discovers it
612 // by watching the guest's DHCP exchange — while rec.IP is known before the 541 // by watching the guest's DHCP exchange — while rec.IP is known before the
613 // guest boots. 542 // guest boots.
@@ -643,8 +572,7 @@ func (e *Engine) SeedLedger(recs map[string]state.Record) {
643 // quotaCheckLocked returns a non-empty reason when booting spec would exceed a 572 // quotaCheckLocked returns a non-empty reason when booting spec would exceed a
644 // configured host cap, else "". Caller holds e.mu. It sums the committed specs 573 // configured host cap, else "". Caller holds e.mu. It sums the committed specs
645 // (excluding vmID itself, so a retry does not double-count) and adds spec's 574 // (excluding vmID itself, so a retry does not double-count) and adds spec's
646 // request. Quarantined VMs are absent from committed, so they are excluded for 575 // request.
647 // free. The binding dimension is named so the operator sees which cap was hit.
648 func (e *Engine) quotaCheckLocked(vmID string, spec state.VMSpec) string { 576 func (e *Engine) quotaCheckLocked(vmID string, spec state.VMSpec) string {
649 if e.MaxVCPUs == 0 && e.MaxMemMB == 0 && e.MaxDiskGB == 0 { 577 if e.MaxVCPUs == 0 && e.MaxMemMB == 0 && e.MaxDiskGB == 0 {
650 return "" // no caps configured — unlimited 578 return "" // no caps configured — unlimited
@@ -687,30 +615,17 @@ func (e *Engine) create(ctx context.Context, d *pb.VMSpec, rec state.Record, ok
687 615
688 spec := specFromWire(d) 616 spec := specFromWire(d)
689 617
690 // Fix 2: if the desired spec differs from the stored spec, the user edited the
691 // VM definition. Reset CreateAttempts so the new spec gets a fresh retry budget
692 // instead of being permanently terminal-failed due to the old spec's failures.
693 // VMSpec is fully comparable (all fields are strings/ints/bool), so == is safe. 618 // VMSpec is fully comparable (all fields are strings/ints/bool), so == is safe.
694 if ok && spec != rec.Spec { 619 if ok && spec != rec.Spec {
695 rec.CreateAttempts = 0 620 rec.CreateAttempts = 0
696 rec.LastError = "" 621 rec.LastError = ""
697 } 622 }
698 623
699 // Terminal check: if we've hit MaxCreateAttempts, stop retrying.
700 if ok && rec.CreateAttempts >= e.MaxCreateAttempts { 624 if ok && rec.CreateAttempts >= e.MaxCreateAttempts {
701 res.report(d.VmId, recAddrs(rec), "stopped", "failed", rec.LastError) 625 res.report(d.VmId, recAddrs(rec), "stopped", "failed", rec.LastError)
702 return 626 return
703 } 627 }
704 628
705 // The guest's host key is generated here and stays here. The control plane
706 // receives the public half, signs it, and sends the certificate back down;
707 // the private half goes straight into this VM's seed and nothing above this
708 // host ever holds it.
709 //
710 // Ahead of the retry budget deliberately. Waiting for a certificate is the
711 // round trip working, not an attempt failing — charging it would spend a
712 // VM's whole budget in three ticks against a control plane that is merely
713 // slow to sign, and terminal-fail every VM on the fleet at once.
714 var hostKey state.HostKey 629 var hostKey state.HostKey
715 if d.HostCertRequired { 630 if d.HostCertRequired {
716 rec.Spec = spec // SaveVM keys off the record's own spec 631 rec.Spec = spec // SaveVM keys off the record's own spec
@@ -728,17 +643,11 @@ func (e *Engine) create(ctx context.Context, d *pb.VMSpec, rec state.Record, ok
728 } 643 }
729 res.hostPubKey = rec.HostPubKey 644 res.hostPubKey = rec.HostPubKey
730 if d.SshHostCert == "" { 645 if d.SshHostCert == "" {
731 // Reported, not yet certified. Publish the public key and wait:
732 // booting now would hand the guest a host key every client refuses,
733 // and converge never rebuilds a seed, so it would stay that way.
734 res.report(d.VmId, recAddrs(rec), "stopped", "creating", "awaiting host certificate") 646 res.report(d.VmId, recAddrs(rec), "stopped", "creating", "awaiting host certificate")
735 return 647 return
736 } 648 }
737 } 649 }
738 650
739 // Serialized admission: compute quota under one lock. A refusal is
740 // NON-TERMINAL — it returns before touching CreateAttempts, so once room
741 // frees the next tick retries and boots.
742 rec.Spec = spec 651 rec.Spec = spec
743 if quotaMsg := e.admit(d.VmId, spec); quotaMsg != "" { 652 if quotaMsg := e.admit(d.VmId, spec); quotaMsg != "" {
744 res.report(d.VmId, recAddrs(rec), "stopped", "failed", quotaMsg) 653 res.report(d.VmId, recAddrs(rec), "stopped", "failed", quotaMsg)
@@ -780,17 +689,12 @@ func (e *Engine) create(ctx context.Context, d *pb.VMSpec, rec state.Record, ok
780 pub(interim) 689 pub(interim)
781 } 690 }
782 691
783 // Ask the backend whether this host can run a guest before spending anything
784 // on finding out. Ahead of the throttle as well as the fetch: a create that
785 // can only be refused should not queue for a slot that a create which might
786 // succeed could be using.
787 if err := e.Prov.Preflight(ctx); err != nil { 692 if err := e.Prov.Preflight(ctx); err != nil {
788 e.failCreate(ctx, rec, err, res) 693 e.failCreate(ctx, rec, err, res)
789 return 694 return
790 } 695 }
791 696
792 // Throttle: cap how many VMs on this host may be inside the I/O-heavy part 697 // Per-VM workers made these concurrent — N simultaneous
793 // of create at once. Per-VM workers made these concurrent — N simultaneous
794 // creates mean N image downloads, N image decodes and N multi-GB disk 698 // creates mean N image downloads, N image decodes and N multi-GB disk
795 // copies against one device, which can starve the state dir that Step scans 699 // copies against one device, which can starve the state dir that Step scans
796 // every tick (the heartbeat's one remaining blocking path) and can exhaust 700 // every tick (the heartbeat's one remaining blocking path) and can exhaust
@@ -798,7 +702,6 @@ func (e *Engine) create(ctx context.Context, d *pb.VMSpec, rec state.Record, ok
798 // is busy, so it publishes nothing and the report keeps its last-known row. 702 // is busy, so it publishes nothing and the report keeps its last-known row.
799 // A wait that outlives VMTimeout fails the attempt, and failCreate REFUNDS a 703 // A wait that outlives VMTimeout fails the attempt, and failCreate REFUNDS a
800 // ctx-expiry failure, so a queued VM never burns retry budget for waiting. 704 // ctx-expiry failure, so a queued VM never burns retry budget for waiting.
801 // Held through Boot, which is cheap but immediately I/O-heavy in the guest.
802 say("waiting for a create slot on this host") 705 say("waiting for a create slot on this host")
803 release, err := e.acquireCreateSlot(ctx) 706 release, err := e.acquireCreateSlot(ctx)
804 if err != nil { 707 if err != nil {
@@ -807,10 +710,6 @@ func (e *Engine) create(ctx context.Context, d *pb.VMSpec, rec state.Record, ok
807 } 710 }
808 defer release() 711 defer release()
809 712
810 // Resolve base image. The download is the longest thing a create does and
811 // the one an operator most often wants a number for, so it narrates itself
812 // as it goes; a cached image passes straight through and the line stands for
813 // only as long as the check takes.
814 say("downloading image") 713 say("downloading image")
815 basePath, err := e.Images(ctx, d.ImageUrl, d.ImageSha256, func(done, total int64) { 714 basePath, err := e.Images(ctx, d.ImageUrl, d.ImageSha256, func(done, total int64) {
816 say(downloadDetail(done, total)) 715 say(downloadDetail(done, total))
@@ -820,7 +719,6 @@ func (e *Engine) create(ctx context.Context, d *pb.VMSpec, rec state.Record, ok
820 return 719 return
821 } 720 }
822 721
823 // Prepare root disk.
824 say("preparing root disk") 722 say("preparing root disk")
825 if err := e.Prov.PrepareRootDisk(ctx, rec.Spec, basePath); err != nil { 723 if err := e.Prov.PrepareRootDisk(ctx, rec.Spec, basePath); err != nil {
826 e.failCreate(ctx, rec, err, res) 724 e.failCreate(ctx, rec, err, res)
@@ -836,26 +734,16 @@ func (e *Engine) create(ctx context.Context, d *pb.VMSpec, rec state.Record, ok
836 SSHAuthorizedKey: d.SshAuthorizedKey, 734 SSHAuthorizedKey: d.SshAuthorizedKey,
837 UserData: d.CloudInit, 735 UserData: d.CloudInit,
838 SSHUserCAAuthorizedKey: joinCALines(d.GetSshUserCaAuthorizedKeys()), 736 SSHUserCAAuthorizedKey: joinCALines(d.GetSshUserCaAuthorizedKeys()),
839 // Zero-valued unless this fleet has a CA, in which case it is the key 737 SSHHostKeyPEM: hostKey.PrivatePEM,
840 // generated above and held on this host. Nothing else can supply one. 738 SSHHostCert: d.SshHostCert,
841 SSHHostKeyPEM: hostKey.PrivatePEM, 739 MAC: state.MAC(d.VmId),
842 SSHHostCert: d.SshHostCert, 740 NetworkMAC: netMACIfNetworked(rec.Spec),
843 // The guest's NICs, so its netplan matches each by the address the
844 // hypervisor gave it. The second one exists only for a VM that asked for
845 // a named network; both are derived from the VM id, so the seed and the
846 // host agree without either asking the hypervisor what it ended up with.
847 MAC: state.MAC(d.VmId),
848 NetworkMAC: netMACIfNetworked(rec.Spec),
849 }); err != nil { 741 }); err != nil {
850 e.failCreate(ctx, rec, err, res) 742 e.failCreate(ctx, rec, err, res)
851 return 743 return
852 } 744 }
853 745
854 // Boot if desired running. The backend attaches the network inside Boot, so 746 // Boot if desired running.
855 // ask it for the address the moment Boot returns: a backend that knows it
856 // immediately records it in this same pass — which is what keeps the create
857 // report row carrying the address, as it always has — and one that learns
858 // it later fills it in on a converge poll.
859 if d.PowerState == "running" { 747 if d.PowerState == "running" {
860 say("booting") 748 say("booting")
861 if err := e.Prov.Boot(ctx, d.VmId, rec.Spec); err != nil { 749 if err := e.Prov.Boot(ctx, d.VmId, rec.Spec); err != nil {
@@ -888,13 +776,7 @@ func permanent(err error) bool {
888 return errors.As(err, &p) && p.Permanent() 776 return errors.As(err, &p) && p.Permanent()
889 } 777 }
890 778
891 // failCreate records a failed create attempt and appends a report row. When 779 // failCreate records a failed create attempt and appends a report row. Ordering: the ctx
892 // this VM's pass context has expired, the failure belongs to the watchdog, not
893 // the VM: the attempt is refunded so a wedged pass can never drive a healthy VM
894 // to terminal failed (see TestWatchdogExpiryDoesNotBurnCreateAttempts). A
895 // Permanent() error spends the whole budget at once — retrying a permanent
896 // misconfiguration only wastes image/disk work across three ticks (pinned by
897 // TestPermanentCreateErrorFailsTerminallyInOneAttempt). Ordering: the ctx
898 // refund wins over permanence — a permanent error surfacing under an expired 780 // refund wins over permanence — a permanent error surfacing under an expired
899 // ctx is refunded this tick and, being deterministic, terminal-fails on the 781 // ctx is refunded this tick and, being deterministic, terminal-fails on the
900 // next tick's fresh ctx. Keeps the watchdog invariant unconditional. 782 // next tick's fresh ctx. Keeps the watchdog invariant unconditional.
@@ -914,15 +796,6 @@ func (e *Engine) failCreate(ctx context.Context, rec state.Record, err error, re
914 res.report(rec.Spec.VMID, recAddrs(rec), "stopped", phase, rec.LastError) 796 res.report(rec.Spec.VMID, recAddrs(rec), "stopped", phase, rec.LastError)
915 } 797 }
916 798
917 // failConverge is the shared epilogue for the converge restart/boot paths:
918 // persist err on the record and report it stopped/failed. Unlike failCreate
919 // there is no create-attempt budget — a converge (already-created VM) failure
920 // is terminal-for-this-tick and reported failed immediately.
921 //
922 // The report carries whatever the hypervisor said on its way out, because the
923 // error alone rarely names the cause: booting is spawning a process, so a guest
924 // handed an image its host cannot execute starts cleanly and dies with its
925 // complaint in a log file nobody reads.
926 func (e *Engine) failConverge(rec state.Record, err error, res *vmResult) { 799 func (e *Engine) failConverge(rec state.Record, err error, res *vmResult) {
927 rec.LastError = err.Error() 800 rec.LastError = err.Error()
928 if why := e.Prov.FailureReason(rec.Spec.VMID); why != "" { 801 if why := e.Prov.FailureReason(rec.Spec.VMID); why != "" {
@@ -943,8 +816,7 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMSpec, rec state.Record, r
943 bootID := e.BootID() 816 bootID := e.BootID()
944 817
945 // The address is a polled fact — a backend whose host OS hands it out only 818 // The address is a polled fact — a backend whose host OS hands it out only
946 // learns it once the guest has asked. Persist a first (or changed) answer 819 // learns it once the guest has asked. On a
947 // straight away so a tick that takes no other action still records it. On a
948 // backend that allocates before boot this never fires after the first pass. 820 // backend that allocates before boot this never fires after the first pass.
949 if e.noteAddress(&rec) { 821 if e.noteAddress(&rec) {
950 _ = e.St.SaveVM(rec) 822 _ = e.St.SaveVM(rec)
@@ -994,9 +866,7 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMSpec, rec state.Record, r
994 _ = e.St.SaveVM(rec) 866 _ = e.St.SaveVM(rec)
995 res.report(d.VmId, recAddrs(rec), "running", "ready", "") 867 res.report(d.VmId, recAddrs(rec), "running", "ready", "")
996 } else if d.PowerState == "stopped" && running { 868 } else if d.PowerState == "stopped" && running {
997 // Stop the VM. Record stop BEFORE side effects so a crash between 869 // Stop the VM.
998 // SaveVM and Shutdown is recoverable (the persisted StopRequested
999 // prevents the VM from being treated as "lost" on next reconcile).
1000 // Fix 5: if SaveVM fails, skip Shutdown — the durability guarantee 870 // Fix 5: if SaveVM fails, skip Shutdown — the durability guarantee
1001 // (record stop BEFORE stopping) must hold; stopping without a durable 871 // (record stop BEFORE stopping) must hold; stopping without a durable
1002 // record would cause the VM to be treated as lost after a crash. 872 // record would cause the VM to be treated as lost after a crash.
@@ -1021,9 +891,6 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMSpec, rec state.Record, r
1021 } 891 }
1022 } 892 }
1023 893
1024 // quarantinedEntry builds a QuarantinedVM proto from a record and its grace
1025 // duration. Used by both reapVM and the fence-path report (Fix 4) so
1026 // the JSON shape is identical in both places.
1027 func quarantinedEntry(rec state.Record, grace time.Duration) *pb.QuarantinedVM { 894 func quarantinedEntry(rec state.Record, grace time.Duration) *pb.QuarantinedVM {
1028 specJSON, _ := json.Marshal(rec.Spec) 895 specJSON, _ := json.Marshal(rec.Spec)
1029 destroyAt := rec.QuarantinedAt.Add(grace).Unix() 896 destroyAt := rec.QuarantinedAt.Add(grace).Unix()
@@ -1042,21 +909,12 @@ func quarantinedEntry(rec state.Record, grace time.Duration) *pb.QuarantinedVM {
1042 type vmResult struct { 909 type vmResult struct {
1043 vm *pb.VMStatus 910 vm *pb.VMStatus
1044 quarantined *pb.QuarantinedVM 911 quarantined *pb.QuarantinedVM
1045 // hostPubKey rides every row this VM contributes rather than being passed 912 hostPubKey string
1046 // to report at each of its call sites. The public key is level-triggered —
1047 // it must be in EVERY report for as long as the VM exists, so that a lost
1048 // snapshot or a control-plane restart re-certifies with no operator step —
1049 // and stamping it once in merge is what makes that true without asking six
1050 // call sites to remember.
1051 hostPubKey string
1052 } 913 }
1053 914
1054 // addrs is where a VM is: its address on its host's NAT underlay, which every 915 // addrs is where a VM is: its address on its host's NAT underlay, which every
1055 // guest has from boot, and — for a guest with a second NIC on a named host 916 // guest has from boot, and — for a guest with a second NIC on a named host
1056 // network — the address the site's own DHCP server granted that one. They 917 // network — the address the site's own DHCP server granted that one.
1057 // travel together because every report of one is a report of the other, and a
1058 // row that carried only the first would blank a networked guest's LAN address
1059 // on the fleet's side every time the host said anything about it.
1060 type addrs struct{ ip, networkIP string } 918 type addrs struct{ ip, networkIP string }
1061 919
1062 // recAddrs reads a record's pair. The record is the only place both are known 920 // recAddrs reads a record's pair. The record is the only place both are known
@@ -1096,12 +954,6 @@ func (r *vmResult) merge(rep *pb.Report) {
1096 } 954 }
1097 955
1098 // downloadDetail words how far an image download has come. 956 // downloadDetail words how far an image download has come.
1099 //
1100 // Both numbers are scaled by the SAME unit, chosen from the larger of them, so
1101 // the pair can be read against each other at a glance — "0.4/3.7 GiB" is a
1102 // fraction; "419.4 MiB/3.7 GiB" is arithmetic. A server that declared no length
1103 // (chunked, total <= 0) gets the only half it can vouch for: how far, with
1104 // nothing said about how far there is to go.
1105 func downloadDetail(done, total int64) string { 957 func downloadDetail(done, total int64) string {
1106 scale, unit := byteScale(max(done, total)) 958 scale, unit := byteScale(max(done, total))
1107 if total <= 0 { 959 if total <= 0 {
@@ -1145,9 +997,6 @@ func joinCALines(lines []string) string {
1145 return b.String() 997 return b.String()
1146 } 998 }
1147 999
1148 // netMACIfNetworked returns the second NIC's MAC for a VM that asked for a
1149 // named network, and "" for one that did not — which is what makes the seed
1150 // write the single-NIC network-config every guest running today already has.
1151 func netMACIfNetworked(spec state.VMSpec) string { 1000 func netMACIfNetworked(spec state.VMSpec) string {
1152 if spec.Network == "" { 1001 if spec.Network == "" {
1153 return "" 1002 return ""
internal/agent/reconcile/worker.go
Old New
@@ -14,9 +14,6 @@ import (
14 // Concurrency contract: 14 // Concurrency contract:
15 // - one goroutine per VM, so every operation on a single VM is serialized for 15 // - one goroutine per VM, so every operation on a single VM is serialized for
16 // free: the worker IS the lock, and there is no per-VM mutex; 16 // free: the worker IS the lock, and there is no per-VM mutex;
17 // - a worker holds its own lock only to swap an assignment in or a result
18 // out, never across a reconcile pass, so deliver and collect never wait on
19 // slow work — that is what protects the heartbeat;
20 // - anything shared BETWEEN VMs is guarded where it lives: the admission 17 // - anything shared BETWEEN VMs is guarded where it lives: the admission
21 // ledger under Engine.mu (see admit), whatever the host backend shares 18 // ledger under Engine.mu (see admit), whatever the host backend shares
22 // between VMs, under the backend's own locking, and the state store by one 19 // between VMs, under the backend's own locking, and the state store by one
@@ -24,11 +21,8 @@ import (
24 type manager struct { 21 type manager struct {
25 eng *Engine 22 eng *Engine
26 23
27 mu sync.Mutex 24 mu sync.Mutex
28 workers map[string]*worker 25 workers map[string]*worker
29 // tombstoned is the delete set from the most recent accepted snapshot. The
30 // destroy ack is level-triggered from it, so aggregate needs it even when
31 // it is called without a fresh snapshot.
32 tombstoned map[string]bool 26 tombstoned map[string]bool
33 stopped bool 27 stopped bool
34 } 28 }
@@ -46,9 +40,6 @@ func (e *Engine) manager() *manager {
46 } 40 }
47 41
48 // deliver hands a VM its latest assignment, spawning its worker on first sight. 42 // deliver hands a VM its latest assignment, spawning its worker on first sight.
49 // It never blocks on the worker: an assignment arriving while the worker is
50 // mid-pass simply replaces any unconsumed one (coalescing — latest desired
51 // wins) and is picked up when the current pass ends.
52 func (m *manager) deliver(id string, a assignment) { 43 func (m *manager) deliver(id string, a assignment) {
53 m.mu.Lock() 44 m.mu.Lock()
54 if m.stopped { 45 if m.stopped {
@@ -70,8 +61,7 @@ func (m *manager) deliver(id string, a assignment) {
70 } 61 }
71 62
72 // reapAbsent stops the workers for VMs present in neither desired state nor 63 // reapAbsent stops the workers for VMs present in neither desired state nor
73 // local records — nothing is left to reconcile. The result of a reaped worker 64 // local records — nothing is left to reconcile.
74 // is dropped with it, which is correct: a destroyed VM contributes no report row.
75 // 65 //
76 // Only IDLE workers are reaped. A worker still holding an in-flight (or pending) 66 // Only IDLE workers are reaped. A worker still holding an in-flight (or pending)
77 // pass owns that VM, and dropping it from the map would let the next deliver for 67 // pass owns that VM, and dropping it from the map would let the next deliver for
@@ -157,11 +147,8 @@ type worker struct {
157 eng *Engine 147 eng *Engine
158 id string 148 id string
159 149
160 mu sync.Mutex 150 mu sync.Mutex
161 cond *sync.Cond 151 cond *sync.Cond
162 // pending is the latest assignment waiting to be reconciled. A newer one
163 // overwrites an unconsumed one — a burst of desired-state updates collapses
164 // into a single pass against the newest.
165 pending *assignment 152 pending *assignment
166 // busy reports that a pass is running (with mu released). 153 // busy reports that a pass is running (with mu released).
167 busy bool 154 busy bool
@@ -212,18 +199,9 @@ func (w *worker) run() {
212 func(interim vmResult) { w.publish(gen, interim) }) 199 func(interim vmResult) { w.publish(gen, interim) })
213 200
214 w.mu.Lock() 201 w.mu.Lock()
215 // Publish only a pass that ran. A skipped pass (this VM's record could
216 // not be read) has observed nothing, and publishing its empty result
217 // would blank the VM out of every host report until the next pass
218 // succeeds — the control plane would see the VM as gone from this host.
219 // Keeping the previous result reports the VM's last known state instead,
220 // which is what a level-triggered loop should do while it retries.
221 if ok { 202 if ok {
222 w.result = res 203 w.result = res
223 } 204 }
224 // Clearing busy under the same lock that lands the result is what
225 // closes the pass to further publishes: a straggler that takes the
226 // lock after this point finds no pass in flight and is dropped.
227 w.busy = false 205 w.busy = false
228 w.cond.Broadcast() 206 w.cond.Broadcast()
229 w.mu.Unlock() 207 w.mu.Unlock()
internal/agent/run/cli.go
Old New
@@ -39,8 +39,7 @@ import (
39 39
40 // hostRunner is the production one-shot command runner injected into the 40 // hostRunner is the production one-shot command runner injected into the
41 // host-touching agent packages — which ones exist depends on the platform. It 41 // host-touching agent packages — which ones exist depends on the platform. It
42 // spawns name+args, waits, and returns their combined stdout/stderr. It lives 42 // lives in the composition root because it is a wiring value: constructing the
43 // in the composition root because it is a wiring value: constructing the
44 // concrete dependency is what a root is for, and keeping it here means no 43 // concrete dependency is what a root is for, and keeping it here means no
45 // platform's wiring has to reach into another platform's provisioner for it. 44 // platform's wiring has to reach into another platform's provisioner for it.
46 func hostRunner(ctx context.Context, name string, args ...string) (string, error) { 45 func hostRunner(ctx context.Context, name string, args ...string) (string, error) {
@@ -95,9 +94,6 @@ func RunCLI(args []string) error {
95 return serve(st, cfg) 94 return serve(st, cfg)
96 } 95 }
97 96
98 // parseConfig defines and parses the agent flags for THIS process's platform —
99 // a thin wrapper over parseConfigOn(runtime.GOOS, args), which exists so a
100 // test can drive the darwin refusal path without a GOOS-tagged build.
101 func parseConfig(args []string) (Config, []string, error) { 97 func parseConfig(args []string) (Config, []string, error) {
102 return parseConfigOn(runtime.GOOS, args) 98 return parseConfigOn(runtime.GOOS, args)
103 } 99 }
@@ -229,7 +225,6 @@ func join(st *state.Store, cfg Config, blob string) error {
229 } 225 }
230 f, err := joinblob.Decode(blob) 226 f, err := joinblob.Decode(blob)
231 if err != nil { 227 if err != nil {
232 // Never echo the blob itself — it carries a bearer token.
233 return fmt.Errorf("invalid join blob: %w", err) 228 return fmt.Errorf("invalid join blob: %w", err)
234 } 229 }
235 230
internal/agent/run/guestcidr.go
Old New
@@ -9,27 +9,10 @@ import (
9 // resolveGuestCIDR decides which subnet this host's guests are on, and persists 9 // resolveGuestCIDR decides which subnet this host's guests are on, and persists
10 // the answer the first time it produces one the identity lacks. 10 // the answer the first time it produces one the identity lacks.
11 // 11 //
12 // Order, fixed:
13 //
14 // 1. Identity.BridgeCIDR — the durable record, and it wins whenever present.
15 // 2. --bridge-cidr — the operator's explicit opinion at first start.
16 // 3. suggestion — what the control plane offered at enrollment (empty at any
17 // other time; the suggestion only exists in the enroll response).
18 //
19 // Rule 1 outranking the flag is the whole of "re-homing an existing host is not
20 // something this makes easy". Letting a flag override a persisted identity
21 // would move the bridge under live guests, and every one of them would lose its
22 // address on the next restart. A deliberate re-home is an edit to identity.json.
23 //
24 // join and serve both call this, which is the point: they are separate 12 // join and serve both call this, which is the point: they are separate
25 // processes that used to source the subnet differently — join never had one and 13 // processes that used to source the subnet differently — join never had one and
26 // serve read the identity — so `eitri-agent join --bridge-cidr X` followed by a 14 // serve read the identity — so `eitri-agent join --bridge-cidr X` followed by a
27 // plain `eitri-agent` could silently re-home the host on the next start. 15 // plain `eitri-agent` could silently re-home the host on the next start.
28 //
29 // It returns "" when nothing has an opinion. That is a real answer on a
30 // platform whose OS owns the guest network and takes no suggestion; a platform
31 // that must BUILD one supplies its own last resort, because a fabricated subnet
32 // is only ever right where something is going to create it.
33 func resolveGuestCIDR(st *state.Store, cfg Config, suggestion string) string { 16 func resolveGuestCIDR(st *state.Store, cfg Config, suggestion string) string {
34 if id, ok := st.Identity(); ok && id.BridgeCIDR != "" { 17 if id, ok := st.Identity(); ok && id.BridgeCIDR != "" {
35 return id.BridgeCIDR 18 return id.BridgeCIDR
@@ -44,10 +27,8 @@ func resolveGuestCIDR(st *state.Store, cfg Config, suggestion string) string {
44 return "" 27 return ""
45 } 28 }
46 29
47 // persistGuestCIDR writes the resolved subnet into the durable identity so the 30 // Best-effort: a failed write leaves the agent running on the value it
48 // next start reads it from rule 1 rather than re-deriving it. Best-effort: a 31 // resolved, and the next start resolves the same way from the same inputs.
49 // failed write leaves the agent running on the value it resolved, and the next
50 // start resolves the same way from the same inputs.
51 func persistGuestCIDR(st *state.Store, cidr string) { 32 func persistGuestCIDR(st *state.Store, cidr string) {
52 id, ok := st.Identity() 33 id, ok := st.Identity()
53 if !ok || id.BridgeCIDR == cidr { 34 if !ok || id.BridgeCIDR == cidr {
internal/agent/run/reservations.go
Old New
@@ -10,12 +10,6 @@ import (
10 // pinning one guest's address. Consumer-owned and minimal (arch R5) so a 10 // pinning one guest's address. Consumer-owned and minimal (arch R5) so a
11 // platform whose networking is nothing like a bridge and a DHCP responder can 11 // platform whose networking is nothing like a bridge and a DHCP responder can
12 // still be handed the surviving guests' addresses. 12 // still be handed the surviving guests' addresses.
13 //
14 // Two verbs because a guest can have two NICs and they rebuild differently:
15 // every guest re-pins the DHCP reservation this host serves its NAT NIC, and a
16 // guest that also has a named-network NIC re-seeds the last address seen there
17 // — never this host's to give — and re-arms the discovery that will confirm or
18 // replace it. A networked guest needs both calls, not one instead of the other.
19 type reservations interface { 13 type reservations interface {
20 AddReservation(vmID, ip string) 14 AddReservation(vmID, ip string)
21 AdoptNetwork(vmID, network, ip string) 15 AdoptNetwork(vmID, network, ip string)
@@ -26,10 +20,7 @@ type reservations interface {
26 // surviving guest's renewal is never answered from an empty table (fail-closed 20 // surviving guest's renewal is never answered from an empty table (fail-closed
27 // plus this preload close the gap). It is also the only thing that carries an 21 // plus this preload close the gap). It is also the only thing that carries an
28 // address across an agent restart: the table is in memory, so a guest whose 22 // address across an agent restart: the table is in memory, so a guest whose
29 // reservation is not replayed is renumbered at its next boot. A record carrying 23 // reservation is not replayed is renumbered at its next boot.
30 // no address has nothing to pin. A failed load warns and returns rather than
31 // aborting startup — a host that renumbers its guests still beats one that
32 // refuses to come up.
33 func replayReservations(st *state.Store, net reservations) { 24 func replayReservations(st *state.Store, net reservations) {
34 recs, err := st.LoadVMs() 25 recs, err := st.LoadVMs()
35 if err != nil { 26 if err != nil {
@@ -40,9 +31,6 @@ func replayReservations(st *state.Store, net reservations) {
40 if rec.IP != "" { 31 if rec.IP != "" {
41 net.AddReservation(rec.Spec.VMID, rec.IP) 32 net.AddReservation(rec.Spec.VMID, rec.IP)
42 } 33 }
43 // A networked guest is adopted even with no recorded LAN address: the
44 // attachment itself is what has to be rebuilt, or nothing re-arms the
45 // discovery that reports the address the site's DHCP server granted it.
46 if rec.Spec.Network != "" { 34 if rec.Spec.Network != "" {
47 net.AdoptNetwork(rec.Spec.VMID, rec.Spec.Network, rec.NetworkIP) 35 net.AdoptNetwork(rec.Spec.VMID, rec.Spec.Network, rec.NetworkIP)
48 } 36 }
internal/agent/run/wire_linux.go
Old New
@@ -60,10 +60,6 @@ type platform struct {
60 HostNetworks []string 60 HostNetworks []string
61 } 61 }
62 62
63 // proposeGuestCIDR is what this platform offers the fleet at enrollment: the
64 // operator's flag, or no opinion at all. Nil takes the fleet's suggestion,
65 // which is what a Linux host without --bridge-cidr wants.
66 //
67 // Per-platform because parseConfig is deliberately platform-neutral — every 63 // Per-platform because parseConfig is deliberately platform-neutral — every
68 // flag is accepted everywhere and ignored by hosts that do not run them — so a 64 // flag is accepted everywhere and ignored by hosts that do not run them — so a
69 // shared reader of cfg.BridgeCIDR would have a Mac propose a fabricated Linux 65 // shared reader of cfg.BridgeCIDR would have a Mac propose a fabricated Linux
internal/agent/vfkit/console.go
Old New
@@ -117,7 +117,6 @@ type vmInspect struct {
117 } `json:"devices"` 117 } `json:"devices"`
118 } 118 }
119 119
120 // serialKind is how vfkit spells the virtio-serial device in its JSON.
121 const serialKind = "virtioserial" 120 const serialKind = "virtioserial"
122 121
123 // inspectPTY asks the VM on sock where its serial PTY is. 122 // inspectPTY asks the VM on sock where its serial PTY is.
@@ -147,18 +146,11 @@ func inspectPTY(client *http.Client, sock string) (string, error) {
147 return d.PtyName, nil 146 return d.PtyName, nil
148 } 147 }
149 } 148 }
150 // vfkit fills ptyName in while it builds the VM, so an empty one is the
151 // normal answer for the first moments after Boot, not a broken config.
152 return "", fmt.Errorf("vfkit inspect: no serial PTY yet") 149 return "", fmt.Errorf("vfkit inspect: no serial PTY yet")
153 } 150 }
154 151
155 // sockKey carries the unix socket a REST request must dial, on the request's
156 // context. It travels there rather than in a per-VM dialer because one client
157 // serves every VM on the host, and the host in the URL — which is what the
158 // connection pool keys on — is the same placeholder for all of them.
159 type sockKey struct{} 152 type sockKey struct{}
160 153
161 // withSocket marks ctx as belonging to the vfkit listening on sock.
162 func withSocket(ctx context.Context, sock string) context.Context { 154 func withSocket(ctx context.Context, sock string) context.Context {
163 return context.WithValue(ctx, sockKey{}, sock) 155 return context.WithValue(ctx, sockKey{}, sock)
164 } 156 }
internal/agent/vfkit/guestnet.go
Old New
@@ -77,7 +77,6 @@ func hostNetworks() []netip.Prefix {
77 return out 77 return out
78 } 78 }
79 79
80 // prefixOf reduces one interface address to the IPv4 network it sits on.
81 func prefixOf(a net.Addr) (netip.Prefix, bool) { 80 func prefixOf(a net.Addr) (netip.Prefix, bool) {
82 ipnet, ok := a.(*net.IPNet) 81 ipnet, ok := a.(*net.IPNet)
83 if !ok || ipnet.IP.To4() == nil { 82 if !ok || ipnet.IP.To4() == nil {
internal/agent/vfkit/leases.go
Old New
@@ -95,12 +95,6 @@ func leaseAddress(leases, mac string) string {
95 return found 95 return found
96 } 96 }
97 97
98 // parseIP keeps a lease's address only if it is one. The field is written by a
99 // system daemon the agent does not control, and hw_address is already checked
100 // exhaustively; without the same treatment here, a stanza carrying this VM's
101 // MAC and any non-empty junk becomes the VM's recorded address, and the damage
102 // surfaces a layer away as a guest nothing can reach. Returns "" for anything
103 // that is not an address, which the caller reads as "this stanza has none".
104 func parseIP(s string) string { 98 func parseIP(s string) string {
105 addr, err := netip.ParseAddr(strings.TrimSpace(s)) 99 addr, err := netip.ParseAddr(strings.TrimSpace(s))
106 if err != nil { 100 if err != nil {
@@ -112,12 +106,10 @@ func parseIP(s string) string {
112 // normalizeMAC reduces a hardware address to one comparable form, because the 106 // normalizeMAC reduces a hardware address to one comparable form, because the
113 // two sides spell the same address differently: state.MAC pads every octet 107 // two sides spell the same address differently: state.MAC pads every octet
114 // ("52:54:00:0a:…") while bootpd writes them bare and prefixes the ARP 108 // ("52:54:00:0a:…") while bootpd writes them bare and prefixes the ARP
115 // hardware type ("1,52:54:0:a:…"). Comparing the strings as written would miss 109 // hardware type ("1,52:54:0:a:…").
116 // every lease. Returns "" for anything that is not six hex octets, so garbage
117 // in the file can never match a real MAC.
118 func normalizeMAC(s string) string { 110 func normalizeMAC(s string) string {
119 if _, rest, ok := strings.Cut(s, ","); ok { 111 if _, rest, ok := strings.Cut(s, ","); ok {
120 s = rest // drop bootpd's hardware-type prefix 112 s = rest
121 } 113 }
122 octets := strings.Split(strings.TrimSpace(s), ":") 114 octets := strings.Split(strings.TrimSpace(s), ":")
123 if len(octets) != 6 { 115 if len(octets) != 6 {
internal/agent/vfkit/vfkit.go
Old New
@@ -43,8 +43,7 @@ const logMode = 0o600
43 43
44 // PumpHooks is the serial-console pump lifecycle the provisioner drives 44 // PumpHooks is the serial-console pump lifecycle the provisioner drives
45 // (consumer-owned; the concrete implementation is *serialpump.Manager, wired 45 // (consumer-owned; the concrete implementation is *serialpump.Manager, wired
46 // by the composition root — vfkit must not import serialpump). nil disables 46 // by the composition root — vfkit must not import serialpump).
47 // the hooks.
48 type PumpHooks interface { 47 type PumpHooks interface {
49 Ensure(vmID string) 48 Ensure(vmID string)
50 Stop(vmID string) 49 Stop(vmID string)
@@ -64,12 +63,8 @@ type Provisioner struct {
64 bin string // vfkit binary: a path, or a bare name resolved on $PATH 63 bin string // vfkit binary: a path, or a bare name resolved on $PATH
65 run agentexec.Runner 64 run agentexec.Runner
66 65
67 // lookPath resolves bin for Preflight. Injected so the refusal path is
68 // testable without depending on what happens to be installed on the box
69 // running the tests.
70 lookPath func(file string) (string, error) 66 lookPath func(file string) (string, error)
71 67
72 // leasesPath is macOS's DHCP lease database. Injected for the same reason.
73 leasesPath string 68 leasesPath string
74 69
75 // bootID identifies the host boot a VM's process was started in. It is the 70 // bootID identifies the host boot a VM's process was started in. It is the
@@ -78,17 +73,12 @@ type Provisioner struct {
78 // without rebooting the box running it. 73 // without rebooting the box running it.
79 bootID func() string 74 bootID func() string
80 75
81 // signal delivers a signal to a pid. Injected because the branch that
82 // matters — a kill the kernel refuses — cannot otherwise be reached without
83 // depending on what the box running the tests is allowed to signal.
84 signal func(pid int, sig syscall.Signal) error 76 signal func(pid int, sig syscall.Signal) error
85 77
86 // rest is the one client every vfkit REST call on this host goes through. 78 // rest is the one client every vfkit REST call on this host goes through.
87 // One, deliberately: see newRESTClient. 79 // One, deliberately: see newRESTClient.
88 rest *http.Client 80 rest *http.Client
89 81
90 // Pumps receives serial-pump lifecycle calls at Boot/Shutdown/kill — a VM's
91 // pump lives exactly as long as its guest is powered on. nil = no-op.
92 Pumps PumpHooks 82 Pumps PumpHooks
93 } 83 }
94 84
@@ -189,9 +179,6 @@ func (p *Provisioner) disks(spec state.VMSpec) []state.Disk {
189 } 179 }
190 } 180 }
191 181
192 // buildArgs returns the vfkit command line for spec. It is pure — createVarStore
193 // is passed in rather than stat'ed here — so the whole argument contract is
194 // unit-testable without a filesystem or a Mac.
195 func (p *Provisioner) buildArgs(spec state.VMSpec, createVarStore bool) []string { 182 func (p *Provisioner) buildArgs(spec state.VMSpec, createVarStore bool) []string {
196 vmID := spec.VMID 183 vmID := spec.VMID
197 184
@@ -303,8 +290,6 @@ func (p *Provisioner) clone(ctx context.Context, src, dst string) error {
303 // wired to it: tying a guest's lifetime to the agent's would power off every VM 290 // wired to it: tying a guest's lifetime to the agent's would power off every VM
304 // on a graceful agent stop. Stopping a VM is Shutdown/Destroy's job alone. 291 // on a graceful agent stop. Stopping a VM is Shutdown/Destroy's job alone.
305 func (p *Provisioner) Boot(_ context.Context, vmID string, spec state.VMSpec) error { 292 func (p *Provisioner) Boot(_ context.Context, vmID string, spec state.VMSpec) error {
306 // vfkit removes its socket on a clean exit only, so a crash, a SIGKILL or a
307 // host reboot leaves one behind that the next bind would trip over.
308 _ = os.Remove(p.sockPath(vmID)) 293 _ = os.Remove(p.sockPath(vmID))
309 294
310 // Anything short of a successful stat means "initialise a store". Matching 295 // Anything short of a successful stat means "initialise a store". Matching
@@ -423,7 +408,7 @@ func (p *Provisioner) powerOff(ctx context.Context, vmID string) error {
423 func (p *Provisioner) sigterm(vmID string) error { 408 func (p *Provisioner) sigterm(vmID string) error {
424 pid := p.ownedPID(vmID) 409 pid := p.ownedPID(vmID)
425 if pid == 0 { 410 if pid == 0 {
426 return nil // already gone 411 return nil
427 } 412 }
428 if err := p.signal(pid, syscall.SIGTERM); err != nil && err != syscall.ESRCH { 413 if err := p.signal(pid, syscall.SIGTERM); err != nil && err != syscall.ESRCH {
429 return fmt.Errorf("SIGTERM %s (pid %d): %w", vmID, pid, err) 414 return fmt.Errorf("SIGTERM %s (pid %d): %w", vmID, pid, err)
@@ -461,14 +446,9 @@ func (p *Provisioner) kill(vmID string) error {
461 if p.Pumps != nil { 446 if p.Pumps != nil {
462 p.Pumps.Stop(vmID) 447 p.Pumps.Stop(vmID)
463 } 448 }
464 // A pid this agent may not claim — none recorded, or one from an earlier
465 // host boot — leaves nothing to kill and nothing to keep the record for.
466 pid := p.ownedPID(vmID) 449 pid := p.ownedPID(vmID)
467 if pid != 0 { 450 if pid != 0 {
468 if err := p.signal(pid, syscall.SIGKILL); err != nil && err != syscall.ESRCH { 451 if err := p.signal(pid, syscall.SIGKILL); err != nil && err != syscall.ESRCH {
469 // Keep the pidfile and report: a SIGKILL that failed with anything
470 // but ESRCH has not proved the process gone, and the pidfile is the
471 // only record of which process it is.
472 return fmt.Errorf("SIGKILL %s (pid %d): %w", vmID, pid, err) 452 return fmt.Errorf("SIGKILL %s (pid %d): %w", vmID, pid, err)
473 } 453 }
474 } 454 }
internal/server/api/api.go
Old New
@@ -28,9 +28,6 @@ import (
28 "github.com/a73x/eitri/internal/server/store" 28 "github.com/a73x/eitri/internal/server/store"
29 ) 29 )
30 30
31 // VM names are validated as RFC-1123 DNS labels, and image digests as hex
32 // SHA-256, via internal/names.
33
34 // DefaultImage is the image applied to one-click VM creates. 31 // DefaultImage is the image applied to one-click VM creates.
35 type DefaultImage struct { 32 type DefaultImage struct {
36 URL string 33 URL string
@@ -219,9 +216,6 @@ func (a *API) StartBackground(ctx context.Context) {
219 // tears down any lingering guest. No config knob — a fixed policy bound. 216 // tears down any lingering guest. No config knob — a fixed policy bound.
220 const abandonedVMReapGrace = 15 * time.Minute 217 const abandonedVMReapGrace = 15 * time.Minute
221 218
222 // sweepAbandonedVMs hard-deletes every VM tombstoned longer than
223 // abandonedVMReapGrace whose host is NOT currently online — the reap the agent
224 // would have acked, done server-side because no agent is connected to do it.
225 // An online host is left alone: its live agent owns the reap and will ack 219 // An online host is left alone: its live agent owns the reap and will ack
226 // through the normal path, and racing it risks flipping the VM onto the longer 220 // through the normal path, and racing it risks flipping the VM onto the longer
227 // vanished-grace clock. Returns true if it removed at least one row. now is 221 // vanished-grace clock. Returns true if it removed at least one row. now is
@@ -237,10 +231,10 @@ func (a *API) sweepAbandonedVMs(now time.Time) bool {
237 continue // live VM — not a delete in progress 231 continue // live VM — not a delete in progress
238 } 232 }
239 if now.Sub(*vm.DeletedAt) < abandonedVMReapGrace { 233 if now.Sub(*vm.DeletedAt) < abandonedVMReapGrace {
240 continue // within grace — leave it for the agent to ack 234 continue
241 } 235 }
242 if st, ok := a.reg.Get(vm.HostID); ok && st.Online { 236 if st, ok := a.reg.Get(vm.HostID); ok && st.Online {
243 continue // agent is live; it owns the reap 237 continue
244 } 238 }
245 if err := a.st.HardDeleteVM(vm.ID); err != nil { 239 if err := a.st.HardDeleteVM(vm.ID); err != nil {
246 slog.Warn("sweep abandoned VM failed", "vm", vm.ID, "host", vm.HostID, "err", err) 240 slog.Warn("sweep abandoned VM failed", "vm", vm.ID, "host", vm.HostID, "err", err)
@@ -408,10 +402,6 @@ func (a *API) handleCreateEnrollToken(w http.ResponseWriter, r *http.Request) {
408 402
409 // --- hosts --- 403 // --- hosts ---
410 404
411 // toHostResponse merges a durable host row with its live registry state into
412 // the wire shape (types.Host). Handler-side facts behind the contract's
413 // sync-health fields: Stale trips when the last report is older than
414 // registry.StaleWindow; AgentVersion is registry-held from the agent's Hello.
415 func toHostResponse(h store.Host, st registry.HostState, ok bool, alloc store.Alloc) types.Host { 405 func toHostResponse(h store.Host, st registry.HostState, ok bool, alloc store.Alloc) types.Host {
416 hr := types.Host{ 406 hr := types.Host{
417 ID: h.ID, 407 ID: h.ID,
@@ -440,16 +430,9 @@ func toHostResponse(h store.Host, st registry.HostState, ok bool, alloc store.Al
440 hr.Stale = st.Stale 430 hr.Stale = st.Stale
441 hr.Sessions = st.Sessions 431 hr.Sessions = st.Sessions
442 hr.AgentVersion = st.AgentVersion 432 hr.AgentVersion = st.AgentVersion
443 // Registry-held like AgentVersion, and for the same reason it is gated
444 // on `ok`: only a host that has connected has said which networks it
445 // serves, and a host this server has not heard from advertises nothing
446 // rather than whatever it once did. The nil check keeps the empty list
447 // above: a connected host that named no networks still serves [].
448 if st.HostNetworks != nil { 433 if st.HostNetworks != nil {
449 hr.HostNetworks = st.HostNetworks 434 hr.HostNetworks = st.HostNetworks
450 } 435 }
451 // LastSeen unset ⇒ connected but never reported: leave the age fields
452 // null rather than emit a bogus "last seen at the zero time".
453 if !st.LastSeen.IsZero() { 436 if !st.LastSeen.IsZero() {
454 seen := st.LastSeen 437 seen := st.LastSeen
455 secs := int64(st.SinceLastSeen.Seconds()) 438 secs := int64(st.SinceLastSeen.Seconds())
@@ -522,7 +505,6 @@ func (a *API) snapshotHosts(p Principal) ([]types.Host, error) {
522 return a.buildHostResponses(hosts, alloc, a.fetchStates(hostIDs(hosts))), nil 505 return a.buildHostResponses(hosts, alloc, a.fetchStates(hostIDs(hosts))), nil
523 } 506 }
524 507
525 // hostIDs projects the host row IDs (registry keys) for a single-fetch states map.
526 func hostIDs(hosts []store.Host) []string { 508 func hostIDs(hosts []store.Host) []string {
527 ids := make([]string, len(hosts)) 509 ids := make([]string, len(hosts))
528 for i, h := range hosts { 510 for i, h := range hosts {
@@ -531,7 +513,6 @@ func hostIDs(hosts []store.Host) []string {
531 return ids 513 return ids
532 } 514 }
533 515
534 // vmHostIDs projects the host IDs referenced by a VM list (registry keys).
535 func vmHostIDs(vms []store.VM) []string { 516 func vmHostIDs(vms []store.VM) []string {
536 ids := make([]string, len(vms)) 517 ids := make([]string, len(vms))
537 for i, vm := range vms { 518 for i, vm := range vms {
@@ -575,8 +556,6 @@ func (a *API) handleListHosts(w http.ResponseWriter, r *http.Request) {
575 556
576 // --- VMs --- 557 // --- VMs ---
577 558
578 // toVMResponse merges a durable VM row with live agent-reported actual-state
579 // into the wire shape (types.VM).
580 func toVMResponse(vm store.VM, actualPower, phase, statusDetail string, destroyAt int64) types.VM { 559 func toVMResponse(vm store.VM, actualPower, phase, statusDetail string, destroyAt int64) types.VM {
581 return types.VM{ 560 return types.VM{
582 ID: vm.ID, 561 ID: vm.ID,
@@ -614,7 +593,6 @@ func deriveLifecycle(vm store.VM, actualPower, phase string) string {
614 if vm.DeletedAt != nil { 593 if vm.DeletedAt != nil {
615 return "deleting" 594 return "deleting"
616 } 595 }
617 // Prefer the agent's live phase; fall back to the desired status.
618 if phase == "" { 596 if phase == "" {
619 phase = vm.Status 597 phase = vm.Status
620 } 598 }
@@ -663,8 +641,6 @@ func (a *API) buildVMResponses(vms []store.VM, states map[string]regState) []typ
663 break 641 break
664 } 642 }
665 } 643 }
666 // A tombstoned VM the agent has stopped and quarantined carries a
667 // hard destroy deadline; surface it so clients can render a countdown.
668 for _, qv := range rs.st.Report.Quarantined { 644 for _, qv := range rs.st.Report.Quarantined {
669 if qv.VMID == vm.ID { 645 if qv.VMID == vm.ID {
670 destroyAt = qv.DestroyAtUnix 646 destroyAt = qv.DestroyAtUnix
@@ -703,7 +679,6 @@ func (a *API) applyVMDefaults(req *types.CreateVMRequest, hostArch string) (stri
703 if req.Name == "" { 679 if req.Name == "" {
704 req.Name = "sandbox-" + random.Hex(3) 680 req.Name = "sandbox-" + random.Hex(3)
705 } 681 }
706 // Image URL+SHA must come as a pair; apply defaults only when BOTH are empty.
707 if req.ImageURL == "" && req.ImageSHA256 == "" { 682 if req.ImageURL == "" && req.ImageSHA256 == "" {
708 img, ok := a.cfg.DefaultImages[hostArch] 683 img, ok := a.cfg.DefaultImages[hostArch]
709 if !ok { 684 if !ok {
@@ -749,7 +724,6 @@ func validateCreateVM(req *types.CreateVMRequest) (string, int) {
749 if !names.IsSHA256Hex(req.ImageSHA256) { 724 if !names.IsSHA256Hex(req.ImageSHA256) {
750 return "invalid image_sha256", http.StatusBadRequest 725 return "invalid image_sha256", http.StatusBadRequest
751 } 726 }
752 // Resource floors (post-defaults, so a zero has already become the default).
753 // Negative values are never meaningful; a too-small disk_gb is also rejected 727 // Negative values are never meaningful; a too-small disk_gb is also rejected
754 // by the agent's never-shrink guard, but nonsense should fail at create time. 728 // by the agent's never-shrink guard, but nonsense should fail at create time.
755 if req.VCPUs < 1 || req.MemMB < 1 || req.DiskGB < 1 { 729 if req.VCPUs < 1 || req.MemMB < 1 || req.DiskGB < 1 {
@@ -810,7 +784,6 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) {
810 return 784 return
811 } 785 }
812 if !mayActAs(principalFromContext(r), host.Tenant) { 786 if !mayActAs(principalFromContext(r), host.Tenant) {
813 // Indistinguishable from absent: don't confirm foreign hosts exist.
814 http.Error(w, "unknown host_id", http.StatusBadRequest) 787 http.Error(w, "unknown host_id", http.StatusBadRequest)
815 return 788 return
816 } 789 }
@@ -972,8 +945,6 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) {
972 InjectedKeyFP: keyFP, 945 InjectedKeyFP: keyFP,
973 InjectedKeyComment: keyComment, 946 InjectedKeyComment: keyComment,
974 947
975 // Frozen from the set the refusal above just checked, so the row records
976 // exactly what permitted it to exist.
977 TrustedCAs: store.FreezeCAs(tenantCAs), 948 TrustedCAs: store.FreezeCAs(tenantCAs),
978 949
979 VCPUs: req.VCPUs, 950 VCPUs: req.VCPUs,
@@ -1018,9 +989,6 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) {
1018 // vanishes post-mutate, since the mutation itself already succeeded). 989 // vanishes post-mutate, since the mutation itself already succeeded).
1019 // Finally SSE watchers are notified and the response is written as 204, 990 // Finally SSE watchers are notified and the response is written as 204,
1020 // unconditionally. 991 // unconditionally.
1021 //
1022 // Ownership is gated before any mutation runs: a foreign-tenant VM answers
1023 // exactly like a missing one — existence is not leaked across tenants.
1024 func (a *API) mutateVM(w http.ResponseWriter, r *http.Request, id string, mutate func(id string) error, 992 func (a *API) mutateVM(w http.ResponseWriter, r *http.Request, id string, mutate func(id string) error,
1025 notFoundMsg string, notFoundStatus int, 993 notFoundMsg string, notFoundStatus int,
1026 auditAction string, auditDetail func(vm store.VM) map[string]string) { 994 auditAction string, auditDetail func(vm store.VM) map[string]string) {
internal/server/api/auth.go
Old New
@@ -111,8 +111,6 @@ func (af *authFlow) ensureProvider(ctx context.Context) error {
111 // single-box quickstart, where the browser and server share localhost. 111 // single-box quickstart, where the browser and server share localhost.
112 func (af *authFlow) secure() bool { return strings.HasPrefix(af.cfg.PublicURL, "https://") } 112 func (af *authFlow) secure() bool { return strings.HasPrefix(af.cfg.PublicURL, "https://") }
113 113
114 // handleLogin mints a state + PKCE verifier, stashes both in short-lived
115 // HttpOnly cookies, and redirects the browser to the issuer's authorize endpoint.
116 func (af *authFlow) handleLogin(w http.ResponseWriter, r *http.Request) { 114 func (af *authFlow) handleLogin(w http.ResponseWriter, r *http.Request) {
117 if err := af.ensureProvider(r.Context()); err != nil { 115 if err := af.ensureProvider(r.Context()); err != nil {
118 slog.Warn("oidc discovery failed", "err", err) 116 slog.Warn("oidc discovery failed", "err", err)
@@ -130,8 +128,6 @@ func (af *authFlow) handleLogin(w http.ResponseWriter, r *http.Request) {
130 http.Redirect(w, r, af.oauth.AuthCodeURL(state, oauth2.S256ChallengeOption(verifier)), http.StatusFound) 128 http.Redirect(w, r, af.oauth.AuthCodeURL(state, oauth2.S256ChallengeOption(verifier)), http.StatusFound)
131 } 129 }
132 130
133 // handleCallback verifies the state, exchanges the code, verifies the id_token,
134 // resolves the identity to a tenant, mints a session, and redirects to "/".
135 func (af *authFlow) handleCallback(w http.ResponseWriter, r *http.Request) { 131 func (af *authFlow) handleCallback(w http.ResponseWriter, r *http.Request) {
136 if err := af.ensureProvider(r.Context()); err != nil { 132 if err := af.ensureProvider(r.Context()); err != nil {
137 slog.Warn("oidc discovery failed", "err", err) 133 slog.Warn("oidc discovery failed", "err", err)
@@ -308,7 +304,6 @@ func (af *authFlow) clearCookie(w http.ResponseWriter, name string) {
308 }) 304 })
309 } 305 }
310 306
311 // randomHex returns n random bytes as a 2n-char lowercase hex string.
312 func randomHex(n int) (string, error) { 307 func randomHex(n int) (string, error) {
313 b := make([]byte, n) 308 b := make([]byte, n)
314 if _, err := rand.Read(b); err != nil { 309 if _, err := rand.Read(b); err != nil {
internal/server/api/authreject.go
Old New
@@ -44,8 +44,6 @@ const (
44 // only ever reads back to explain a failure. 44 // only ever reads back to explain a failure.
45 const stateOriginSep = "." 45 const stateOriginSep = "."
46 46
47 // newState mints a state value carrying the origin the browser used to reach
48 // /auth/login.
49 func newState(origin string) (string, error) { 47 func newState(origin string) (string, error) {
50 r, err := randomHex(16) // 32 hex chars 48 r, err := randomHex(16) // 32 hex chars
51 if err != nil { 49 if err != nil {
@@ -130,9 +128,6 @@ type stateRejection struct {
130 console string // public_url, trailing slash trimmed 128 console string // public_url, trailing slash trimmed
131 } 129 }
132 130
133 // diagnoseCallback runs the state check and, on failure, says which way it
134 // failed. It returns the PKCE verifier when every check passes.
135 //
136 // The order is the order of the evidence, not of the checks it replaces. A 131 // The order is the order of the evidence, not of the checks it replaces. A
137 // callback with no state parameter at all was never sent by an identity 132 // callback with no state parameter at all was never sent by an identity
138 // provider. Then the cookie: when it did not arrive, an origin that is not 133 // provider. Then the cookie: when it did not arrive, an origin that is not
internal/server/api/console.go
Old New
@@ -67,7 +67,6 @@ func (a *API) handleConsoleWS(w http.ResponseWriter, r *http.Request) {
67 stream, err := a.console.OpenConsole(openCtx, vm.HostID, id) 67 stream, err := a.console.OpenConsole(openCtx, vm.HostID, id)
68 cancel() 68 cancel()
69 if err != nil { 69 if err != nil {
70 // Agent leg failed (offline host, refused VM): close with the reason.
71 reason := "console unavailable: " + err.Error() 70 reason := "console unavailable: " + err.Error()
72 if len(reason) > 120 { // close reasons are capped at 123 bytes 71 if len(reason) > 120 { // close reasons are capped at 123 bytes
73 // The byte trim can bisect a multi-byte rune and RFC 6455 requires 72 // The byte trim can bisect a multi-byte rune and RFC 6455 requires
@@ -78,11 +77,10 @@ func (a *API) handleConsoleWS(w http.ResponseWriter, r *http.Request) {
78 return 77 return
79 } 78 }
80 79
81 // Two pumps, raw bytes. NetConn adapts the WS to net.Conn (binary frames).
82 nc := websocket.NetConn(r.Context(), c, websocket.MessageBinary) 80 nc := websocket.NetConn(r.Context(), c, websocket.MessageBinary)
83 done := make(chan struct{}, 2) 81 done := make(chan struct{}, 2)
84 go func() { _, _ = io.Copy(stream, nc); stream.Close(); done <- struct{}{} }() // keystrokes → guest 82 go func() { _, _ = io.Copy(stream, nc); stream.Close(); done <- struct{}{} }()
85 go func() { _, _ = io.Copy(nc, stream); nc.Close(); done <- struct{}{} }() // guest → browser 83 go func() { _, _ = io.Copy(nc, stream); nc.Close(); done <- struct{}{} }()
86 <-done 84 <-done
87 <-done 85 <-done
88 _ = c.Close(websocket.StatusNormalClosure, "") 86 _ = c.Close(websocket.StatusNormalClosure, "")
internal/server/api/events.go
Old New
@@ -103,8 +103,6 @@ func forceParam(r *http.Request) bool {
103 // until the operator re-enrolls it with a fresh join blob. 103 // until the operator re-enrolls it with a fresh join blob.
104 func (a *API) handleRevokeCredential(w http.ResponseWriter, r *http.Request) { 104 func (a *API) handleRevokeCredential(w http.ResponseWriter, r *http.Request) {
105 id := r.PathValue("id") 105 id := r.PathValue("id")
106 // Ownership gate before any mutation: a foreign-tenant host answers exactly
107 // like a missing one (no existence leak), matching the mayActAs convention.
108 h, err := a.st.GetHost(id) 106 h, err := a.st.GetHost(id)
109 switch { 107 switch {
110 case errors.Is(err, sql.ErrNoRows): 108 case errors.Is(err, sql.ErrNoRows):
@@ -239,8 +237,6 @@ func (a *API) handleEvents(w http.ResponseWriter, r *http.Request) {
239 w.Header().Set("Cache-Control", "no-cache") 237 w.Header().Set("Cache-Control", "no-cache")
240 w.Header().Set("Connection", "keep-alive") 238 w.Header().Set("Connection", "keep-alive")
241 239
242 // Subscribe scoped to the ticket's tenant: the hub fans this connection only
243 // its tenant's snapshot bytes, so no cross-tenant host/VM/metric ever reaches it.
244 snaps, unsub := a.snap.subscribe(tenant) 240 snaps, unsub := a.snap.subscribe(tenant)
245 defer unsub() 241 defer unsub()
246 242
internal/server/api/exposures.go
Old New
@@ -93,8 +93,6 @@ func toExposureSessions(s *registry.ExposureSessions) *types.ExposureSessions {
93 return &types.ExposureSessions{Active: s.Active, Refused: s.Refused, Dropped: s.Dropped} 93 return &types.ExposureSessions{Active: s.Active, Refused: s.Refused, Dropped: s.Dropped}
94 } 94 }
95 95
96 // toExposureResponse merges a durable exposure row with its host's address and
97 // live reported state into the wire shape.
98 func toExposureResponse(e store.Exposure, hostAddr string, statuses []registry.ExposureStatus) types.Exposure { 96 func toExposureResponse(e store.Exposure, hostAddr string, statuses []registry.ExposureStatus) types.Exposure {
99 state, reason, sessions := exposureState(statuses, e.ID) 97 state, reason, sessions := exposureState(statuses, e.ID)
100 return types.Exposure{ 98 return types.Exposure{
@@ -120,7 +118,6 @@ func (a *API) hostFacing(hostID string) (string, []registry.ExposureStatus) {
120 return addr, nil 118 return addr, nil
121 } 119 }
122 120
123 // handleCreateExposure publishes one guest port of one VM on that VM's host.
124 func (a *API) handleCreateExposure(w http.ResponseWriter, r *http.Request) { 121 func (a *API) handleCreateExposure(w http.ResponseWriter, r *http.Request) {
125 vm, ok := a.exposureVM(w, r) 122 vm, ok := a.exposureVM(w, r)
126 if !ok { 123 if !ok {
@@ -178,7 +175,6 @@ func (a *API) handleCreateExposure(w http.ResponseWriter, r *http.Request) {
178 writeJSON(w, http.StatusCreated, toExposureResponse(e, addr, statuses)) 175 writeJSON(w, http.StatusCreated, toExposureResponse(e, addr, statuses))
179 } 176 }
180 177
181 // handleListExposures lists one VM's published ports, lowest host port first.
182 func (a *API) handleListExposures(w http.ResponseWriter, r *http.Request) { 178 func (a *API) handleListExposures(w http.ResponseWriter, r *http.Request) {
183 vm, ok := a.exposureVM(w, r) 179 vm, ok := a.exposureVM(w, r)
184 if !ok { 180 if !ok {
internal/server/api/principal.go
Old New
@@ -50,7 +50,6 @@ func mayActAs(p Principal, tenant string) bool {
50 return tenant != "" && p.Tenant == tenant 50 return tenant != "" && p.Tenant == tenant
51 } 51 }
52 52
53 // filterVMs returns only the VMs p may act on.
54 func filterVMs(p Principal, vms []store.VM) []store.VM { 53 func filterVMs(p Principal, vms []store.VM) []store.VM {
55 out := make([]store.VM, 0, len(vms)) 54 out := make([]store.VM, 0, len(vms))
56 for _, vm := range vms { 55 for _, vm := range vms {
internal/server/api/ratelimit.go
Old New
@@ -39,7 +39,6 @@ func newIPLimiter(now func() time.Time) *ipLimiter {
39 return &ipLimiter{buckets: map[string]*bucket{}, now: now} 39 return &ipLimiter{buckets: map[string]*bucket{}, now: now}
40 } 40 }
41 41
42 // allow reports whether ip may proceed, consuming one token if so.
43 func (l *ipLimiter) allow(ip string) bool { 42 func (l *ipLimiter) allow(ip string) bool {
44 l.mu.Lock() 43 l.mu.Lock()
45 defer l.mu.Unlock() 44 defer l.mu.Unlock()
@@ -64,7 +63,6 @@ func (l *ipLimiter) allow(ip string) bool {
64 return true 63 return true
65 } 64 }
66 65
67 // prune drops buckets idle past a full refill; reports whether any were freed.
68 func (l *ipLimiter) prune(now time.Time) bool { 66 func (l *ipLimiter) prune(now time.Time) bool {
69 idle := time.Duration(enrollBurst) * enrollRefillEvery 67 idle := time.Duration(enrollBurst) * enrollRefillEvery
70 freed := false 68 freed := false
internal/server/api/sshcert.go
Old New
@@ -83,8 +83,6 @@ func (a *API) handleRevokeSSHCert(w http.ResponseWriter, r *http.Request) {
83 http.Error(w, "not a certificate", http.StatusBadRequest) 83 http.Error(w, "not a certificate", http.StatusBadRequest)
84 return 84 return
85 } 85 }
86 // If the cert is provably signed by another tenant's registered CA, this
87 // is a cross-tenant revoke: answer 404, exactly like a foreign VM.
88 if owner, ok, err := a.st.TenantForUserCA(sshca.AuthorizedKeyLine(cert.SignatureKey)); err != nil { 86 if owner, ok, err := a.st.TenantForUserCA(sshca.AuthorizedKeyLine(cert.SignatureKey)); err != nil {
89 http.Error(w, "internal error", http.StatusInternalServerError) 87 http.Error(w, "internal error", http.StatusInternalServerError)
90 return 88 return
internal/server/api/ticket.go
Old New
@@ -65,7 +65,7 @@ func (t *ticketStore) consume(tick string) (tenant string, ok bool) {
65 if !found { 65 if !found {
66 return "", false 66 return "", false
67 } 67 }
68 delete(t.tickets, tick) // one-time, even when expired 68 delete(t.tickets, tick)
69 if t.now().After(e.exp) { 69 if t.now().After(e.exp) {
70 return "", false 70 return "", false
71 } 71 }
internal/server/api/types/lifecycle_projection_test.go
Old New
@@ -0,0 +1,64 @@
1 package types_test
2
3 import (
4 "go/ast"
5 "go/parser"
6 "go/token"
7 "os"
8 "path/filepath"
9 "strings"
10 "testing"
11 )
12
13 // controlLoopDirs are the packages that own a VM's real state axes: the agent
14 // side that converges guests, and the server side that stores and reaps them.
15 var controlLoopDirs = []string{
16 "internal/agent",
17 "internal/server/store",
18 "internal/server/syncsvc",
19 "internal/server/hub",
20 }
21
22 // TestLifecycleIsNeverReadByAControlLoop pins the one claim types.VM.Lifecycle
23 // makes that nothing else can catch: it is a lossy projection computed on read,
24 // like a Pod's status.phase, and the loops that decide what a VM should do must
25 // branch on the axes it is derived from (deleted / phase / power) rather than on
26 // the rollup. A loop that reads Lifecycle would be deciding from a summary the
27 // server wrote for clients, and would silently disagree with itself the moment
28 // the projection changed. api.go DERIVES the field and mcpserver READS it, and
29 // both are correct — one is the producer, the other a client.
30 func TestLifecycleIsNeverReadByAControlLoop(t *testing.T) {
31 root := repoRoot(t)
32 fset := token.NewFileSet()
33 walked := 0
34
35 for _, dir := range controlLoopDirs {
36 err := filepath.WalkDir(filepath.Join(root, dir), func(path string, d os.DirEntry, err error) error {
37 if err != nil || d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
38 return err
39 }
40 walked++
41 f, perr := parser.ParseFile(fset, path, nil, 0)
42 if perr != nil {
43 t.Fatalf("parse %s: %v", path, perr)
44 }
45 ast.Inspect(f, func(n ast.Node) bool {
46 sel, ok := n.(*ast.SelectorExpr)
47 if !ok || sel.Sel.Name != "Lifecycle" {
48 return true
49 }
50 rel, _ := filepath.Rel(root, path)
51 t.Errorf("%s:%d reads .Lifecycle — it is a read-time projection for clients, not state; branch on the VM's deleted/phase/power axes instead",
52 rel, fset.Position(sel.Pos()).Line)
53 return true
54 })
55 return nil
56 })
57 if err != nil {
58 t.Fatalf("walk %s: %v", dir, err)
59 }
60 }
61 if walked == 0 {
62 t.Fatal("walked no control-loop sources — the package layout changed and this test stopped guarding anything")
63 }
64 }
internal/server/api/types/types.go
Old New
@@ -15,9 +15,8 @@ import (
15 "time" 15 "time"
16 ) 16 )
17 17
18 // Capacity is the snake_case wire form of a host resource triple. It appears 18 // Capacity appears twice per Host: the host's TOTALS (capacity) and the amount
19 // twice per Host in GET /api/v1/hosts and the SSE snapshot: as the host's 19 // committed to live VMs (allocated).
20 // TOTALS (capacity) and as the amount committed to live VMs (allocated).
21 type Capacity struct { 20 type Capacity struct {
22 VCPUs int64 `json:"vcpus"` 21 VCPUs int64 `json:"vcpus"`
23 MemMB int64 `json:"mem_mb"` 22 MemMB int64 `json:"mem_mb"`
@@ -49,9 +48,7 @@ type PendingUpgrade struct {
49 AgeS int64 `json:"age_s"` 48 AgeS int64 `json:"age_s"`
50 } 49 }
51 50
52 // Host is the explicit snake_case wire representation of a host, served by 51 // Host is served by GET /api/v1/hosts and the SSE snapshot.
53 // GET /api/v1/hosts and the SSE snapshot. Every field is spelled out — no
54 // struct embedding — to prevent PascalCase field leakage.
55 type Host struct { 52 type Host struct {
56 ID string `json:"id"` 53 ID string `json:"id"`
57 Name string `json:"name"` 54 Name string `json:"name"`
@@ -104,10 +101,8 @@ type Host struct {
104 Metrics *Metrics `json:"metrics"` 101 Metrics *Metrics `json:"metrics"`
105 } 102 }
106 103
107 // VM is the explicit snake_case wire representation of a VM, served by 104 // VM is served by GET /api/v1/vms and the SSE snapshot. Write-only fields —
108 // GET /api/v1/vms and the SSE snapshot. Write-only fields — image_sha256, 105 // image_sha256, cloud_init, ssh_authorized_key — are deliberately excluded.
109 // cloud_init, ssh_authorized_key — are deliberately excluded. Every field is
110 // spelled out — no struct embedding.
111 type VM struct { 106 type VM struct {
112 ID string `json:"id"` 107 ID string `json:"id"`
113 HostID string `json:"host_id"` 108 HostID string `json:"host_id"`
@@ -150,13 +145,10 @@ type VM struct {
150 // guest stopped, awaiting the tombstone grace window); 0 in the normal case. 145 // guest stopped, awaiting the tombstone grace window); 0 in the normal case.
151 // Clients render a countdown instead of an opaque "deleting". 146 // Clients render a countdown instead of an opaque "deleting".
152 DestroyAt int64 `json:"destroy_at"` 147 DestroyAt int64 `json:"destroy_at"`
153 // Lifecycle is a server-derived rollup of the orthogonal state axes above 148 // Lifecycle rolls the axes above (deleted / phase / power) into one coarse
154 // (deleted / phase / power) into a single coarse word, so 149 // word so every client agrees on "what is this VM doing" without
155 // every client agrees on "what is this VM doing" without re-deriving it. 150 // re-deriving it: creating | ready | stopped | failed | deleting. A lossy
156 // It is NOT authoritative and NOT stored: like a Kubernetes Pod's 151 // read-time projection, never state — see TestLifecycleIsNeverReadByAControlLoop.
157 // status.phase, it is a lossy projection computed on read — the control
158 // loops (reconciler, reaper, agent) read and write the underlying axes,
159 // never this field. Values: creating | ready | stopped | failed | deleting.
160 Lifecycle string `json:"lifecycle"` 152 Lifecycle string `json:"lifecycle"`
161 // InjectedKey describes the authorized key EITRI installed in this guest at 153 // InjectedKey describes the authorized key EITRI installed in this guest at
162 // create, or null when it installed none. It is a description, not the key: 154 // create, or null when it installed none. It is a description, not the key:
@@ -176,10 +168,6 @@ type VM struct {
176 // different from an empty list, which cannot occur — create refuses a 168 // different from an empty list, which cannot occur — create refuses a
177 // tenant with no CA — so clients must not conflate them. 169 // tenant with no CA — so clients must not conflate them.
178 // 170 //
179 // A POINTER to the slice, for that distinction alone: a plain nil slice
180 // also marshals to null, but the spec generator reads nullability off the
181 // pointer, so a bare slice would be published as an always-present array
182 // and the contract would promise something the server does not deliver.
183 TrustedCAs *[]TrustedCA `json:"trusted_cas"` 171 TrustedCAs *[]TrustedCA `json:"trusted_cas"`
184 } 172 }
185 173
@@ -225,9 +213,12 @@ type AuditEvent struct {
225 } 213 }
226 214
227 // RevokedCert is the wire form of one revoked SSH cert, served by 215 // RevokedCert is the wire form of one revoked SSH cert, served by
228 // GET /api/v1/ssh-certs/revoked. Serial is a STRING, not a JSON number: a 216 // GET /api/v1/ssh-certs/revoked.
229 // uint64 serial routinely exceeds 2^53 and would lose precision in a 217 //
230 // JavaScript client that parsed it as a double. 218 // Serial is a string because SSH serials are uint64 and routinely exceed 2^53,
219 // which any client parsing JSON numbers as a double silently rounds. Widening
220 // it to uint64 is a compile error at the call sites, and the compiler cannot
221 // say why — so it says it here.
231 type RevokedCert struct { 222 type RevokedCert struct {
232 Serial string `json:"serial"` 223 Serial string `json:"serial"`
233 RevokedAt time.Time `json:"revoked_at"` 224 RevokedAt time.Time `json:"revoked_at"`
internal/server/api/usercas.go
Old New
@@ -80,9 +80,6 @@ func (a *API) handleUploadUserCA(w http.ResponseWriter, r *http.Request) {
80 writeJSON(w, http.StatusCreated, types.UserCAUploadResponse{Fingerprint: ssh.FingerprintSHA256(pub)}) 80 writeJSON(w, http.StatusCreated, types.UserCAUploadResponse{Fingerprint: ssh.FingerprintSHA256(pub)})
81 } 81 }
82 82
83 // handleListUserCAs lists a tenant's registered user CAs (pubkey + label + fp)
84 // — the {tenant} path segment on the explicit route, or the caller's own tenant
85 // on the tenant-less sibling.
86 func (a *API) handleListUserCAs(w http.ResponseWriter, r *http.Request) { 83 func (a *API) handleListUserCAs(w http.ResponseWriter, r *http.Request) {
87 tenant, ok := a.userCATenant(w, r) 84 tenant, ok := a.userCATenant(w, r)
88 if !ok { 85 if !ok {
internal/server/store/exposures.go
Old New
@@ -49,8 +49,6 @@ var ErrNoFreeHostPort = errors.New("no free host port in the reserved range")
49 // of the contract each `from` clause below keeps. 49 // of the contract each `from` clause below keeps.
50 const exposureColumns = `e.id, e.tenant, e.vm_id, e.host_id, e.guest_port, e.host_port, e.protocol, e.scope, e.created_at` 50 const exposureColumns = `e.id, e.tenant, e.vm_id, e.host_id, e.guest_port, e.host_port, e.protocol, e.scope, e.created_at`
51 51
52 // scanExposure's column order must match exposureColumns exactly — it is
53 // positional, not name-based.
54 func scanExposure(rows *sql.Rows) (Exposure, error) { 52 func scanExposure(rows *sql.Rows) (Exposure, error) {
55 var e Exposure 53 var e Exposure
56 var createdAt string 54 var createdAt string
@@ -97,9 +95,6 @@ func (s *Store) CreateExposure(vmID string, guestPort, hostPort int64, protocol
97 } 95 }
98 defer tx.Rollback() 96 defer tx.Rollback()
99 97
100 // The VM read is the derivation point: an exposure's tenant is ALWAYS its
101 // VM's, and its host is the host the VM is placed on. A tombstoned VM is
102 // treated as absent — its guest is on its way out.
103 var tenant, hostID string 98 var tenant, hostID string
104 switch err := tx.QueryRow( 99 switch err := tx.QueryRow(
105 `SELECT tenant, host_id FROM vms WHERE id=? AND deleted_at IS NULL`, vmID, 100 `SELECT tenant, host_id FROM vms WHERE id=? AND deleted_at IS NULL`, vmID,
internal/server/store/identity.go
Old New
@@ -87,9 +87,6 @@ func (s *Store) CreateTenantForIdentity(issuer, subject, email string) (Tenant,
87 base := handleFromEmail(email) 87 base := handleFromEmail(email)
88 now := time.Now().UTC().Format(time.RFC3339) 88 now := time.Now().UTC().Format(time.RFC3339)
89 89
90 // Try handle, handle-2, handle-3… on id (PRIMARY KEY, errno 1555) collision;
91 // SystemTenant is reserved so we skip it. A collision on the identity index
92 // (errno 2067) is terminal: the identity already owns a tenant.
93 for attempt := 1; ; attempt++ { 90 for attempt := 1; ; attempt++ {
94 candidate := base 91 candidate := base
95 if attempt > 1 { 92 if attempt > 1 {
internal/server/store/store.go
Old New
@@ -23,14 +23,11 @@ import (
23 sqlite "modernc.org/sqlite" 23 sqlite "modernc.org/sqlite"
24 ) 24 )
25 25
26 // ErrNameTaken is returned by CreateVM when the name is already in use by a live VM in the same tenant.
27 var ErrNameTaken = errors.New("vm name already in use") 26 var ErrNameTaken = errors.New("vm name already in use")
28 27
29 // ErrHostNotFound is returned by CreateVM when the host_id does not exist. 28 // ErrHostNotFound is returned by CreateVM when the host_id does not exist.
30 var ErrHostNotFound = errors.New("host not found") 29 var ErrHostNotFound = errors.New("host not found")
31 30
32 // ErrHostNotEnrolled is returned by CreateVM when the target host exists but is
33 // not accepting new VMs (e.g. it is decommissioning).
34 var ErrHostNotEnrolled = errors.New("host not accepting new VMs") 31 var ErrHostNotEnrolled = errors.New("host not accepting new VMs")
35 32
36 // SystemTenant is the audit scope for events with no resolvable tenant — a 33 // SystemTenant is the audit scope for events with no resolvable tenant — a
@@ -207,7 +204,6 @@ CREATE TABLE IF NOT EXISTS vms (
207 tenant TEXT NOT NULL DEFAULT 'default' REFERENCES tenants(id) 204 tenant TEXT NOT NULL DEFAULT 'default' REFERENCES tenants(id)
208 ); 205 );
209 206
210 -- vms name uniqueness is per-tenant.
211 CREATE UNIQUE INDEX IF NOT EXISTS vms_tenant_name ON vms(tenant, name) WHERE deleted_at IS NULL; 207 CREATE UNIQUE INDEX IF NOT EXISTS vms_tenant_name ON vms(tenant, name) WHERE deleted_at IS NULL;
212 208
213 -- revoked SSH user certs: a tenant can revoke a specific user cert by its serial 209 -- revoked SSH user certs: a tenant can revoke a specific user cert by its serial
@@ -261,14 +257,6 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_tenant_user_cas_pubkey ON tenant_user_cas(
261 -- table, and the unique index IS the collision check — a port is claimed by 257 -- table, and the unique index IS the collision check — a port is claimed by
262 -- whoever inserts first, and the second insert fails at the storage layer 258 -- whoever inserts first, and the second insert fails at the storage layer
263 -- rather than after a read-then-write race. 259 -- rather than after a read-then-write race.
264 --
265 -- protocol is 'tcp' or 'udp' — two different ports on the same number, which
266 -- is why it is part of the unique index. scope is 'lan', stored so 'public' is
267 -- an additive value later rather than a migration.
268 --
269 -- ON DELETE CASCADE against vms is what keeps the promise that no listener
270 -- outlives the thing it pointed at: hard-deleting a reaped VM takes its
271 -- exposures with it, with no sweep to write and none to forget.
272 CREATE TABLE IF NOT EXISTS exposures ( 260 CREATE TABLE IF NOT EXISTS exposures (
273 id TEXT PRIMARY KEY, 261 id TEXT PRIMARY KEY,
274 tenant TEXT NOT NULL REFERENCES tenants(id), 262 tenant TEXT NOT NULL REFERENCES tenants(id),
@@ -583,8 +571,6 @@ func (s *Store) RedeemEnrollmentToken(tok string, f EnrollFacts) (Host, error) {
583 return Host{}, fmt.Errorf("token invalid, expired, or already used") 571 return Host{}, fmt.Errorf("token invalid, expired, or already used")
584 } 572 }
585 573
586 // The consumed token's tenant becomes the host's — the derivation chain
587 // (token → host → VM) starts here.
588 var tenant string 574 var tenant string
589 if err := tx.QueryRow(`SELECT tenant FROM enrollment_tokens WHERE token_hash=?`, hash).Scan(&tenant); err != nil { 575 if err := tx.QueryRow(`SELECT tenant FROM enrollment_tokens WHERE token_hash=?`, hash).Scan(&tenant); err != nil {
590 return Host{}, fmt.Errorf("read token tenant: %w", err) 576 return Host{}, fmt.Errorf("read token tenant: %w", err)
@@ -845,10 +831,6 @@ func (s *Store) CreateVM(vm VM) error {
845 vm.ID = random.Hex(16) 831 vm.ID = random.Hex(16)
846 } 832 }
847 833
848 // Refuse to place a VM on a host that is not enrolled, and read the host's
849 // tenant in the SAME tx: a VM's tenant is ALWAYS its host's — derived
850 // here, never accepted from the caller (the API handler cannot override
851 // it; this is the enforcement point for the partition invariant).
852 var hostStatus, hostTenant string 834 var hostStatus, hostTenant string
853 switch err := tx.QueryRow(`SELECT status, tenant FROM hosts WHERE id=?`, vm.HostID).Scan(&hostStatus, &hostTenant); { 835 switch err := tx.QueryRow(`SELECT status, tenant FROM hosts WHERE id=?`, vm.HostID).Scan(&hostStatus, &hostTenant); {
854 case errors.Is(err, sql.ErrNoRows): 836 case errors.Is(err, sql.ErrNoRows):
@@ -1143,8 +1125,6 @@ func (s *Store) ListVMEvents(tenant, vmID string, limit int) ([]AuditEntry, erro
1143 // were removed. Retention keeps the append-only log bounded; the caller 1125 // were removed. Retention keeps the append-only log bounded; the caller
1144 // (eitri-server) runs it at startup and daily. 1126 // (eitri-server) runs it at startup and daily.
1145 func (s *Store) PruneAudit(olderThan time.Duration) (int64, error) { 1127 func (s *Store) PruneAudit(olderThan time.Duration) (int64, error) {
1146 // Guard the contract, not just the caller: a zero/negative window would
1147 // compute a now-or-future cutoff and wipe the entire forensic trail.
1148 if olderThan <= 0 { 1128 if olderThan <= 0 {
1149 return 0, nil 1129 return 0, nil
1150 } 1130 }
@@ -1509,8 +1489,6 @@ const vmColumns = `id, host_id, name, tenant, image_url, image_sha256, cloud_ini
1509 vcpus, mem_mb, disk_gb, power_state, network, status, last_error, assigned_ip, network_ip, 1489 vcpus, mem_mb, disk_gb, power_state, network, status, last_error, assigned_ip, network_ip,
1510 created_at, deleted_at` 1490 created_at, deleted_at`
1511 1491
1512 // scanVM's column order must match vmColumns exactly — it is positional, not
1513 // name-based.
1514 func scanVM(rows *sql.Rows) (VM, error) { 1492 func scanVM(rows *sql.Rows) (VM, error) {
1515 var vm VM 1493 var vm VM
1516 var createdAt string 1494 var createdAt string
internal/server/syncsvc/hostcert.go
Old New
@@ -9,8 +9,7 @@ import (
9 ) 9 )
10 10
11 // hostCertSigner signs a guest's public host key for the principal the control 11 // hostCertSigner signs a guest's public host key for the principal the control
12 // plane chose. Nil when the fleet has no SSH CA — there is then no certificate 12 // plane chose.
13 // to issue, and buildSnapshot tells hosts not to wait for one.
14 type hostCertSigner interface { 13 type hostCertSigner interface {
15 SignHostCert(pub ssh.PublicKey, principal string) (certLine string, err error) 14 SignHostCert(pub ssh.PublicKey, principal string) (certLine string, err error)
16 } 15 }
@@ -21,12 +20,6 @@ type hostCertSigner interface {
21 func (s *Service) SetHostCertSigner(c hostCertSigner) { s.certs = c } 20 func (s *Service) SetHostCertSigner(c hostCertSigner) { s.certs = c }
22 21
23 // signAndRecordHostCert certifies one guest's host key. 22 // signAndRecordHostCert certifies one guest's host key.
24 //
25 // The AGENT SUPPLIES A KEY, NEVER A NAME. The principal is derived here, from
26 // the VM's own row — so a host cannot obtain a certificate for a name it does
27 // not own, and the host_id predicate inside RecordVMHostKey means it cannot
28 // obtain one for another host's VM either. Those two together are the whole
29 // authorization story for this exchange.
30 func (s *Service) signAndRecordHostCert(hostID, vmID, pubLine string) error { 23 func (s *Service) signAndRecordHostCert(hostID, vmID, pubLine string) error {
31 vm, err := s.st.GetVM(vmID) 24 vm, err := s.st.GetVM(vmID)
32 if err != nil { 25 if err != nil {
@@ -42,8 +35,6 @@ func (s *Service) signAndRecordHostCert(hostID, vmID, pubLine string) error {
42 if err != nil { 35 if err != nil {
43 return fmt.Errorf("parse reported host key: %w", err) 36 return fmt.Errorf("parse reported host key: %w", err)
44 } 37 }
45 // <tenant>.<name> is the connect name clients dial and verify, the same
46 // principal the gate resolves a VM by.
47 cert, err := s.certs.SignHostCert(pub, vm.Tenant+"."+vm.Name) 38 cert, err := s.certs.SignHostCert(pub, vm.Tenant+"."+vm.Name)
48 if err != nil { 39 if err != nil {
49 return fmt.Errorf("sign host cert: %w", err) 40 return fmt.Errorf("sign host cert: %w", err)
@@ -51,8 +42,6 @@ func (s *Service) signAndRecordHostCert(hostID, vmID, pubLine string) error {
51 if err := s.st.RecordVMHostKey(vmID, hostID, pubLine, cert); err != nil { 42 if err := s.st.RecordVMHostKey(vmID, hostID, pubLine, cert); err != nil {
52 return fmt.Errorf("record host cert: %w", err) 43 return fmt.Errorf("record host cert: %w", err)
53 } 44 }
54 // Push rather than wait for the next tick: the guest is not booting until
55 // this certificate reaches it.
56 s.hub.Poke(hostID) 45 s.hub.Poke(hostID)
57 return nil 46 return nil
58 } 47 }
internal/server/syncsvc/syncsvc.go
Old New
@@ -20,10 +20,6 @@ import (
20 "github.com/quic-go/quic-go" 20 "github.com/quic-go/quic-go"
21 ) 21 )
22 22
23 // defaultWriteTimeout bounds each down-stream snapshot write so a stalled or
24 // malicious agent (one that keeps the QUIC connection alive but stops reading
25 // the down-stream) cannot pin server goroutines once the flow-control window
26 // fills.
27 const defaultWriteTimeout = 30 * time.Second 23 const defaultWriteTimeout = 30 * time.Second
28 24
29 // vmStatusRecorder is the durable-write seam applyReport uses to record VM 25 // vmStatusRecorder is the durable-write seam applyReport uses to record VM
@@ -31,7 +27,6 @@ const defaultWriteTimeout = 30 * time.Second
31 // direct s.st call) lets tests substitute a counting fake to prove that 27 // direct s.st call) lets tests substitute a counting fake to prove that
32 // unchanged reports perform no write. 28 // unchanged reports perform no write.
33 type vmStatusRecorder interface { 29 type vmStatusRecorder interface {
34 // RecordVMStatus returns the address it wrote, empty when it wrote none.
35 RecordVMStatus(id, status, lastErr, ip string) (string, error) 30 RecordVMStatus(id, status, lastErr, ip string) (string, error)
36 } 31 }
37 32
@@ -48,8 +43,7 @@ type Service struct {
48 recorder vmStatusRecorder 43 recorder vmStatusRecorder
49 tracker *statusTracker 44 tracker *statusTracker
50 netTrack *netTracker 45 netTrack *netTracker
51 // certs signs a guest's reported host key. Nil when the fleet has no SSH 46 // certs signs a guest's reported host key.
52 // CA: there is then no certificate to issue and none to require.
53 certs hostCertSigner 47 certs hostCertSigner
54 // certTrack remembers the public key each VM was last certified for, so the 48 // certTrack remembers the public key each VM was last certified for, so the
55 // steady state — every host repeating every VM's key every tick, forever — 49 // steady state — every host repeating every VM's key every tick, forever —
@@ -179,7 +173,6 @@ func (s *Service) handleConn(ctx context.Context, conn quic.Connection) {
179 _ = conn.CloseWithError(transport.CodeAuthRejected, "first frame must be Hello") 173 _ = conn.CloseWithError(transport.CodeAuthRejected, "first frame must be Hello")
180 return 174 return
181 } 175 }
182 // Auth: the Hello carries a Bearer host credential (string credential = 9).
183 cred := h.GetCredential() 176 cred := h.GetCredential()
184 // host_id in Hello is advisory; the authenticated identity comes from the credential (hosttoken.Verify), so a spoofed Hello.HostId cannot mislead us. 177 // host_id in Hello is advisory; the authenticated identity comes from the credential (hosttoken.Verify), so a spoofed Hello.HostId cannot mislead us.
185 claims, ok := hosttoken.Verify(s.secret, cred) 178 claims, ok := hosttoken.Verify(s.secret, cred)
@@ -229,14 +222,6 @@ func (s *Service) handleConn(ctx context.Context, conn quic.Connection) {
229 // Console reachability: only now — with the down-stream open — is it safe 222 // Console reachability: only now — with the down-stream open — is it safe
230 // for OpenConsole to add streams to this connection (stream-order 223 // for OpenConsole to add streams to this connection (stream-order
231 // invariant: the snapshot down-stream is always the first accepted). 224 // invariant: the snapshot down-stream is always the first accepted).
232 //
233 // One host, one session: a second Hello for a host already connected wins,
234 // and the session it displaces is closed rather than abandoned. Two agents
235 // sharing an identity are a misconfiguration, and the newest connection is
236 // the one that just proved it holds the credential — but the displaced side
237 // must SEE the close, or it sits there reporting into a stream that has no
238 // reader. CodeSuperseded puts it on the normal reconnect backoff, so a host
239 // whose agent was merely restarted comes straight back.
240 s.consoleMu.Lock() 225 s.consoleMu.Lock()
241 displaced := s.conns[hostID] 226 displaced := s.conns[hostID]
242 s.conns[hostID] = conn 227 s.conns[hostID] = conn
@@ -249,7 +234,6 @@ func (s *Service) handleConn(ctx context.Context, conn quic.Connection) {
249 } 234 }
250 defer func() { 235 defer func() {
251 s.consoleMu.Lock() 236 s.consoleMu.Lock()
252 // Only deregister OUR conn: a reconnect may already have replaced it.
253 if s.conns[hostID] == conn { 237 if s.conns[hostID] == conn {
254 delete(s.conns, hostID) 238 delete(s.conns, hostID)
255 } 239 }
@@ -267,7 +251,6 @@ func (s *Service) handleConn(ctx context.Context, conn quic.Connection) {
267 defer cancel() 251 defer cancel()
268 sendErr := make(chan error, 1) 252 sendErr := make(chan error, 1)
269 go func() { 253 go func() {
270 // Initial snapshot first (single-sender rule), then one per poke.
271 if err := s.pushSnapshot(down, hostID); err != nil { 254 if err := s.pushSnapshot(down, hostID); err != nil {
272 s.failWrite(conn, hostID, err) 255 s.failWrite(conn, hostID, err)
273 sendErr <- err 256 sendErr <- err
@@ -293,9 +276,6 @@ func (s *Service) handleConn(ctx context.Context, conn quic.Connection) {
293 return 276 return
294 } 277 }
295 if rep := msg.GetReport(); rep != nil { 278 if rep := msg.GetReport(); rep != nil {
296 // Re-check the credential each report so a revoke (or max-age
297 // expiry) lands within one tick (~10s) instead of waiting for the
298 // session to drop naturally.
299 row, err := s.st.GetHost(hostID) 279 row, err := s.st.GetHost(hostID)
300 if err != nil { 280 if err != nil {
301 // Transient store failure — NOT an auth verdict. Close with a 281 // Transient store failure — NOT an auth verdict. Close with a
@@ -334,21 +314,11 @@ func (s *Service) buildSnapshot(hostID string) (*pb.Snapshot, error) {
334 snap.AgentUpgrade = s.offerFor(hostID) 314 snap.AgentUpgrade = s.offerFor(hostID)
335 caCache := map[string][]string{} // tenant -> canonical CA lines, legacy rows only 315 caCache := map[string][]string{} // tenant -> canonical CA lines, legacy rows only
336 for _, v := range vms { 316 for _, v := range vms {
337 // The CA set a guest is built to trust is the one frozen onto its row
338 // at create. Serving it from here — rather than re-reading the tenant's
339 // CURRENT set on every push, as this once did — is what makes that
340 // trust decided at create in fact and not just in the documentation: a
341 // CA registered after a VM exists no longer reaches it, however long
342 // that VM has been waiting on an image download or a host certificate.
343 cas := make([]string, 0, len(v.TrustedCAs)) 317 cas := make([]string, 0, len(v.TrustedCAs))
344 for _, c := range v.TrustedCAs { 318 for _, c := range v.TrustedCAs {
345 cas = append(cas, c.AuthorizedKey) 319 cas = append(cas, c.AuthorizedKey)
346 } 320 }
347 321
348 // A row that recorded nothing has no set to serve — so fall back to the
349 // tenant's live set, which is exactly what this VM would have been sent
350 // before the freeze existed.
351 //
352 // This is now vestigial: store.Open freezes a set onto every unrecorded 322 // This is now vestigial: store.Open freezes a set onto every unrecorded
353 // row it finds, so by the time a snapshot is built there are none left. 323 // row it finds, so by the time a snapshot is built there are none left.
354 // It survives as a safety net for the one row the backfill cannot have 324 // It survives as a safety net for the one row the backfill cannot have
@@ -377,11 +347,6 @@ func (s *Service) buildSnapshot(hostID string) (*pb.Snapshot, error) {
377 snap.Vms = append(snap.Vms, &pb.VMSpec{ 347 snap.Vms = append(snap.Vms, &pb.VMSpec{
378 VmId: v.ID, Name: v.Name, ImageUrl: v.ImageURL, ImageSha256: v.ImageSHA256, 348 VmId: v.ID, Name: v.Name, ImageUrl: v.ImageURL, ImageSha256: v.ImageSHA256,
379 CloudInit: v.CloudInit, Vcpus: v.VCPUs, MemMb: v.MemMB, DiskGb: v.DiskGB, 349 CloudInit: v.CloudInit, Vcpus: v.VCPUs, MemMb: v.MemMB, DiskGb: v.DiskGB,
380 // Always true, and no longer read from anywhere: every VM is
381 // persistent, so there is nothing left to look up. It stays on the
382 // wire for the agents that still read it — a v0.0.5 agent handed the
383 // proto3 default takes absence for "ephemeral" and marks a guest
384 // failed forever the first time its host reboots. See sync.proto.
385 Persistent: true, PowerState: v.PowerState, Tombstoned: v.DeletedAt != nil, 350 Persistent: true, PowerState: v.PowerState, Tombstoned: v.DeletedAt != nil,
386 SshAuthorizedKey: v.SSHAuthorizedKey, 351 SshAuthorizedKey: v.SSHAuthorizedKey,
387 SshUserCaAuthorizedKeys: cas, 352 SshUserCaAuthorizedKeys: cas,
@@ -421,10 +386,6 @@ func (s *Service) pushSnapshot(down quic.Stream, hostID string) error {
421 if err != nil { 386 if err != nil {
422 return err 387 return err
423 } 388 }
424 // Bound the write: if a stalled agent stops reading the down-stream but keeps
425 // the connection alive, the flow-control window fills and an unbounded Write
426 // would block forever, pinning this goroutine. The deadline turns that into a
427 // write error, which the caller uses to close the connection.
428 if err := down.SetWriteDeadline(time.Now().Add(s.writeTimeout)); err != nil { 389 if err := down.SetWriteDeadline(time.Now().Add(s.writeTimeout)); err != nil {
429 return fmt.Errorf("set write deadline: %w", err) 390 return fmt.Errorf("set write deadline: %w", err)
430 } 391 }
@@ -454,11 +415,6 @@ func (s *Service) offerFor(hostID string) *pb.AgentUpgrade {
454 // PendingAgentUpgrade reports hostID's outstanding offer: the version offered 415 // PendingAgentUpgrade reports hostID's outstanding offer: the version offered
455 // and how long it has been standing. ok is false when there is none. 416 // and how long it has been standing. ok is false when there is none.
456 // 417 //
457 // The age is derived here, at read time, rather than served as an absolute
458 // instant — the same choice registry.HostState makes for SinceLastSeen, and for
459 // the same reason: a browser comparing our timestamp against its own clock
460 // would be reporting the difference between two clocks as well as the wait.
461 //
462 // Offers live in memory only. A server restart forgets them, so the pending 418 // Offers live in memory only. A server restart forgets them, so the pending
463 // state disappears and the button comes back — which is the honest answer, 419 // state disappears and the button comes back — which is the honest answer,
464 // because a restarted server has also forgotten to put the offer in the next 420 // because a restarted server has also forgotten to put the offer in the next
@@ -473,17 +429,12 @@ func (s *Service) PendingAgentUpgrade(hostID string) (version string, age time.D
473 return o.up.GetVersion(), s.now().Sub(o.offeredAt), true 429 return o.up.GetVersion(), s.now().Sub(o.offeredAt), true
474 } 430 }
475 431
476 // ClearAgentUpgrade drops any pending offer for hostID unconditionally —
477 // called when the host leaves the fleet (decommission/purge) so a stale offer
478 // cannot outlive its host.
479 func (s *Service) ClearAgentUpgrade(hostID string) { 432 func (s *Service) ClearAgentUpgrade(hostID string) {
480 s.offersMu.Lock() 433 s.offersMu.Lock()
481 defer s.offersMu.Unlock() 434 defer s.offersMu.Unlock()
482 delete(s.offers, hostID) 435 delete(s.offers, hostID)
483 } 436 }
484 437
485 // clearOfferIfDone drops hostID's pending offer once the agent reports the
486 // target version (from its Hello) — the offer has converged.
487 func (s *Service) clearOfferIfDone(hostID, reportedVersion string) { 438 func (s *Service) clearOfferIfDone(hostID, reportedVersion string) {
488 s.offersMu.Lock() 439 s.offersMu.Lock()
489 defer s.offersMu.Unlock() 440 defer s.offersMu.Unlock()
@@ -521,34 +472,10 @@ func (s *Service) applyReport(hostID string, rep *pb.Report) {
521 472
522 s.reg.UpdateReport(hostID, r) 473 s.reg.UpdateReport(hostID, r)
523 474
524 // Certify any guest host key this host has newly generated. Ahead of the
525 // status loop because it is a precondition for the phases that loop cares
526 // about: a VM awaiting its certificate has not booted yet.
527 s.certifyReportedHostKeys(hostID, rep.GetVms()) 475 s.certifyReportedHostKeys(hostID, rep.GetVms())
528 476
529 // Write-through durable status for lifecycle phases ready/failed only.
530 // Skip the SELECT+UPDATE when the durable triple (status, last_error,
531 // effective assigned_ip) is unchanged from the last write we recorded.
532 // writeThrough runs decide→write→commit atomically per VM so overlapping
533 // reports for the same host can never reorder the write and the cache commit;
534 // the cache is updated only after a successful write, so a rejected write
535 // never suppresses the next retry.
536 for _, v := range rep.GetVms() { 477 for _, v := range rep.GetVms() {
537 vmID := v.GetVmId() 478 vmID := v.GetVmId()
538 // The address the site's own DHCP server granted this guest on its
539 // named network. Recorded in every phase, not just the two below: the
540 // lease can land while the guest is still booting, and it is a fact
541 // about the guest's second NIC rather than about its lifecycle. Empty
542 // is "not discovered", never "gone" — the rule assigned_ip and
543 // guest_cidr both follow, enforced in the UPDATE itself.
544 //
545 // The cache below commits the RAW reported netIP, while the store may
546 // still drop it as unusable and keep the prior value — unlike
547 // statusTracker, this tracker never learns what the store actually
548 // wrote. That divergence is benign: a later good address still reads
549 // as a change and lands, and a repeat of the same unusable address is
550 // just a no-op skip, which is no different from what the store would
551 // have done with it anyway.
552 if netIP := v.GetNetworkIp(); netIP != "" { 479 if netIP := v.GetNetworkIp(); netIP != "" {
553 if err := s.netIPTrack.writeThrough(vmID, netIP, func() error { 480 if err := s.netIPTrack.writeThrough(vmID, netIP, func() error {
554 return s.st.RecordVMNetworkIP(vmID, hostID, netIP) 481 return s.st.RecordVMNetworkIP(vmID, hostID, netIP)
@@ -561,23 +488,14 @@ func (s *Service) applyReport(hostID string, rep *pb.Report) {
561 if phase != "ready" && phase != "failed" { 488 if phase != "ready" && phase != "failed" {
562 continue 489 continue
563 } 490 }
564 // Pass the RAW reported ip to RecordVMStatus to preserve its
565 // empty-ip-keeps-prior UPDATE semantics, and cache the address it
566 // reports back rather than the one we sent it.
567 err := s.tracker.writeThrough(vmID, phase, v.GetLastError(), v.GetIp(), func() (string, error) { 491 err := s.tracker.writeThrough(vmID, phase, v.GetLastError(), v.GetIp(), func() (string, error) {
568 return s.recorder.RecordVMStatus(vmID, phase, v.GetLastError(), v.GetIp()) 492 return s.recorder.RecordVMStatus(vmID, phase, v.GetLastError(), v.GetIp())
569 }) 493 })
570 if err != nil { 494 if err != nil {
571 // ErrNoRows (row gone) or a validation error: the write did not land
572 // and the cache is unchanged, so the next report retries.
573 slog.Warn("RecordVMStatus rejected", "vm", vmID, "host", hostID, "err", err) 495 slog.Warn("RecordVMStatus rejected", "vm", vmID, "host", hostID, "err", err)
574 } 496 }
575 } 497 }
576 498
577 // The subnet this host says its guests are on. Empty is "no answer", never
578 // "no network" — the same rule noteAddress follows for a VM's address, and
579 // for the same reason: a host that cannot see its own network yet must not
580 // erase what it told us when it could.
581 if cidr := rep.GetGuestCidr(); cidr != "" { 499 if cidr := rep.GetGuestCidr(); cidr != "" {
582 if err := s.netTrack.writeThrough(hostID, cidr, func() error { 500 if err := s.netTrack.writeThrough(hostID, cidr, func() error {
583 return s.st.RecordHostNetwork(hostID, cidr) 501 return s.st.RecordHostNetwork(hostID, cidr)
@@ -586,8 +504,6 @@ func (s *Service) applyReport(hostID string, rep *pb.Report) {
586 } 504 }
587 } 505 }
588 506
589 // The address this host says it answers on. Empty is "no answer", never
590 // "no address" — the same rule guest_cidr follows, and for the same reason.
591 if addr := rep.GetHostUplinkAddr(); addr != "" { 507 if addr := rep.GetHostUplinkAddr(); addr != "" {
592 if err := s.uplinkTrack.writeThrough(hostID, addr, func() error { 508 if err := s.uplinkTrack.writeThrough(hostID, addr, func() error {
593 return s.st.RecordHostUplink(hostID, addr) 509 return s.st.RecordHostUplink(hostID, addr)
@@ -602,10 +518,6 @@ func (s *Service) applyReport(hostID string, rep *pb.Report) {
602 "host", hostID, "agent_epoch", rep.GetLastSeenEpoch()) 518 "host", hostID, "agent_epoch", rep.GetLastSeenEpoch())
603 } 519 }
604 520
605 // Hard-delete each VM the agent has confirmed destroyed (level-triggered acks).
606 // Track whether any delete succeeded so we can poke the agent once after the
607 // loop — otherwise the agent holds a stale snapshot containing the tombstone
608 // and re-acks every tick forever (log spam) until some other edit pokes it.
609 // A reaped VM shares its host's tenant; resolve it once so the terminal 521 // A reaped VM shares its host's tenant; resolve it once so the terminal
610 // vm.reap row is tenant-scoped like the rest of the VM's timeline. The host 522 // vm.reap row is tenant-scoped like the rest of the VM's timeline. The host
611 // row still exists during graceful reap (RemoveHost waits for the drain); 523 // row still exists during graceful reap (RemoveHost waits for the drain);
@@ -702,10 +614,6 @@ func toRegistryExposures(in []*pb.ExposureStatus) []registry.ExposureStatus {
702 return out 614 return out
703 } 615 }
704 616
705 // toRegistrySessions maps an exposure's reported counters, keeping the one
706 // distinction the whole field exists for: an agent that reports no counters
707 // (nil) is not an exposure that has counted zero, and it must not become one on
708 // the way through.
709 func toRegistrySessions(s *pb.ExposureSessions) *registry.ExposureSessions { 617 func toRegistrySessions(s *pb.ExposureSessions) *registry.ExposureSessions {
710 if s == nil { 618 if s == nil {
711 return nil 619 return nil
@@ -727,9 +635,6 @@ func toRegistryCapacity(c *pb.Capacity) registry.Capacity {
727 // provisioner is a top-level Hello field rather than one of HostFacts, but it 635 // provisioner is a top-level Hello field rather than one of HostFacts, but it
728 // is the same KIND of fact — slow-changing host identity, refreshed on every 636 // is the same KIND of fact — slow-changing host identity, refreshed on every
729 // reconnect — so it is written by the same path instead of a second one. 637 // reconnect — so it is written by the same path instead of a second one.
730 //
731 // A nil facts block yields zero values, which UpdateHostFacts reads as "said
732 // nothing" and leaves the row alone.
733 func toStoreFacts(h *pb.Hello) store.HostFacts { 638 func toStoreFacts(h *pb.Hello) store.HostFacts {
734 f := h.GetFacts() 639 f := h.GetFacts()
735 return store.HostFacts{ 640 return store.HostFacts{
internal/server/syncsvc/tracker.go
Old New
@@ -2,18 +2,12 @@ package syncsvc
2 2
3 import "sync" 3 import "sync"
4 4
5 // vmStatus is the durable status triple a successful RecordVMStatus persists:
6 // the (status, last_error, assigned_ip) that ends up stored in the vms row.
7 type vmStatus struct { 5 type vmStatus struct {
8 status string 6 status string
9 lastErr string 7 lastErr string
10 ip string 8 ip string
11 } 9 }
12 10
13 // statusTracker remembers the last DURABLY-WRITTEN status triple per VM so
14 // applyReport can skip the SELECT+UPDATE that RecordVMStatus does when nothing
15 // changed. Multiple host read-loop goroutines call into it concurrently, so all
16 // access is guarded by mu. Keyed by vmID (globally unique).
17 type statusTracker struct { 11 type statusTracker struct {
18 mu sync.Mutex 12 mu sync.Mutex
19 last map[string]vmStatus 13 last map[string]vmStatus
@@ -23,38 +17,10 @@ func newStatusTracker() *statusTracker {
23 return &statusTracker{last: map[string]vmStatus{}} 17 return &statusTracker{last: map[string]vmStatus{}}
24 } 18 }
25 19
26 // writeThrough runs the decide→durable-write→commit sequence for one VM under a
27 // single lock, so it is ATOMIC per VM: concurrent reports for the same host
28 // (e.g. an agent reconnect where the old and new sessions both deliver a report)
29 // cannot interleave the write and the cache update and leave the cache
30 // disagreeing with the row — a divergence that would then suppress every later
31 // identical report indefinitely.
32 //
33 // write performs the durable RecordVMStatus call and RETURNS THE ADDRESS IT
34 // WROTE; it runs ONLY when the report could still change the row, and the cache
35 // is updated ONLY if write returns nil, so a rejected write never suppresses the
36 // next retry. write receives no arguments: the caller passes the RAW reported ip
37 // to RecordVMStatus (whose UPDATE keeps the prior ip on empty).
38 //
39 // The tracker never decides what an address means. It caches what the write says
40 // it stored, folded with that UPDATE's own empty-keeps-prior rule:
41 //
42 // assigned_ip = CASE WHEN ?='' THEN assigned_ip ELSE ? END
43 //
44 // so the cache holds what the row holds. An address the store DROPPED leaves the
45 // cache on the prior value rather than the rejected one, and the next report
46 // carrying a good address reads as a change and lands. Deciding the effective ip
47 // here instead — from the value the agent REPORTED rather than the one the store
48 // WROTE — is how a permanently blank assigned_ip survived every report that
49 // would have filled it.
50 func (t *statusTracker) writeThrough(vmID, status, lastErr, ip string, write func() (string, error)) error { 20 func (t *statusTracker) writeThrough(vmID, status, lastErr, ip string, write func() (string, error)) error {
51 t.mu.Lock() 21 t.mu.Lock()
52 defer t.mu.Unlock() 22 defer t.mu.Unlock()
53 prior, ok := t.last[vmID] 23 prior, ok := t.last[vmID]
54 // Decide on the RAW ip. Whether the store keeps it is the store's business;
55 // all this needs is that a SKIPPED report is one that provably changes
56 // nothing — an empty ip leaves assigned_ip alone, and an ip already stored
57 // rewrites it to itself.
58 if ok && prior.status == status && prior.lastErr == lastErr && 24 if ok && prior.status == status && prior.lastErr == lastErr &&
59 (ip == "" || ip == prior.ip) { 25 (ip == "" || ip == prior.ip) {
60 return nil // unchanged — skip the write 26 return nil // unchanged — skip the write
@@ -70,26 +36,12 @@ func (t *statusTracker) writeThrough(vmID, status, lastErr, ip string, write fun
70 return nil 36 return nil
71 } 37 }
72 38
73 // forget drops a VM's cached state to bound memory. Called when the agent acks
74 // a VM as destroyed (the row is being hard-deleted).
75 func (t *statusTracker) forget(vmID string) { 39 func (t *statusTracker) forget(vmID string) {
76 t.mu.Lock() 40 t.mu.Lock()
77 defer t.mu.Unlock() 41 defer t.mu.Unlock()
78 delete(t.last, vmID) 42 delete(t.last, vmID)
79 } 43 }
80 44
81 // netTracker remembers the last value durably WRITTEN per host — one instance
82 // per fact, currently the host's guest subnet and the address it answers on —
83 // so a report that repeats it costs nothing. The guard is not premature: every
84 // host reports every tick — ten seconds by default — forever, and the store runs
85 // on a single connection, so an unguarded write spends a round trip on that
86 // connection for every host for the life of the fleet. Each write's own
87 // `WHERE <column><>?` makes a concurrent reconnect harmless; this makes the
88 // steady state free.
89 //
90 // Keyed by hostID. Only a SUCCESSFUL write is remembered, so a rejected one is
91 // retried by the next report rather than suppressed — the same rule the status
92 // tracker follows, for the same reason.
93 type netTracker struct { 45 type netTracker struct {
94 mu sync.Mutex 46 mu sync.Mutex
95 last map[string]string 47 last map[string]string
@@ -113,8 +65,6 @@ func (t *netTracker) writeThrough(hostID, value string, write func() error) erro
113 return nil 65 return nil
114 } 66 }
115 67
116 // forget drops a host's cached value, so a host that leaves and returns is
117 // written afresh rather than trusted to a memory of a row that may be gone.
118 func (t *netTracker) forget(hostID string) { 68 func (t *netTracker) forget(hostID string) {
119 t.mu.Lock() 69 t.mu.Lock()
120 defer t.mu.Unlock() 70 defer t.mu.Unlock()