a73x

f52ac6a1

networks: a VM joins the operator's LAN by name, and keeps its NAT life

a73x   2026-08-14 16:49

Commit message
networks: a VM joins the operator's LAN by name, and keeps its NAT life

A host advertises the bridge-backed networks its operator declared
(--host-network lan=br0, repeatable; the bridge is the operator's — eitri
attaches taps, never creates or addresses one). A create names a network and
the guest gains a second NIC there, beside the NAT NIC every guest keeps: the
gate, the console and published ports still aim at the private address, while
the operator's own DHCP server addresses the guest on their LAN. The lease is
discovered, not allocated — a snoop on the guest's tap reads the site's DHCP
ACK, believes only frames leaving toward the guest (a guest cannot mint its
own address), and a bridge-family nft rule stops any guest answering leases
onto the operator's network at all.

Admission refuses what a host does not advertise, with the remedy worded from
the host's state — a Mac is told its OS cannot serve named networks, and its
agent refuses the flag at startup rather than dropping it. The console offers
a host's networks at create and never invents vocabulary: one Network row
carries the operator's own name for the network and the address it granted.
The MCP surface takes and reports the same fields, and default placement
considers the requested network rather than manufacturing a refusal.

The second NIC's MAC is 0x56 plus five hash bytes — locally administered,
disjoint from the NAT space and the QEMU convention on the same wire. Route
metrics keep the default route private on the default image. Docs tell the
feature: quickstart recipe, FAQ answer, assumption entries for the metric and
DHCP-trust claims, and a decision recording the naming.

ROADMAP.md
Old New
@@ -8,9 +8,10 @@ What shipped in each version is in the
8 - Connectors: browser clients (claude.ai among them) sign in to the MCP 8 - Connectors: browser clients (claude.ai among them) sign in to the MCP
9 endpoint with OAuth rather than a pasted token—dynamic client registration 9 endpoint with OAuth rather than a pasted token—dynamic client registration
10 on eitri-oidc, so a connector enrols itself 10 on eitri-oidc, so a connector enrols itself
11 - Bridged guest networking: a Linux host puts its guests on the operator's 11 - Per-VM named networks: a host advertises the bridges its operator declared
12 own network—taps on a named bridge, the site's DHCP addresses the guests, 12 (`--host-network lan=br0`), a VM requests one by name, and admission matches
13 the agent discovers the result 13 or refuses—taps on the operator's bridge, the site's DHCP addresses the
14 guests, the agent discovers the result
14 - Public gateway: route inbound traffic to guest services through eitri, 15 - Public gateway: route inbound traffic to guest services through eitri,
15 building on published ports 16 building on published ports
16 - Hot-reload trusted CAs into running guests: a CA added to the tenant reaches 17 - Hot-reload trusted CAs into running guests: a CA added to the tenant reaches
docs/assumptions.md
Old New
@@ -570,3 +570,80 @@ refusing at boot — a `--max-vcpus` cap (no report carries a core count), a
570 failed probe, and a cap set at exactly the machine's total. False the moment 570 failed probe, and a cap set at exactly the machine's total. False the moment
571 the wire says which a number is; the refusal should then judge the declared 571 the wire says which a number is; the refusal should then judge the declared
572 capacity directly and this arithmetic goes away. 572 capacity directly and this arithmetic goes away.
573
574 ### A networked guest is flat on the LAN, and still on its host's private fabric
575
576 A guest on a named network is a machine on that network: every port, every
577 device, no gate in front of that NIC. That is the feature — the stance the
578 exposure proxy already takes for one published port ("the LAN address is the
579 feature"), here unbounded — and it underpins putting guests on a named network
580 with nothing of eitri's in front of them. NAT-only stays the default; flatness
581 is asked for by name, per VM. **True by construction**, and false as an
582 expectation the moment a tenant assumes eitri-enforced isolation between
583 guests sharing a LAN — that is VLAN filtering at the site's switch,
584 deliberately out of scope.
585
586 The second half is the price of the NIC being additive: the same guest also
587 sits on `eitri0` beside every other guest on its host, because that is where
588 its reservation, its gate splice and its egress live. VM-to-VM on the host is
589 exactly why the NIC is additive rather than a replacement — and it means
590 compromising a LAN-facing guest is a foothold on the host's private L2, not
591 only on the network that guest was exposed to. Stated rather than mitigated:
592 the alternative was a guest that trades one attachment for the other, which is
593 two networking models to ship and reason about instead of one.
594
595 ### A named network's LAN serves DHCP and tolerates new MACs
596
597 A guest's named NIC asks the site's DHCP server for an address and shows up at
598 the switch as one more MAC. Underpins allocating nothing for that NIC: the
599 site's DHCP addresses it, the agent discovers the result. **Falsified at the
600 switch**: 802.1X or MAC ACLs refuse the guest with the switch's diagnostics,
601 not eitri's. The soft half is a guest configured static — it never DHCPs, so
602 it is reachable there but unreported and `network_ip` stays empty, a
603 documented limitation rather than a fault.
604
605 Tolerating new MACs is not tolerating new servers, and eitri does not leave the
606 site to tell them apart. A bridge-family nft rule drops UDP source port 67 at
607 the ingress of every named tap, so a guest of eitri's cannot answer a lease —
608 not to the operator's real machines, not to a sibling guest across the bridge,
609 not to the switch beyond it. The rule is programmed only on hosts that serve
610 named networks, and a kernel that cannot take it fails agent startup rather
611 than bridging unfiltered. DHCPv6 (source port 547) and router advertisements
612 are deliberately unfiltered, and that is the design rather than a gap in it: a
613 bridged guest is meant to be a full peer, an operator may run a router guest on
614 purpose, and neither v6 path can poison what eitri records, since the snoop
615 reads IPv4 ACKs only. **True by construction** for v4; false as an expectation
616 the moment a tenant reads it as eitri policing the LAN's v6 too.
617
618 The blast radius of both halves is bounded by the NAT NIC being unconditional.
619 Whatever the site's DHCP does or refuses to do, `assigned_ip` is allocated
620 before the guest boots, so the gate, the console and every published port
621 still reach it: a network that never answers costs the guest its LAN identity,
622 not its existence.
623
624 One downgrade residue rides with this. Admission keeps networked VMs off agents
625 that never advertised the network — a pre-field agent advertises nothing and
626 is refused — but a host that advertised and was then downgraded to a pre-field
627 binary ignores the wire field and gives those guests the NAT NIC alone — no
628 second NIC, no LAN address, the one outcome admission exists to forbid. Closing
629 it takes a release floor at snapshot time, the `FirstDatagramExposures` pattern;
630 accepted for now because downgrading an advertising host is an operator's own
631 deliberate act, and rare.
632
633 ### The guest honours the route metrics the seed asks for
634
635 A two-NIC guest is handed `dhcp4-overrides.route-metric` in its network-config
636 v2 seed — 100 on the NAT NIC, 200 on the named one — and that is the whole
637 mechanism putting the default route on the NAT NIC. Underpins the egress
638 promise: outbound leaves through the host's masquerade whatever the operator's
639 network is doing, and either attachment can be down without taking the other
640 with it. **True on the Ubuntu cloud image**, where cloud-init renders v2 to
641 netplan and the metrics arrive verbatim. **Unverified off netplan**:
642 cloud-init's other renderers, sysconfig and NetworkManager, have historically
643 ignored `dhcp4-overrides.route-metric` when translating v2, which leaves both
644 NICs at the distro's default metrics and the default route undetermined — a
645 guest that egresses over the LAN, or that swaps between the two on a lease
646 renewal.
647 Falsified by an operator pointing the default image at a non-netplan distro,
648 and quietly: both NICs still address, the guest still boots, and nothing in
649 eitri reads a route table to notice.
docs/decisions.md
Old New
@@ -150,3 +150,54 @@ an MCP server by self-hosting the plane, and a public front would target this
150 same endpoint rather than being a second thing. Reversed by a genuine air-gapped 150 same endpoint rather than being a second thing. Reversed by a genuine air-gapped
151 need: guests to drive with no plane to reach. 151 need: guests to drive with no plane to reach.
152 Details in [mcp.md](mcp.md). 152 Details in [mcp.md](mcp.md).
153
154 ### Per-VM named networks
155
156 A host advertises the bridge-backed networks its operator declared
157 (`--host-network name=bridge`); a VM requests one by name; admission places it
158 on a host advertising that name or refuses at create. Instead of a per-host
159 bridged mode, which could not express the actual fleet — one host serving a
160 LAN pet and NAT'd cattle at once. And never a silent fallback: an agent from
161 before the field advertises nothing and would quietly NAT the guest, so
162 silence refuses — a VM that asked for the LAN either gets it or does not
163 exist.
164
165 The operator owns the bridge; eitri uses it and never creates it. The
166 precedent is unanimous: runtime-created network objects are how everyone
167 builds NAT bridges and macvlan attachments — docker0, virbr0, eitri's own
168 eitri0, Docker's macvlan — while a bridge that enslaves the uplink is
169 persistent host configuration everywhere one exists — Proxmox's vmbr0,
170 libvirt's bridge mode. eitri copies the only precedent that exists. And the
171 address is discovered, never allocated: the agent snoops the guest's own tap
172 for the DHCP ACK keyed to its deterministic MAC — the site's DHCP addresses
173 the guests, the agent discovers the result.
174
175 ### A named network adds a NIC, it does not replace one
176
177 Every guest keeps its NAT attachment: `eth0` on `eitri0`, its reservation
178 allocated before boot, the gate's splice target, its egress and its way to the
179 other guests on that host. A named network is a SECOND NIC on the operator's
180 bridge, addressed by the site's own DHCP, and the VM reports both addresses —
181 `assigned_ip` from boot, `network_ip` once the network answers. The default
182 route stays on the NAT NIC (metric 100 against 200, as the default Ubuntu
183 image's netplan renders them), so egress is deterministic and either side can
184 be down without taking the other with it.
185
186 This is multipass's shape, which is the shape the daily driver already runs:
187 NAT primary with the default route, LAN secondary, and which address a service
188 advertises left to that service. The replace-shaped version — LAN *instead of*
189 NAT — was built first and rejected: it shipped two networking models, with a
190 guest's reachability, its gate path and its VM-to-VM story all forking on
191 which one it was, and it opened a discovery window on the gate path that the
192 NAT-only guest never had. One model costs a stated blast radius instead (see
193 [assumptions.md](assumptions.md) — a LAN-facing guest is also on the host's
194 private L2), and that trade was taken deliberately.
195
196 ### A named network is called by the operator's name for it
197
198 The API and wire already agreed with themselves — `network`, `network_ip`,
199 `--host-network` — but the console had coined a second word, "LAN", for the
200 same idea, on a second row for it. Both are gone: one Network row now carries
201 the operator's own name as its value, joined to the discovered address once
202 the network's DHCP answers (`lan · 192.168.0.37`). eitri displays the
203 operator's vocabulary rather than inventing its own.
docs/faq.md
Old New
@@ -3,14 +3,21 @@
3 ## How do VMs get network access? 3 ## How do VMs get network access?
4 4
5 Outbound works out of the box: guests are NAT'd through their host and reach 5 Outbound works out of the box: guests are NAT'd through their host and reach
6 the internet like any process on it. Inbound is deliberately minimal: guests 6 the internet like any process on it. That NIC is unconditional—the guest's
7 live on private per-host bridges, and the only built-in way in is the SSH 7 egress, its way to the other guests on that host, and where the SSH
8 [jump gate](ssh-access.md). 8 [jump gate](ssh-access.md) meets it.
9 9
10 To serve traffic from a VM, or to reach one directly from another network, 10 Inbound has two answers. The gate works from anywhere: it authenticates every
11 install [Tailscale](https://tailscale.com) (or WireGuard) inside it—it is 11 connection against your tenant's CA and tunnels you in. The other is to put the
12 a normal Linux machine. A public gateway is on the 12 VM on your own network—declare a bridge on a Linux host, name it to the agent
13 [roadmap](../ROADMAP.md). 13 (`--host-network lan=br0`), pick that name at create. Your own DHCP server
14 addresses the guest's second NIC, and it is then a machine on the network:
15 every port, every device, nothing of eitri's in front of that NIC. The
16 quickstart's [Put a VM on your LAN](quickstart.md) is the recipe.
17
18 For reach beyond that LAN it is still a normal Linux machine, so install
19 [Tailscale](https://tailscale.com) (or WireGuard) inside it. A public gateway
20 is on the [roadmap](../ROADMAP.md).
14 21
15 ## Is an exposed port authenticated? 22 ## Is an exposed port authenticated?
16 23
docs/mcp.md
Old New
@@ -15,9 +15,9 @@ self-hosted eitri is a self-hosted MCP server, with nothing extra to install.
15 15
16 | Tool | Purpose | 16 | Tool | Purpose |
17 |---|---| 17 |---|---|
18 | `vm_create` | Create a VM; by default waits for `lifecycle=ready` + IP, then for guest SSH and `cloud-init status --wait` to finish. Refused while the tenant has no registered SSH user CA—call `ca_upload` first, or the guest would trust nothing. | 18 | `vm_create` | Create a VM; by default waits for `lifecycle=ready` + IP, then for guest SSH and `cloud-init status --wait` to finish. Takes an optional `network`, the named host network to give the guest a second NIC on; a host serves only the names it advertises, and naming another refuses the create. Returns `network` and `network_ip`—the address that network's own DHCP granted, empty at ready when it has not answered yet. Refused while the tenant has no registered SSH user CA—call `ca_upload` first, or the guest would trust nothing. |
19 | `vm_list` | List all VMs on the fleet (id, name, lifecycle, IP, size). | 19 | `vm_list` | List all VMs on the fleet (id, name, lifecycle, IP, size, `network` and `network_ip`). |
20 | `vm_info` | Show one VM's state plus a ready-to-use `ssh` command. | 20 | `vm_info` | Show one VM's state, both addresses, plus a ready-to-use `ssh` command. Where to read `network_ip` back when it was still empty at create. |
21 | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. | 21 | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. |
22 | `vm_write_file` | Write content to a path in a VM over SFTP (parent dirs created). | 22 | `vm_write_file` | Write content to a path in a VM over SFTP (parent dirs created). |
23 | `vm_read_file` | Read a file from a VM over SFTP (capped at 1 MiB, truncation flagged). | 23 | `vm_read_file` | Read a file from a VM over SFTP (capped at 1 MiB, truncation flagged). |
docs/openapi.json
Old New
@@ -140,6 +140,9 @@
140 "name": { 140 "name": {
141 "type": "string" 141 "type": "string"
142 }, 142 },
143 "network": {
144 "type": "string"
145 },
143 "power_state": { 146 "power_state": {
144 "type": "string" 147 "type": "string"
145 }, 148 },
@@ -400,6 +403,12 @@
400 "format": "date-time", 403 "format": "date-time",
401 "type": "string" 404 "type": "string"
402 }, 405 },
406 "host_networks": {
407 "items": {
408 "type": "string"
409 },
410 "type": "array"
411 },
403 "id": { 412 "id": {
404 "type": "string" 413 "type": "string"
405 }, 414 },
@@ -485,6 +494,7 @@
485 "capacity", 494 "capacity",
486 "cpu_model", 495 "cpu_model",
487 "enrolled_at", 496 "enrolled_at",
497 "host_networks",
488 "id", 498 "id",
489 "kernel", 499 "kernel",
490 "name", 500 "name",
@@ -798,6 +808,12 @@
798 "name": { 808 "name": {
799 "type": "string" 809 "type": "string"
800 }, 810 },
811 "network": {
812 "type": "string"
813 },
814 "network_ip": {
815 "type": "string"
816 },
801 "phase": { 817 "phase": {
802 "type": "string" 818 "type": "string"
803 }, 819 },
@@ -837,6 +853,8 @@
837 "lifecycle", 853 "lifecycle",
838 "mem_mb", 854 "mem_mb",
839 "name", 855 "name",
856 "network",
857 "network_ip",
840 "phase", 858 "phase",
841 "power_state", 859 "power_state",
842 "status", 860 "status",
docs/quickstart.md
Old New
@@ -245,6 +245,58 @@ more reason to publish only onto a network you are content to serve.
245 Remove the row to take the port down. Deleting the VM takes its exposures with 245 Remove the row to take the port down. Deleting the VM takes its exposures with
246 it. 246 it.
247 247
248 ### Put a VM on your LAN
249
250 Every guest lives on a private bridge behind its host, NAT'd out. A VM can
251 *also* sit on your own network: declare a bridge on a Linux host, name it to
252 the agent, pick that name at create—a second NIC there, and it keeps the first.
253
254 The bridge is yours, declared in the host's own network config; eitri attaches
255 taps and never creates or addresses one. netplan shown, networkd and
256 NetworkManager have equivalents:
257
258 ```yaml
259 network:
260 ethernets:
261 enp4s0: {}
262 bridges:
263 br0:
264 interfaces: [enp4s0]
265 dhcp4: true # or the host's static address, verbatim
266 ```
267
268 The host's address moves onto `br0` at boot, and so must anything bound to the
269 uplink by name—a PXE dnsmasq's `interface=enp4s0` becomes `interface=br0`.
270
271 Name the network to the agent in `/etc/default/eitri-agent`, then restart it:
272
273 ```sh
274 EITRI_AGENT_FLAGS="--host-network lan=br0"
275 ```
276
277 The flag repeats. Names are 1–32 of `[a-z0-9-]`, no leading
278 or trailing hyphen; `nat` is reserved. Linux hosts only. An agent whose named
279 bridge is missing refuses to start, naming it.
280
281 Create the VM on that network—the console's dialog shows a **Network** select
282 for a host advertising any, and naming one its host does not is refused.
283
284 The VM page then shows both addresses. **IP** is the private one the gate, the
285 console and published ports aim at; **Network** is your own name for the
286 network, and gains the address your DHCP server granted the second NIC once
287 that lease lands. A guest configured static there never DHCPs—reachable all
288 the same, but **Network** shows the name alone.
289
290 The private NIC keeps the default route (route metric 100, against 200 on the
291 second NIC), so outbound leaves through the host's masquerade whatever your
292 network is doing—on the default Ubuntu image, where netplan honours those
293 metrics. Which address a service advertises is that service's own business.
294
295 A guest on your network is flat on it: every port, every device, no gate in
296 front of that NIC—attach what you are content to have live there, the rest
297 NAT-only. It is also still on the host's private bridge beside that host's
298 other guests, which is what lets them talk and what a compromise of it reaches.
299
248 ## Self-hosting 300 ## Self-hosting
249 301
250 Run the whole control plane yourself—everything below is your own box. 302 Run the whole control plane yourself—everything below is your own box.
docs/shape.html
Old New
@@ -208,11 +208,18 @@
208 "imports": [ 208 "imports": [
209 "internal/agent/dhcp", 209 "internal/agent/dhcp",
210 "internal/agent/exec", 210 "internal/agent/exec",
211 "internal/agent/netsnoop",
211 "internal/agent/permanent", 212 "internal/agent/permanent",
212 "internal/agent/state" 213 "internal/agent/state"
213 ] 214 ]
214 }, 215 },
215 { 216 {
217 "importPath": "internal/agent/netsnoop",
218 "plane": "data",
219 "synopsis": "Package netsnoop discovers the address a guest's named-network NIC was granted, by watching that NIC's own tap for its DHCP exchange.",
220 "imports": []
221 },
222 {
216 "importPath": "internal/agent/permanent", 223 "importPath": "internal/agent/permanent",
217 "plane": "data", 224 "plane": "data",
218 "synopsis": "Package permanent mints agent errors that no retry can fix.", 225 "synopsis": "Package permanent mints agent errors that no retry can fix.",
@@ -253,7 +260,8 @@
253 "internal/agent/statelock", 260 "internal/agent/statelock",
254 "internal/agent/syncclient", 261 "internal/agent/syncclient",
255 "internal/covsnap", 262 "internal/covsnap",
256 "internal/joinblob" 263 "internal/joinblob",
264 "internal/names"
257 ] 265 ]
258 }, 266 },
259 { 267 {
docs/shape.json
Old New
@@ -157,11 +157,18 @@
157 "imports": [ 157 "imports": [
158 "internal/agent/dhcp", 158 "internal/agent/dhcp",
159 "internal/agent/exec", 159 "internal/agent/exec",
160 "internal/agent/netsnoop",
160 "internal/agent/permanent", 161 "internal/agent/permanent",
161 "internal/agent/state" 162 "internal/agent/state"
162 ] 163 ]
163 }, 164 },
164 { 165 {
166 "importPath": "internal/agent/netsnoop",
167 "plane": "data",
168 "synopsis": "Package netsnoop discovers the address a guest's named-network NIC was granted, by watching that NIC's own tap for its DHCP exchange.",
169 "imports": []
170 },
171 {
165 "importPath": "internal/agent/permanent", 172 "importPath": "internal/agent/permanent",
166 "plane": "data", 173 "plane": "data",
167 "synopsis": "Package permanent mints agent errors that no retry can fix.", 174 "synopsis": "Package permanent mints agent errors that no retry can fix.",
@@ -202,7 +209,8 @@
202 "internal/agent/statelock", 209 "internal/agent/statelock",
203 "internal/agent/syncclient", 210 "internal/agent/syncclient",
204 "internal/covsnap", 211 "internal/covsnap",
205 "internal/joinblob" 212 "internal/joinblob",
213 "internal/names"
206 ] 214 ]
207 }, 215 },
208 { 216 {
go.mod
Old New
@@ -14,6 +14,7 @@ require (
14 github.com/stretchr/testify v1.11.1 14 github.com/stretchr/testify v1.11.1
15 github.com/yuin/goldmark v1.8.4 15 github.com/yuin/goldmark v1.8.4
16 golang.org/x/crypto v0.54.0 16 golang.org/x/crypto v0.54.0
17 golang.org/x/net v0.56.0
17 golang.org/x/oauth2 v0.36.0 18 golang.org/x/oauth2 v0.36.0
18 golang.org/x/sync v0.20.0 19 golang.org/x/sync v0.20.0
19 golang.org/x/sys v0.47.0 20 golang.org/x/sys v0.47.0
@@ -55,7 +56,6 @@ require (
55 go.uber.org/mock v0.4.0 // indirect 56 go.uber.org/mock v0.4.0 // indirect
56 golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect 57 golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect
57 golang.org/x/mod v0.33.0 // indirect 58 golang.org/x/mod v0.33.0 // indirect
58 golang.org/x/net v0.56.0 // indirect
59 golang.org/x/tools v0.42.0 // indirect 59 golang.org/x/tools v0.42.0 // indirect
60 modernc.org/libc v1.72.3 // indirect 60 modernc.org/libc v1.72.3 // indirect
61 modernc.org/mathutil v1.7.1 // indirect 61 modernc.org/mathutil v1.7.1 // indirect
internal/agent/cloudhv/cloudhv.go
Old New
@@ -45,19 +45,30 @@ type PumpHooks interface {
45 // responder) is the composition root's job and never appears here: it happens 45 // responder) is the composition root's job and never appears here: it happens
46 // once at agent start, not once per guest. 46 // once at agent start, not once per guest.
47 type Network interface { 47 type Network interface {
48 // ReserveIP returns this VM's sticky address, recording its DHCP 48 // ReserveIP returns this VM's sticky address on the NAT underlay,
49 // reservation. 49 // recording its DHCP reservation. Every guest has one, whatever else its
50 // spec asks for.
50 ReserveIP(vmID string) (string, error) 51 ReserveIP(vmID string) (string, error)
51 // Address returns the address already reserved for vmID, or "" if none. 52 // Address returns the NAT address already reserved for vmID, or "" if none.
52 // It never allocates. 53 // It never allocates.
53 Address(vmID string) string 54 Address(vmID string) string
54 // CreateTap creates the VM's tap device, enslaves it to the bridge and 55 // NetworkAddress returns the address the site's own DHCP server granted
55 // pins ip as the guest's DHCP reservation. Idempotent. 56 // this VM's named-network NIC, or "" — for a guest with no such NIC, and
56 CreateTap(ctx context.Context, vmID, ip string) error 57 // for one whose guest has not finished asking yet. The host learns it by
57 // DeleteTap removes the reservation and the tap. Idempotent. 58 // watching, so unlike Address it is not known at boot.
59 NetworkAddress(vmID string) string
60 // CreateTap creates the VM's tap on the NAT bridge and pins ip as its
61 // reservation. When network is non-empty it ALSO creates a second tap on
62 // that named network's bridge — additional, never instead. Idempotent.
63 CreateTap(ctx context.Context, vmID, ip, network string) error
64 // DeleteTap removes the reservation and both taps. Idempotent.
58 DeleteTap(ctx context.Context, vmID string) error 65 DeleteTap(ctx context.Context, vmID string) error
59 // TapName is the device name that goes in cloud-hypervisor's --net argument. 66 // TapName is the device name of the NAT NIC's tap — the first --net
67 // argument, and therefore the guest's eth0.
60 TapName(vmID string) string 68 TapName(vmID string) string
69 // NetTapName is the device name of the named-network NIC's tap, used only
70 // by a VM whose spec asked for one.
71 NetTapName(vmID string) string
61 } 72 }
62 73
63 // Provisioner manages cloud-hypervisor processes for all VMs on this host. 74 // Provisioner manages cloud-hypervisor processes for all VMs on this host.
@@ -149,8 +160,17 @@ func (p *Provisioner) buildArgs(spec state.VMSpec) []string {
149 } 160 }
150 args = append(args, fmt.Sprintf("path=%s,image_type=raw", d.Path)) 161 args = append(args, fmt.Sprintf("path=%s,image_type=raw", d.Path))
151 } 162 }
163 // NIC ORDER IS THE GUEST ABI. The NAT NIC is first and therefore eth0: it
164 // is the management fabric every guest has, the address the gate splices
165 // to, and what the seed's primary netplan stanza expects to find. A guest
166 // that also asked for a named network gets that NIC second — same MAC
167 // determinism, its own tap, its own DHCP server answering it.
168 args = append(args, "--net", fmt.Sprintf("tap=%s,mac=%s", tap, mac))
169 if spec.Network != "" {
170 args = append(args, "--net",
171 fmt.Sprintf("tap=%s,mac=%s", p.net.NetTapName(vmID), state.NetMAC(vmID)))
172 }
152 return append(args, 173 return append(args,
153 "--net", fmt.Sprintf("tap=%s,mac=%s", tap, mac),
154 "--serial", fmt.Sprintf("socket=%s", p.st.SerialSocketPath(vmID)), 174 "--serial", fmt.Sprintf("socket=%s", p.st.SerialSocketPath(vmID)),
155 "--console", "off", 175 "--console", "off",
156 ) 176 )
@@ -241,7 +261,7 @@ func (p *Provisioner) Boot(ctx context.Context, vmID string, spec state.VMSpec)
241 // fail instead yields an illegible cloud-hypervisor error for the same 261 // fail instead yields an illegible cloud-hypervisor error for the same
242 // root cause — and %w keeps a Permanent() marker (a tap name collision) 262 // root cause — and %w keeps a Permanent() marker (a tap name collision)
243 // unwrappable, so reconcile still terminal-fails it in one attempt. 263 // unwrappable, so reconcile still terminal-fails it in one attempt.
244 if err := p.attachNet(ctx, vmID); err != nil { 264 if err := p.attachNet(ctx, spec); err != nil {
245 return fmt.Errorf("attach network %s: %w", vmID, err) 265 return fmt.Errorf("attach network %s: %w", vmID, err)
246 } 266 }
247 267
@@ -303,12 +323,17 @@ func (p *Provisioner) Boot(ctx context.Context, vmID string, spec state.VMSpec)
303 // carries the DHCP reservation, so the address has to be known first. 323 // carries the DHCP reservation, so the address has to be known first.
304 // Idempotent: both halves tolerate a re-run, which is what makes a Boot retry 324 // Idempotent: both halves tolerate a re-run, which is what makes a Boot retry
305 // and a restart-after-host-reboot the same code path. 325 // and a restart-after-host-reboot the same code path.
306 func (p *Provisioner) attachNet(ctx context.Context, vmID string) error { 326 //
307 ip, err := p.net.ReserveIP(vmID) 327 // It takes the whole spec rather than the id because whether the VM gets a
328 // second NIC is the spec's to say: the network name the control plane sent
329 // rides here and nowhere else. The reservation is unconditional either way —
330 // every guest is on the NAT underlay.
331 func (p *Provisioner) attachNet(ctx context.Context, spec state.VMSpec) error {
332 ip, err := p.net.ReserveIP(spec.VMID)
308 if err != nil { 333 if err != nil {
309 return err 334 return err
310 } 335 }
311 return p.net.CreateTap(ctx, vmID, ip) 336 return p.net.CreateTap(ctx, spec.VMID, ip, spec.Network)
312 } 337 }
313 338
314 // ownedPID is the pid of the cloud-hypervisor process THIS agent started for 339 // ownedPID is the pid of the cloud-hypervisor process THIS agent started for
@@ -354,6 +379,12 @@ func (p *Provisioner) Running(vmID string) bool {
354 // backend whose host OS hands out addresses cannot answer that early. 379 // backend whose host OS hands out addresses cannot answer that early.
355 func (p *Provisioner) Address(vmID string) string { return p.net.Address(vmID) } 380 func (p *Provisioner) Address(vmID string) string { return p.net.Address(vmID) }
356 381
382 // NetworkAddress returns what the site's DHCP server granted this VM's
383 // named-network NIC, or "" when it has none or has not been heard from yet.
384 // Unlike Address it is genuinely polled: the host discovers it by watching the
385 // guest's own exchange, which cannot have happened before the guest boots.
386 func (p *Provisioner) NetworkAddress(vmID string) string { return p.net.NetworkAddress(vmID) }
387
357 // socketClient returns an *http.Client whose transport dials over the VM's 388 // socketClient returns an *http.Client whose transport dials over the VM's
358 // Unix socket. 389 // Unix socket.
359 func (p *Provisioner) socketClient(vmID string) *http.Client { 390 func (p *Provisioner) socketClient(vmID string) *http.Client {
internal/agent/cloudhv/cloudhv_test.go
Old New
@@ -41,6 +41,29 @@ func TestBuildArgs(t *testing.T) {
41 assert.Contains(t, joined, st.DiskPath("vm1")) 41 assert.Contains(t, joined, st.DiskPath("vm1"))
42 assert.Contains(t, joined, st.SeedPath("vm1")) 42 assert.Contains(t, joined, st.SeedPath("vm1"))
43 assert.Contains(t, joined, "tap=eit-vm1,mac="+state.MAC("vm1")) 43 assert.Contains(t, joined, "tap=eit-vm1,mac="+state.MAC("vm1"))
44 assert.Equal(t, 1, strings.Count(joined, "--net"), "a guest with no named network has one NIC")
45 }
46
47 // TestBuildArgsGivesANetworkedGuestTwoNICsNATFirst pins the guest ABI: the NAT
48 // NIC's --net comes first, so it is eth0 in every guest whether or not the VM
49 // also sits on the operator's LAN. The second NIC is the named one, on its own
50 // tap, with its own deterministic MAC — the one the host's snoop listens for
51 // and the one the seed's second netplan stanza matches.
52 func TestBuildArgsGivesANetworkedGuestTwoNICsNATFirst(t *testing.T) {
53 st, err := state.Open(t.TempDir())
54 require.NoError(t, err)
55 p := New(st, "ch", "fw", nil, newFakeNet())
56
57 args := p.buildArgs(state.VMSpec{VMID: "vm1", Network: "lan", VCPUs: 1, MemMB: 512})
58 joined := strings.Join(args, " ")
59
60 nat := "tap=eit-vm1,mac=" + state.MAC("vm1")
61 named := "tap=eil-vm1,mac=" + state.NetMAC("vm1")
62 assert.Contains(t, joined, nat)
63 assert.Contains(t, joined, named)
64 assert.Equal(t, 2, strings.Count(joined, "--net"), "two NICs, no more")
65 assert.Less(t, strings.Index(joined, nat), strings.Index(joined, named),
66 "NIC order is the guest ABI: eth0 is the NAT NIC")
44 } 67 }
45 68
46 // TestBuildArgsDeclaresRawImageType pins image_type=raw on BOTH disk entries. 69 // TestBuildArgsDeclaresRawImageType pins image_type=raw on BOTH disk entries.
@@ -464,10 +487,20 @@ type fakeNet struct {
464 reserveErr error 487 reserveErr error
465 tapErr error 488 tapErr error
466 delErr error 489 delErr error
490 // tapNetwork records the network CreateTap was passed, so a test can see
491 // that the spec's request reached the networking layer intact without a
492 // bridge on the test machine. discovered stands in for what a snoop heard.
493 tapNetwork map[string]string
494 tapIP map[string]string
495 discovered map[string]string
467 } 496 }
468 497
469 func newFakeNet() *fakeNet { 498 func newFakeNet() *fakeNet {
470 return &fakeNet{reserved: map[string]string{}, taps: map[string]bool{}} 499 return &fakeNet{
500 reserved: map[string]string{}, taps: map[string]bool{},
501 tapNetwork: map[string]string{}, tapIP: map[string]string{},
502 discovered: map[string]string{},
503 }
471 } 504 }
472 505
473 func (f *fakeNet) ReserveIP(vmID string) (string, error) { 506 func (f *fakeNet) ReserveIP(vmID string) (string, error) {
@@ -484,7 +517,11 @@ func (f *fakeNet) ReserveIP(vmID string) (string, error) {
484 517
485 func (f *fakeNet) Address(vmID string) string { return f.reserved[vmID] } 518 func (f *fakeNet) Address(vmID string) string { return f.reserved[vmID] }
486 519
487 func (f *fakeNet) CreateTap(_ context.Context, vmID, _ string) error { 520 func (f *fakeNet) NetworkAddress(vmID string) string { return f.discovered[vmID] }
521
522 func (f *fakeNet) CreateTap(_ context.Context, vmID, ip, network string) error {
523 f.tapNetwork[vmID] = network
524 f.tapIP[vmID] = ip
488 if f.tapErr != nil { 525 if f.tapErr != nil {
489 return f.tapErr 526 return f.tapErr
490 } 527 }
@@ -498,14 +535,19 @@ func (f *fakeNet) DeleteTap(_ context.Context, vmID string) error {
498 } 535 }
499 delete(f.reserved, vmID) 536 delete(f.reserved, vmID)
500 delete(f.taps, vmID) 537 delete(f.taps, vmID)
538 delete(f.discovered, vmID)
501 return nil 539 return nil
502 } 540 }
503 541
504 func (f *fakeNet) TapName(vmID string) string { 542 func (f *fakeNet) TapName(vmID string) string { return "eit-" + shortID(vmID) }
543
544 func (f *fakeNet) NetTapName(vmID string) string { return "eil-" + shortID(vmID) }
545
546 func shortID(vmID string) string {
505 if len(vmID) > 8 { 547 if len(vmID) > 8 {
506 vmID = vmID[:8] 548 return vmID[:8]
507 } 549 }
508 return "eit-" + vmID 550 return vmID
509 } 551 }
510 552
511 // TestBootAttachesTheNetworkBeforeLaunching pins the fold: the --net argument 553 // TestBootAttachesTheNetworkBeforeLaunching pins the fold: the --net argument
@@ -529,6 +571,70 @@ func TestBootAttachesTheNetworkBeforeLaunching(t *testing.T) {
529 assert.Equal(t, "10.77.1.2", p.Address(vmID), "Boot must reserve the VM's address") 571 assert.Equal(t, "10.77.1.2", p.Address(vmID), "Boot must reserve the VM's address")
530 } 572 }
531 573
574 // TestBootNetworkedVMReservesAndPassesTheNetwork pins what this package owes
575 // the networking layer for a guest that asked for a named network: the address
576 // is still reserved (the NAT NIC is unconditional), and the network name
577 // reaches CreateTap unaltered so the second NIC can be built.
578 func TestBootNetworkedVMReservesAndPassesTheNetwork(t *testing.T) {
579 st, err := state.Open(t.TempDir())
580 require.NoError(t, err)
581 vmID := "vm-lan"
582 spec := state.VMSpec{VMID: vmID, Network: "lan", VCPUs: 1, MemMB: 128}
583 require.NoError(t, st.SaveVM(state.Record{Spec: spec}))
584
585 fakeCH := filepath.Join(t.TempDir(), "fake-ch")
586 require.NoError(t, os.WriteFile(fakeCH, []byte("#!/bin/sh\nexec sleep 60\n"), 0o755))
587
588 fn := newFakeNet()
589 p := New(st, fakeCH, "fw", nil, fn)
590 require.NoError(t, p.Boot(context.Background(), vmID, spec))
591 t.Cleanup(func() { _ = p.Destroy(context.Background(), vmID) })
592
593 assert.Equal(t, "lan", fn.tapNetwork[vmID])
594 assert.Equal(t, "10.77.1.2", fn.tapIP[vmID],
595 "a networked guest is on the NAT underlay too — it has an address before it boots")
596 assert.Equal(t, "10.77.1.2", p.Address(vmID))
597 assert.Empty(t, p.NetworkAddress(vmID), "and nothing on the named NIC until its guest asks")
598 assert.True(t, fn.taps[vmID], "the tap is still Boot's to create")
599 }
600
601 // TestNetworkAddressIsPolledFromTheNetworkingLayer pins the second address's
602 // only route upward: whatever the host discovered, unaltered and not confused
603 // with the reservation.
604 func TestNetworkAddressIsPolledFromTheNetworkingLayer(t *testing.T) {
605 st, err := state.Open(t.TempDir())
606 require.NoError(t, err)
607 fn := newFakeNet()
608 p := New(st, "ch", "fw", nil, fn)
609
610 assert.Empty(t, p.NetworkAddress("vm-lan"))
611 fn.discovered["vm-lan"] = "192.168.0.42"
612 assert.Equal(t, "192.168.0.42", p.NetworkAddress("vm-lan"))
613 assert.Empty(t, p.Address("vm-lan"), "which is never mistaken for the NAT address")
614 }
615
616 // TestBootNATVMPassesNoNetwork is the counterpart, and the regression guard for
617 // every host running today: an unnamed spec asks for no second NIC and gets the
618 // address this host allocated.
619 func TestBootNATVMPassesNoNetwork(t *testing.T) {
620 st, err := state.Open(t.TempDir())
621 require.NoError(t, err)
622 vmID := "vm-nat"
623 spec := state.VMSpec{VMID: vmID, VCPUs: 1, MemMB: 128}
624 require.NoError(t, st.SaveVM(state.Record{Spec: spec}))
625
626 fakeCH := filepath.Join(t.TempDir(), "fake-ch")
627 require.NoError(t, os.WriteFile(fakeCH, []byte("#!/bin/sh\nexec sleep 60\n"), 0o755))
628
629 fn := newFakeNet()
630 p := New(st, fakeCH, "fw", nil, fn)
631 require.NoError(t, p.Boot(context.Background(), vmID, spec))
632 t.Cleanup(func() { _ = p.Destroy(context.Background(), vmID) })
633
634 assert.Empty(t, fn.tapNetwork[vmID])
635 assert.Equal(t, "10.77.1.2", fn.tapIP[vmID], "the allocated address is what the tap pins")
636 }
637
532 // TestBootKeepsTheAddressTheVMAlreadyHolds pins that a reboot is not a renumber: 638 // TestBootKeepsTheAddressTheVMAlreadyHolds pins that a reboot is not a renumber:
533 // the reservation the VM already holds is what Boot re-attaches it to. 639 // the reservation the VM already holds is what Boot re-attaches it to.
534 func TestBootKeepsTheAddressTheVMAlreadyHolds(t *testing.T) { 640 func TestBootKeepsTheAddressTheVMAlreadyHolds(t *testing.T) {
internal/agent/netenv/named.go
Old New
@@ -0,0 +1,282 @@
1 package netenv
2
3 import (
4 "context"
5 "fmt"
6 "log/slog"
7 "net"
8 "os"
9 "path/filepath"
10 "strings"
11
12 "github.com/a73x/eitri/internal/agent/permanent"
13 "github.com/a73x/eitri/internal/agent/state"
14 )
15
16 // sysfsIsBridge reports whether name is a bridge device:
17 // /sys/class/net/<name>/bridge is a directory that exists for exactly those.
18 // Bridge-ness is checked rather than assumed because attaching a tap to a link
19 // that is not a bridge fails later with a raw RTNETLINK error naming neither
20 // the network nor the flag that asked for it.
21 func sysfsIsBridge(name string) bool { return sysfsIsBridgeAt("/sys/class/net", name) }
22
23 // sysfsIsBridgeAt is sysfsIsBridge with an injectable sysfs root (for tests).
24 func sysfsIsBridgeAt(root, name string) bool {
25 fi, err := os.Stat(filepath.Join(root, name, "bridge"))
26 return err == nil && fi.IsDir()
27 }
28
29 // VerifyNetworks confirms every configured network's bridge exists and is a
30 // bridge, before any VM can be asked for. eitri never creates these — a missing
31 // one is the operator's netplan/networkd change not yet made, and the only
32 // honest response is a startup failure that names it. Coming up anyway would
33 // advertise a network this host cannot honor, and every VM placed on it would
34 // fail one at a time instead.
35 func (n *Net) VerifyNetworks() error {
36 for name, br := range n.networks {
37 if !n.isBridge(br) {
38 return fmt.Errorf("--host-network %s=%s: %s is not an existing bridge on this host; "+
39 "declare it in the host's own network config first (eitri attaches to bridges, it does not create them)",
40 name, br, br)
41 }
42 }
43 return nil
44 }
45
46 // blockGuestDHCP installs the one rule a host that lends out an operator bridge
47 // owes it: a frame entering that bridge from an eitri guest tap with UDP source
48 // port 67 — DHCPv4 server talk — is dropped. Called from EnsureBridge, so it is
49 // reprogrammed on every agent start; a host with no configured network runs
50 // none of it and its ruleset stays exactly what it was.
51 //
52 // One rule covers a host serving any number of named networks, because it keys
53 // on the tap-name prefix and not on the bridge: every named tap this agent
54 // creates is eil-something, whichever operator bridge it was enslaved to.
55 //
56 // It buys two separate things. The first is the half of the address snoop that
57 // no socket can settle by itself: internal/agent/netsnoop accepts only frames
58 // the kernel marks outgoing, which proves a guest cannot mint its own address,
59 // but an ACK one guest unicasts at a neighbour's MAC is forwarded by the bridge
60 // and leaves the neighbour tap outgoing too, indistinguishable there from the
61 // site answering. Dropping it at ingress settles it before the question can be
62 // asked — the frame never reaches the bridge, so there is nothing to forward.
63 // The second is plain protection for the LAN: a guest handing out leases would
64 // otherwise address the operator's real machines, and eitri is what put it on
65 // their network.
66 //
67 // Ingress at the tap is the hook point precisely because it is upstream of
68 // every path the frame could take. A drop at prerouting happens before the
69 // bridge picks a port, so the forged lease reaches neither a sibling guest nor
70 // the physical uplink and the switch beyond it.
71 //
72 // DHCPv6 (source port 547) and IPv6 router advertisements are deliberately not
73 // filtered, and the omission is the design rather than a gap in it: a bridged
74 // guest is meant to be a full peer on the operator's LAN, an operator may run a
75 // router guest on purpose, and neither v6 path can poison anything eitri
76 // records, since the snoop reads IPv4 ACKs only.
77 func (n *Net) blockGuestDHCP(ctx context.Context) error {
78 if len(n.networks) == 0 {
79 return nil
80 }
81 // Same idempotency as the NAT chain: add table/add chain are no-ops when the
82 // object is already there, and the flush before the rule keeps restarts from
83 // stacking copies of it. Two invocations are two transactions, though, so a
84 // restart on a host with live taps leaves the chain empty for the moment
85 // between them; the honest fix is one nft -f transaction, which the NAT chain
86 // needs identically and which belongs to its own change rather than this one.
87 steps := [][]string{
88 {"nft", "add", "table", "bridge", "eitri"},
89 // The priority is the number and not the name filter, which is -200 in the
90 // bridge family but 0 in the ip family the NAT chain above lives in: the
91 // number reads the same whichever table a reader has in mind.
92 {"nft", "add", "chain", "bridge", "eitri", "prerouting",
93 "{ type filter hook prerouting priority -200 ; }"},
94 {"nft", "flush", "chain", "bridge", "eitri", "prerouting"},
95 // The quotes around the pattern are nft syntax, not shell quoting that a
96 // runner without a shell would have eaten: nft re-lexes its arguments
97 // joined, and an unquoted trailing star is a parse error there.
98 {"nft", "add", "rule", "bridge", "eitri", "prerouting",
99 "iifname", fmt.Sprintf("%q", netTapPrefix+"*"), "udp", "sport", "67", "drop"},
100 }
101 for _, s := range steps {
102 if _, err := n.run(ctx, s[0], s[1:]...); err != nil {
103 return fmt.Errorf("%s %s: %w", s[0], strings.Join(s[1:], " "), err)
104 }
105 }
106 return nil
107 }
108
109 // createNamedTap is CreateTap's second half for a guest that asked for a named
110 // network: a SECOND tap -> enslave to the operator's bridge -> up -> start the
111 // address snoop. No reservation, no masquerade, no forced DNS — the site's DHCP
112 // server owns all three on this NIC. The guest's first tap, on eitri0, was
113 // created before this ran and is untouched by it.
114 //
115 // The network is known to be configured: CreateTap refuses an unknown one
116 // before it creates anything at all.
117 func (n *Net) createNamedTap(ctx context.Context, vmID, network string) error {
118 bridge := n.networks[network]
119 tap := n.NetTapName(vmID)
120 if _, err := n.run(ctx, "ip", "link", "show", "dev", tap); err != nil {
121 if _, err := n.best(ctx, "ip", "tuntap", "add", "dev", tap, "mode", "tap"); err != nil {
122 return err
123 }
124 } else if !n.isTap(tap) {
125 return permanent.Errorf("link %s exists but is not a TAP device — name collision or stale interface; remove it or rename the VM", tap)
126 }
127 if _, err := n.best(ctx, "ip", "link", "set", tap, "master", bridge); err != nil {
128 return err
129 }
130 if _, err := n.best(ctx, "ip", "link", "set", tap, "up"); err != nil {
131 return err
132 }
133 n.mu.Lock()
134 n.attachLocked(vmID)
135 n.mu.Unlock()
136 // A snoop that will not start fails the create. The named NIC's address
137 // arrives only this way, so a guest booted without one is a guest whose LAN
138 // address nothing above the host can ever learn — visibly failing the
139 // create beats reporting a networked guest that is silently half-reported.
140 // The one refusal that is not a failure is the VM being deleted while this
141 // ran: that takes the attachment with it, and startSnoop leaves a guest on
142 // its way out unwatched rather than arming a goroutine for a dead tap.
143 //nolint:contextcheck // the snoop's lifetime is the tap's, not this call's: inheriting ctx would end discovery the moment CreateTap returned
144 return n.startSnoop(vmID, tap)
145 }
146
147 // attachLocked returns vmID's attachment, creating it if the VM has none.
148 // Called with n.mu held.
149 //
150 // It keeps any existing entry rather than replacing it, because both callers
151 // can arrive at a VM that is already attached: a Boot retry re-runs
152 // createNamedTap on a guest whose snoop is live, and neither that snoop nor the
153 // address it has already found may be dropped on the floor by the re-attach.
154 func (n *Net) attachLocked(vmID string) *attachment {
155 a, live := n.attached[vmID]
156 if !live {
157 a = &attachment{}
158 n.attached[vmID] = a
159 }
160 return a
161 }
162
163 // startSnoop begins DHCP-ACK discovery for vmID on tap, the VM's named-network
164 // one. It keys on state.NetMAC — the second NIC's address — so the guest's NAT
165 // lease, which crosses a different tap and this host granted itself, can never
166 // be mistaken for what the site's server said. Idempotent per VM: a second call
167 // while a snoop is live is a no-op, which is what makes it safe on a Boot retry
168 // and on the restart replay. A VM with no attachment has no NIC to watch, and
169 // gets no listener.
170 func (n *Net) startSnoop(vmID, tap string) error {
171 n.mu.Lock()
172 a, live := n.attached[vmID]
173 if !live || a.stopSnoop != nil {
174 n.mu.Unlock()
175 return nil
176 }
177 mac, err := net.ParseMAC(state.NetMAC(vmID))
178 if err != nil {
179 n.mu.Unlock()
180 return err
181 }
182 ctx, cancel := context.WithCancel(context.Background())
183 a.stopSnoop = cancel
184 n.mu.Unlock()
185
186 if err := n.listen(ctx, tap, mac, n.noteDiscovered(vmID)); err != nil {
187 // Leave nothing armed behind a failure: the cancel above would make the
188 // retry believe a snoop is already running and skip starting one. Only
189 // this attachment is disarmed — a VM deleted and re-created while the
190 // listener was refusing has a new entry, and that one's snoop is not
191 // this failure's to un-arm.
192 n.mu.Lock()
193 if cur, live := n.attached[vmID]; live && cur == a {
194 cur.stopSnoop = nil
195 }
196 n.mu.Unlock()
197 cancel()
198 return err
199 }
200 return nil
201 }
202
203 // NetworkAddress returns the address the site's DHCP server granted this VM's
204 // named-network NIC, or "" — for a VM that has no such NIC, and for a VM whose
205 // guest has not yet completed a DHCP exchange on it. Empty is "not yet known",
206 // never "unreachable": the VM's NAT address (Address) is what eitri itself
207 // reaches it by, and that one exists from boot.
208 func (n *Net) NetworkAddress(vmID string) string {
209 n.mu.Lock()
210 defer n.mu.Unlock()
211 if a, live := n.attached[vmID]; live {
212 return a.discovered
213 }
214 return ""
215 }
216
217 // noteDiscovered builds vmID's snoop callback. It records what the ACK granted,
218 // but only while the VM is still attached: the listener checks its context
219 // between reads, so a frame that arrived in the last poll window (~1s) can call
220 // back after DeleteTap already cancelled it, and a deleted VM must not come
221 // back holding an address.
222 //
223 // The callback closes over the id and looks the attachment up on every ACK,
224 // never over the entry itself: an entry DeleteTap dropped is a place nothing
225 // may still be writing to, and only the map can say whether it is still there.
226 func (n *Net) noteDiscovered(vmID string) func(ip string) {
227 return func(ip string) {
228 n.mu.Lock()
229 defer n.mu.Unlock()
230 if a, live := n.attached[vmID]; live {
231 a.discovered = ip
232 }
233 }
234 }
235
236 // AdoptNetwork rebuilds a networked VM's in-memory discovery state after an
237 // agent restart: the record's last known named-NIC address (so the fleet keeps
238 // reading it before the next DHCP renewal is snooped) and, when the tap is
239 // still up, a fresh snoop. It says nothing about the VM's NAT attachment —
240 // that half is AddReservation's, and the replay calls both for the same guest.
241 //
242 // It adopts the VM whether or not this agent still serves the network named in
243 // the record, and that is the kind answer rather than an oversight: an agent
244 // restarted without one of its --host-network flags is looking at a guest that
245 // is still running, still holding the tap the previous agent enslaved, still
246 // answering on the address the site gave it. Forgetting the attachment would
247 // take that address off the VM's report and disarm the snoop that keeps it
248 // current, punishing the guest for a change made on the host.
249 //
250 // What it will not do is stay quiet about it. The VM survives on borrowed
251 // configuration — CreateTap refuses the network permanently now, so the guest
252 // fails the next time it is booted — and the only moment an operator can be
253 // told before that happens is this one, so the drift is warned here.
254 func (n *Net) AdoptNetwork(vmID, network, ip string) {
255 n.mu.Lock()
256 a := n.attachLocked(vmID)
257 if ip != "" {
258 a.discovered = ip
259 }
260 n.mu.Unlock()
261 // networks is the agent's command line, fixed at construction, so it is read
262 // without the lock exactly as CreateTap reads it.
263 if _, served := n.networks[network]; !served {
264 slog.Warn("adopting a guest onto a named network this host no longer serves; it keeps its NIC and address, but its next reboot fails until --host-network names this network again",
265 "vm_id", vmID, "network", network)
266 }
267 tap := n.NetTapName(vmID)
268 if n.isTap(tap) {
269 // No tap means the VM is not running; Boot creates both it and the
270 // snoop. A snoop that refuses to start here is not worth failing the
271 // agent's startup over: the address above is already reported, and the
272 // VM's own converge is where a broken tap becomes a story.
273 //
274 // The cost is bounded but real: startSnoop leaves nothing armed after a
275 // failure, so a VM that is later booted re-arms — but a VM already
276 // running keeps no re-arm path until it reboots or the agent restarts,
277 // and reports the replayed address until then. That address is almost
278 // certainly still the guest's: the MAC is derived from the VM id and
279 // the LAN's lease table hands the same one back.
280 _ = n.startSnoop(vmID, tap)
281 }
282 }
internal/agent/netenv/named_test.go
Old New
@@ -0,0 +1,554 @@
1 package netenv
2
3 import (
4 "bytes"
5 "context"
6 "errors"
7 "log/slog"
8 "net"
9 "os"
10 "strings"
11 "sync"
12 "testing"
13
14 "github.com/a73x/eitri/internal/agent/exec"
15 "github.com/stretchr/testify/assert"
16 "github.com/stretchr/testify/require"
17 )
18
19 // snoopCall is one Listen the Net asked for. The context and the callback are
20 // kept so a test can play the part of the kernel: cancel like a teardown, or
21 // deliver an ACK like a guest.
22 type snoopCall struct {
23 ifname string
24 mac string
25 ctx context.Context
26 found func(ip string)
27 }
28
29 // fakeListen stands in for the AF_PACKET snoop. Production needs CAP_NET_RAW
30 // and a real tap; what this package owns is what it does with what the snoop
31 // hears, so the socket is injected out exactly like isTap and ifaces.
32 type fakeListen struct {
33 mu sync.Mutex
34 calls []snoopCall
35 err error
36 }
37
38 func (f *fakeListen) listen(ctx context.Context, ifname string, mac net.HardwareAddr, found func(ip string)) error {
39 f.mu.Lock()
40 defer f.mu.Unlock()
41 if f.err != nil {
42 return f.err
43 }
44 f.calls = append(f.calls, snoopCall{ifname: ifname, mac: mac.String(), ctx: ctx, found: found})
45 return nil
46 }
47
48 func (f *fakeListen) last(t *testing.T) snoopCall {
49 t.Helper()
50 f.mu.Lock()
51 defer f.mu.Unlock()
52 require.NotEmpty(t, f.calls, "expected a snoop to have been started")
53 return f.calls[len(f.calls)-1]
54 }
55
56 func (f *fakeListen) count() int {
57 f.mu.Lock()
58 defer f.mu.Unlock()
59 return len(f.calls)
60 }
61
62 // bridgedNet builds a Net on a host that declares networks and whose bridges
63 // all exist, with the snoop faked out.
64 func bridgedNet(t *testing.T, run exec.Runner, networks map[string]string) (*Net, *fakeListen) {
65 t.Helper()
66 n, err := New(run, "10.77.1.0/24", networks)
67 require.NoError(t, err)
68 n.isBridge = func(string) bool { return true }
69 snoop := &fakeListen{}
70 n.listen = snoop.listen
71 return n, snoop
72 }
73
74 // TestVerifyNetworksRefusesMissingBridge pins the startup gate: eitri attaches
75 // to bridges the operator declared and creates none, so a name whose link is
76 // absent is a host that cannot honor what it is about to advertise.
77 func TestVerifyNetworksRefusesMissingBridge(t *testing.T) {
78 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
79 n, err := New(noop, "10.77.1.0/24", map[string]string{"lan": "br0"})
80 require.NoError(t, err)
81 n.isBridge = func(string) bool { return false }
82
83 err = n.VerifyNetworks()
84 require.Error(t, err)
85 assert.Contains(t, err.Error(), "lan", "the refusal must name the network")
86 assert.Contains(t, err.Error(), "br0", "and the link the operator has to declare")
87
88 n.isBridge = func(name string) bool { return name == "br0" }
89 assert.NoError(t, n.VerifyNetworks(), "a declared bridge passes")
90 }
91
92 // TestVerifyNetworksPassesWithNoNetworks pins that a host with no --host-network
93 // flags — every host today — starts as it always did.
94 func TestVerifyNetworksPassesWithNoNetworks(t *testing.T) {
95 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
96 n, err := New(noop, "10.77.1.0/24", nil)
97 require.NoError(t, err)
98 assert.NoError(t, n.VerifyNetworks())
99 }
100
101 // networkedNet builds a Net whose vm1 is about to get both taps: neither link
102 // exists yet, so both `ip link show` probes fail and both adds run.
103 func networkedNet(t *testing.T, networks map[string]string) (*Net, *fakeListen, *[]call) {
104 t.Helper()
105 run, calls := recorder(nil, map[string]error{
106 "ip link show dev eit-vm1": errors.New(`Device "eit-vm1" does not exist.`),
107 "ip link show dev eil-vm1": errors.New(`Device "eil-vm1" does not exist.`),
108 })
109 n, snoop := bridgedNet(t, run, networks)
110 return n, snoop, calls
111 }
112
113 // TestNamedNetworkAddsASecondNIC is the shape of the whole feature: the guest
114 // keeps everything a NAT-only guest has — its tap on eitri0, its reservation,
115 // its address — and GAINS a second tap on the operator's bridge, watched for
116 // the address the site's own DHCP server grants it.
117 func TestNamedNetworkAddsASecondNIC(t *testing.T) {
118 n, snoop, calls := networkedNet(t, map[string]string{"lan": "br0"})
119
120 ip, err := n.ReserveIP("vm1")
121 require.NoError(t, err)
122 require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan"))
123
124 all := joinCalls(calls)
125 assert.Contains(t, all, "ip tuntap add dev eit-vm1 mode tap")
126 assert.Contains(t, all, "ip link set eit-vm1 master eitri0", "the NAT NIC is unconditional")
127 assert.Contains(t, all, "ip link set eit-vm1 up")
128 assert.Contains(t, all, "ip tuntap add dev eil-vm1 mode tap")
129 assert.Contains(t, all, "ip link set eil-vm1 master br0", "and the named NIC is additional")
130 assert.Contains(t, all, "ip link set eil-vm1 up")
131
132 mac, _ := net.ParseMAC(stateMAC("vm1"))
133 reserved, ok := n.dhcp.Lookup(mac)
134 require.True(t, ok, "a networked guest still holds a NAT reservation")
135 assert.Equal(t, ip, reserved.String())
136 assert.Equal(t, ip, n.Address("vm1"), "known before the guest boots, as ever")
137
138 started := snoop.last(t)
139 assert.Equal(t, "eil-vm1", started.ifname, "the snoop watches the named NIC's tap, not the NAT one")
140 netMAC, _ := net.ParseMAC(stateNetMAC("vm1"))
141 assert.Equal(t, netMAC.String(), started.mac,
142 "and listens for the second NIC's MAC — the NAT lease crosses the other tap and is this host's own answer")
143 }
144
145 // TestNICOrderIsTheNATTapFirst pins the guest ABI at the layer that creates the
146 // devices: eth0 is the NAT NIC, so its tap is made and enslaved before the
147 // named one exists. A guest whose NICs arrive in the other order has its
148 // management fabric on eth1 and everything that assumes eth0 is wrong.
149 func TestNICOrderIsTheNATTapFirst(t *testing.T) {
150 n, _, calls := networkedNet(t, map[string]string{"lan": "br0"})
151 require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan"))
152
153 all := joinCalls(calls)
154 assert.Less(t, strings.Index(all, "eit-vm1"), strings.Index(all, "eil-vm1"),
155 "the NAT tap is created first")
156 }
157
158 // TestUnknownNetworkIsPermanentAndCreatesNothing pins the one outcome the
159 // design forbids: an agent restarted without the flag fails the VM by name
160 // rather than quietly booting it with the NIC it asked for missing. It fails
161 // permanently — no retry on this agent can change the answer — and it fails
162 // before the first device is created, so the host is left as it was found.
163 func TestUnknownNetworkIsPermanentAndCreatesNothing(t *testing.T) {
164 run, calls := recorder(nil, nil)
165 n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"})
166
167 err := n.CreateTap(context.Background(), "vm1", "10.77.1.2", "ghost")
168 require.Error(t, err)
169 assert.Contains(t, err.Error(), "ghost")
170 assert.Contains(t, err.Error(), "--host-network")
171 var p interface{ Permanent() bool }
172 assert.True(t, errors.As(err, &p) && p.Permanent(), "no retry on this agent can find the network")
173 assert.Empty(t, joinCalls(calls), "an unknown network creates nothing at all — not even the NAT tap")
174 }
175
176 // TestCreateTapFailsWhenSnoopWillNotStart pins the deliberate choice to fail the
177 // create: a named NIC's address arrives only through the snoop, so a guest
178 // booted without one is a guest whose LAN address nothing can ever learn.
179 func TestCreateTapFailsWhenSnoopWillNotStart(t *testing.T) {
180 n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"})
181 snoop.err = errors.New("operation not permitted")
182
183 err := n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan")
184 require.Error(t, err)
185 assert.Contains(t, err.Error(), "operation not permitted")
186
187 // Nothing armed behind the failure, or the create-retry would believe a
188 // snoop is already running and never start one.
189 snoop.err = nil
190 require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan"))
191 assert.Equal(t, 1, snoop.count(), "the retry starts the snoop the first attempt could not")
192 }
193
194 // TestNetworkAddressComesFromDiscovery pins the poll-until-known contract for
195 // the named NIC — and, in the same breath, that it never leaks into the address
196 // the rest of eitri uses. Two NICs, two addresses, two questions.
197 func TestNetworkAddressComesFromDiscovery(t *testing.T) {
198 n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"})
199
200 ip, err := n.ReserveIP("vm1")
201 require.NoError(t, err)
202 require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan"))
203 assert.Empty(t, n.NetworkAddress("vm1"), "no ACK seen yet — the honest answer is nothing")
204 assert.Equal(t, ip, n.Address("vm1"), "which never delays the address the gate splices to")
205
206 snoop.last(t).found("192.168.0.42")
207 assert.Equal(t, "192.168.0.42", n.NetworkAddress("vm1"))
208 assert.Equal(t, ip, n.Address("vm1"), "the LAN address must never be reported as the NAT one")
209
210 // A renewal that grants a different address is the guest's truth, not a
211 // conflict: the LAN moved it, and the report follows.
212 snoop.last(t).found("192.168.0.43")
213 assert.Equal(t, "192.168.0.43", n.NetworkAddress("vm1"))
214 }
215
216 // TestAdoptNetworkSeedsAddressAndRearmsSnoop pins the restart replay: the last
217 // known LAN address is readable again before the guest's next renewal is
218 // snooped, and the snoop is running again for when that renewal comes.
219 func TestAdoptNetworkSeedsAddressAndRearmsSnoop(t *testing.T) {
220 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
221 n, snoop := bridgedNet(t, noop, map[string]string{"lan": "br0"})
222 n.isTap = func(string) bool { return true } // the guest survived the restart
223
224 n.AdoptNetwork("vm1", "lan", "192.168.0.7")
225
226 assert.Equal(t, "192.168.0.7", n.NetworkAddress("vm1"))
227 assert.Equal(t, "eil-vm1", snoop.last(t).ifname, "the re-armed snoop watches the named NIC")
228 assert.Empty(t, n.Address("vm1"),
229 "the NAT half of the replay is AddReservation's, and this must not fake it")
230 }
231
232 // captureLogs redirects the default logger into a buffer for the duration of
233 // the test — a replay that adopts a guest onto a network this host no longer
234 // serves has no caller to tell, and reports it the only way it can.
235 func captureLogs(t *testing.T) *bytes.Buffer {
236 t.Helper()
237 var logs bytes.Buffer
238 prev := slog.Default()
239 slog.SetDefault(slog.New(slog.NewTextHandler(&logs, nil)))
240 t.Cleanup(func() { slog.SetDefault(prev) })
241 return &logs
242 }
243
244 // TestAdoptNetworkWarnsWhenTheHostNoLongerServesTheNetwork pins the one moment
245 // an operator can be told about a dropped --host-network flag before it costs
246 // them a guest: the agent restarts, the running VM is adopted onto a network
247 // this host no longer has, and everything keeps working until the reboot that
248 // fails. The warning has to carry the consequence, because the state it
249 // describes looks entirely healthy from every other angle.
250 func TestAdoptNetworkWarnsWhenTheHostNoLongerServesTheNetwork(t *testing.T) {
251 logs := captureLogs(t)
252 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
253 n, _ := bridgedNet(t, noop, map[string]string{"lan": "br0"})
254 n.isTap = func(string) bool { return true }
255
256 n.AdoptNetwork("vm1", "office", "192.168.0.7")
257
258 warned := logs.String()
259 assert.Contains(t, warned, "level=WARN")
260 assert.Contains(t, warned, "vm_id=vm1", "the operator has to know which guest")
261 assert.Contains(t, warned, "network=office", "and which network went missing")
262 assert.Contains(t, warned, "--host-network", "the flag that would put it back")
263 assert.Contains(t, warned, "next reboot fails", "and what happens to this guest if they do not")
264
265 assert.Equal(t, "192.168.0.7", n.NetworkAddress("vm1"),
266 "the running guest keeps the address it has — the warning is instead of dropping it, not before")
267 }
268
269 // TestAdoptNetworkIsSilentForANetworkThisHostServes pins the other side: the
270 // ordinary restart, every guest adopted back onto a network the flags still
271 // name, says nothing. A warning that fires on the healthy path is one nobody
272 // reads on the unhealthy one.
273 func TestAdoptNetworkIsSilentForANetworkThisHostServes(t *testing.T) {
274 logs := captureLogs(t)
275 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
276 n, _ := bridgedNet(t, noop, map[string]string{"lan": "br0"})
277 n.isTap = func(string) bool { return true }
278
279 n.AdoptNetwork("vm1", "lan", "192.168.0.7")
280
281 assert.Empty(t, logs.String())
282 }
283
284 // TestReAttachKeepsWhatTheAttachmentAlreadyFound pins that re-attaching a VM
285 // that is already attached — a Boot retry, or a replay racing one — adds the
286 // network to what is there rather than starting the guest's state over: the
287 // address the snoop already found survives, and so does the snoop.
288 func TestReAttachKeepsWhatTheAttachmentAlreadyFound(t *testing.T) {
289 n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"})
290 require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan"))
291 snoop.last(t).found("192.168.0.42")
292 first := snoop.last(t)
293
294 n.isTap = func(string) bool { return true }
295 n.AdoptNetwork("vm1", "lan", "")
296 require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan"))
297
298 assert.Equal(t, "192.168.0.42", n.NetworkAddress("vm1"), "a re-attach is not a forgetting")
299 assert.Equal(t, 1, snoop.count(), "and the live snoop is still the only one")
300 require.NoError(t, n.DeleteTap(context.Background(), "vm1"))
301 assert.Error(t, first.ctx.Err(), "still the one teardown cancels")
302 }
303
304 // TestAdoptNetworkWithoutATapStartsNoSnoop pins the other half: a VM that is not
305 // running has no tap to watch, and Boot creates both.
306 func TestAdoptNetworkWithoutATapStartsNoSnoop(t *testing.T) {
307 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
308 n, snoop := bridgedNet(t, noop, map[string]string{"lan": "br0"})
309 n.isTap = func(string) bool { return false }
310
311 n.AdoptNetwork("vm1", "lan", "")
312
313 assert.Zero(t, snoop.count(), "nothing to snoop on a VM with no tap")
314 assert.Empty(t, n.NetworkAddress("vm1"), "and no address to claim on its behalf")
315 }
316
317 // TestDeleteTapTearsDownBothNICs pins the teardown of everything a networked
318 // guest held: both taps (a named one left behind holds a port on the operator's
319 // own bridge), the reservation, the snoop goroutine, and the discovered
320 // address. It also pins the race the listener concedes: it checks its context
321 // between reads, so an ACK received in the last poll window can call back after
322 // the cancel, and a deleted VM must not come back holding an address.
323 func TestDeleteTapTearsDownBothNICs(t *testing.T) {
324 n, snoop, calls := networkedNet(t, map[string]string{"lan": "br0"})
325 ip, err := n.ReserveIP("vm1")
326 require.NoError(t, err)
327 require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan"))
328 started := snoop.last(t)
329 started.found("192.168.0.42")
330
331 require.NoError(t, n.DeleteTap(context.Background(), "vm1"))
332
333 all := joinCalls(calls)
334 assert.Contains(t, all, "ip link del eit-vm1")
335 assert.Contains(t, all, "ip link del eil-vm1", "the named NIC's tap goes too")
336 mac, _ := net.ParseMAC(stateMAC("vm1"))
337 _, reserved := n.dhcp.Lookup(mac)
338 assert.False(t, reserved, "the NAT reservation is released")
339 assert.Error(t, started.ctx.Err(), "the snoop's context must be cancelled")
340 assert.Empty(t, n.NetworkAddress("vm1"), "a deleted VM holds no address")
341
342 started.found("192.168.0.99") // the late frame from the last poll window
343
344 // Re-attach the id before reading, because that is the only way to see the
345 // write the guard prevents: with no entry for the VM, NetworkAddress answers
346 // "" whether or not the late ACK was recorded. A VM re-created under the same
347 // id would attach exactly like this, and it must not inherit the dead one's
348 // address.
349 n.AdoptNetwork("vm1", "lan", "")
350 assert.Empty(t, n.NetworkAddress("vm1"), "a late ACK must not resurrect a deleted VM")
351 }
352
353 // TestDeleteTapRemovesANamedTapThisAgentNeverSaw pins that the teardown asks
354 // the device, not the bookkeeping: the map of who has a named NIC is in-memory
355 // state, and a tap that outlives its guest holds a port on a bridge eitri does
356 // not own.
357 func TestDeleteTapRemovesANamedTapThisAgentNeverSaw(t *testing.T) {
358 run, calls := recorder(nil, nil)
359 n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"})
360
361 require.NoError(t, n.DeleteTap(context.Background(), "vm1"))
362
363 assert.Contains(t, joinCalls(calls), "ip link del eil-vm1")
364 }
365
366 // TestDeleteTapRemovesTheNamedTapEvenWhenTheNATOneFails pins the ordering
367 // promise the comment makes: a named tap must never outlive its guest, so a
368 // NAT tap that refuses to go must not be allowed to keep the operator's bridge
369 // holding a port. Both are asked; both failures are reported.
370 func TestDeleteTapRemovesTheNamedTapEvenWhenTheNATOneFails(t *testing.T) {
371 run, calls := recorder(nil, map[string]error{
372 "ip link del eit-vm1": errors.New("RTNETLINK answers: Operation not permitted"),
373 })
374 n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"})
375
376 err := n.DeleteTap(context.Background(), "vm1")
377 require.Error(t, err, "the NAT tap's failure is still reported")
378 assert.Contains(t, err.Error(), "Operation not permitted")
379 assert.Contains(t, joinCalls(calls), "ip link del eil-vm1",
380 "the named tap is asked regardless — it is on a bridge eitri does not own")
381 }
382
383 // TestCreateTapRetryStartsNoSecondSnoop pins the idempotency a Boot retry and
384 // the startup replay both lean on: a VM whose snoop is already running gets no
385 // second listener. Two goroutines on one tap would report every ACK twice and
386 // only one of them would answer the cancel — the other would outlive the guest.
387 func TestCreateTapRetryStartsNoSecondSnoop(t *testing.T) {
388 n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"})
389 ip, err := n.ReserveIP("vm1")
390 require.NoError(t, err)
391 require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan"))
392 require.Equal(t, 1, snoop.count())
393 first := snoop.last(t)
394
395 require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan"))
396
397 assert.Equal(t, 1, snoop.count(), "the live snoop is left alone")
398 require.NoError(t, n.DeleteTap(context.Background(), "vm1"))
399 assert.Error(t, first.ctx.Err(), "and it is still the one teardown cancels")
400 }
401
402 // TestCreateNamedTapRejectsANonTapDevice is TestCreateTapRejectsNonTapDevice's
403 // twin for the second NIC: a name collision on the operator's own bridge must
404 // fail here, legibly and permanently, rather than as an illegible
405 // cloud-hypervisor error after eitri has enslaved somebody else's device.
406 func TestCreateNamedTapRejectsANonTapDevice(t *testing.T) {
407 run, _ := recorder(nil, map[string]error{
408 "ip link show dev eit-vm1": errors.New(`Device "eit-vm1" does not exist.`),
409 })
410 n, snoop := bridgedNet(t, run, map[string]string{"lan": "br0"})
411 n.isTap = func(name string) bool { return name != "eil-vm1" } // eil-vm1 exists and is something else
412
413 err := n.CreateTap(context.Background(), "vm1", "10.77.1.5", "lan")
414
415 require.Error(t, err)
416 assert.Contains(t, err.Error(), "not a TAP device")
417 assert.Contains(t, err.Error(), "eil-vm1")
418 var p interface{ Permanent() bool }
419 assert.True(t, errors.As(err, &p) && p.Permanent(), "a collision never self-resolves")
420 assert.Zero(t, snoop.count(), "and nothing is watched on a device that is not the guest's")
421 }
422
423 // TestNATOnlyCreateTapStartsNoSnoopOrSecondTap pins that the guest without a
424 // named network gained nothing: one tap, no listener, and the reservation
425 // pinned exactly as before.
426 func TestNATOnlyCreateTapStartsNoSnoopOrSecondTap(t *testing.T) {
427 errs := map[string]error{
428 "ip link show dev eit-vm2": errors.New(`Device "eit-vm2" does not exist.`),
429 }
430 run, calls := recorder(nil, errs)
431 n, snoop := bridgedNet(t, run, map[string]string{"lan": "br0"})
432
433 ip, err := n.ReserveIP("vm2")
434 require.NoError(t, err)
435 require.NoError(t, n.CreateTap(context.Background(), "vm2", ip, ""))
436
437 assert.Zero(t, snoop.count(), "a guest with one NIC has nothing to discover")
438 all := joinCalls(calls)
439 assert.Contains(t, all, "ip link set eit-vm2 master eitri0")
440 assert.NotContains(t, all, "eil-vm2", "and no second tap is created for it")
441 assert.Equal(t, ip, n.Address("vm2"))
442 assert.Empty(t, n.NetworkAddress("vm2"))
443 }
444
445 // bridgeAddrShow is the post-condition output EnsureBridge insists on before it
446 // gets as far as any nft command.
447 var bridgeAddrShow = map[string]string{
448 "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0",
449 }
450
451 // nftCalls returns the nft commands a run issued, in order and without the
452 // program name — the whole ruleset this host asked the kernel for.
453 func nftCalls(calls *[]call) []string {
454 var out []string
455 for _, c := range *calls {
456 if c.name == "nft" {
457 out = append(out, c.args)
458 }
459 }
460 return out
461 }
462
463 // bridgeFamily narrows nftCalls to the bridge-family half.
464 func bridgeFamily(all []string) []string {
465 var out []string
466 for _, c := range all {
467 if strings.Contains(c, "bridge eitri") {
468 out = append(out, c)
469 }
470 }
471 return out
472 }
473
474 // TestEnsureBridgeDropsGuestDHCPOnNamedNetworkHost pins the rule that stops a
475 // guest being a DHCP server on the operator's LAN: without it a guest can both
476 // answer its neighbours (minting an address the snoop cannot tell from the
477 // site's) and lease to the operator's real machines.
478 func TestEnsureBridgeDropsGuestDHCPOnNamedNetworkHost(t *testing.T) {
479 run, calls := recorder(bridgeAddrShow, nil)
480 n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"})
481 require.NoError(t, n.EnsureBridge(context.Background()))
482
483 assert.Equal(t, []string{
484 "add table bridge eitri",
485 "add chain bridge eitri prerouting { type filter hook prerouting priority -200 ; }",
486 "flush chain bridge eitri prerouting",
487 `add rule bridge eitri prerouting iifname "` + netTapPrefix + `*" udp sport 67 drop`,
488 }, bridgeFamily(nftCalls(calls)),
489 "the flush precedes the rule, so restarts cannot stack copies of it")
490 assert.True(t, strings.HasPrefix(n.NetTapName("vm1"), netTapPrefix),
491 "and the pattern matches the taps this host actually creates")
492 }
493
494 // TestEnsureBridgeIssuesNoBridgeFamilyRuleWithoutNetworks pins the other side of
495 // it: a host with no --host-network flag — every host today — asks the kernel
496 // for exactly the ruleset it always did.
497 func TestEnsureBridgeIssuesNoBridgeFamilyRuleWithoutNetworks(t *testing.T) {
498 run, calls := recorder(bridgeAddrShow, nil)
499 n, err := New(run, "10.77.1.0/24", nil)
500 require.NoError(t, err)
501 require.NoError(t, n.EnsureBridge(context.Background()))
502
503 assert.Equal(t, []string{
504 "add table ip eitri",
505 "add chain ip eitri postrouting { type nat hook postrouting priority srcnat ; }",
506 "flush chain ip eitri postrouting",
507 "add rule ip eitri postrouting ip saddr 10.77.1.0/24 masquerade",
508 }, nftCalls(calls), "a NAT-only host's ruleset is untouched by named networks")
509 assert.Empty(t, bridgeFamily(nftCalls(calls)))
510 }
511
512 // TestEnsureBridgeSurfacesGuestDHCPRuleFailure pins that a kernel which will not
513 // take the rule stops the agent naming the command, rather than starting a host
514 // that serves named networks with the block missing.
515 func TestEnsureBridgeSurfacesGuestDHCPRuleFailure(t *testing.T) {
516 rule := `nft add rule bridge eitri prerouting iifname "` + netTapPrefix + `*" udp sport 67 drop`
517 run, _ := recorder(bridgeAddrShow, map[string]error{
518 rule: errors.New("Error: Could not process rule: Operation not supported"),
519 })
520 n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"})
521
522 err := n.EnsureBridge(context.Background())
523 require.Error(t, err)
524 assert.Contains(t, err.Error(), "udp sport 67 drop", "the refusal names the command")
525 assert.Contains(t, err.Error(), "Operation not supported", "and what the kernel said")
526 }
527
528 // TestNetTapNameIsTruncatedBelowIFNAMSIZ pins the second device name against
529 // the same limit as the first, and against ever colliding with it.
530 func TestNetTapNameIsTruncatedBelowIFNAMSIZ(t *testing.T) {
531 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
532 n, err := New(noop, "10.77.1.0/24", nil)
533 require.NoError(t, err)
534
535 assert.Equal(t, "eil-vm1", n.NetTapName("vm1"))
536 assert.Equal(t, "eil-abcdefgh", n.NetTapName("abcdefghijklmnop"))
537 assert.LessOrEqual(t, len(n.NetTapName("abcdefghijklmnop")), 15, "IFNAMSIZ is 16 including NUL")
538 assert.NotEqual(t, n.TapName("vm1"), n.NetTapName("vm1"), "one guest, two devices, two names")
539 }
540
541 // TestSysfsIsBridge exercises the production probe against a fabricated /sys
542 // tree: a bridge has a bridge/ directory, an ordinary link does not.
543 func TestSysfsIsBridge(t *testing.T) {
544 root := t.TempDir()
545 require.NoError(t, os.MkdirAll(root+"/br0/bridge", 0o755))
546 require.NoError(t, os.MkdirAll(root+"/eth0", 0o755))
547 require.NoError(t, os.MkdirAll(root+"/notabr", 0o755))
548 require.NoError(t, os.WriteFile(root+"/notabr/bridge", []byte("x"), 0o644))
549
550 assert.True(t, sysfsIsBridgeAt(root, "br0"))
551 assert.False(t, sysfsIsBridgeAt(root, "eth0"), "an ordinary link is not a bridge")
552 assert.False(t, sysfsIsBridgeAt(root, "notabr"), "a file named bridge is not the directory")
553 assert.False(t, sysfsIsBridgeAt(root, "absent"), "a missing link is not a bridge")
554 }
internal/agent/netenv/netenv.go
Old New
@@ -2,10 +2,18 @@
2 // the host as .1 gateway, per-VM taps, and NAT for outbound internet. The 2 // the host as .1 gateway, per-VM taps, and NAT for outbound internet. The
3 // bridge is a pure masqueraded underlay; inbound admin reachability is via the 3 // bridge is a pure masqueraded underlay; inbound admin reachability is via the
4 // server's SSH-CA jump gate (internal/server/sshgate), not any guest overlay. 4 // server's SSH-CA jump gate (internal/server/sshgate), not any guest overlay.
5 //
6 // A VM whose spec names one of the host's configured networks gets a SECOND
7 // tap on top of all that, attached to a bridge the operator already owns (see
8 // named.go). None of the machinery above touches it — the site's own DHCP
9 // server addresses that NIC and its own gateway routes it — and none of it is
10 // withheld from the guest either: every VM keeps its NAT attachment, its
11 // reservation and its place on eitri0 beside its siblings.
5 package netenv 12 package netenv
6 13
7 import ( 14 import (
8 "context" 15 "context"
16 "errors"
9 "fmt" 17 "fmt"
10 "net" 18 "net"
11 "net/netip" 19 "net/netip"
@@ -13,6 +21,7 @@ import (
13 "path/filepath" 21 "path/filepath"
14 "strconv" 22 "strconv"
15 "strings" 23 "strings"
24 "sync"
16 "time" 25 "time"
17 26
18 "github.com/a73x/eitri/internal/agent/dhcp" 27 "github.com/a73x/eitri/internal/agent/dhcp"
@@ -37,6 +46,49 @@ type Net struct {
37 // the kernel, and the logic above it should be testable without one. 46 // the kernel, and the logic above it should be testable without one.
38 ifaces func() ([]Iface, error) 47 ifaces func() ([]Iface, error)
39 dhcp *dhcp.Server 48 dhcp *dhcp.Server
49
50 // networks maps a configured network name to its operator-owned bridge.
51 // Fixed at construction: it is the agent's command line, and changing it
52 // is a restart.
53 networks map[string]string
54 // isBridge reports whether a link is a bridge. Injectable like isTap;
55 // production reads /sys/class/net/<name>/bridge, a directory that exists
56 // exactly for bridge devices.
57 isBridge func(name string) bool
58 // listen watches a tap for the DHCP ACK addressed to a guest's MAC.
59 // Injectable like isTap, and for a second reason: the production snoop is
60 // a Linux packet socket, so the default is chosen per platform.
61 listen func(ctx context.Context, ifname string, mac net.HardwareAddr, found func(ip string)) error
62
63 mu sync.Mutex
64 // attached holds one entry per VM that has a second NIC on a named network,
65 // and holding an entry is what being attached means: it is created when the
66 // NIC is (CreateTap, or AdoptNetwork replaying one that survived a restart)
67 // and deleted once, whole, when the NIC goes (DeleteTap). Everything the
68 // attachment accumulates — the address the site granted it, the goroutine
69 // watching for the next one — lives and dies with that single entry, so
70 // there is no state left over to speak for a VM this map has forgotten. In
71 // memory, rebuilt at startup by the replay, exactly like the DHCP
72 // reservation table.
73 attached map[string]*attachment
74 }
75
76 // attachment is one VM's named-network NIC as this agent holds it. Which
77 // network that is does not appear here: the record the control plane placed is
78 // what says so, and the agent re-reads it on every replay, so a copy in memory
79 // would only be a second answer to a question already settled. What is here is
80 // what nothing else can answer — the address the site handed this NIC, and the
81 // goroutine listening for the next one. Both arrive after the attachment does,
82 // and both end with it, because the entry does.
83 type attachment struct {
84 // discovered is the last address the site's DHCP server granted this NIC.
85 // Never the NAT address: that one this host allocates, and it lives in the
86 // reservation table. Empty until an ACK is snooped or a restart replays one.
87 discovered string
88 // stopSnoop ends the goroutine watching this NIC's tap, and its presence is
89 // what says one is running: nil for a VM adopted with no tap to watch, and
90 // nil again after a snoop that would not start (so a retry re-arms).
91 stopSnoop context.CancelFunc
40 } 92 }
41 93
42 // guestDNS is handed to guests as their DHCP DNS servers (option 6). Public 94 // guestDNS is handed to guests as their DHCP DNS servers (option 6). Public
@@ -45,7 +97,11 @@ type Net struct {
45 var guestDNS = []net.IP{net.IPv4(1, 1, 1, 1), net.IPv4(9, 9, 9, 9)} 97 var guestDNS = []net.IP{net.IPv4(1, 1, 1, 1), net.IPv4(9, 9, 9, 9)}
46 98
47 // New constructs a Net. cidr must be a valid IPv4 prefix (e.g. "10.77.1.0/24"). 99 // New constructs a Net. cidr must be a valid IPv4 prefix (e.g. "10.77.1.0/24").
48 func New(run exec.Runner, cidr string) (*Net, error) { 100 // networks maps a named guest network to the operator-owned bridge backing it;
101 // nil or empty means this host serves only the NAT underlay. The names are not
102 // checked against the host's links here — VerifyNetworks does that, so a
103 // caller decides whether a missing bridge stops the agent.
104 func New(run exec.Runner, cidr string, networks map[string]string) (*Net, error) {
49 p, err := netip.ParsePrefix(cidr) 105 p, err := netip.ParsePrefix(cidr)
50 if err != nil { 106 if err != nil {
51 return nil, err 107 return nil, err
@@ -53,7 +109,11 @@ func New(run exec.Runner, cidr string) (*Net, error) {
53 if !p.Addr().Is4() { 109 if !p.Addr().Is4() {
54 return nil, fmt.Errorf("bridge CIDR must be IPv4, got %s", cidr) 110 return nil, fmt.Errorf("bridge CIDR must be IPv4, got %s", cidr)
55 } 111 }
56 n := &Net{run: run, cidr: p, isTap: sysfsIsTap, ifaces: hostIfaces} 112 n := &Net{
113 run: run, cidr: p, isTap: sysfsIsTap, ifaces: hostIfaces,
114 networks: networks, isBridge: sysfsIsBridge, listen: defaultListen,
115 attached: map[string]*attachment{},
116 }
57 gw := net.ParseIP(n.Gateway()) 117 gw := net.ParseIP(n.Gateway())
58 mask := net.CIDRMask(p.Bits(), 32) 118 mask := net.CIDRMask(p.Bits(), 32)
59 n.dhcp = dhcp.NewServer(Bridge, n.cidr.String(), gw, mask, guestDNS, 12*time.Hour) 119 n.dhcp = dhcp.NewServer(Bridge, n.cidr.String(), gw, mask, guestDNS, 12*time.Hour)
@@ -88,12 +148,30 @@ func (n *Net) Gateway() string { return n.cidr.Masked().Addr().Next().String() }
88 // IFNAMSIZ limit. It is a method rather than a package function because it is 148 // IFNAMSIZ limit. It is a method rather than a package function because it is
89 // part of what a host networking backend answers for — a backend with no tap 149 // part of what a host networking backend answers for — a backend with no tap
90 // device has no such name, and nothing above the platform line may assume one. 150 // device has no such name, and nothing above the platform line may assume one.
91 func (n *Net) TapName(vmID string) string { 151 func (n *Net) TapName(vmID string) string { return tapName("eit-", vmID) }
92 prefix := vmID 152
93 if len(prefix) > 8 { 153 // netTapPrefix begins the name of every named-network tap, and is the one place
94 prefix = prefix[:8] 154 // that shape is written down. The nftables rule in named.go matches devices by
155 // it, so a rename that missed the rule would leave the bridge judging a name no
156 // tap has any more — hence one home, and both readers derived from it.
157 const netTapPrefix = "eil-"
158
159 // NetTapName returns the device name of the VM's SECOND tap, the one on a
160 // named host network: "eil-" plus the same 8 characters, so the two taps of
161 // one guest sort together and neither can be mistaken for the other in
162 // `ip link`. The "l" is just that marker — the second-NIC tap, distinct from
163 // "eit-" — not short for anything the console says; live devices already wear
164 // the prefix, so it stays even though the word it once stood for did not.
165 // Same 12-character budget as TapName.
166 func (n *Net) NetTapName(vmID string) string { return tapName(netTapPrefix, vmID) }
167
168 // tapName is the shared truncation both device names owe their IFNAMSIZ safety
169 // to: one rule, so a second NIC can never be the one that overruns it.
170 func tapName(prefix, vmID string) string {
171 if len(vmID) > 8 {
172 vmID = vmID[:8]
95 } 173 }
96 return "eit-" + prefix 174 return prefix + vmID
97 } 175 }
98 176
99 // ReserveIP returns a sticky IP for vmID, allocated from the bridge CIDR and 177 // ReserveIP returns a sticky IP for vmID, allocated from the bridge CIDR and
@@ -101,6 +179,11 @@ func (n *Net) TapName(vmID string) string {
101 // MAC). Idempotent: the same VM gets the same address across reboots and agent 179 // MAC). Idempotent: the same VM gets the same address across reboots and agent
102 // restarts. The table lives in memory, so an address survives an agent restart 180 // restarts. The table lives in memory, so an address survives an agent restart
103 // only by way of the startup replay that rebuilds it from durable records. 181 // only by way of the startup replay that rebuilds it from durable records.
182 //
183 // Every VM gets one, whatever else its spec asks for: the NAT NIC is the
184 // management fabric — the gate's splice target, the guests' way to each other,
185 // the egress — and a guest that also sits on the operator's LAN is a guest with
186 // two NICs, not a guest that gave this one up.
104 func (n *Net) ReserveIP(vmID string) (string, error) { 187 func (n *Net) ReserveIP(vmID string) (string, error) {
105 mac, err := net.ParseMAC(state.MAC(vmID)) 188 mac, err := net.ParseMAC(state.MAC(vmID))
106 if err != nil { 189 if err != nil {
@@ -113,9 +196,13 @@ func (n *Net) ReserveIP(vmID string) (string, error) {
113 return ip.String(), nil 196 return ip.String(), nil
114 } 197 }
115 198
116 // Address returns the address currently reserved for vmID, or "" when the VM 199 // Address returns the address currently reserved for vmID on the NAT underlay,
117 // holds no reservation. Unlike ReserveIP it never allocates: it is the polled 200 // or "" when the VM holds no reservation. Unlike ReserveIP it never allocates:
118 // read of what this host believes the guest's address is. 201 // it is the polled read of what this host believes the guest's address is.
202 //
203 // It is unaffected by a named network. A guest's second NIC has its own address
204 // and its own question (NetworkAddress); this one is the address everything
205 // eitri does with a VM goes through, and it is known before the guest boots.
119 func (n *Net) Address(vmID string) string { 206 func (n *Net) Address(vmID string) string {
120 mac, err := net.ParseMAC(state.MAC(vmID)) 207 mac, err := net.ParseMAC(state.MAC(vmID))
121 if err != nil { 208 if err != nil {
@@ -158,9 +245,13 @@ func (n *Net) best(ctx context.Context, name string, args ...string) (string, er
158 // forwarding, and installs an nftables NAT rule that masquerades VM outbound 245 // forwarding, and installs an nftables NAT rule that masquerades VM outbound
159 // traffic on every interface. 246 // traffic on every interface.
160 // 247 //
248 // On a host that serves named networks it also programs the bridge-family rule
249 // those networks owe the operator (blockGuestDHCP). A host with none issues no
250 // bridge-family command at all: its ruleset is what it always was.
251 //
161 // The function is safe to call on every agent restart: the nft chain is 252 // The function is safe to call on every agent restart: the nft chain is
162 // flushed before the masquerade rule is added, so rules never accumulate 253 // flushed before the masquerade rule is added, so rules never accumulate
163 // across restarts. 254 // across restarts. The same holds for the bridge-family chain.
164 func (n *Net) EnsureBridge(ctx context.Context) error { 255 func (n *Net) EnsureBridge(ctx context.Context) error {
165 gw := n.Gateway() 256 gw := n.Gateway()
166 bits := n.cidr.Bits() 257 bits := n.cidr.Bits()
@@ -224,7 +315,7 @@ func (n *Net) EnsureBridge(ctx context.Context) error {
224 return fmt.Errorf("nft add rule: %w", err) 315 return fmt.Errorf("nft add rule: %w", err)
225 } 316 }
226 317
227 return nil 318 return n.blockGuestDHCP(ctx)
228 } 319 }
229 320
230 // StartDHCP begins serving reserved leases on the bridge. Call after 321 // StartDHCP begins serving reserved leases on the bridge. Call after
@@ -256,7 +347,25 @@ func (n *Net) AddReservation(vmID, ip string) {
256 // allow-list can never chase across iproute2 versions. A create-retry hitting 347 // allow-list can never chase across iproute2 versions. A create-retry hitting
257 // the previous attempt's tap must not mask the retry's real error. Pinned by 348 // the previous attempt's tap must not mask the retry's real error. Pinned by
258 // TestCreateTapIdempotentWhenTapExists. 349 // TestCreateTapIdempotentWhenTapExists.
259 func (n *Net) CreateTap(ctx context.Context, vmID, ip string) error { 350 //
351 // network, when non-empty, adds a SECOND tap on the operator's own bridge (see
352 // createNamedTap) after this one — the guest keeps everything above and gains a
353 // NIC on the named network. It is validated before anything is created, so a
354 // network this agent does not serve leaves the host exactly as it found it.
355 func (n *Net) CreateTap(ctx context.Context, vmID, ip, network string) error {
356 if network != "" {
357 if _, ok := n.networks[network]; !ok {
358 // The control plane only places a network on a host that advertised
359 // it, so getting here means this agent restarted without the flag.
360 // Permanent, and refused before the NAT tap exists: converge must
361 // fail the VM legibly, never boot it with the NIC it asked for
362 // missing — a guest silently reachable on the private underlay
363 // alone is a guest nothing that was talking to it can find.
364 return permanent.Errorf("VM wants network %q but this agent is not configured with it; "+
365 "restart the agent with --host-network %s=<bridge> (its NIC is never silently dropped)",
366 network, network)
367 }
368 }
260 tap := n.TapName(vmID) 369 tap := n.TapName(vmID)
261 if _, err := n.run(ctx, "ip", "link", "show", "dev", tap); err != nil { 370 if _, err := n.run(ctx, "ip", "link", "show", "dev", tap); err != nil {
262 if _, err := n.best(ctx, "ip", "tuntap", "add", "dev", tap, "mode", "tap"); err != nil { 371 if _, err := n.best(ctx, "ip", "tuntap", "add", "dev", tap, "mode", "tap"); err != nil {
@@ -275,22 +384,53 @@ func (n *Net) CreateTap(ctx context.Context, vmID, ip string) error {
275 return err 384 return err
276 } 385 }
277 n.AddReservation(vmID, ip) 386 n.AddReservation(vmID, ip)
278 return nil 387 if network == "" {
388 return nil
389 }
390 return n.createNamedTap(ctx, vmID, network)
279 } 391 }
280 392
281 // DeleteTap removes the VM's DHCP reservation and TAP device. Idempotent: 393 // DeleteTap removes the VM's DHCP reservation and BOTH of its TAP devices, plus
282 // a missing device is tolerated. 394 // the snoop and discovered address a named NIC leaves behind. Idempotent: a
395 // missing device is tolerated.
396 //
397 // The named tap is deleted whether or not this agent remembers the VM having
398 // one. The memory of it is in-memory state rebuilt from records at startup, and
399 // a tap outliving its guest holds a port on the operator's bridge — so the
400 // device, not the bookkeeping, is what gets asked.
401 //
402 // Both deletes are attempted and their errors joined, for the same reason: an
403 // early return on the NAT tap would leave the named one on the operator's
404 // bridge, alive, with nothing left that remembers to ask again.
283 func (n *Net) DeleteTap(ctx context.Context, vmID string) error { 405 func (n *Net) DeleteTap(ctx context.Context, vmID string) error {
284 tap := n.TapName(vmID) 406 n.mu.Lock()
407 var cancel context.CancelFunc
408 if a, live := n.attached[vmID]; live {
409 cancel = a.stopSnoop
410 }
411 delete(n.attached, vmID)
412 n.mu.Unlock()
413 if cancel != nil {
414 cancel()
415 }
285 if mac, err := net.ParseMAC(state.MAC(vmID)); err == nil { 416 if mac, err := net.ParseMAC(state.MAC(vmID)); err == nil {
286 n.dhcp.RemoveReservation(mac) 417 n.dhcp.RemoveReservation(mac)
287 } 418 }
288 out, err := n.run(ctx, "ip", "link", "del", tap) 419 return errors.Join(
420 n.deleteLink(ctx, n.TapName(vmID)),
421 n.deleteLink(ctx, n.NetTapName(vmID)),
422 )
423 }
424
425 // deleteLink removes one device, tolerating its absence — the shape both taps
426 // are torn down in.
427 func (n *Net) deleteLink(ctx context.Context, dev string) error {
428 out, err := n.run(ctx, "ip", "link", "del", dev)
289 if err != nil { 429 if err != nil {
290 if tolerated(out, err, "Cannot find device") { 430 if tolerated(out, err, "Cannot find device") {
291 return nil 431 return nil
292 } 432 }
293 return fmt.Errorf("ip link del %s: %w", tap, err) 433 return fmt.Errorf("ip link del %s: %w", dev, err)
294 } 434 }
295 return nil 435 return nil
296 } 436 }
internal/agent/netenv/netenv_test.go
Old New
@@ -17,6 +17,9 @@ import (
17 // stateMAC re-exposes state.MAC for the test without a separate import block. 17 // stateMAC re-exposes state.MAC for the test without a separate import block.
18 func stateMAC(vmID string) string { return state.MAC(vmID) } 18 func stateMAC(vmID string) string { return state.MAC(vmID) }
19 19
20 // stateNetMAC re-exposes state.NetMAC — the second NIC's address — likewise.
21 func stateNetMAC(vmID string) string { return state.NetMAC(vmID) }
22
20 type call struct { 23 type call struct {
21 name string 24 name string
22 args string 25 args string
@@ -48,11 +51,8 @@ func TestEnsureBridgeSetsUpGatewayForwardingAndNAT(t *testing.T) {
48 errs := map[string]error{ 51 errs := map[string]error{
49 "ip link show dev eitri0": errors.New("Device \"eitri0\" does not exist."), 52 "ip link show dev eitri0": errors.New("Device \"eitri0\" does not exist."),
50 } 53 }
51 out := map[string]string{ 54 run, calls := recorder(bridgeAddrShow, errs)
52 "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0", 55 n, err := New(run, "10.77.1.0/24", nil)
53 }
54 run, calls := recorder(out, errs)
55 n, err := New(run, "10.77.1.0/24")
56 require.NoError(t, err) 56 require.NoError(t, err)
57 require.NoError(t, n.EnsureBridge(context.Background())) 57 require.NoError(t, n.EnsureBridge(context.Background()))
58 58
@@ -82,7 +82,7 @@ func TestEnsureBridgeIdempotentOnRestart(t *testing.T) {
82 "ip addr add 10.77.1.1/24 dev eitri0": errors.New("Error: ipv4: Address already assigned."), 82 "ip addr add 10.77.1.1/24 dev eitri0": errors.New("Error: ipv4: Address already assigned."),
83 } 83 }
84 run, calls := recorder(out, errs) 84 run, calls := recorder(out, errs)
85 n, err := New(run, "10.77.1.0/24") 85 n, err := New(run, "10.77.1.0/24", nil)
86 require.NoError(t, err) 86 require.NoError(t, err)
87 require.NoError(t, n.EnsureBridge(context.Background()), 87 require.NoError(t, n.EnsureBridge(context.Background()),
88 "restart with existing bridge+address must succeed") 88 "restart with existing bridge+address must succeed")
@@ -99,8 +99,8 @@ func TestTapLifecycle(t *testing.T) {
99 "ip link show dev eit-abc123": errors.New(`Device "eit-abc123" does not exist.`), 99 "ip link show dev eit-abc123": errors.New(`Device "eit-abc123" does not exist.`),
100 } 100 }
101 run, calls := recorder(nil, errs) 101 run, calls := recorder(nil, errs)
102 n, _ := New(run, "10.77.1.0/24") 102 n, _ := New(run, "10.77.1.0/24", nil)
103 require.NoError(t, n.CreateTap(context.Background(), "abc123", "10.77.1.5")) 103 require.NoError(t, n.CreateTap(context.Background(), "abc123", "10.77.1.5", ""))
104 all := joinCalls(calls) 104 all := joinCalls(calls)
105 assert.Contains(t, all, "ip tuntap add dev eit-abc123 mode tap") 105 assert.Contains(t, all, "ip tuntap add dev eit-abc123 mode tap")
106 assert.Contains(t, all, "ip link set eit-abc123 master eitri0") 106 assert.Contains(t, all, "ip link set eit-abc123 master eitri0")
@@ -116,7 +116,7 @@ func TestEnsureBridgeFlushPrecedesRuleAdd(t *testing.T) {
116 "ip -o addr show dev eitri0": addrShowOut, 116 "ip -o addr show dev eitri0": addrShowOut,
117 } 117 }
118 run, calls := recorder(out, nil) 118 run, calls := recorder(out, nil)
119 n, err := New(run, "10.77.1.0/24") 119 n, err := New(run, "10.77.1.0/24", nil)
120 require.NoError(t, err) 120 require.NoError(t, err)
121 121
122 // Call EnsureBridge twice — simulating two agent restarts. 122 // Call EnsureBridge twice — simulating two agent restarts.
@@ -156,7 +156,7 @@ func TestEnsureBridgeGatewayConflictDetected(t *testing.T) {
156 "ip -o addr show dev eitri0": "2: eitri0 inet scope global eitri0", 156 "ip -o addr show dev eitri0": "2: eitri0 inet scope global eitri0",
157 } 157 }
158 run, _ := recorder(out, nil) 158 run, _ := recorder(out, nil)
159 n, err := New(run, "10.77.1.0/24") 159 n, err := New(run, "10.77.1.0/24", nil)
160 require.NoError(t, err) 160 require.NoError(t, err)
161 161
162 err = n.EnsureBridge(context.Background()) 162 err = n.EnsureBridge(context.Background())
@@ -171,7 +171,7 @@ func TestEnsureBridgePlainMasquerade(t *testing.T) {
171 "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0", 171 "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0",
172 } 172 }
173 run, calls := recorder(out, nil) 173 run, calls := recorder(out, nil)
174 n, err := New(run, "10.77.1.0/24") 174 n, err := New(run, "10.77.1.0/24", nil)
175 require.NoError(t, err) 175 require.NoError(t, err)
176 require.NoError(t, n.EnsureBridge(context.Background())) 176 require.NoError(t, n.EnsureBridge(context.Background()))
177 177
@@ -204,8 +204,8 @@ func TestCreateTapToleratesAlreadyExists(t *testing.T) {
204 "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`), 204 "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`),
205 }, 205 },
206 ) 206 )
207 n, _ := New(run, "10.77.1.0/24") 207 n, _ := New(run, "10.77.1.0/24", nil)
208 require.NoError(t, n.CreateTap(context.Background(), "x", "10.77.1.5"), 208 require.NoError(t, n.CreateTap(context.Background(), "x", "10.77.1.5", ""),
209 "already-exists must be tolerated for idempotency") 209 "already-exists must be tolerated for idempotency")
210 }) 210 })
211 } 211 }
@@ -218,8 +218,8 @@ func TestCreateTapPropagatesRealError(t *testing.T) {
218 key: errors.New("Operation not permitted"), 218 key: errors.New("Operation not permitted"),
219 "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`), 219 "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`),
220 }) 220 })
221 n, _ := New(run, "10.77.1.0/24") 221 n, _ := New(run, "10.77.1.0/24", nil)
222 err := n.CreateTap(context.Background(), "x", "10.77.1.5") 222 err := n.CreateTap(context.Background(), "x", "10.77.1.5", "")
223 require.Error(t, err) 223 require.Error(t, err)
224 assert.Contains(t, err.Error(), "Operation not permitted") 224 assert.Contains(t, err.Error(), "Operation not permitted")
225 } 225 }
@@ -229,15 +229,15 @@ func TestCreateTapPropagatesRealError(t *testing.T) {
229 func TestDeleteTapToleratesMissingDevice(t *testing.T) { 229 func TestDeleteTapToleratesMissingDevice(t *testing.T) {
230 key := "ip link del eit-x" 230 key := "ip link del eit-x"
231 run, _ := recorder(nil, map[string]error{key: errors.New(`Cannot find device "eit-x"`)}) 231 run, _ := recorder(nil, map[string]error{key: errors.New(`Cannot find device "eit-x"`)})
232 n, _ := New(run, "10.77.1.0/24") 232 n, _ := New(run, "10.77.1.0/24", nil)
233 require.NoError(t, n.DeleteTap(context.Background(), "x")) 233 require.NoError(t, n.DeleteTap(context.Background(), "x"))
234 234
235 run2, _ := recorder(map[string]string{key: "Cannot find device"}, map[string]error{key: errors.New("exit status 1")}) 235 run2, _ := recorder(map[string]string{key: "Cannot find device"}, map[string]error{key: errors.New("exit status 1")})
236 n2, _ := New(run2, "10.77.1.0/24") 236 n2, _ := New(run2, "10.77.1.0/24", nil)
237 require.NoError(t, n2.DeleteTap(context.Background(), "x")) 237 require.NoError(t, n2.DeleteTap(context.Background(), "x"))
238 238
239 run3, _ := recorder(nil, map[string]error{key: errors.New("RTNETLINK answers: Operation not permitted")}) 239 run3, _ := recorder(nil, map[string]error{key: errors.New("RTNETLINK answers: Operation not permitted")})
240 n3, _ := New(run3, "10.77.1.0/24") 240 n3, _ := New(run3, "10.77.1.0/24", nil)
241 require.Error(t, n3.DeleteTap(context.Background(), "x")) 241 require.Error(t, n3.DeleteTap(context.Background(), "x"))
242 } 242 }
243 243
@@ -258,10 +258,10 @@ func TestCreateTapIdempotentWhenTapExists(t *testing.T) {
258 "ip tuntap add dev eit-busy mode tap": errors.New("ioctl(TUNSETIFF): Device or resource busy"), 258 "ip tuntap add dev eit-busy mode tap": errors.New("ioctl(TUNSETIFF): Device or resource busy"),
259 } 259 }
260 run, calls := recorder(out, errs) 260 run, calls := recorder(out, errs)
261 n, err := New(run, "10.77.1.0/24") 261 n, err := New(run, "10.77.1.0/24", nil)
262 require.NoError(t, err) 262 require.NoError(t, err)
263 n.isTap = func(string) bool { return true } // the leftover IS a real tap 263 n.isTap = func(string) bool { return true } // the leftover IS a real tap
264 require.NoError(t, n.CreateTap(context.Background(), "busy", "10.77.1.5"), 264 require.NoError(t, n.CreateTap(context.Background(), "busy", "10.77.1.5", ""),
265 "CreateTap on an existing tap must succeed") 265 "CreateTap on an existing tap must succeed")
266 266
267 all := joinCalls(calls) 267 all := joinCalls(calls)
@@ -279,11 +279,11 @@ func TestCreateTapRejectsNonTapDevice(t *testing.T) {
279 "ip link show dev eit-clash": "5: eit-clash: <BROADCAST> state DOWN", // exists 279 "ip link show dev eit-clash": "5: eit-clash: <BROADCAST> state DOWN", // exists
280 } 280 }
281 run, calls := recorder(out, nil) 281 run, calls := recorder(out, nil)
282 n, err := New(run, "10.77.1.0/24") 282 n, err := New(run, "10.77.1.0/24", nil)
283 require.NoError(t, err) 283 require.NoError(t, err)
284 n.isTap = func(tap string) bool { return false } // not a tun/tap device 284 n.isTap = func(tap string) bool { return false } // not a tun/tap device
285 285
286 err = n.CreateTap(context.Background(), "clash", "10.77.1.5") 286 err = n.CreateTap(context.Background(), "clash", "10.77.1.5", "")
287 require.Error(t, err) 287 require.Error(t, err)
288 assert.Contains(t, err.Error(), "not a TAP device") 288 assert.Contains(t, err.Error(), "not a TAP device")
289 var p interface{ Permanent() bool } 289 var p interface{ Permanent() bool }
@@ -300,11 +300,11 @@ func TestCreateTapAcceptsExistingRealTap(t *testing.T) {
300 "ip link show dev eit-ok": "5: eit-ok: <NO-CARRIER,BROADCAST,MULTICAST,UP> master eitri0", 300 "ip link show dev eit-ok": "5: eit-ok: <NO-CARRIER,BROADCAST,MULTICAST,UP> master eitri0",
301 } 301 }
302 run, calls := recorder(out, nil) 302 run, calls := recorder(out, nil)
303 n, err := New(run, "10.77.1.0/24") 303 n, err := New(run, "10.77.1.0/24", nil)
304 require.NoError(t, err) 304 require.NoError(t, err)
305 n.isTap = func(tap string) bool { return true } 305 n.isTap = func(tap string) bool { return true }
306 306
307 require.NoError(t, n.CreateTap(context.Background(), "ok", "10.77.1.5")) 307 require.NoError(t, n.CreateTap(context.Background(), "ok", "10.77.1.5", ""))
308 all := joinCalls(calls) 308 all := joinCalls(calls)
309 assert.NotContains(t, all, "ip tuntap add") 309 assert.NotContains(t, all, "ip tuntap add")
310 assert.Contains(t, all, "ip link set eit-ok master eitri0") 310 assert.Contains(t, all, "ip link set eit-ok master eitri0")
@@ -342,10 +342,10 @@ func TestCreateTapAddsReservationAndTapCommands(t *testing.T) {
342 "ip link show dev eit-vm-abc12": errors.New("does not exist"), 342 "ip link show dev eit-vm-abc12": errors.New("does not exist"),
343 } 343 }
344 run, calls := recorder(nil, errs) 344 run, calls := recorder(nil, errs)
345 n, err := New(run, "10.77.1.0/24") 345 n, err := New(run, "10.77.1.0/24", nil)
346 require.NoError(t, err) 346 require.NoError(t, err)
347 347
348 require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7")) 348 require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7", ""))
349 349
350 all := joinCalls(calls) 350 all := joinCalls(calls)
351 assert.Contains(t, all, "ip tuntap add dev eit-vm-abc12 mode tap") 351 assert.Contains(t, all, "ip tuntap add dev eit-vm-abc12 mode tap")
@@ -359,7 +359,7 @@ func TestCreateTapAddsReservationAndTapCommands(t *testing.T) {
359 359
360 func TestReserveIPAllocatesAndRecordsReservation(t *testing.T) { 360 func TestReserveIPAllocatesAndRecordsReservation(t *testing.T) {
361 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } 361 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
362 n, err := New(noop, "10.77.1.0/24") 362 n, err := New(noop, "10.77.1.0/24", nil)
363 require.NoError(t, err) 363 require.NoError(t, err)
364 364
365 ip, err := n.ReserveIP("vm-alpha") 365 ip, err := n.ReserveIP("vm-alpha")
@@ -381,9 +381,9 @@ func TestDeleteTapRemovesReservationAndTap(t *testing.T) {
381 "ip link show dev eit-vm-abc12": errors.New("does not exist"), 381 "ip link show dev eit-vm-abc12": errors.New("does not exist"),
382 } 382 }
383 run, calls := recorder(nil, errs) 383 run, calls := recorder(nil, errs)
384 n, err := New(run, "10.77.1.0/24") 384 n, err := New(run, "10.77.1.0/24", nil)
385 require.NoError(t, err) 385 require.NoError(t, err)
386 require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7")) 386 require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7", ""))
387 387
388 require.NoError(t, n.DeleteTap(context.Background(), "vm-abc12345")) 388 require.NoError(t, n.DeleteTap(context.Background(), "vm-abc12345"))
389 389
@@ -395,7 +395,7 @@ func TestDeleteTapRemovesReservationAndTap(t *testing.T) {
395 395
396 func TestAddressReportsTheReservationWithoutAllocating(t *testing.T) { 396 func TestAddressReportsTheReservationWithoutAllocating(t *testing.T) {
397 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } 397 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
398 n, err := New(noop, "10.77.1.0/24") 398 n, err := New(noop, "10.77.1.0/24", nil)
399 require.NoError(t, err) 399 require.NoError(t, err)
400 400
401 assert.Empty(t, n.Address("vm-alpha"), "an unreserved VM has no address") 401 assert.Empty(t, n.Address("vm-alpha"), "an unreserved VM has no address")
@@ -415,7 +415,7 @@ func TestAddressReportsTheReservationWithoutAllocating(t *testing.T) {
415 415
416 func TestReserveIPKeepsAPreloadedAddress(t *testing.T) { 416 func TestReserveIPKeepsAPreloadedAddress(t *testing.T) {
417 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } 417 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
418 n, err := New(noop, "10.77.1.0/24") 418 n, err := New(noop, "10.77.1.0/24", nil)
419 require.NoError(t, err) 419 require.NoError(t, err)
420 420
421 // The reservation table is in-memory and preloaded at agent startup from 421 // The reservation table is in-memory and preloaded at agent startup from
@@ -437,7 +437,7 @@ func TestReserveIPKeepsAPreloadedAddress(t *testing.T) {
437 437
438 func TestTapNameIsTruncatedBelowIFNAMSIZ(t *testing.T) { 438 func TestTapNameIsTruncatedBelowIFNAMSIZ(t *testing.T) {
439 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } 439 noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil }
440 n, err := New(noop, "10.77.1.0/24") 440 n, err := New(noop, "10.77.1.0/24", nil)
441 require.NoError(t, err) 441 require.NoError(t, err)
442 442
443 assert.Equal(t, "eit-vm1", n.TapName("vm1")) 443 assert.Equal(t, "eit-vm1", n.TapName("vm1"))
internal/agent/netenv/snoop_linux.go
Old New
@@ -0,0 +1,12 @@
1 //go:build linux
2
3 package netenv
4
5 import "github.com/a73x/eitri/internal/agent/netsnoop"
6
7 // defaultListen is production address discovery for a guest's second NIC: a
8 // packet socket on that NIC's own tap, reading the DHCP ACK the site's server
9 // sends it. A variable behind a build tag rather than a direct call because
10 // the snoop needs AF_PACKET and this package is compiled on platforms that
11 // have none.
12 var defaultListen = netsnoop.Listen
internal/agent/netenv/snoop_other.go
Old New
@@ -0,0 +1,18 @@
1 //go:build !linux
2
3 package netenv
4
5 import (
6 "context"
7 "fmt"
8 "net"
9 )
10
11 // defaultListen refuses where there is no packet socket to snoop with. Nothing
12 // reaches it — named networks come from a flag only a Linux host acts on, and
13 // their taps are created by the Linux backend — so this exists to keep the
14 // package buildable everywhere the agent is, and to fail loudly rather than
15 // silently report no address if that ever stops being true.
16 func defaultListen(_ context.Context, ifname string, _ net.HardwareAddr, _ func(ip string)) error {
17 return fmt.Errorf("snoop %s: named networks require linux", ifname)
18 }
internal/agent/netenv/uplink_test.go
Old New
@@ -26,7 +26,7 @@ func ifaceFrom(t *testing.T, name string, cidrs ...string) Iface {
26 // The command runner is nil: this check reads interfaces and runs nothing. 26 // The command runner is nil: this check reads interfaces and runs nothing.
27 func newNetFor(t *testing.T, cidr string, ifaces ...Iface) *Net { 27 func newNetFor(t *testing.T, cidr string, ifaces ...Iface) *Net {
28 t.Helper() 28 t.Helper()
29 n, err := New(nil, cidr) 29 n, err := New(nil, cidr, nil)
30 if err != nil { 30 if err != nil {
31 t.Fatalf("New(%q): %v", cidr, err) 31 t.Fatalf("New(%q): %v", cidr, err)
32 } 32 }
@@ -105,7 +105,7 @@ func TestCheckUplinkCollisionSkipsAddressesItCannotCompare(t *testing.T) {
105 // could not be run would turn a guard against unreachability into a cause of 105 // could not be run would turn a guard against unreachability into a cause of
106 // it. 106 // it.
107 func TestCheckUplinkCollisionAllowsWhenItCannotLook(t *testing.T) { 107 func TestCheckUplinkCollisionAllowsWhenItCannotLook(t *testing.T) {
108 n, err := New(nil, "10.77.1.0/24") 108 n, err := New(nil, "10.77.1.0/24", nil)
109 if err != nil { 109 if err != nil {
110 t.Fatal(err) 110 t.Fatal(err)
111 } 111 }
internal/agent/netsnoop/listen_linux.go
Old New
@@ -0,0 +1,218 @@
1 //go:build linux
2
3 package netsnoop
4
5 import (
6 "bytes"
7 "context"
8 "errors"
9 "fmt"
10 "net"
11 "time"
12
13 "golang.org/x/net/bpf"
14 "golang.org/x/sys/unix"
15 )
16
17 // pollInterval bounds how long the receive loop can sit in the kernel before
18 // it looks at ctx again — that is, how long after DeleteTap the goroutine and
19 // its socket survive. Short enough that teardown is not noticeable, long
20 // enough that an idle tap costs one wakeup a second.
21 const pollInterval = time.Second
22
23 // snaplen is how much of a frame the snoop keeps. The filter tells the kernel
24 // to copy this much and the receive buffer is cut to the same number, so the
25 // two cannot drift apart into a kernel that copies more than userspace reads.
26 //
27 // It is far more than a lease needs — the fixed DHCP header is 236 bytes and
28 // the options a site adds to it do not approach the rest — and everything that
29 // reaches the buffer has already passed the filter, so anything truncated here
30 // would be a truncated lease rather than traffic being skipped. Such a frame
31 // fails the DHCP decode and goes unreported, which is the trade this package
32 // makes everywhere: a missing address, never a wrong one.
33 const snaplen = 2048
34
35 // dhcpReplyFilter is the shape of a lease as the kernel can recognise it:
36 // IPv4, not a fragment, UDP, from port 67 to port 68. Everything else on a
37 // busy tap is dropped before it ever reaches userspace, so a guest cannot make
38 // the snoop work by talking loudly.
39 //
40 // It is a classifier, not a guard. A guest can emit frames that match every
41 // line of it — that is exactly what forging a lease looks like — so the
42 // question of whose lease it is gets settled later, by direction, in lease.
43 //
44 // It must never be narrower than ParseACK: a frame the filter drops is one the
45 // parser never sees, so a filter that is stricter anywhere is a real lease the
46 // snoop silently never reports. TestFilterIsNoNarrowerThanParser holds the two
47 // to the same corpus.
48 //
49 // The IPv4 header is variable length, so the ports cannot be read at a fixed
50 // offset: LoadMemShift puts 4*IHL in X and the two LoadIndirect reads are
51 // relative to it. Every absolute offset here already includes the 14 bytes of
52 // Ethernet header the socket keeps.
53 var dhcpReplyFilter = []bpf.Instruction{
54 bpf.LoadAbsolute{Off: 12, Size: 2}, // ethertype
55 bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 0x0800, SkipTrue: 10}, // not IPv4
56 bpf.LoadAbsolute{Off: 14 + 9, Size: 1}, // IP protocol
57 bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 17, SkipTrue: 8}, // not UDP
58 bpf.LoadAbsolute{Off: 14 + 6, Size: 2}, // flags and fragment offset
59 bpf.JumpIf{Cond: bpf.JumpBitsSet, Val: 0x1fff, SkipTrue: 6}, // a later fragment has no ports
60 bpf.LoadMemShift{Off: 14}, // X = 4*IHL
61 bpf.LoadIndirect{Off: 14 + 0, Size: 2}, // UDP source port
62 bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 67, SkipTrue: 3}, // not from a DHCP server
63 bpf.LoadIndirect{Off: 14 + 2, Size: 2}, // UDP destination port
64 bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 68, SkipTrue: 1}, // not to a DHCP client
65 bpf.RetConstant{Val: snaplen}, // keep, as much of it as the buffer takes
66 bpf.RetConstant{Val: 0}, // drop
67 }
68
69 // rawDHCPReplyFilter is the same program in the form the kernel takes it. The
70 // program is a constant, so assembly can only fail if the constant above is
71 // malformed — a fact worth learning when the agent starts rather than when the
72 // first named-network VM is created.
73 var rawDHCPReplyFilter = mustAssemble(dhcpReplyFilter)
74
75 func mustAssemble(prog []bpf.Instruction) []unix.SockFilter {
76 raw, err := bpf.Assemble(prog)
77 if err != nil {
78 panic(fmt.Sprintf("netsnoop: DHCP reply filter does not assemble: %v", err))
79 }
80 filter := make([]unix.SockFilter, len(raw))
81 for i, ins := range raw {
82 filter[i] = unix.SockFilter{Code: ins.Op, Jt: ins.Jt, Jf: ins.Jf, K: ins.K}
83 }
84 return filter
85 }
86
87 // lease reports the address one frame grants to want, and is the judgement the
88 // receive loop is made of — kept out of the goroutine, which needs CAP_NET_RAW
89 // and a live tap to reach, so that the judgement itself can be tested.
90 //
91 // pkttype is what makes an answer attributable, and it is worth being exact
92 // about how far that reaches. A tap carries both directions and the kernel
93 // says which is which: a reply from a DHCP server arrives at the host and is
94 // written towards the guest, so the packet socket sees it on its way out —
95 // PACKET_OUTGOING. Anything the guest itself puts on the wire enters the host
96 // through the tap and is delivered as one of the inbound types instead.
97 //
98 // So this closes self-forgery, which is the attack a guest can mount alone: it
99 // cannot mint its own address by writing ACKs at its own NIC, however correct
100 // it makes the headers look. It does not close forgery aimed at a neighbour.
101 // The taps of one named network are enslaved to the same operator bridge, so
102 // an ACK a guest unicasts at another guest's MAC is forwarded by the bridge and
103 // leaves that guest's tap outgoing, indistinguishable from the site answering
104 // by the time it arrives here. Nothing this socket can see separates those two.
105 // Only the bridge can, by refusing UDP source port 67 from a guest tap outright
106 // so that no guest is ever a DHCP server; that rule lives in netenv, which owns
107 // the bridge (blockGuestDHCP), and not in this package.
108 func lease(pkttype uint8, frame []byte, want net.HardwareAddr) (ip string, ok bool) {
109 if pkttype != unix.PACKET_OUTGOING {
110 return "", false
111 }
112 mac, granted, ok := ParseACK(frame)
113 if !ok || !bytes.Equal(mac, want) {
114 return "", false
115 }
116 return granted.String(), true
117 }
118
119 // htons converts a uint16 to network byte order. AF_PACKET takes its protocol
120 // that way in both socket(2) and bind(2) — x/sys passes SockaddrLinklayer's
121 // Protocol straight through to the raw sockaddr without swapping it.
122 func htons(v uint16) uint16 { return v<<8 | v>>8 }
123
124 // Listen opens an AF_PACKET socket bound to ifname and calls found with every
125 // address a DHCP ACK grants to mac, until ctx ends. It returns once the socket
126 // is open and bound, so a caller learns about a bad interface immediately;
127 // errors after that point end the loop silently (the tap is being torn down,
128 // and the VM's converge owns that story).
129 //
130 // found runs on the receive path — it must not block, or it stalls the snoop
131 // — and it fires on every ACK, not only the first: a guest renewing its lease
132 // re-reports the same address, and one that is moved reports a new one, so the
133 // caller decides what a repeat means.
134 //
135 // One goroutine per named NIC, started by CreateTap and stopped by DeleteTap
136 // — the listener's lifetime is the tap's, mirroring serialpump's: started at
137 // create, stopped at delete. That is the only parallel; netsnoop has no
138 // Manager and no registry of its own to mirror.
139 func Listen(ctx context.Context, ifname string, mac net.HardwareAddr, found func(ip string)) error {
140 if len(mac) == 0 {
141 // A snoop with no MAC to match would run forever and report nothing;
142 // that is a wiring bug, not a runtime condition.
143 return fmt.Errorf("snoop %s: no MAC to watch for", ifname)
144 }
145 iface, err := net.InterfaceByName(ifname)
146 if err != nil {
147 return fmt.Errorf("snoop %s: %w", ifname, err)
148 }
149 // SOCK_RAW keeps the Ethernet header, which ParseACK and the filter both
150 // expect; SOCK_DGRAM would strip it. Opening with protocol 0 opens the
151 // socket to nothing: it is attached to no protocol and no interface, and
152 // the kernel queues nothing on it until the bind below. That is the only
153 // moment at which the filter can be installed with no gap behind it —
154 // filtering after the fact leaves a window in which unfiltered frames are
155 // already sitting in the receive queue, and a guest that floods its tap at
156 // boot fills exactly that window.
157 fd, err := unix.Socket(unix.AF_PACKET, unix.SOCK_RAW|unix.SOCK_CLOEXEC, 0)
158 if err != nil {
159 return fmt.Errorf("snoop %s: %w", ifname, err)
160 }
161 if err := unix.SetsockoptSockFprog(fd, unix.SOL_SOCKET, unix.SO_ATTACH_FILTER, &unix.SockFprog{
162 Len: uint16(len(rawDHCPReplyFilter)), Filter: &rawDHCPReplyFilter[0],
163 }); err != nil {
164 unix.Close(fd)
165 return fmt.Errorf("snoop filter %s: %w", ifname, err)
166 }
167 // The bind narrows the socket to this VM's tap and, by naming a protocol at
168 // last, starts delivery.
169 if err := unix.Bind(fd, &unix.SockaddrLinklayer{
170 Protocol: htons(unix.ETH_P_ALL), Ifindex: iface.Index,
171 }); err != nil {
172 unix.Close(fd)
173 return fmt.Errorf("snoop bind %s: %w", ifname, err)
174 }
175 // A receive timeout, rather than closing the fd from the cancel path, is
176 // what stops this loop: Linux does not reliably wake a thread blocked in
177 // recvfrom when another thread closes the descriptor, so a closing
178 // canceller can leave the reader parked forever on a descriptor number
179 // the kernel has already handed to someone else. With a timeout, the
180 // reading goroutine is the only owner of fd and the only closer of it,
181 // and cancellation is observed by looking rather than by being hit.
182 if err := unix.SetsockoptTimeval(fd, unix.SOL_SOCKET, unix.SO_RCVTIMEO,
183 &unix.Timeval{Sec: int64(pollInterval / time.Second)}); err != nil {
184 unix.Close(fd)
185 return fmt.Errorf("snoop timeout %s: %w", ifname, err)
186 }
187 go func() {
188 defer unix.Close(fd)
189 buf := make([]byte, snaplen)
190 for ctx.Err() == nil {
191 n, from, err := unix.Recvfrom(fd, buf, 0)
192 if err != nil {
193 // EAGAIN is the receive timeout expiring, EINTR a signal
194 // landing on this thread: both mean "nothing yet", not
195 // "stop".
196 if errors.Is(err, unix.EAGAIN) || errors.Is(err, unix.EINTR) {
197 continue
198 }
199 return
200 }
201 // An AF_PACKET read always comes with a link-layer address, and
202 // its packet type is what lease judges the frame by; a frame that
203 // arrived without one could not be judged at all.
204 ll, ok := from.(*unix.SockaddrLinklayer)
205 if !ok {
206 continue
207 }
208 // found is called from here rather than from inside lease so that
209 // it stays on the receive path in plain sight, where the contract
210 // documented above — it must not block, or the snoop stalls behind
211 // it — is a fact about this loop and not about a helper.
212 if ip, ok := lease(ll.Pkttype, buf[:n], mac); ok {
213 found(ip)
214 }
215 }
216 }()
217 return nil
218 }
internal/agent/netsnoop/listen_linux_test.go
Old New
@@ -0,0 +1,226 @@
1 //go:build linux
2
3 package netsnoop
4
5 import (
6 "context"
7 "errors"
8 "net"
9 "testing"
10
11 "github.com/insomniacslk/dhcp/dhcpv4"
12 "golang.org/x/net/bpf"
13 "golang.org/x/sys/unix"
14 )
15
16 // The kernel wants the packet protocol in network byte order in both socket(2)
17 // and bind(2); getting the swap wrong binds to a protocol nothing matches and
18 // the snoop goes quiet instead of failing, so pin the value.
19 func TestHtons(t *testing.T) {
20 if got := htons(unix.ETH_P_ALL); got != 0x0300 {
21 t.Errorf("htons(ETH_P_ALL=%#x) = %#04x, want 0x0300", unix.ETH_P_ALL, got)
22 }
23 if got := htons(0x0800); got != 0x0008 {
24 t.Errorf("htons(ETH_P_IP) = %#04x, want 0x0008", got)
25 }
26 }
27
28 // Listen reports setup failures to its caller rather than swallowing them in
29 // the goroutine: a tap that cannot be watched is a VM that will never get an
30 // address, and CreateTap should hear about it.
31 func TestListenRejectsBadSetup(t *testing.T) {
32 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
33 if err := Listen(t.Context(), "eitri-no-such-tap", mac, func(string) {}); err == nil {
34 t.Error("Listen on a missing interface returned nil, want error")
35 }
36 if err := Listen(t.Context(), "lo", nil, func(string) {}); err == nil {
37 t.Error("Listen with no MAC returned nil, want error")
38 }
39 }
40
41 // Opening the socket needs CAP_NET_RAW, so this only verifies that setup
42 // under an already-cancelled context does not hang or panic. The receive
43 // loop itself is proven on real hardware, not here.
44 func TestListenOnAlreadyCancelledContext(t *testing.T) {
45 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
46 ctx, cancel := context.WithCancel(t.Context())
47 cancel()
48 err := Listen(ctx, "lo", mac, func(string) {})
49 if err != nil && !errors.Is(err, unix.EPERM) && !errors.Is(err, unix.EACCES) {
50 t.Fatalf("Listen on lo: %v", err)
51 }
52 }
53
54 // A real lease reaches the guest through the tap, so the packet socket sees it
55 // outgoing, and lease reports the address it grants.
56 func TestLeaseReportsOutgoingACK(t *testing.T) {
57 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
58 ack := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42"))
59
60 ip, ok := lease(unix.PACKET_OUTGOING, ack, mac)
61 if !ok || ip != "192.168.0.42" {
62 t.Fatalf("lease = %q %v, want 192.168.0.42 true", ip, ok)
63 }
64 }
65
66 // The forgery the direction check exists to close, and the reason it cannot be
67 // deleted: this is the byte-for-byte frame the test above accepts, put on the
68 // wire by the guest instead of by the site. Every check inside the frame
69 // passes — right ports, boot reply, ACK, the guest's own MAC — and only the
70 // packet type separates the two. A guest that got past this would have eitri
71 // publishing an address the guest invented for itself.
72 //
73 // What this does not prove, because it is not true, is that a guest cannot
74 // forge a lease for a NEIGHBOUR. Such a frame crosses the operator bridge and
75 // arrives outgoing on the neighbour's tap, which is the accepted case above.
76 // Only the bridge can refuse it; see the comment on lease.
77 func TestLeaseIgnoresGuestOriginatedACK(t *testing.T) {
78 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
79 ack := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "10.66.66.66"))
80
81 for name, pkttype := range map[string]uint8{
82 "unicast to host": unix.PACKET_HOST,
83 "broadcast": unix.PACKET_BROADCAST,
84 "at a neighbour": unix.PACKET_OTHERHOST,
85 "multicast": unix.PACKET_MULTICAST,
86 } {
87 if ip, ok := lease(pkttype, ack, mac); ok {
88 t.Errorf("%s: lease reported %s from a guest-originated frame", name, ip)
89 }
90 }
91 }
92
93 // The tap carries the whole named network, so an outgoing ACK is routinely
94 // somebody else's; only this VM's MAC is this VM's address.
95 func TestLeaseIgnoresOtherFrames(t *testing.T) {
96 mine, _ := net.ParseMAC("52:54:00:aa:bb:cc")
97 neighbour, _ := net.ParseMAC("52:54:00:11:22:33")
98
99 for name, b := range map[string][]byte{
100 "a lease for another VM": frame(t, reply(t, neighbour, dhcpv4.MessageTypeAck, "192.168.0.43")),
101 "not a lease": frame(t, []byte{1, 2, 3, 4}),
102 "empty": {},
103 } {
104 if ip, ok := lease(unix.PACKET_OUTGOING, b, mine); ok {
105 t.Errorf("%s: lease reported %s", name, ip)
106 }
107 }
108 }
109
110 // The kernel runs the filter before the parser sees anything, so a filter that
111 // is stricter than the parser anywhere is a real lease silently never
112 // reported — the worst failure this package has, because it looks exactly like
113 // a quiet network. Hold both halves to the one corpus of real leases.
114 //
115 // The filter cannot be exercised without CAP_NET_RAW, but the program is the
116 // same either way, so it runs here in the userspace VM.
117 func TestFilterIsNoNarrowerThanParser(t *testing.T) {
118 vm, err := bpf.NewVM(dhcpReplyFilter)
119 if err != nil {
120 t.Fatal(err)
121 }
122 for name, want := range grantedFrames(t) {
123 if _, _, ok := ParseACK(want.frame); !ok {
124 t.Errorf("%s: ParseACK rejects a frame the corpus calls a lease", name)
125 continue
126 }
127 n, err := vm.Run(want.frame)
128 if err != nil {
129 t.Fatalf("%s: %v", name, err)
130 }
131 if n == 0 {
132 t.Errorf("%s: filter drops a frame ParseACK accepts", name)
133 }
134 }
135 }
136
137 // What the filter is allowed to drop. It may be wider than the parser — it
138 // says nothing about opcodes — but everything it drops here the parser rejects
139 // too, so nothing is lost to the kernel that userspace would have wanted.
140 func TestFilterDropsWhatIsNotALease(t *testing.T) {
141 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
142 ack := reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42")
143
144 notIPv4 := frame(t, ack)
145 notIPv4[12], notIPv4[13] = 0x08, 0x06 // ARP
146
147 tcp := frame(t, ack)
148 tcp[14+9] = 6
149
150 vm, err := bpf.NewVM(dhcpReplyFilter)
151 if err != nil {
152 t.Fatal(err)
153 }
154 for name, b := range map[string][]byte{
155 "not IPv4": notIPv4,
156 "TCP": tcp,
157 "guest to server": framePorts(t, ack, 68, 67),
158 "unrelated UDP": framePorts(t, ack, 12345, 54321),
159 "runt": {0, 1, 2},
160 } {
161 n, err := vm.Run(b)
162 if err != nil {
163 t.Fatalf("%s: %v", name, err)
164 }
165 if n != 0 {
166 t.Errorf("%s: filter returned %d, want it dropped", name, n)
167 }
168 if _, _, ok := ParseACK(b); ok {
169 t.Errorf("%s: filter drops it but ParseACK accepts it", name)
170 }
171 }
172 }
173
174 // The one place the kernel is deliberately stricter than the parser, and the
175 // reason the fragment line is in the filter at all.
176 //
177 // ParseACK reads the ports at 4*IHL without asking whether this is the first
178 // fragment. In a later fragment those bytes are not a UDP header, they are
179 // payload continuation — so a guest that fragments a packet such that the
180 // continuation happens to read as 67 to 68 followed by valid DHCP bytes gets
181 // an address believed by the parser alone. This frame is exactly that, and the
182 // assertion below records that the parser does fall for it: the filter is what
183 // stops it reaching userspace.
184 //
185 // A real ACK is a few hundred bytes and is never fragmented, so dropping later
186 // fragments costs no lease. Teaching ParseACK the same guard would be the
187 // belt to this braces, and is worth doing if the filter is ever bypassed.
188 func TestFilterDropsFragmentsThatFoolTheParser(t *testing.T) {
189 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
190 fragment := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "10.66.66.66"))
191 fragment[14+6], fragment[14+7] = 0x00, 0x25 // a later fragment, offset 37
192
193 vm, err := bpf.NewVM(dhcpReplyFilter)
194 if err != nil {
195 t.Fatal(err)
196 }
197 n, err := vm.Run(fragment)
198 if err != nil {
199 t.Fatal(err)
200 }
201 if n != 0 {
202 t.Errorf("filter returned %d for a later fragment, want it dropped", n)
203 }
204 if _, _, ok := ParseACK(fragment); !ok {
205 t.Log("ParseACK now rejects later fragments too; the filter is no longer the only guard")
206 }
207 }
208
209 // The kernel takes the program as SockFilter, not as x/net instructions, and
210 // a truncated or reordered conversion would attach a filter that means
211 // something else entirely.
212 func TestRawFilterMatchesProgram(t *testing.T) {
213 raw, err := bpf.Assemble(dhcpReplyFilter)
214 if err != nil {
215 t.Fatal(err)
216 }
217 if len(rawDHCPReplyFilter) != len(raw) {
218 t.Fatalf("assembled filter has %d instructions, want %d", len(rawDHCPReplyFilter), len(raw))
219 }
220 for i, ins := range raw {
221 got := rawDHCPReplyFilter[i]
222 if got.Code != ins.Op || got.Jt != ins.Jt || got.Jf != ins.Jf || got.K != ins.K {
223 t.Errorf("instruction %d = %+v, want %+v", i, got, ins)
224 }
225 }
226 }
internal/agent/netsnoop/netsnoop.go
Old New
@@ -0,0 +1,72 @@
1 // Package netsnoop discovers the address a guest's named-network NIC was
2 // granted, by watching that NIC's own tap for its DHCP exchange. On a named
3 // network the site's own server assigns the address, so the agent is not the
4 // authority it is on the eitri bridge (internal/agent/dhcp) and cannot simply
5 // report what it handed out. With no in-guest presence — a boundary this
6 // project holds, the guest owns the guest — the tap is the only place the
7 // answer appears: it carries every frame that NIC sends or receives, so
8 // initial leases and unicast renewals are both visible without touching the
9 // guest or the site's server.
10 //
11 // The snoop reads plain untagged Ethernet frames. A guest that does its own
12 // 802.1Q tagging DHCPs inside the tag and is not discovered here; it keeps
13 // whatever address it negotiates, and eitri reports none — the guest owns the
14 // guest, and a wrong address would be worse than no address.
15 package netsnoop
16
17 import (
18 "net"
19 "net/netip"
20
21 "github.com/insomniacslk/dhcp/dhcpv4"
22 )
23
24 // ParseACK inspects one Ethernet frame and, when it is a DHCP ACK, returns
25 // the client MAC it is addressed to and the address it grants. ok=false for
26 // every other frame — this is called on all tap traffic, including whatever a
27 // hostile guest chooses to emit, so it must be cheap and unshockable: it
28 // indexes only after a length check and treats a parse failure as "not an
29 // ACK", never as an error worth reporting.
30 func ParseACK(frame []byte) (mac net.HardwareAddr, ip netip.Addr, ok bool) {
31 const ethHdr = 14
32 if len(frame) < ethHdr+20+8 {
33 return nil, netip.Addr{}, false
34 }
35 if frame[12] != 0x08 || frame[13] != 0x00 { // not IPv4
36 return nil, netip.Addr{}, false
37 }
38 ipb := frame[ethHdr:]
39 if ipb[0]>>4 != 4 {
40 return nil, netip.Addr{}, false
41 }
42 ihl := int(ipb[0]&0x0f) * 4
43 if ihl < 20 || len(ipb) < ihl+8 || ipb[9] != 17 { // not UDP
44 return nil, netip.Addr{}, false
45 }
46 udp := ipb[ihl:]
47 srcPort := int(udp[0])<<8 | int(udp[1])
48 dstPort := int(udp[2])<<8 | int(udp[3])
49 if srcPort != 67 || dstPort != 68 { // not server->client DHCP
50 return nil, netip.Addr{}, false
51 }
52 // The opcode check sorts frames, it does not defend against them. A guest
53 // can write a boot reply on these ports as easily as a server can, so
54 // nothing read out of the frame says who sent it. What makes an address
55 // believable is the direction the frame was travelling on the tap, which
56 // the receive path settles before it calls here — see lease in
57 // listen_linux.go, which is also honest about what direction does not
58 // cover.
59 msg, err := dhcpv4.FromBytes(udp[8:])
60 if err != nil || msg.OpCode != dhcpv4.OpcodeBootReply ||
61 msg.MessageType() != dhcpv4.MessageTypeAck {
62 return nil, netip.Addr{}, false
63 }
64 // A wire yiaddr is four bytes, so To4 is non-nil in practice; the
65 // AddrFromSlice check is the belt to that braces. The live arm is
66 // IsUnspecified: a reply that grants 0.0.0.0 is not an address.
67 granted, okAddr := netip.AddrFromSlice(msg.YourIPAddr.To4())
68 if !okAddr || granted.IsUnspecified() {
69 return nil, netip.Addr{}, false
70 }
71 return msg.ClientHWAddr, granted, true
72 }
internal/agent/netsnoop/netsnoop_test.go
Old New
@@ -0,0 +1,214 @@
1 package netsnoop
2
3 import (
4 "bytes"
5 "net"
6 "testing"
7
8 "github.com/insomniacslk/dhcp/dhcpv4"
9 )
10
11 // frame wraps a DHCP payload in Ethernet/IPv4/UDP headers the way it crosses
12 // a guest tap: server 67 -> client 68.
13 func frame(t *testing.T, payload []byte) []byte {
14 t.Helper()
15 return framePorts(t, payload, 67, 68)
16 }
17
18 // framePorts is frame with the UDP ports under the test's control, so the
19 // port guard can be exercised without hand-rolling a second builder.
20 func framePorts(t *testing.T, payload []byte, src, dst int) []byte {
21 t.Helper()
22 udp := make([]byte, 8+len(payload))
23 udp[0], udp[1] = byte(src>>8), byte(src)
24 udp[2], udp[3] = byte(dst>>8), byte(dst)
25 udp[4], udp[5] = byte(len(udp)>>8), byte(len(udp))
26 copy(udp[8:], payload)
27 ip := make([]byte, 20+len(udp))
28 ip[0] = 0x45 // v4, ihl 5
29 ip[2], ip[3] = byte(len(ip)>>8), byte(len(ip))
30 ip[8] = 64
31 ip[9] = 17 // UDP
32 copy(ip[20:], udp)
33 eth := make([]byte, 14+len(ip))
34 eth[12], eth[13] = 0x08, 0x00 // IPv4
35 copy(eth[14:], ip)
36 return eth
37 }
38
39 // newReply builds a server->client DHCP message. dhcpv4.New defaults the
40 // opcode to BootRequest (the client direction), so replies set it explicitly.
41 func newReply(t *testing.T, mods ...dhcpv4.Modifier) *dhcpv4.DHCPv4 {
42 t.Helper()
43 msg, err := dhcpv4.New(mods...)
44 if err != nil {
45 t.Fatal(err)
46 }
47 msg.OpCode = dhcpv4.OpcodeBootReply
48 return msg
49 }
50
51 // reply builds a server->client DHCP message of the given type for mac,
52 // granting yiaddr ("" leaves yiaddr unset).
53 func reply(t *testing.T, mac net.HardwareAddr, typ dhcpv4.MessageType, yiaddr string) []byte {
54 t.Helper()
55 mods := []dhcpv4.Modifier{
56 dhcpv4.WithMessageType(typ),
57 dhcpv4.WithHwAddr(mac),
58 }
59 if yiaddr != "" {
60 mods = append(mods, dhcpv4.WithYourIP(net.ParseIP(yiaddr)))
61 }
62 return newReply(t, mods...).ToBytes()
63 }
64
65 // withIPOptions splices four bytes of IPv4 option (NOP, NOP, NOP, EOL) into a
66 // frame's IP header and fixes up IHL and total length, so that the UDP header
67 // starts at 4*IHL rather than at a fixed 20. It catches anyone reading the
68 // ports at a constant offset — in the parser or in the kernel filter.
69 func withIPOptions(t *testing.T, plain []byte) []byte {
70 t.Helper()
71 out := make([]byte, 0, len(plain)+4)
72 out = append(out, plain[:14+20]...)
73 out = append(out, 1, 1, 1, 0)
74 out = append(out, plain[14+20:]...)
75 out[14] = 0x46 // v4, ihl 6
76 total := len(out) - 14
77 out[16], out[17] = byte(total>>8), byte(total)
78 return out
79 }
80
81 // granted is one frame that carries a real lease, with what must be read out
82 // of it.
83 type granted struct {
84 frame []byte
85 mac net.HardwareAddr
86 ip string
87 }
88
89 // grantedFrames is the corpus of real leases: every shape a site's server
90 // grants one in that has to survive the receive path intact. Both halves of
91 // that path answer to it — ParseACK must report each of these, and the kernel
92 // filter must pass each of them — because the filter runs first, so a frame it
93 // drops is one the parser never sees.
94 func grantedFrames(t *testing.T) map[string]granted {
95 t.Helper()
96 first, _ := net.ParseMAC("52:54:00:aa:bb:cc")
97 renewing, _ := net.ParseMAC("52:54:00:de:ad:01")
98 plain := frame(t, reply(t, first, dhcpv4.MessageTypeAck, "192.168.0.42"))
99
100 // A renewal is unicast to the guest and carries the options a real server
101 // sends a lease with, after the fixed header. Renewals are the only chance
102 // to notice a site server moving a guest, so they have to be read exactly
103 // like the broadcast first lease.
104 renewal := newReply(t,
105 dhcpv4.WithMessageType(dhcpv4.MessageTypeAck),
106 dhcpv4.WithHwAddr(renewing),
107 dhcpv4.WithYourIP(net.ParseIP("10.4.5.6")),
108 dhcpv4.WithNetmask(net.CIDRMask(24, 32)),
109 dhcpv4.WithRouter(net.ParseIP("10.4.5.1")),
110 dhcpv4.WithDNS(net.ParseIP("10.4.5.1")),
111 )
112
113 return map[string]granted{
114 "first lease": {plain, first, "192.168.0.42"},
115 "renewal with options": {frame(t, renewal.ToBytes()), renewing, "10.4.5.6"},
116 "IP header with options": {withIPOptions(t, plain), first, "192.168.0.42"},
117 }
118 }
119
120 func TestParseACKAcceptsEveryGrantedLease(t *testing.T) {
121 for name, want := range grantedFrames(t) {
122 gotMAC, gotIP, ok := ParseACK(want.frame)
123 if !ok || !bytes.Equal(gotMAC, want.mac) || gotIP.String() != want.ip {
124 t.Errorf("%s: ParseACK = %v %v %v, want %v %s true",
125 name, gotMAC, gotIP, ok, want.mac, want.ip)
126 }
127 }
128 }
129
130 func TestParseACKRejects(t *testing.T) {
131 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
132 ack := reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42")
133
134 // A request the guest sends: client -> server, and a BootRequest (New's
135 // default opcode).
136 req, err := dhcpv4.New(
137 dhcpv4.WithMessageType(dhcpv4.MessageTypeRequest),
138 dhcpv4.WithHwAddr(mac),
139 )
140 if err != nil {
141 t.Fatal(err)
142 }
143
144 truncatedIP := frame(t, ack)[:14+12]
145 badIHL := frame(t, ack)
146 badIHL[14] = 0x44 // v4, ihl 4 — shorter than the fixed header allows
147
148 notUDP := frame(t, ack)
149 notUDP[14+9] = 6 // TCP
150
151 // A guest that forges an ACK on the right ports still fails the opcode
152 // check: only a boot reply grants an address.
153 forged := newReply(t,
154 dhcpv4.WithMessageType(dhcpv4.MessageTypeAck),
155 dhcpv4.WithHwAddr(mac),
156 dhcpv4.WithYourIP(net.ParseIP("192.168.0.42")),
157 )
158 forged.OpCode = dhcpv4.OpcodeBootRequest
159
160 for name, b := range map[string][]byte{
161 "offer-not-ack": frame(t, reply(t, mac, dhcpv4.MessageTypeOffer, "192.168.0.42")),
162 "nak-not-ack": frame(t, reply(t, mac, dhcpv4.MessageTypeNak, "")),
163 "guest-request": framePorts(t, req.ToBytes(), 68, 67),
164 "opcode-request": frame(t, forged.ToBytes()),
165 "ack-wrong-dst": framePorts(t, ack, 67, 12345),
166 "ack-wrong-src": framePorts(t, ack, 12345, 68),
167 "zero-yiaddr": frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "")),
168 "truncated-ip": truncatedIP,
169 "bad-ihl": badIHL,
170 "not-udp": notUDP,
171 "garbage-dhcp": frame(t, []byte{1, 2, 3, 4}),
172 "short": {0, 1, 2},
173 "not-ipv4": append(make([]byte, 14), 0xde),
174 "arp-ethertype": append([]byte{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x08, 0x06}, make([]byte, 60)...),
175 "ipv6-in-v4-eth": func() []byte { f := frame(t, ack); f[14] = 0x65; return f }(),
176 "empty": {},
177 "nil": nil,
178 } {
179 if _, _, ok := ParseACK(b); ok {
180 t.Errorf("%s: ParseACK accepted, want reject", name)
181 }
182 }
183 }
184
185 // A DHCPv4 yiaddr is four bytes on the wire, so anything ParseACK decodes
186 // has a non-nil To4 and the To4==nil arm of the address guard is unreachable
187 // from a real frame. This pins that invariant: if the decoder ever started
188 // handing back a wider IP, the guard would become live and this test would be
189 // the record of why it exists.
190 func TestDecodedYourIPIsAlwaysV4(t *testing.T) {
191 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
192 msg, err := dhcpv4.FromBytes(reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42"))
193 if err != nil {
194 t.Fatal(err)
195 }
196 if msg.YourIPAddr.To4() == nil {
197 t.Fatalf("decoded yiaddr %v is not IPv4-shaped", msg.YourIPAddr)
198 }
199 }
200
201 // The snoop runs on every frame a busy tap carries, so it must survive
202 // arbitrary bytes without panicking — the only failure mode allowed is ok=false.
203 func TestParseACKFuzzResistant(t *testing.T) {
204 mac, _ := net.ParseMAC("52:54:00:aa:bb:cc")
205 full := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42"))
206 for i := range full {
207 ParseACK(full[:i]) // every truncation of a real frame
208 }
209 for i := range full {
210 mutated := append([]byte(nil), full...)
211 mutated[i] ^= 0xff
212 ParseACK(mutated)
213 }
214 }
internal/agent/reconcile/reconcile.go
Old New
@@ -109,6 +109,18 @@ type Provisioner interface {
109 // and Boot must not block inside the create slot waiting for that. Empty 109 // and Boot must not block inside the create slot waiting for that. Empty
110 // means "no answer", never "no address" — see noteAddress. 110 // means "no answer", never "no address" — see noteAddress.
111 Address(vmID string) string 111 Address(vmID string) string
112
113 // NetworkAddress returns the address the VM's SECOND NIC — the one on the
114 // named host network its spec asked for — was granted, or "" when it has no
115 // such NIC or the backend has not learned one yet. A backend that serves no
116 // named networks answers "" always, which is every Mac and every Linux host
117 // with no --host-network flag.
118 //
119 // Separate from Address rather than replacing it because the two are
120 // different facts with different timings: Address is allocated by the host
121 // and known at boot, this one is granted by someone else's DHCP server and
122 // discovered afterwards. Empty means "not yet known", never "unreachable".
123 NetworkAddress(vmID string) string
112 } 124 }
113 125
114 // Engine is the reconcile loop. All fields must be set before calling Step. 126 // Engine is the reconcile loop. All fields must be set before calling Step.
@@ -349,7 +361,7 @@ func (e *Engine) fenceReport(currentEpoch uint64) *pb.ActualStateReport {
349 if rec.LastError != "" { 361 if rec.LastError != "" {
350 phase = "failed" 362 phase = "failed"
351 } 363 }
352 res.report(rec.Spec.VMID, rec.IP, power, phase, rec.LastError) 364 res.report(rec.Spec.VMID, recAddrs(rec), power, phase, rec.LastError)
353 res.merge(rep) 365 res.merge(rep)
354 } 366 }
355 return rep 367 return rep
@@ -589,18 +601,27 @@ func (e *Engine) releaseCompute(vmID string) {
589 delete(e.committed, vmID) 601 delete(e.committed, vmID)
590 } 602 }
591 603
592 // noteAddress folds the backend's current answer for this VM's address into 604 // noteAddress folds the backend's current answers for this VM's addresses into
593 // rec, reporting whether it changed. An empty answer never clears a known 605 // rec, reporting whether either changed. An empty answer never clears a known
594 // address: "I don't know yet" is not "it has none", and blanking rec.IP would 606 // address: "I don't know yet" is not "it has none", and blanking rec.IP would
595 // break the guest SSH tunnel (syncclient refuses an empty address) for a VM 607 // break the guest SSH tunnel (syncclient refuses an empty address) for a VM
596 // that is perfectly reachable. 608 // that is perfectly reachable.
609 //
610 // Both NICs are asked on every poll, and the same rule covers both. The named
611 // NIC's address is the one that genuinely arrives late — the host discovers it
612 // by watching the guest's DHCP exchange — while rec.IP is known before the
613 // guest boots.
597 func (e *Engine) noteAddress(rec *state.Record) bool { 614 func (e *Engine) noteAddress(rec *state.Record) bool {
598 ip := e.Prov.Address(rec.Spec.VMID) 615 changed := false
599 if ip == "" || ip == rec.IP { 616 if ip := e.Prov.Address(rec.Spec.VMID); ip != "" && ip != rec.IP {
600 return false 617 rec.IP = ip
618 changed = true
601 } 619 }
602 rec.IP = ip 620 if netIP := e.Prov.NetworkAddress(rec.Spec.VMID); netIP != "" && netIP != rec.NetworkIP {
603 return true 621 rec.NetworkIP = netIP
622 changed = true
623 }
624 return changed
604 } 625 }
605 626
606 // SeedLedger rebuilds the compute ledger from persisted records at startup so 627 // SeedLedger rebuilds the compute ledger from persisted records at startup so
@@ -660,7 +681,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
660 // as cheap insurance for a caller that later threads a cancellable context 681 // as cheap insurance for a caller that later threads a cancellable context
661 // into a pass. Report and let the next tick do the work. 682 // into a pass. Report and let the next tick do the work.
662 if err := ctx.Err(); err != nil { 683 if err := ctx.Err(); err != nil {
663 res.report(d.VmId, rec.IP, "stopped", "creating", "reconcile aborted: "+err.Error()) 684 res.report(d.VmId, recAddrs(rec), "stopped", "creating", "reconcile aborted: "+err.Error())
664 return 685 return
665 } 686 }
666 687
@@ -677,7 +698,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
677 698
678 // Terminal check: if we've hit MaxCreateAttempts, stop retrying. 699 // Terminal check: if we've hit MaxCreateAttempts, stop retrying.
679 if ok && rec.CreateAttempts >= e.MaxCreateAttempts { 700 if ok && rec.CreateAttempts >= e.MaxCreateAttempts {
680 res.report(d.VmId, rec.IP, "stopped", "failed", rec.LastError) 701 res.report(d.VmId, recAddrs(rec), "stopped", "failed", rec.LastError)
681 return 702 return
682 } 703 }
683 704
@@ -710,7 +731,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
710 // Reported, not yet certified. Publish the public key and wait: 731 // Reported, not yet certified. Publish the public key and wait:
711 // booting now would hand the guest a host key every client refuses, 732 // booting now would hand the guest a host key every client refuses,
712 // and converge never rebuilds a seed, so it would stay that way. 733 // and converge never rebuilds a seed, so it would stay that way.
713 res.report(d.VmId, rec.IP, "stopped", "creating", "awaiting host certificate") 734 res.report(d.VmId, recAddrs(rec), "stopped", "creating", "awaiting host certificate")
714 return 735 return
715 } 736 }
716 } 737 }
@@ -720,7 +741,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
720 // frees the next tick retries and boots. 741 // frees the next tick retries and boots.
721 rec.Spec = spec 742 rec.Spec = spec
722 if quotaMsg := e.admit(d.VmId, spec); quotaMsg != "" { 743 if quotaMsg := e.admit(d.VmId, spec); quotaMsg != "" {
723 res.report(d.VmId, rec.IP, "stopped", "failed", quotaMsg) 744 res.report(d.VmId, recAddrs(rec), "stopped", "failed", quotaMsg)
724 return 745 return
725 } 746 }
726 rec.CreateAttempts++ 747 rec.CreateAttempts++
@@ -754,7 +775,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
754 } 775 }
755 var interim vmResult 776 var interim vmResult
756 interim.hostPubKey = res.hostPubKey 777 interim.hostPubKey = res.hostPubKey
757 interim.report(d.VmId, rec.IP, "stopped", "creating", "") 778 interim.report(d.VmId, recAddrs(rec), "stopped", "creating", "")
758 interim.vm.StatusDetail = detail 779 interim.vm.StatusDetail = detail
759 pub(interim) 780 pub(interim)
760 } 781 }
@@ -819,6 +840,12 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
819 // generated above and held on this host. Nothing else can supply one. 840 // generated above and held on this host. Nothing else can supply one.
820 SSHHostKeyPEM: hostKey.PrivatePEM, 841 SSHHostKeyPEM: hostKey.PrivatePEM,
821 SSHHostCert: d.SshHostCert, 842 SSHHostCert: d.SshHostCert,
843 // The guest's NICs, so its netplan matches each by the address the
844 // hypervisor gave it. The second one exists only for a VM that asked for
845 // a named network; both are derived from the VM id, so the seed and the
846 // host agree without either asking the hypervisor what it ended up with.
847 MAC: state.MAC(d.VmId),
848 NetworkMAC: netMACIfNetworked(rec.Spec),
822 }); err != nil { 849 }); err != nil {
823 e.failCreate(ctx, rec, err, res) 850 e.failCreate(ctx, rec, err, res)
824 return 851 return
@@ -848,7 +875,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record,
848 if d.PowerState != "running" { 875 if d.PowerState != "running" {
849 power = "stopped" 876 power = "stopped"
850 } 877 }
851 res.report(d.VmId, rec.IP, power, "ready", "") 878 res.report(d.VmId, recAddrs(rec), power, "ready", "")
852 } 879 }
853 880
854 // permanent reports whether err (anywhere in its chain) carries the 881 // permanent reports whether err (anywhere in its chain) carries the
@@ -884,7 +911,7 @@ func (e *Engine) failCreate(ctx context.Context, rec state.Record, err error, re
884 if rec.CreateAttempts >= e.MaxCreateAttempts { 911 if rec.CreateAttempts >= e.MaxCreateAttempts {
885 phase = "failed" 912 phase = "failed"
886 } 913 }
887 res.report(rec.Spec.VMID, rec.IP, "stopped", phase, rec.LastError) 914 res.report(rec.Spec.VMID, recAddrs(rec), "stopped", phase, rec.LastError)
888 } 915 }
889 916
890 // failConverge is the shared epilogue for the converge restart/boot paths: 917 // failConverge is the shared epilogue for the converge restart/boot paths:
@@ -902,7 +929,7 @@ func (e *Engine) failConverge(rec state.Record, err error, res *vmResult) {
902 rec.LastError += ": " + why 929 rec.LastError += ": " + why
903 } 930 }
904 _ = e.St.SaveVM(rec) 931 _ = e.St.SaveVM(rec)
905 res.report(rec.Spec.VMID, rec.IP, "stopped", "failed", rec.LastError) 932 res.report(rec.Spec.VMID, recAddrs(rec), "stopped", "failed", rec.LastError)
906 } 933 }
907 934
908 // converge drives an existing VM toward its desired power state, 935 // converge drives an existing VM toward its desired power state,
@@ -943,13 +970,13 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record
943 rec.StopRequested = false 970 rec.StopRequested = false
944 rec.LastError = "" // Fix 3: clear stale error on successful restart 971 rec.LastError = "" // Fix 3: clear stale error on successful restart
945 _ = e.St.SaveVM(rec) 972 _ = e.St.SaveVM(rec)
946 res.report(d.VmId, rec.IP, "running", "ready", "") 973 res.report(d.VmId, recAddrs(rec), "running", "ready", "")
947 } else { 974 } else {
948 // Lost + desired stopped: update boot ID, mark stop recorded. 975 // Lost + desired stopped: update boot ID, mark stop recorded.
949 rec.BootID = bootID 976 rec.BootID = bootID
950 rec.StopRequested = true 977 rec.StopRequested = true
951 _ = e.St.SaveVM(rec) 978 _ = e.St.SaveVM(rec)
952 res.report(d.VmId, rec.IP, "stopped", "ready", "") 979 res.report(d.VmId, recAddrs(rec), "stopped", "ready", "")
953 } 980 }
954 return 981 return
955 } 982 }
@@ -965,7 +992,7 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record
965 rec.StopRequested = false 992 rec.StopRequested = false
966 rec.LastError = "" // Fix 3: clear stale error on successful boot 993 rec.LastError = "" // Fix 3: clear stale error on successful boot
967 _ = e.St.SaveVM(rec) 994 _ = e.St.SaveVM(rec)
968 res.report(d.VmId, rec.IP, "running", "ready", "") 995 res.report(d.VmId, recAddrs(rec), "running", "ready", "")
969 } else if d.PowerState == "stopped" && running { 996 } else if d.PowerState == "stopped" && running {
970 // Stop the VM. Record stop BEFORE side effects so a crash between 997 // Stop the VM. Record stop BEFORE side effects so a crash between
971 // SaveVM and Shutdown is recoverable (the persisted StopRequested 998 // SaveVM and Shutdown is recoverable (the persisted StopRequested
@@ -977,11 +1004,11 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record
977 if err := e.St.SaveVM(rec); err != nil { 1004 if err := e.St.SaveVM(rec); err != nil {
978 // Cannot durably record the stop intent; skip Shutdown this cycle. 1005 // Cannot durably record the stop intent; skip Shutdown this cycle.
979 // The next reconcile will retry once the store recovers. 1006 // The next reconcile will retry once the store recovers.
980 res.report(d.VmId, rec.IP, "running", "failed", err.Error()) 1007 res.report(d.VmId, recAddrs(rec), "running", "failed", err.Error())
981 return 1008 return
982 } 1009 }
983 _ = e.Prov.Shutdown(ctx, d.VmId) 1010 _ = e.Prov.Shutdown(ctx, d.VmId)
984 res.report(d.VmId, rec.IP, "stopped", "ready", "") 1011 res.report(d.VmId, recAddrs(rec), "stopped", "ready", "")
985 } else { 1012 } else {
986 // Already at desired state. 1013 // Already at desired state.
987 power := "stopped" 1014 power := "stopped"
@@ -990,7 +1017,7 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record
990 } 1017 }
991 // Preserve last error in the report field but phase stays ready 1018 // Preserve last error in the report field but phase stays ready
992 // (the VM is converged; the error is informational history). 1019 // (the VM is converged; the error is informational history).
993 res.report(d.VmId, rec.IP, power, "ready", rec.LastError) 1020 res.report(d.VmId, recAddrs(rec), power, "ready", rec.LastError)
994 } 1021 }
995 } 1022 }
996 1023
@@ -1024,10 +1051,22 @@ type vmResult struct {
1024 hostPubKey string 1051 hostPubKey string
1025 } 1052 }
1026 1053
1054 // addrs is where a VM is: its address on its host's NAT underlay, which every
1055 // guest has from boot, and — for a guest with a second NIC on a named host
1056 // network — the address the site's own DHCP server granted that one. They
1057 // travel together because every report of one is a report of the other, and a
1058 // row that carried only the first would blank a networked guest's LAN address
1059 // on the fleet's side every time the host said anything about it.
1060 type addrs struct{ ip, networkIP string }
1061
1062 // recAddrs reads a record's pair. The record is the only place both are known
1063 // at once: the backend answers them one question at a time.
1064 func recAddrs(rec state.Record) addrs { return addrs{ip: rec.IP, networkIP: rec.NetworkIP} }
1065
1027 // report records this VM's actual row. A VM contributes at most one row, so a 1066 // report records this VM's actual row. A VM contributes at most one row, so a
1028 // later call in the same reconcile replaces an earlier one. 1067 // later call in the same reconcile replaces an earlier one.
1029 func (r *vmResult) report(vmID, ip, power, phase, lastError string) { 1068 func (r *vmResult) report(vmID string, at addrs, power, phase, lastError string) {
1030 r.vm = newActualVM(vmID, ip, power, phase, lastError) 1069 r.vm = newActualVM(vmID, at, power, phase, lastError)
1031 } 1070 }
1032 1071
1033 // clone returns a deep copy, so the caller's report owns its rows outright. 1072 // clone returns a deep copy, so the caller's report owns its rows outright.
@@ -1084,10 +1123,11 @@ func byteScale(n int64) (float64, string) {
1084 // newActualVM builds one ActualVM row from what a reconcile pass observed. 1123 // newActualVM builds one ActualVM row from what a reconcile pass observed.
1085 // The row's remaining field, ssh_host_pubkey, is stamped by merge — see 1124 // The row's remaining field, ssh_host_pubkey, is stamped by merge — see
1086 // vmResult.hostPubKey. Unset values are the proto zero-value "". 1125 // vmResult.hostPubKey. Unset values are the proto zero-value "".
1087 func newActualVM(vmID, ip, power, phase, lastError string) *pb.ActualVM { 1126 func newActualVM(vmID string, at addrs, power, phase, lastError string) *pb.ActualVM {
1088 return &pb.ActualVM{ 1127 return &pb.ActualVM{
1089 VmId: vmID, 1128 VmId: vmID,
1090 Ip: ip, 1129 Ip: at.ip,
1130 NetworkIp: at.networkIP,
1091 Power: power, 1131 Power: power,
1092 Phase: phase, 1132 Phase: phase,
1093 LastError: lastError, 1133 LastError: lastError,
@@ -1105,6 +1145,16 @@ func joinCALines(lines []string) string {
1105 return b.String() 1145 return b.String()
1106 } 1146 }
1107 1147
1148 // netMACIfNetworked returns the second NIC's MAC for a VM that asked for a
1149 // named network, and "" for one that did not — which is what makes the seed
1150 // write the single-NIC network-config every guest running today already has.
1151 func netMACIfNetworked(spec state.VMSpec) string {
1152 if spec.Network == "" {
1153 return ""
1154 }
1155 return state.NetMAC(spec.VMID)
1156 }
1157
1108 // specFromDesired maps a pb.VMDesired to state.VMSpec. 1158 // specFromDesired maps a pb.VMDesired to state.VMSpec.
1109 func specFromDesired(d *pb.VMDesired) state.VMSpec { 1159 func specFromDesired(d *pb.VMDesired) state.VMSpec {
1110 return state.VMSpec{ 1160 return state.VMSpec{
@@ -1114,6 +1164,7 @@ func specFromDesired(d *pb.VMDesired) state.VMSpec {
1114 ImageSHA256: d.ImageSha256, 1164 ImageSHA256: d.ImageSha256,
1115 CloudInit: d.CloudInit, 1165 CloudInit: d.CloudInit,
1116 SSHAuthorizedKey: d.SshAuthorizedKey, 1166 SSHAuthorizedKey: d.SshAuthorizedKey,
1167 Network: d.Network,
1117 VCPUs: d.Vcpus, 1168 VCPUs: d.Vcpus,
1118 MemMB: d.MemMb, 1169 MemMB: d.MemMb,
1119 DiskGB: d.DiskGb, 1170 DiskGB: d.DiskGb,
internal/agent/reconcile/reconcile_test.go
Old New
@@ -43,6 +43,10 @@ type fakeProv struct {
43 43
44 cidr string 44 cidr string
45 addrs map[string]string // vmID -> ip (sticky, mirrors the DHCP table) 45 addrs map[string]string // vmID -> ip (sticky, mirrors the DHCP table)
46 // netAddrs is what a snoop on the named NIC heard, per VM. Never populated
47 // by Boot: this address comes from someone else's DHCP server, so a test
48 // that wants one plays the guest with answerNetworkAddress.
49 netAddrs map[string]string
46 50
47 // failReason is what the fake's "hypervisor" left behind; empty is a 51 // failReason is what the fake's "hypervisor" left behind; empty is a
48 // backend with nothing to add, which is the common case. 52 // backend with nothing to add, which is the common case.
@@ -56,9 +60,10 @@ type fakeProv struct {
56 60
57 func newFakeProv() *fakeProv { 61 func newFakeProv() *fakeProv {
58 return &fakeProv{ 62 return &fakeProv{
59 running: map[string]bool{}, 63 running: map[string]bool{},
60 addrs: map[string]string{}, 64 addrs: map[string]string{},
61 cidr: "10.77.1.0/24", 65 netAddrs: map[string]string{},
66 cidr: "10.77.1.0/24",
62 } 67 }
63 } 68 }
64 69
@@ -123,6 +128,21 @@ func (f *fakeProv) Address(id string) string {
123 return f.addrs[id] 128 return f.addrs[id]
124 } 129 }
125 130
131 func (f *fakeProv) NetworkAddress(id string) string {
132 f.mu.Lock()
133 defer f.mu.Unlock()
134 return f.netAddrs[id]
135 }
136
137 // answerNetworkAddress plays the site's own DHCP server: the guest's second NIC
138 // has now been granted an address, and the host's snoop heard it. Always late —
139 // there is no path by which this is known when Boot returns.
140 func (f *fakeProv) answerNetworkAddress(id, ip string) {
141 f.mu.Lock()
142 defer f.mu.Unlock()
143 f.netAddrs[id] = ip
144 }
145
126 // answerAddress starts answering for id: the guest has now booted and asked the 146 // answerAddress starts answering for id: the guest has now booted and asked the
127 // host's DHCP server, seconds after Boot returned. Only meaningful under 147 // host's DHCP server, seconds after Boot returned. Only meaningful under
128 // lateAddress. 148 // lateAddress.
@@ -826,6 +846,80 @@ func TestAddressDiscoveredAfterBootIsPersisted(t *testing.T) {
826 assert.Equal(t, "10.77.1.2", recs["vm1"].IP, "and persist it, not just report it") 846 assert.Equal(t, "10.77.1.2", recs["vm1"].IP, "and persist it, not just report it")
827 } 847 }
828 848
849 // TestNetworkedVMReportsBothAddresses pins the additive shape at the layer that
850 // reports it: the NAT address is there from the first tick, as it is for every
851 // guest, and the named NIC's address joins it — separately, later, without ever
852 // displacing it. The gate path uses the first, so it has no discovery window at
853 // all; the LAN address arrives when the site's DHCP server says so.
854 func TestNetworkedVMReportsBothAddresses(t *testing.T) {
855 f := setup(t)
856 lan := func(v *pb.VMDesired) { v.Network = "lan" }
857
858 rep := f.step(snap(1, vm("vm1", lan)))
859 av := findVM(rep, "vm1")
860 require.NotNil(t, av)
861 assert.Equal(t, "10.77.1.2", av.Ip, "the NAT NIC is unconditional and known at boot")
862 assert.Empty(t, av.NetworkIp, "the LAN address is not known until the guest asks for it")
863
864 f.prov.answerNetworkAddress("vm1", "192.168.0.42") // the site's server answered
865
866 rep = f.step(snap(2, vm("vm1", lan)))
867 av = findVM(rep, "vm1")
868 require.NotNil(t, av)
869 assert.Equal(t, "10.77.1.2", av.Ip, "which never moves")
870 assert.Equal(t, "192.168.0.42", av.NetworkIp)
871
872 recs, _ := f.st.LoadVMs()
873 require.Contains(t, recs, "vm1")
874 assert.Equal(t, "10.77.1.2", recs["vm1"].IP)
875 assert.Equal(t, "192.168.0.42", recs["vm1"].NetworkIP, "persisted, not just reported")
876 assert.Equal(t, "lan", recs["vm1"].Spec.Network, "and the spec that asked survives the restart")
877
878 // The empty guard covers this address too: a snoop that has heard nothing
879 // since the agent restarted must not erase the fleet's record of where the
880 // guest is on the LAN.
881 f.prov.answerNetworkAddress("vm1", "")
882 rep = f.step(snap(3, vm("vm1", lan)))
883 av = findVM(rep, "vm1")
884 require.NotNil(t, av)
885 assert.Equal(t, "192.168.0.42", av.NetworkIp, "an empty answer is 'not known', not 'gone'")
886 }
887
888 // TestNATOnlyVMReportsNoNetworkAddress is the guard for every guest running
889 // today: nothing about the second NIC leaks into a VM that never asked for one,
890 // even when the backend is asked about it every tick.
891 func TestNATOnlyVMReportsNoNetworkAddress(t *testing.T) {
892 f := setup(t)
893
894 rep := f.step(snap(1, vm("vm1")))
895 av := findVM(rep, "vm1")
896 require.NotNil(t, av)
897 assert.Equal(t, "10.77.1.2", av.Ip)
898 assert.Empty(t, av.NetworkIp)
899
900 recs, _ := f.st.LoadVMs()
901 assert.Empty(t, recs["vm1"].NetworkIP)
902 }
903
904 // TestSeedCarriesBothNICsOnlyForANetworkedGuest pins what the guest is told
905 // about its own hardware: a networked guest's seed names both MACs so netplan
906 // can match each NIC on its own, and a NAT-only guest's names no second one at
907 // all — which is what keeps its network-config the single-stanza document every
908 // running guest already booted with.
909 func TestSeedCarriesBothNICsOnlyForANetworkedGuest(t *testing.T) {
910 f := setup(t)
911 var got seed.Params
912 f.eng.Seed = func(_ string, p seed.Params) error { got = p; return nil }
913
914 f.step(snap(1, vm("vm1", func(v *pb.VMDesired) { v.Network = "lan" })))
915 assert.Equal(t, state.MAC("vm1"), got.MAC)
916 assert.Equal(t, state.NetMAC("vm1"), got.NetworkMAC)
917
918 f.step(snap(2, vm("vm2")))
919 assert.Equal(t, state.MAC("vm2"), got.MAC)
920 assert.Empty(t, got.NetworkMAC, "a guest that asked for no network is told of no second NIC")
921 }
922
829 // TestEmptyAddressAnswerKeepsTheKnownOne pins noteAddress's empty guard: "I have 923 // TestEmptyAddressAnswerKeepsTheKnownOne pins noteAddress's empty guard: "I have
830 // no answer" is not "it has no address". Blanking rec.IP would cut the guest SSH 924 // no answer" is not "it has no address". Blanking rec.IP would cut the guest SSH
831 // tunnel to a VM that is reachable and running. 925 // tunnel to a VM that is reachable and running.
internal/agent/run/cli.go
Old New
@@ -19,6 +19,7 @@ import (
19 "os/exec" 19 "os/exec"
20 "os/signal" 20 "os/signal"
21 "runtime" 21 "runtime"
22 "strings"
22 "syscall" 23 "syscall"
23 "time" 24 "time"
24 25
@@ -33,6 +34,7 @@ import (
33 "github.com/a73x/eitri/internal/agent/syncclient" 34 "github.com/a73x/eitri/internal/agent/syncclient"
34 "github.com/a73x/eitri/internal/covsnap" 35 "github.com/a73x/eitri/internal/covsnap"
35 "github.com/a73x/eitri/internal/joinblob" 36 "github.com/a73x/eitri/internal/joinblob"
37 "github.com/a73x/eitri/internal/names"
36 ) 38 )
37 39
38 // hostRunner is the production one-shot command runner injected into the 40 // hostRunner is the production one-shot command runner injected into the
@@ -61,6 +63,10 @@ type Config struct {
61 // MaxVCPUs, MaxMemMB, MaxDiskGB cap the host resources this agent offers the 63 // MaxVCPUs, MaxMemMB, MaxDiskGB cap the host resources this agent offers the
62 // fleet (0 = unlimited): advertised to the server AND enforced at VM boot. 64 // fleet (0 = unlimited): advertised to the server AND enforced at VM boot.
63 MaxVCPUs, MaxMemMB, MaxDiskGB int64 65 MaxVCPUs, MaxMemMB, MaxDiskGB int64
66 // HostNetworks maps a named guest network to the bridge this host's
67 // operator already declared for it (Linux hosts only). eitri attaches taps
68 // to the bridge; it never creates or addresses it.
69 HostNetworks map[string]string
64 } 70 }
65 71
66 // RunCLI dispatches the eitri-agent command line (everything after the binary 72 // RunCLI dispatches the eitri-agent command line (everything after the binary
@@ -89,11 +95,24 @@ func RunCLI(args []string) error {
89 return serve(st, cfg) 95 return serve(st, cfg)
90 } 96 }
91 97
92 // parseConfig defines and parses the agent flags, returning the assembled 98 // parseConfig defines and parses the agent flags for THIS process's platform —
99 // a thin wrapper over parseConfigOn(runtime.GOOS, args), which exists so a
100 // test can drive the darwin refusal path without a GOOS-tagged build.
101 func parseConfig(args []string) (Config, []string, error) {
102 return parseConfigOn(runtime.GOOS, args)
103 }
104
105 // parseConfigOn defines and parses the agent flags, returning the assembled
93 // Config and any positional arguments (the "join <blob>" subcommand). Resource 106 // Config and any positional arguments (the "join <blob>" subcommand). Resource
94 // caps are validated here so a negative cap is rejected before any state is 107 // caps are validated here so a negative cap is rejected before any state is
95 // touched. 108 // touched. goos decides whether --host-network is refused (see
96 func parseConfig(args []string) (Config, []string, error) { 109 // validateHostNetworks); production always passes runtime.GOOS, and it is a
110 // parameter — not a read of runtime.GOOS inline — so this refusal is provable
111 // on every platform a test runs on, not only the one the flag actually parses
112 // under. goos governs ONLY that one refusal: defaultStateDir and every other
113 // per-platform choice below still come from the build-tagged wire_*.go for
114 // this binary's real GOOS — passing "darwin" here does not simulate a Mac.
115 func parseConfigOn(goos string, args []string) (Config, []string, error) {
97 fs := flag.NewFlagSet("eitri-agent", flag.ContinueOnError) 116 fs := flag.NewFlagSet("eitri-agent", flag.ContinueOnError)
98 // The default is per-platform: a Linux host keeps state where its 117 // The default is per-platform: a Linux host keeps state where its
99 // root-run unit can create it, a Mac in a dotdir the running account owns. 118 // root-run unit can create it, a Mac in a dotdir the running account owns.
@@ -124,10 +143,32 @@ func parseConfig(args []string) (Config, []string, error) {
124 maxVCPUs := fs.Int64("max-vcpus", 0, "cap the total vCPUs this host offers the fleet (0 = unlimited; reserves headroom, advertised + enforced at boot)") 143 maxVCPUs := fs.Int64("max-vcpus", 0, "cap the total vCPUs this host offers the fleet (0 = unlimited; reserves headroom, advertised + enforced at boot)")
125 maxMemMB := fs.Int64("max-mem-mb", 0, "cap the total memory (MB) this host offers the fleet (0 = unlimited)") 144 maxMemMB := fs.Int64("max-mem-mb", 0, "cap the total memory (MB) this host offers the fleet (0 = unlimited)")
126 maxDiskGB := fs.Int64("max-disk-gb", 0, "cap the total disk (GB) this host offers the fleet (0 = unlimited)") 145 maxDiskGB := fs.Int64("max-disk-gb", 0, "cap the total disk (GB) this host offers the fleet (0 = unlimited)")
146 hostNetworks := map[string]string{}
147 fs.Func("host-network", "a named guest network backed by a bridge this host's "+
148 "operator already declared, as name=bridge (repeatable; Linux hosts only — "+
149 "eitri attaches taps to the bridge, it never creates or addresses it)",
150 func(v string) error {
151 name, bridge, ok := strings.Cut(v, "=")
152 if !ok || bridge == "" {
153 return fmt.Errorf("want name=bridge, got %q", v)
154 }
155 if !names.IsNetworkName(name) {
156 return fmt.Errorf("invalid network name %q (1-32 of [a-z0-9-], no leading/trailing hyphen, \"nat\" reserved)", name)
157 }
158 if _, dup := hostNetworks[name]; dup {
159 return fmt.Errorf("network %q declared twice", name)
160 }
161 hostNetworks[name] = bridge
162 return nil
163 })
127 if err := fs.Parse(args); err != nil { 164 if err := fs.Parse(args); err != nil {
128 return Config{}, nil, err 165 return Config{}, nil, err
129 } 166 }
130 167
168 if err := validateHostNetworks(names.OSServesNamedNetworks(goos), hostNetworks); err != nil {
169 return Config{}, nil, err
170 }
171
131 // Fixed order so the reported cap is deterministic when more than one is bad. 172 // Fixed order so the reported cap is deterministic when more than one is bad.
132 caps := []struct { 173 caps := []struct {
133 flag string 174 flag string
@@ -158,9 +199,27 @@ func parseConfig(args []string) (Config, []string, error) {
158 MaxVCPUs: *maxVCPUs, 199 MaxVCPUs: *maxVCPUs,
159 MaxMemMB: *maxMemMB, 200 MaxMemMB: *maxMemMB,
160 MaxDiskGB: *maxDiskGB, 201 MaxDiskGB: *maxDiskGB,
202 HostNetworks: hostNetworks,
161 }, fs.Args(), nil 203 }, fs.Args(), nil
162 } 204 }
163 205
206 // validateHostNetworks refuses --host-network on a host that cannot serve it.
207 // It takes the platform's support as a bool rather than a goos string so the
208 // one fact (names.OSServesNamedNetworks) has a single call site in this file,
209 // in parseConfigOn.
210 //
211 // Refusing here — at parse time, before state is opened or a bridge touched —
212 // matches this project's fail-loudly-at-boot precedent: a Mac that swallowed
213 // the flag would advertise no named networks anyway, and the operator's next
214 // signal would be a 409 telling them to add the exact flag they already
215 // passed. That is a dead end forever; this is a dead end at startup.
216 func validateHostNetworks(platformSupportsHostNetworks bool, hostNetworks map[string]string) error {
217 if len(hostNetworks) == 0 || platformSupportsHostNetworks {
218 return nil
219 }
220 return errors.New("--host-network: named networks are served by Linux hosts only; remove the flag (bridged guests need one of those)")
221 }
222
164 // join handles the "join <blob>" subcommand: decode the join blob, enroll, 223 // join handles the "join <blob>" subcommand: decode the join blob, enroll,
165 // and persist identity — pinning the server cert from the blob (the enroll 224 // and persist identity — pinning the server cert from the blob (the enroll
166 // response's fingerprint is ignored, so the blob is the sole trust root). 225 // response's fingerprint is ignored, so the blob is the sole trust root).
@@ -317,19 +376,20 @@ func serve(st *state.Store, cfg Config) error {
317 } 376 }
318 377
319 client := &syncclient.Client{ 378 client := &syncclient.Client{
320 Engine: engine, 379 Engine: engine,
321 St: st, 380 St: st,
322 Identity: id, 381 Identity: id,
323 StateDir: cfg.StateDir, 382 StateDir: cfg.StateDir,
324 Provisioner: platformProvisioner, 383 Provisioner: platformProvisioner,
325 GuestCIDR: plat.GuestCIDR, 384 GuestCIDR: plat.GuestCIDR,
326 Runner: hostRunner, 385 HostNetworks: plat.HostNetworks,
327 Console: pumps, 386 Runner: hostRunner,
328 Exposures: proxy, 387 Console: pumps,
329 UplinkAddr: func() string { return hostinfo.UplinkAddr(uplinkVia) }, 388 Exposures: proxy,
330 MaxVCPUs: cfg.MaxVCPUs, 389 UplinkAddr: func() string { return hostinfo.UplinkAddr(uplinkVia) },
331 MaxMemMB: cfg.MaxMemMB, 390 MaxVCPUs: cfg.MaxVCPUs,
332 MaxDiskGB: cfg.MaxDiskGB, 391 MaxMemMB: cfg.MaxMemMB,
392 MaxDiskGB: cfg.MaxDiskGB,
333 } 393 }
334 394
335 slog.Info("agent started", "host_id", id.HostID, "guest_cidr", plat.GuestCIDR()) 395 slog.Info("agent started", "host_id", id.HostID, "guest_cidr", plat.GuestCIDR())
internal/agent/run/cli_test.go
Old New
@@ -49,6 +49,7 @@ func TestParseConfigDefaults(t *testing.T) {
49 assert.Equal(t, int64(0), cfg.MaxVCPUs) 49 assert.Equal(t, int64(0), cfg.MaxVCPUs)
50 assert.Equal(t, int64(0), cfg.MaxMemMB) 50 assert.Equal(t, int64(0), cfg.MaxMemMB)
51 assert.Equal(t, int64(0), cfg.MaxDiskGB) 51 assert.Equal(t, int64(0), cfg.MaxDiskGB)
52 assert.Equal(t, map[string]string{}, cfg.HostNetworks)
52 } 53 }
53 54
54 // TestParseConfigOverrides confirms every flag threads through to the Config. 55 // TestParseConfigOverrides confirms every flag threads through to the Config.
@@ -108,6 +109,76 @@ func TestParseConfigBadFlag(t *testing.T) {
108 require.Error(t, err) 109 require.Error(t, err)
109 } 110 }
110 111
112 // TestParseHostNetworkFlag confirms --host-network is repeatable and each
113 // name=bridge pair lands in Config.HostNetworks.
114 func TestParseHostNetworkFlag(t *testing.T) {
115 cfg, _, err := parseConfig([]string{"--host-network", "lan=br0", "--host-network", "lab=br1"})
116 require.NoError(t, err)
117 assert.Equal(t, map[string]string{"lan": "br0", "lab": "br1"}, cfg.HostNetworks)
118 }
119
120 // TestParseHostNetworkFlagRefusals rejects every malformed or ambiguous
121 // --host-network value: missing/empty bridge, the reserved "nat" name, a name
122 // that fails the shared grammar, and a duplicate declaration.
123 func TestParseHostNetworkFlagRefusals(t *testing.T) {
124 for _, args := range [][]string{
125 {"--host-network", "lan"}, // no bridge
126 {"--host-network", "lan="}, // empty bridge
127 {"--host-network", "nat=br0"}, // reserved name
128 {"--host-network", "LAN=br0"}, // bad grammar
129 {"--host-network", "lan=br0", "--host-network", "lan=br1"}, // duplicate
130 } {
131 _, _, err := parseConfig(args)
132 assert.Error(t, err, "parseConfig(%v) accepted, want error", args)
133 }
134 }
135
136 // TestParseConfigOnRefusesHostNetworkOnUnsupportedGOOS proves the refusal is
137 // actually wired into the parse path, not just unit-tested in isolation:
138 // deleting the validateHostNetworks call in parseConfigOn makes this fail,
139 // where a test of validateHostNetworks alone would not notice.
140 func TestParseConfigOnRefusesHostNetworkOnUnsupportedGOOS(t *testing.T) {
141 _, _, err := parseConfigOn("darwin", []string{"--host-network", "lan=br0"})
142 require.Error(t, err)
143 assert.Contains(t, err.Error(), "--host-network")
144 assert.Contains(t, err.Error(), "Linux")
145 }
146
147 // TestParseConfigOnAllowsUnsupportedGOOSWithoutTheFlag confirms a Mac agent
148 // with no --host-network at all still parses cleanly — the refusal is for the
149 // flag, not the platform.
150 func TestParseConfigOnAllowsUnsupportedGOOSWithoutTheFlag(t *testing.T) {
151 cfg, _, err := parseConfigOn("darwin", []string{"--max-vcpus=2"})
152 require.NoError(t, err)
153 assert.Equal(t, int64(2), cfg.MaxVCPUs)
154 }
155
156 // TestValidateHostNetworksRefusesUnsupportedPlatform rejects --host-network on
157 // a platform that cannot serve it (Mac today), naming the remedy rather than
158 // leaving the operator to rediscover it from a later 409. The message says
159 // "Linux" once, not twice — the fact ("Linux hosts only") already carries the
160 // remedy's platform, so the remedy itself need not repeat it.
161 func TestValidateHostNetworksRefusesUnsupportedPlatform(t *testing.T) {
162 err := validateHostNetworks(false, map[string]string{"lan": "br0"})
163 require.Error(t, err)
164 assert.Contains(t, err.Error(), "--host-network")
165 assert.Equal(t, 1, strings.Count(err.Error(), "Linux"), "message %q should name Linux once, not repeat it", err.Error())
166 }
167
168 // TestValidateHostNetworksAllowsSupportedPlatform lets --host-network through
169 // on a platform that can serve it (Linux today).
170 func TestValidateHostNetworksAllowsSupportedPlatform(t *testing.T) {
171 err := validateHostNetworks(true, map[string]string{"lan": "br0"})
172 require.NoError(t, err)
173 }
174
175 // TestValidateHostNetworksVacuousWithoutFlag is fine on any platform: an
176 // operator who never passed --host-network has nothing to be refused for.
177 func TestValidateHostNetworksVacuousWithoutFlag(t *testing.T) {
178 err := validateHostNetworks(false, map[string]string{})
179 require.NoError(t, err)
180 }
181
111 // TestJoinEmptyBlob rejects a missing blob before any decode or network call; 182 // TestJoinEmptyBlob rejects a missing blob before any decode or network call;
112 // st is never touched, so nil is safe. 183 // st is never touched, so nil is safe.
113 func TestJoinEmptyBlob(t *testing.T) { 184 func TestJoinEmptyBlob(t *testing.T) {
internal/agent/run/exposures_test.go
Old New
@@ -4,6 +4,7 @@ import (
4 "testing" 4 "testing"
5 5
6 "github.com/a73x/eitri/internal/agent/state" 6 "github.com/a73x/eitri/internal/agent/state"
7 "github.com/stretchr/testify/assert"
7 "github.com/stretchr/testify/require" 8 "github.com/stretchr/testify/require"
8 ) 9 )
9 10
@@ -30,3 +31,23 @@ func TestGuestAddrAnswersFromTheAgentsOwnRecords(t *testing.T) {
30 t.Errorf("guestAddr(vm2) = %q, want empty", got) 31 t.Errorf("guestAddr(vm2) = %q, want empty", got)
31 } 32 }
32 } 33 }
34
35 // TestGuestAddrIsTheNATAddressForANetworkedGuest pins which of a two-NIC
36 // guest's addresses this host reaches it by. Exposures and the SSH tunnel both
37 // run over the private bridge — that address exists before the guest boots and
38 // needs no site DHCP server to have answered — so a guest on the operator's LAN
39 // is published exactly like its NAT-only siblings. Answering with the LAN
40 // address instead would make a published port depend on someone else's network
41 // being up.
42 func TestGuestAddrIsTheNATAddressForANetworkedGuest(t *testing.T) {
43 st, err := state.Open(t.TempDir())
44 require.NoError(t, err)
45
46 require.NoError(t, st.SaveVM(state.Record{
47 Spec: state.VMSpec{VMID: "vm-lan", Network: "lan"},
48 IP: "10.77.1.5",
49 NetworkIP: "192.168.0.42",
50 }))
51
52 assert.Equal(t, "10.77.1.5", guestAddr(st, "vm-lan"))
53 }
internal/agent/run/reservations.go
Old New
@@ -10,8 +10,15 @@ import (
10 // pinning one guest's address. Consumer-owned and minimal (arch R5) so a 10 // pinning one guest's address. Consumer-owned and minimal (arch R5) so a
11 // platform whose networking is nothing like a bridge and a DHCP responder can 11 // platform whose networking is nothing like a bridge and a DHCP responder can
12 // still be handed the surviving guests' addresses. 12 // still be handed the surviving guests' addresses.
13 //
14 // Two verbs because a guest can have two NICs and they rebuild differently:
15 // every guest re-pins the DHCP reservation this host serves its NAT NIC, and a
16 // guest that also has a named-network NIC re-seeds the last address seen there
17 // — never this host's to give — and re-arms the discovery that will confirm or
18 // replace it. A networked guest needs both calls, not one instead of the other.
13 type reservations interface { 19 type reservations interface {
14 AddReservation(vmID, ip string) 20 AddReservation(vmID, ip string)
21 AdoptNetwork(vmID, network, ip string)
15 } 22 }
16 23
17 // replayReservations rebuilds net's reservation table from durable records. A 24 // replayReservations rebuilds net's reservation table from durable records. A
@@ -33,5 +40,11 @@ func replayReservations(st *state.Store, net reservations) {
33 if rec.IP != "" { 40 if rec.IP != "" {
34 net.AddReservation(rec.Spec.VMID, rec.IP) 41 net.AddReservation(rec.Spec.VMID, rec.IP)
35 } 42 }
43 // A networked guest is adopted even with no recorded LAN address: the
44 // attachment itself is what has to be rebuilt, or nothing re-arms the
45 // discovery that reports the address the site's DHCP server granted it.
46 if rec.Spec.Network != "" {
47 net.AdoptNetwork(rec.Spec.VMID, rec.Spec.Network, rec.NetworkIP)
48 }
36 } 49 }
37 } 50 }
internal/agent/run/reservations_test.go
Old New
@@ -12,16 +12,24 @@ import (
12 "github.com/stretchr/testify/require" 12 "github.com/stretchr/testify/require"
13 ) 13 )
14 14
15 // fakeReservations records what the replay pinned. The real table lives behind 15 // fakeReservations records what the replay pinned and what it adopted. The real
16 // a DHCP responder on a host bridge; the replay only has to decide what to pin. 16 // table lives behind a DHCP responder on a host bridge; the replay only has to
17 type fakeReservations struct{ pinned map[string]string } 17 // decide which of the two a surviving guest needs.
18 type fakeReservations struct {
19 pinned map[string]string
20 adopted map[string][2]string // vmID -> {network, ip}
21 }
18 22
19 func newFakeReservations() *fakeReservations { 23 func newFakeReservations() *fakeReservations {
20 return &fakeReservations{pinned: map[string]string{}} 24 return &fakeReservations{pinned: map[string]string{}, adopted: map[string][2]string{}}
21 } 25 }
22 26
23 func (f *fakeReservations) AddReservation(vmID, ip string) { f.pinned[vmID] = ip } 27 func (f *fakeReservations) AddReservation(vmID, ip string) { f.pinned[vmID] = ip }
24 28
29 func (f *fakeReservations) AdoptNetwork(vmID, network, ip string) {
30 f.adopted[vmID] = [2]string{network, ip}
31 }
32
25 // storeWithRecords opens a store in a temp dir and persists recs, returning the 33 // storeWithRecords opens a store in a temp dir and persists recs, returning the
26 // store and its directory. 34 // store and its directory.
27 func storeWithRecords(t *testing.T, recs ...state.Record) (*state.Store, string) { 35 func storeWithRecords(t *testing.T, recs ...state.Record) (*state.Store, string) {
@@ -77,6 +85,52 @@ func TestReplayReservationsSkipsRecordWithoutAddress(t *testing.T) {
77 assert.Equal(t, map[string]string{"vm-addressed": "10.77.1.7"}, table.pinned) 85 assert.Equal(t, map[string]string{"vm-addressed": "10.77.1.7"}, table.pinned)
78 } 86 }
79 87
88 // TestReplayReservationsRebuildsBothNICs pins the additive rebuild: a networked
89 // guest has two NICs and needs both halves back. Its NAT reservation is pinned
90 // like every other guest's — that is the address the gate splices to and the
91 // one it must not be renumbered out of — AND its named NIC is adopted with the
92 // last address the site's DHCP server was seen granting it, which this host
93 // pins nowhere because it was never this host's to hand out.
94 func TestReplayReservationsRebuildsBothNICs(t *testing.T) {
95 st, _ := storeWithRecords(t,
96 state.Record{Spec: state.VMSpec{VMID: "vm-nat"}, IP: "10.77.1.5"},
97 state.Record{
98 Spec: state.VMSpec{VMID: "vm-lan", Network: "lan"},
99 IP: "10.77.1.6",
100 NetworkIP: "192.168.0.7",
101 },
102 // Booted, but its guest has not been heard on the LAN yet: the
103 // attachment still has to be rebuilt, or nothing re-arms the discovery
104 // that will report the address when it comes.
105 state.Record{Spec: state.VMSpec{VMID: "vm-fresh", Network: "lab"}, IP: "10.77.1.7"},
106 )
107
108 table := newFakeReservations()
109 replayReservations(st, table)
110
111 assert.Equal(t, map[string]string{
112 "vm-nat": "10.77.1.5",
113 "vm-lan": "10.77.1.6",
114 "vm-fresh": "10.77.1.7",
115 }, table.pinned, "every guest keeps its NAT reservation, networked or not")
116 assert.Equal(t, map[string][2]string{
117 "vm-lan": {"lan", "192.168.0.7"},
118 "vm-fresh": {"lab", ""},
119 }, table.adopted, "and only a networked guest is adopted, with the LAN address alone")
120 }
121
122 // TestReplayAdoptsNoNetworkForANATOnlyGuest pins the other side: the guest
123 // every host runs today is rebuilt exactly as it was before named networks
124 // existed — pinned, and adopted by nothing.
125 func TestReplayAdoptsNoNetworkForANATOnlyGuest(t *testing.T) {
126 st, _ := storeWithRecords(t, state.Record{Spec: state.VMSpec{VMID: "vm-nat"}, IP: "10.77.1.5"})
127
128 table := newFakeReservations()
129 replayReservations(st, table)
130
131 assert.Empty(t, table.adopted)
132 }
133
80 // TestReplayReservationsSurvivesStateLoadFailure pins the fail-closed 134 // TestReplayReservationsSurvivesStateLoadFailure pins the fail-closed
81 // behaviour: an unreadable state directory warns and returns, leaving the 135 // behaviour: an unreadable state directory warns and returns, leaving the
82 // table empty rather than aborting the agent's startup. 136 // table empty rather than aborting the agent's startup.
internal/agent/run/wire_darwin.go
Old New
@@ -64,6 +64,11 @@ type platform struct {
64 // GuestCIDR reports the subnet this host's guests are on, asked once per 64 // GuestCIDR reports the subnet this host's guests are on, asked once per
65 // report rather than resolved at startup. See the sync client's field. 65 // report rather than resolved at startup. See the sync client's field.
66 GuestCIDR func() string 66 GuestCIDR func() string
67 // HostNetworks is always empty here, and the field exists only because
68 // serve() is platform-neutral. Attaching a guest to a named bridge is a
69 // Linux tap on a Linux bridge; vmnet gives a Mac neither, so this host
70 // advertises none and every create asking for one is refused above it.
71 HostNetworks []string
67 } 72 }
68 73
69 // newPlatform builds the macOS backend: vfkit over Apple's 74 // newPlatform builds the macOS backend: vfkit over Apple's
internal/agent/run/wire_linux.go
Old New
@@ -5,6 +5,8 @@ package run
5 import ( 5 import (
6 "context" 6 "context"
7 "fmt" 7 "fmt"
8 "maps"
9 "slices"
8 10
9 "github.com/a73x/eitri/internal/agent/bootstrap" 11 "github.com/a73x/eitri/internal/agent/bootstrap"
10 "github.com/a73x/eitri/internal/agent/cloudhv" 12 "github.com/a73x/eitri/internal/agent/cloudhv"
@@ -51,6 +53,11 @@ type platform struct {
51 // returns a constant; the poll shape costs it nothing and lets a platform 53 // returns a constant; the poll shape costs it nothing and lets a platform
52 // whose OS owns the network answer late, or not at all. 54 // whose OS owns the network answer late, or not at all.
53 GuestCIDR func() string 55 GuestCIDR func() string
56 // HostNetworks are the named guest networks this host serves, sorted. A
57 // value rather than a poll, unlike GuestCIDR: the set is the agent's
58 // command line, so it changes only with a restart — which is why it rides
59 // in Hello and not in every report.
60 HostNetworks []string
54 } 61 }
55 62
56 // proposeGuestCIDR is what this platform offers the fleet at enrollment: the 63 // proposeGuestCIDR is what this platform offers the fleet at enrollment: the
@@ -93,10 +100,16 @@ func newPlatform(ctx context.Context, cfg Config, st *state.Store) (platform, er
93 bridgeCIDR = lastResortGuestCIDR 100 bridgeCIDR = lastResortGuestCIDR
94 persistGuestCIDR(st, bridgeCIDR) 101 persistGuestCIDR(st, bridgeCIDR)
95 } 102 }
96 net, err := netenv.New(hostRunner, bridgeCIDR) 103 net, err := netenv.New(hostRunner, bridgeCIDR, cfg.HostNetworks)
97 if err != nil { 104 if err != nil {
98 return platform{}, fmt.Errorf("netenv init: %w", err) 105 return platform{}, fmt.Errorf("netenv init: %w", err)
99 } 106 }
107 // Before the NAT bridge and before any VM: a configured network whose
108 // bridge is missing is a host that cannot honor what it is about to
109 // advertise. Refuse to start, in the flag's own vocabulary.
110 if err := net.VerifyNetworks(); err != nil {
111 return platform{}, err
112 }
100 // Before anything is created: a subnet overlapping this host's own uplink 113 // Before anything is created: a subnet overlapping this host's own uplink
101 // would make its address local and stop it answering, and the only machine 114 // would make its address local and stop it answering, and the only machine
102 // positioned to fix that is the one that just lost the route. Refusing here 115 // positioned to fix that is the one that just lost the route. Refusing here
@@ -131,5 +144,8 @@ func newPlatform(ctx context.Context, cfg Config, st *state.Store) (platform, er
131 prov.Pumps = pumps 144 prov.Pumps = pumps
132 145
133 return platform{Prov: prov, Pumps: pumps, 146 return platform{Prov: prov, Pumps: pumps,
134 GuestCIDR: func() string { return bridgeCIDR }}, nil 147 GuestCIDR: func() string { return bridgeCIDR },
148 // Sorted so every Hello from an unchanged host is byte-identical: map
149 // iteration order would make the advertisement churn for no reason.
150 HostNetworks: slices.Sorted(maps.Keys(cfg.HostNetworks))}, nil
135 } 151 }
internal/agent/run/wire_linux_test.go
Old New
@@ -0,0 +1,24 @@
1 //go:build linux
2
3 package run
4
5 import (
6 "testing"
7
8 "github.com/stretchr/testify/assert"
9 "github.com/stretchr/testify/require"
10 )
11
12 // TestProposeGuestCIDRIsTheFlagOrNoOpinion pins what a Linux host tells the
13 // fleet at enrollment about the subnet its guests will sit on. Silence is an
14 // answer here — nil takes the control plane's suggestion, which is what a host
15 // started without --bridge-cidr wants — and it is only ever this platform's to
16 // give, because only a host that CREATES the guest network may name it.
17 func TestProposeGuestCIDRIsTheFlagOrNoOpinion(t *testing.T) {
18 assert.Nil(t, proposeGuestCIDR(Config{}),
19 "no flag is no opinion, not a fabricated subnet")
20
21 got := proposeGuestCIDR(Config{BridgeCIDR: "10.30.0.0/24"})
22 require.NotNil(t, got)
23 assert.Equal(t, "10.30.0.0/24", *got)
24 }
internal/agent/seed/seed.go
Old New
@@ -43,6 +43,15 @@ type Params struct {
43 // CA key. 43 // CA key.
44 SSHHostKeyPEM string 44 SSHHostKeyPEM string
45 SSHHostCert string 45 SSHHostCert string
46 // MAC and NetworkMAC are the guest's NICs, in the order the hypervisor
47 // attaches them: MAC is the NAT NIC every guest has, NetworkMAC the second
48 // one a guest on a named host network gets. NetworkMAC empty — the usual
49 // case — writes the single-NIC network-config verbatim as it has always
50 // been; non-empty requires MAC, because a two-NIC netplan matches both by
51 // hardware address and a stanza with nothing to match would claim the wrong
52 // link or none.
53 MAC string
54 NetworkMAC string
46 } 55 }
47 56
48 // validateParams checks that fields embedded into YAML do not contain newlines 57 // validateParams checks that fields embedded into YAML do not contain newlines
@@ -56,11 +65,19 @@ func validateParams(p Params) error {
56 {"Hostname", p.Hostname}, 65 {"Hostname", p.Hostname},
57 {"InstanceID", p.InstanceID}, 66 {"InstanceID", p.InstanceID},
58 {"SSHAuthorizedKey", p.SSHAuthorizedKey}, 67 {"SSHAuthorizedKey", p.SSHAuthorizedKey},
68 {"MAC", p.MAC},
69 {"NetworkMAC", p.NetworkMAC},
59 } { 70 } {
60 if strings.ContainsAny(f.value, "\n\r") { 71 if strings.ContainsAny(f.value, "\n\r") {
61 return fmt.Errorf("seed: %s must not contain newline or carriage return", f.name) 72 return fmt.Errorf("seed: %s must not contain newline or carriage return", f.name)
62 } 73 }
63 } 74 }
75 // A second NIC with no first one to match is a netplan that claims the
76 // wrong link or no link at all — refuse to build it rather than ship a
77 // guest whose management NIC never comes up.
78 if p.NetworkMAC != "" && p.MAC == "" {
79 return fmt.Errorf("seed: NetworkMAC given without MAC — a two-NIC network-config matches both by hardware address")
80 }
64 return nil 81 return nil
65 } 82 }
66 83
@@ -286,8 +303,17 @@ func metaData(p Params) string {
286 // it under the identifier, so the address becomes unfindable, and worse, an 303 // it under the identifier, so the address becomes unfindable, and worse, an
287 // earlier MAC-keyed offer can linger and be read as current. Linux is unharmed 304 // earlier MAC-keyed offer can linger and be read as current. Linux is unharmed
288 // because our responder keys on the packet's hardware address either way. 305 // because our responder keys on the packet's hardware address either way.
289 func networkConfig() string { 306 //
290 return `network: 307 // A guest with a second NIC on a named host network gets a second stanza, and
308 // then BOTH are matched by MAC rather than by name: two NICs cannot share one
309 // name glob (netplan refuses a link claimed by two definitions), and the MACs
310 // are the only thing the host and the guest already agree on for each NIC
311 // independently. Route metrics decide the default route explicitly — 100 for
312 // the NAT NIC, 200 for the named one — so egress is deterministic and a LAN
313 // outage costs reachability, not the guest's outbound.
314 func networkConfig(p Params) string {
315 if p.NetworkMAC == "" {
316 return `network:
291 version: 2 317 version: 2
292 ethernets: 318 ethernets:
293 primary: 319 primary:
@@ -295,7 +321,28 @@ func networkConfig() string {
295 name: "en*" 321 name: "en*"
296 dhcp4: true 322 dhcp4: true
297 dhcp-identifier: mac 323 dhcp-identifier: mac
324 dhcp4-overrides:
325 route-metric: 100
298 ` 326 `
327 }
328 return fmt.Sprintf(`network:
329 version: 2
330 ethernets:
331 primary:
332 match:
333 macaddress: "%s"
334 dhcp4: true
335 dhcp-identifier: mac
336 dhcp4-overrides:
337 route-metric: 100
338 net1:
339 match:
340 macaddress: "%s"
341 dhcp4: true
342 dhcp-identifier: mac
343 dhcp4-overrides:
344 route-metric: 200
345 `, p.MAC, p.NetworkMAC)
299 } 346 }
300 347
301 // Build creates a cloud-init NoCloud seed ISO at outPath. 348 // Build creates a cloud-init NoCloud seed ISO at outPath.
@@ -367,7 +414,7 @@ func Build(outPath string, p Params) error {
367 files := map[string]string{ 414 files := map[string]string{
368 "/user-data": userData(p), 415 "/user-data": userData(p),
369 "/meta-data": metaData(p), 416 "/meta-data": metaData(p),
370 "/network-config": networkConfig(), 417 "/network-config": networkConfig(p),
371 } 418 }
372 // Vendor-data carries eitri's own config (disk-grow, and the SSH user-CA 419 // Vendor-data carries eitri's own config (disk-grow, and the SSH user-CA
373 // drop-in + host cert when set); it is always non-empty, so always written. 420 // drop-in + host cert when set); it is always non-empty, so always written.
internal/agent/seed/seed_test.go
Old New
@@ -84,7 +84,7 @@ func TestUserDataCustomPassthrough(t *testing.T) {
84 } 84 }
85 85
86 func TestNetworkConfigUsesDHCP(t *testing.T) { 86 func TestNetworkConfigUsesDHCP(t *testing.T) {
87 got := networkConfig() 87 got := networkConfig(Params{})
88 if !strings.Contains(got, "dhcp4: true") { 88 if !strings.Contains(got, "dhcp4: true") {
89 t.Fatalf("network-config should enable DHCP, got:\n%s", got) 89 t.Fatalf("network-config should enable DHCP, got:\n%s", got)
90 } 90 }
@@ -93,8 +93,62 @@ func TestNetworkConfigUsesDHCP(t *testing.T) {
93 } 93 }
94 } 94 }
95 95
96 // TestNetworkConfigSingleNICIsTheGuestEveryHostRunsToday pins the shape a guest
97 // with no named network gets, whole: one stanza, matched by name, DHCP, and an
98 // explicit default-route metric so the two-NIC guest's 100/200 ordering is a
99 // widening of a stated number rather than a guess about netplan's default.
100 func TestNetworkConfigSingleNICIsTheGuestEveryHostRunsToday(t *testing.T) {
101 assert.Equal(t, `network:
102 version: 2
103 ethernets:
104 primary:
105 match:
106 name: "en*"
107 dhcp4: true
108 dhcp-identifier: mac
109 dhcp4-overrides:
110 route-metric: 100
111 `, networkConfig(Params{MAC: "52:54:00:aa:bb:cc"}),
112 "a guest with no named network is matched by name, as it always was")
113 }
114
115 // TestNetworkConfigTwoNICsMatchByMACAndOrderTheDefaultRoute pins the widened
116 // shape: both NICs matched by their own hardware address (a name glob would
117 // claim both links and netplan refuses a link two definitions match), both on
118 // DHCP, and the NAT NIC carrying the default route at metric 100 against the
119 // named NIC's 200 — deterministic egress, with the LAN as fallback.
120 func TestNetworkConfigTwoNICsMatchByMACAndOrderTheDefaultRoute(t *testing.T) {
121 assert.Equal(t, `network:
122 version: 2
123 ethernets:
124 primary:
125 match:
126 macaddress: "52:54:00:aa:bb:cc"
127 dhcp4: true
128 dhcp-identifier: mac
129 dhcp4-overrides:
130 route-metric: 100
131 net1:
132 match:
133 macaddress: "52:54:00:dd:ee:ff"
134 dhcp4: true
135 dhcp-identifier: mac
136 dhcp4-overrides:
137 route-metric: 200
138 `, networkConfig(Params{MAC: "52:54:00:aa:bb:cc", NetworkMAC: "52:54:00:dd:ee:ff"}))
139 }
140
141 // TestBuildRefusesASecondNICWithoutAFirst pins the refusal: a stanza matching
142 // nothing would leave the guest's management NIC unconfigured, which is worse
143 // than no seed at all.
144 func TestBuildRefusesASecondNICWithoutAFirst(t *testing.T) {
145 err := Build(t.TempDir()+"/seed.iso", Params{Hostname: "h", NetworkMAC: "52:54:00:dd:ee:ff"})
146 require.Error(t, err)
147 assert.Contains(t, err.Error(), "NetworkMAC")
148 }
149
96 func TestNetworkConfigIdentifiesTheGuestByItsMAC(t *testing.T) { 150 func TestNetworkConfigIdentifiesTheGuestByItsMAC(t *testing.T) {
97 got := networkConfig() 151 got := networkConfig(Params{})
98 152
99 // Without this, systemd-networkd sends a DUID-based client identifier and 153 // Without this, systemd-networkd sends a DUID-based client identifier and
100 // the DHCP server files the lease under that instead of the hardware 154 // the DHCP server files the lease under that instead of the hardware
internal/agent/state/state.go
Old New
@@ -16,7 +16,14 @@ import (
16 16
17 type VMSpec struct { 17 type VMSpec struct {
18 VMID, Name, ImageURL, ImageSHA256, CloudInit, SSHAuthorizedKey string 18 VMID, Name, ImageURL, ImageSHA256, CloudInit, SSHAuthorizedKey string
19 VCPUs, MemMB, DiskGB int64 19 // Network names the host network this guest gets a SECOND NIC on; "" is a
20 // guest with the NAT NIC alone. Additive, never instead: every guest is on
21 // the host's private bridge whatever this says. Part of the spec, not the
22 // record: it is desired state the control plane sent, and it must survive
23 // an agent restart so the replay can rebuild the VM's in-memory discovery
24 // state.
25 Network string
26 VCPUs, MemMB, DiskGB int64
20 } 27 }
21 28
22 // Disk is one block device attached to a VM, in attachment order. Index 0 is 29 // Disk is one block device attached to a VM, in attachment order. Index 0 is
@@ -28,8 +35,17 @@ type Disk struct {
28 } 35 }
29 36
30 type Record struct { 37 type Record struct {
31 Spec VMSpec 38 Spec VMSpec
32 IP string 39 // IP is the guest's address on the host's NAT underlay — allocated by this
40 // host, known before the guest boots, and what everything that has to reach
41 // the VM uses.
42 IP string
43 // NetworkIP is the address the site's DHCP server granted the guest on its
44 // named-network NIC, as this host snooped it. Empty for a guest with no
45 // such NIC, and until the first ACK crosses its tap. Durable so that an
46 // agent restart can report it again before the guest's next renewal is
47 // seen — the reservation table's twin for an address this host never gave.
48 NetworkIP string
33 BootID string // host boot ID at last start (lost-detection) 49 BootID string // host boot ID at last start (lost-detection)
34 StopRequested bool // set BEFORE stopping: stopped != lost 50 StopRequested bool // set BEFORE stopping: stopped != lost
35 QuarantinedAt *time.Time 51 QuarantinedAt *time.Time
@@ -106,6 +122,34 @@ func MAC(vmID string) string {
106 return fmt.Sprintf("52:54:00:%02x:%02x:%02x", h[0], h[1], h[2]) 122 return fmt.Sprintf("52:54:00:%02x:%02x:%02x", h[0], h[1], h[2])
107 } 123 }
108 124
125 // NetMAC returns the deterministic MAC of the guest's SECOND NIC — the one
126 // attached to a named host network. Derived, like MAC(vmID), from a namespaced
127 // input ("net1:" + vmID) so a guest's two NICs can never collide and neither
128 // can two guests'. MAC(vmID) is left untouched by this: a VM's first NIC keeps
129 // the identity its DHCP reservation, its lease and its host's records were all
130 // filed under.
131 //
132 // It does not share MAC's OUI, on purpose. That NIC used to land on a private
133 // bridge this host alone spoke for; now it lands on the operator's real LAN,
134 // alongside whatever else is there — most of it, if it is QEMU or libvirt,
135 // answering under 52:54:00 too. The first octet here is 0x56: the
136 // locally-administered bit is set and the multicast bit is clear, so it is a
137 // legal unicast address, and because it differs from 0x52 no NetMAC can ever
138 // land in the same space as MAC's or as a neighbour's conventional guest MAC —
139 // not by any hash outcome, the two spaces are disjoint by construction. The
140 // remaining five octets carry SHA-256 of that input, instead of three,
141 // because a LAN is a bigger room than a private bridge: 40 bits of hash keeps
142 // a fleet's worth of guests comfortably below birthday-collision odds where
143 // 24 got uncomfortable at fleet scale.
144 //
145 // It is what the address snoop keys on, and what the guest's netplan matches
146 // the second stanza by, so both sides of a discovery agree without either
147 // asking the hypervisor what it ended up with.
148 func NetMAC(vmID string) string {
149 h := sha256.Sum256([]byte("net1:" + vmID))
150 return fmt.Sprintf("56:%02x:%02x:%02x:%02x:%02x", h[0], h[1], h[2], h[3], h[4])
151 }
152
109 // atomicWrite writes data to path using a tmp file + rename so that readers 153 // atomicWrite writes data to path using a tmp file + rename so that readers
110 // never see a partial write. 154 // never see a partial write.
111 func atomicWrite(path string, data []byte) error { 155 func atomicWrite(path string, data []byte) error {
internal/agent/state/state_test.go
Old New
@@ -3,6 +3,7 @@ package state
3 import ( 3 import (
4 "os" 4 "os"
5 "path/filepath" 5 "path/filepath"
6 "strconv"
6 "testing" 7 "testing"
7 "time" 8 "time"
8 9
@@ -92,6 +93,65 @@ func TestMACIsDeterministicAndLocallyAdministered(t *testing.T) {
92 } 93 }
93 } 94 }
94 95
96 // TestNetMACGoldenValue pins the exact byte layout for one known vmID: octet
97 // 0 is the fixed 0x56, the remaining five are SHA-256("net1:"+vmID)[:5]. A
98 // full revert to the old 3-byte QEMU-OUI form is already caught below by the
99 // ≠0x52 assertion; what only this golden catches is a narrowed hash width —
100 // 0x56 followed by just three hash bytes would pass every property assertion
101 // in this file and fail only here.
102 func TestNetMACGoldenValue(t *testing.T) {
103 assert.Equal(t, "56:78:a6:1f:4f:7e", NetMAC("vm-abc123"))
104 }
105
106 // TestNetMACIsTheOtherNICOfTheSameVM pins the second NIC's identity: same
107 // shape, same determinism, and never the first NIC's address — one guest with
108 // two NICs sharing a MAC is a guest whose bridge learns the wrong port, and a
109 // snoop keyed on it would report the NAT lease as the LAN one. It also pins
110 // the OUI split from MAC(): NetMAC lands on the operator's real LAN now, so
111 // its first octet must carry the locally-administered bit, leave the
112 // multicast bit clear, and differ from 0x52 — disjoint from both MAC's space
113 // and every conventional 52:54:00 QEMU/libvirt guest already on that LAN.
114 func TestNetMACIsTheOtherNICOfTheSameVM(t *testing.T) {
115 got := NetMAC("vm-abc123")
116 assert.Equal(t, got, NetMAC("vm-abc123"), "NetMAC must be deterministic")
117 assert.Len(t, got, 17)
118
119 first, err := strconv.ParseUint(got[:2], 16, 8)
120 require.NoError(t, err)
121 assert.NotZero(t, first&0x02, "locally-administered bit must be set")
122 assert.Zero(t, first&0x01, "multicast bit must be clear (valid unicast LAA)")
123 assert.NotEqual(t, byte(0x52), byte(first), "must not share MAC's OUI octet")
124
125 assert.NotEqual(t, MAC("vm-abc123"), got, "a VM's two NICs must never share a MAC")
126 assert.NotEqual(t, NetMAC("vm-xyz789"), got, "distinct VMs must not collide either")
127 // Not just for one id: every VM's two NICs must differ, or a guest whose id
128 // happened to hash into a collision would black-hole its own traffic.
129 for _, id := range []string{"a", "vm-1", "0ba1a1b6-6a71-4f6f-9e5f-2c0e3a3b1f42", ""} {
130 assert.NotEqual(t, MAC(id), NetMAC(id), "vmID %q", id)
131 }
132 }
133
134 // TestRecordCarriesBothAddresses pins that a networked guest's two addresses
135 // both survive an agent restart: the NAT one to re-pin its reservation, the
136 // named-network one to report while the next DHCP renewal is waited for.
137 func TestRecordCarriesBothAddresses(t *testing.T) {
138 dir := t.TempDir()
139 s, _ := Open(dir)
140 require.NoError(t, s.SaveVM(Record{
141 Spec: VMSpec{VMID: "vm1", Network: "lan"},
142 IP: "10.77.1.2",
143 NetworkIP: "192.168.0.42",
144 }))
145
146 s2, _ := Open(dir)
147 got, err := s2.LoadVMs()
148 require.NoError(t, err)
149 require.Contains(t, got, "vm1")
150 assert.Equal(t, "10.77.1.2", got["vm1"].IP)
151 assert.Equal(t, "192.168.0.42", got["vm1"].NetworkIP)
152 assert.Equal(t, "lan", got["vm1"].Spec.Network)
153 }
154
95 func TestSerialSocketPath(t *testing.T) { 155 func TestSerialSocketPath(t *testing.T) {
96 s := open(t) 156 s := open(t)
97 p := s.SerialSocketPath("vm1") 157 p := s.SerialSocketPath("vm1")
internal/agent/syncclient/client.go
Old New
@@ -76,6 +76,12 @@ type Client struct {
76 // report. Nil, or an empty return, means "no answer" — never "no network". 76 // report. Nil, or an empty return, means "no answer" — never "no network".
77 GuestCIDR func() string 77 GuestCIDR func() string
78 78
79 // HostNetworks are the named guest networks this host is configured to
80 // serve (--host-network), advertised in Hello rather than in the report:
81 // the set changes only with the agent's command line, i.e. a restart. A
82 // host that advertises none takes no VM that asks for a network by name.
83 HostNetworks []string
84
79 // Runner executes host-introspection subprocesses (on Linux, 85 // Runner executes host-introspection subprocesses (on Linux,
80 // systemd-detect-virt via hostinfo). Injected so this data-plane package 86 // systemd-detect-virt via hostinfo). Injected so this data-plane package
81 // never imports os/exec (R6); nil is tolerated (virt reported as unknown). 87 // never imports os/exec (R6); nil is tolerated (virt reported as unknown).
@@ -336,8 +342,9 @@ func (c *Client) session(ctx context.Context) error {
336 HostId: c.Identity.HostID, Hostname: hostname, Os: runtime.GOOS, Arch: runtime.GOARCH, 342 HostId: c.Identity.HostID, Hostname: hostname, Os: runtime.GOOS, Arch: runtime.GOARCH,
337 Provisioner: c.Provisioner, 343 Provisioner: c.Provisioner,
338 LastSeenEpoch: c.St.Epoch(), Capacity: c.advertisedCapacity(stateDir), 344 LastSeenEpoch: c.St.Epoch(), Capacity: c.advertisedCapacity(stateDir),
339 Facts: helloFacts(ctx, c.Runner), 345 Facts: helloFacts(ctx, c.Runner),
340 Credential: c.Identity.Credential, 346 HostNetworks: c.HostNetworks,
347 Credential: c.Identity.Credential,
341 }}} 348 }}}
342 if err := transport.WriteMsg(up, hello); err != nil { 349 if err := transport.WriteMsg(up, hello); err != nil {
343 return classifyErr(err) 350 return classifyErr(err)
internal/agent/syncclient/client_test.go
Old New
@@ -37,6 +37,7 @@ func (noopProv) Shutdown(context.Context, string) error { r
37 func (noopProv) Destroy(context.Context, string) error { return nil } 37 func (noopProv) Destroy(context.Context, string) error { return nil }
38 func (noopProv) Running(string) bool { return false } 38 func (noopProv) Running(string) bool { return false }
39 func (noopProv) Address(string) string { return "10.77.1.2" } 39 func (noopProv) Address(string) string { return "10.77.1.2" }
40 func (noopProv) NetworkAddress(string) string { return "" }
40 func (noopProv) FailureReason(string) string { return "" } 41 func (noopProv) FailureReason(string) string { return "" }
41 42
42 // testQUICIdle is the deliberately-short idle timeout the test listeners use so 43 // testQUICIdle is the deliberately-short idle timeout the test listeners use so
@@ -231,6 +232,51 @@ func newClient(t *testing.T, addr, fp, hostID, cred string) *Client {
231 } 232 }
232 } 233 }
233 234
235 // TestHelloAdvertisesHostNetworks pins the advertisement: the named networks
236 // this agent is configured to serve ride in the Hello, which is where the
237 // control plane's admission reads them. In the handshake and not in the report
238 // because the set is this agent's command line — it changes only with a
239 // restart, and a restart is a new session.
240 func TestHelloAdvertisesHostNetworks(t *testing.T) {
241 certPEM, keyPEM, fp := genTestCert(t)
242 lis := listenTestQUIC(t, "127.0.0.1:0", certPEM, keyPEM)
243 defer lis.Close()
244
245 hellos := make(chan *pb.Hello, 1)
246 acceptCtx, stopAccept := context.WithCancel(context.Background())
247 defer stopAccept()
248 go func() {
249 conn, err := lis.Accept(acceptCtx)
250 if err != nil {
251 return
252 }
253 up, err := conn.AcceptStream(acceptCtx)
254 if err != nil {
255 return
256 }
257 var first pb.AgentMessage
258 if err := transport.ReadMsg(up, &first, transport.DefaultMaxFrame); err != nil {
259 return
260 }
261 hellos <- first.GetHello()
262 _ = conn.CloseWithError(0, "captured")
263 }()
264
265 c := newClient(t, lis.Addr().String(), fp, "host-nets", "host-nets.cred")
266 c.HostNetworks = []string{"lab", "lan"}
267 runCtx, stopRun := context.WithCancel(context.Background())
268 done := make(chan struct{})
269 go func() { defer close(done); c.Run(runCtx) }()
270 defer func() { stopRun(); <-done }()
271
272 select {
273 case h := <-hellos:
274 assert.Equal(t, []string{"lab", "lan"}, h.GetHostNetworks())
275 case <-time.After(10 * time.Second):
276 t.Fatal("no Hello arrived")
277 }
278 }
279
234 func TestReconnectAfterDrop(t *testing.T) { 280 func TestReconnectAfterDrop(t *testing.T) {
235 h := newServerHarness(t) 281 h := newServerHarness(t)
236 hostID, cred := h.enroll() 282 hostID, cred := h.enroll()
internal/agent/vfkit/leases.go
Old New
@@ -32,6 +32,14 @@ func (p *Provisioner) Address(vmID string) string {
32 return leaseAddress(string(raw), state.MAC(vmID)) 32 return leaseAddress(string(raw), state.MAC(vmID))
33 } 33 }
34 34
35 // NetworkAddress is always "" on this backend: a Mac host advertises no named
36 // guest networks (bridged attachment needs the restricted
37 // com.apple.vm.networking entitlement), so no VM placed here can have a second
38 // NIC to report an address for. Answering "" is not a stub — it is the whole
39 // truth for this platform, and the create-time refusal upstream is what keeps
40 // it from ever being asked a different question.
41 func (p *Provisioner) NetworkAddress(string) string { return "" }
42
35 // leaseAddress finds the address most recently leased to mac in the contents 43 // leaseAddress finds the address most recently leased to mac in the contents
36 // of macOS's dhcpd_leases file, or "" if there is none. The file is a series 44 // of macOS's dhcpd_leases file, or "" if there is none. The file is a series
37 // of brace-delimited stanzas of key=value lines: 45 // of brace-delimited stanzas of key=value lines:
internal/mcpserver/api_test.go
Old New
@@ -75,7 +75,7 @@ func TestFirstEligibleHost(t *testing.T) {
75 {"id": "h1", "name": "mewtwo", "online": true, "agent_version": "v0.0.5"}, 75 {"id": "h1", "name": "mewtwo", "online": true, "agent_version": "v0.0.5"},
76 }) 76 })
77 }) 77 })
78 h, err := c.FirstEligibleHost(t.Context()) 78 h, err := c.FirstEligibleHost(t.Context(), "")
79 require.NoError(t, err) 79 require.NoError(t, err)
80 assert.Equal(t, "h1", h.ID) 80 assert.Equal(t, "h1", h.ID)
81 } 81 }
@@ -92,7 +92,7 @@ func TestFirstEligibleHostSkipsPreCSRAgents(t *testing.T) {
92 {"id": "h3", "name": "mewtwo", "online": true, "agent_version": "v0.0.4"}, 92 {"id": "h3", "name": "mewtwo", "online": true, "agent_version": "v0.0.4"},
93 }) 93 })
94 }) 94 })
95 h, err := c.FirstEligibleHost(t.Context()) 95 h, err := c.FirstEligibleHost(t.Context(), "")
96 require.NoError(t, err) 96 require.NoError(t, err)
97 assert.Equal(t, "h3", h.ID) 97 assert.Equal(t, "h3", h.ID)
98 } 98 }
@@ -106,7 +106,7 @@ func TestFirstEligibleHostReportsWhyNoneAreEligible(t *testing.T) {
106 {"id": "h1", "name": "older", "online": true, "agent_version": "v0.0.1"}, 106 {"id": "h1", "name": "older", "online": true, "agent_version": "v0.0.1"},
107 }) 107 })
108 }) 108 })
109 _, err := c.FirstEligibleHost(t.Context()) 109 _, err := c.FirstEligibleHost(t.Context(), "")
110 require.Error(t, err) 110 require.Error(t, err)
111 assert.Contains(t, err.Error(), "2 online host(s)") 111 assert.Contains(t, err.Error(), "2 online host(s)")
112 assert.Contains(t, err.Error(), "predate certified host keys (v0.0.4)") 112 assert.Contains(t, err.Error(), "predate certified host keys (v0.0.4)")
@@ -114,6 +114,60 @@ func TestFirstEligibleHostReportsWhyNoneAreEligible(t *testing.T) {
114 assert.NotContains(t, err.Error(), "no online hosts") 114 assert.NotContains(t, err.Error(), "no online hosts")
115 } 115 }
116 116
117 // Asking for a named network narrows placement to hosts that advertise it: the
118 // first eligible host is only the default when nothing further was asked for.
119 func TestFirstEligibleHostPrefersOneAdvertisingTheNetwork(t *testing.T) {
120 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {
121 json.NewEncoder(w).Encode([]map[string]any{
122 {"id": "h1", "name": "onyx", "online": true, "agent_version": "v0.0.6"},
123 {"id": "h2", "name": "mewtwo", "online": true, "agent_version": "v0.0.6", "host_networks": []string{"lan"}},
124 })
125 })
126 h, err := c.FirstEligibleHost(t.Context(), "lan")
127 require.NoError(t, err)
128 assert.Equal(t, "h2", h.ID)
129
130 // Without the ask, the first eligible host still wins.
131 h, err = c.FirstEligibleHost(t.Context(), "")
132 require.NoError(t, err)
133 assert.Equal(t, "h1", h.ID)
134 }
135
136 // The refusal has to teach: an agent that cannot see the fleet learns the
137 // network names from here, and the operator learns the flag that adds one.
138 func TestFirstEligibleHostForAnUnservedNetworkNamesTheServedOnes(t *testing.T) {
139 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {
140 json.NewEncoder(w).Encode([]map[string]any{
141 {"id": "h1", "name": "onyx", "online": true, "agent_version": "v0.0.6", "host_networks": []string{"lan"}},
142 {"id": "h2", "name": "gone", "online": false, "agent_version": "v0.0.6", "host_networks": []string{"storage"}},
143 {"id": "h3", "name": "ancient", "online": true, "agent_version": "v0.0.3", "host_networks": []string{"storage"}},
144 })
145 })
146 _, err := c.FirstEligibleHost(t.Context(), "storage")
147 require.Error(t, err)
148 assert.Contains(t, err.Error(), `no eligible host advertises network "storage"`)
149 assert.Contains(t, err.Error(), "onyx: lan")
150 assert.NotContains(t, err.Error(), "gone", "an offline host's advertised networks are stale registry state, not an offer")
151 assert.NotContains(t, err.Error(), "ancient",
152 "a host that would refuse the create over its agent version is not an offer either — naming it sends the caller to configure a bridge when the remedy is an upgrade")
153 assert.Contains(t, err.Error(), "--host-network storage=<bridge>")
154 }
155
156 // An agent too old to certify host keys is ruled out before its networks are
157 // ever consulted, and the refusal is the upgrade one: the remedy is the agent,
158 // not the bridge.
159 func TestFirstEligibleHostReportsThePreCSRAgentEvenWhenANetworkWasAsked(t *testing.T) {
160 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {
161 json.NewEncoder(w).Encode([]map[string]any{
162 {"id": "h1", "name": "old", "online": true, "agent_version": "v0.0.3", "host_networks": []string{"lan"}},
163 })
164 })
165 _, err := c.FirstEligibleHost(t.Context(), "lan")
166 require.Error(t, err)
167 assert.Contains(t, err.Error(), "predate certified host keys")
168 assert.NotContains(t, err.Error(), "advertises network")
169 }
170
117 // A fleet with nothing up still says exactly that. 171 // A fleet with nothing up still says exactly that.
118 func TestFirstEligibleHostWithNothingOnline(t *testing.T) { 172 func TestFirstEligibleHostWithNothingOnline(t *testing.T) {
119 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) { 173 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {
@@ -121,7 +175,7 @@ func TestFirstEligibleHostWithNothingOnline(t *testing.T) {
121 {"id": "h0", "name": "down", "online": false, "agent_version": "v0.0.5"}, 175 {"id": "h0", "name": "down", "online": false, "agent_version": "v0.0.5"},
122 }) 176 })
123 }) 177 })
124 _, err := c.FirstEligibleHost(t.Context()) 178 _, err := c.FirstEligibleHost(t.Context(), "")
125 require.ErrorContains(t, err, "no online hosts") 179 require.ErrorContains(t, err, "no online hosts")
126 } 180 }
127 181
internal/mcpserver/server.go
Old New
@@ -58,7 +58,13 @@ type DelegateCompleteIn struct {
58 func NewServer(t *Tools, opts Options) *mcp.Server { 58 func NewServer(t *Tools, opts Options) *mcp.Server {
59 s := mcp.NewServer(&mcp.Implementation{Name: "eitri", Version: "0.1.0"}, 59 s := mcp.NewServer(&mcp.Implementation{Name: "eitri", Version: "0.1.0"},
60 &mcp.ServerOptions{SchemaCache: opts.SchemaCache}) 60 &mcp.ServerOptions{SchemaCache: opts.SchemaCache})
61 register(s, "vm_create", "Create an eitri VM. Waits for ready+cloud-init by default.", t.VMCreate) 61 register(s, "vm_create",
62 "Create an eitri VM. Waits for ready+cloud-init by default. Name a network to also put the guest on one "+
63 "of its host's named networks, where that network's own DHCP addresses it: the result reports that "+
64 "address as network_ip, which may still be EMPTY when the tool returns — readiness is the guest booting, "+
65 "and a lease from the site's DHCP server arrives when it arrives. Read it back with vm_info rather "+
66 "than treating an empty one as a failed create.",
67 t.VMCreate)
62 register(s, "vm_list", "List all VMs on the eitri fleet.", t.VMList) 68 register(s, "vm_list", "List all VMs on the eitri fleet.", t.VMList)
63 register(s, "vm_info", "Show one VM's state and how to reach it.", t.VMInfo) 69 register(s, "vm_info", "Show one VM's state and how to reach it.", t.VMInfo)
64 register(s, "vm_exec", "Run a shell command in a VM over SSH; returns stdout/stderr/exit code.", t.VMExec) 70 register(s, "vm_exec", "Run a shell command in a VM over SSH; returns stdout/stderr/exit code.", t.VMExec)
internal/mcpserver/server_test.go
Old New
@@ -2,6 +2,7 @@ package mcpserver
2 2
3 import ( 3 import (
4 "context" 4 "context"
5 "encoding/json"
5 "errors" 6 "errors"
6 "testing" 7 "testing"
7 8
@@ -190,3 +191,35 @@ func TestCAUploadDescriptionNamesTheFrozenCASet(t *testing.T) {
190 assert.Contains(t, desc, "create new VMs after uploading") 191 assert.Contains(t, desc, "create new VMs after uploading")
191 assert.Contains(t, desc, "before vm_create") 192 assert.Contains(t, desc, "before vm_create")
192 } 193 }
194
195 // A guest on a named host network is addressed by a DHCP server eitri does not
196 // run, so vm_create can return ready with network_ip still empty. A model that
197 // has not been told reads that as a broken create and destroys a working VM, so
198 // the tool's own description says it, and says where to look instead.
199 func TestVMCreateDescriptionSaysTheNetworkAddressCanLagReady(t *testing.T) {
200 desc := describe(t, connect(t, NewServer(&Tools{}, Options{})), "vm_create")
201 require.NotEmpty(t, desc)
202 assert.Contains(t, desc, "network_ip")
203 assert.Contains(t, desc, "EMPTY")
204 assert.Contains(t, desc, "vm_info")
205 }
206
207 // The network field's own description has to carry the rule the control plane
208 // enforces: only names the host advertises, and no silent fallback to NAT for
209 // one it does not.
210 func TestVMCreateNetworkFieldDescribesWhatAHostWillAccept(t *testing.T) {
211 res, err := connect(t, NewServer(&Tools{}, Options{})).ListTools(t.Context(), nil)
212 require.NoError(t, err)
213 var schema string
214 for _, tool := range res.Tools {
215 if tool.Name == "vm_create" {
216 props, err := json.Marshal(tool.InputSchema)
217 require.NoError(t, err)
218 schema = string(props)
219 }
220 }
221 require.NotEmpty(t, schema)
222 assert.Contains(t, schema, "network")
223 assert.Contains(t, schema, "advertises")
224 assert.Contains(t, schema, "NAT underlay")
225 }
internal/mcpserver/tools.go
Old New
@@ -6,6 +6,7 @@ import (
6 "fmt" 6 "fmt"
7 "io/fs" 7 "io/fs"
8 "net" 8 "net"
9 "slices"
9 "strconv" 10 "strconv"
10 "strings" 11 "strings"
11 "time" 12 "time"
@@ -23,7 +24,7 @@ type api interface {
23 CreateVM(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) 24 CreateVM(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error)
24 DeleteVM(ctx context.Context, id string) error 25 DeleteVM(ctx context.Context, id string) error
25 ListHosts(ctx context.Context) ([]client.Host, error) 26 ListHosts(ctx context.Context) ([]client.Host, error)
26 FirstEligibleHost(ctx context.Context) (client.Host, error) 27 FirstEligibleHost(ctx context.Context, network string) (client.Host, error)
27 CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) 28 CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error)
28 ListExposures(ctx context.Context, vmID string) ([]client.Exposure, error) 29 ListExposures(ctx context.Context, vmID string) ([]client.Exposure, error)
29 DeleteExposure(ctx context.Context, id string) error 30 DeleteExposure(ctx context.Context, id string) error
@@ -64,31 +65,67 @@ func (a API) RegisterUserCA(ctx context.Context, caLine, label string) error {
64 } 65 }
65 66
66 // FirstEligibleHost returns the first host a VM can be placed on — the default 67 // FirstEligibleHost returns the first host a VM can be placed on — the default
67 // target when the caller doesn't name one. Eligible means online AND running an 68 // target when the caller doesn't name one. network, when non-empty, is the
68 // agent that certifies its guests' host keys: a VM created anywhere else is 69 // named host network the VM asked for, and narrows the field to hosts serving
69 // unreachable through the gate, and the control plane refuses the create, so 70 // it.
70 // such a host is no more a candidate than one that is down. Ordering is 71 func (a API) FirstEligibleHost(ctx context.Context, network string) (client.Host, error) {
71 // server-defined; callers must not assume stability across calls.
72 //
73 // When every online host is ruled out that way, the refusal says so — "no
74 // online hosts" would send the caller looking for hardware that is in fact up
75 // and one upgrade away from usable.
76 func (a API) FirstEligibleHost(ctx context.Context) (client.Host, error) {
77 hosts, err := a.ListHosts(ctx) 72 hosts, err := a.ListHosts(ctx)
78 if err != nil { 73 if err != nil {
79 return client.Host{}, err 74 return client.Host{}, err
80 } 75 }
81 precsr := 0 76 return eligibleHost(hosts, network)
77 }
78
79 // placeable reports whether a VM can be put on this host at all: it is online
80 // AND runs an agent that certifies its guests' host keys. A VM created anywhere
81 // else is unreachable through the gate, and the control plane refuses the
82 // create, so such a host is no more a candidate than one that is down.
83 //
84 // It is one predicate because two callers must agree: the rule that picks a
85 // host and the message that lists the alternatives. A refusal naming a host
86 // placement would not have chosen sends the caller after the wrong remedy.
87 func placeable(h client.Host) bool {
88 return h.Online && release.CertifiesGuestHostKeys(h.AgentVersion)
89 }
90
91 // eligibleHost is the placement rule itself, over a fleet already read. It is a
92 // function of the listing alone so the tools' fake obeys the same rule the real
93 // client does, rather than a second, kinder one.
94 //
95 // A named network narrows the placeable hosts further — one that does not
96 // advertise it would refuse the create too. Ordering is server-defined; callers
97 // must not assume stability across calls.
98 //
99 // Each way of ruling every host out has its own refusal, because each has its
100 // own remedy. "no online hosts" would send a caller whose fleet is up and one
101 // upgrade away from usable looking for dead hardware; and it would tell a
102 // caller who asked for a network nothing about which hosts carry one.
103 func eligibleHost(hosts []client.Host, network string) (client.Host, error) {
104 precsr, unnetworked := 0, 0
82 for _, h := range hosts { 105 for _, h := range hosts {
83 if !h.Online { 106 if !placeable(h) {
107 // Of the hosts that cannot take a VM, the ones that are UP are the
108 // ones an upgrade would fix; a host that is down needs nothing said
109 // about its agent.
110 if h.Online {
111 precsr++
112 }
84 continue 113 continue
85 } 114 }
86 if !release.CertifiesGuestHostKeys(h.AgentVersion) { 115 if network != "" && !slices.Contains(h.HostNetworks, network) {
87 precsr++ 116 unnetworked++
88 continue 117 continue
89 } 118 }
90 return h, nil 119 return h, nil
91 } 120 }
121 // The network refusal comes first when both apply: it answers the question
122 // the caller actually asked, and it names the fleet's networks, which is the
123 // one place an MCP caller can learn them without provoking a 409.
124 if unnetworked > 0 {
125 return client.Host{}, fmt.Errorf("no eligible host advertises network %q (online hosts advertise: %s); "+
126 "name a host that does, restart an agent with --host-network %s=<bridge>, or omit network for the NAT underlay",
127 network, describeHostNetworks(hosts), network)
128 }
92 if precsr > 0 { 129 if precsr > 0 {
93 return client.Host{}, fmt.Errorf("no eligible hosts: %d online host(s) run agents that predate certified host keys (%s), "+ 130 return client.Host{}, fmt.Errorf("no eligible hosts: %d online host(s) run agents that predate certified host keys (%s), "+
94 "so a VM created on them could not be verified; upgrade an agent (the console's upgrade button, or "+ 131 "so a VM created on them could not be verified; upgrade an agent (the console's upgrade button, or "+
@@ -137,13 +174,25 @@ type VMCreateIn struct {
137 MemMB int64 `json:"mem_mb,omitempty" jsonschema:"default 2048"` 174 MemMB int64 `json:"mem_mb,omitempty" jsonschema:"default 2048"`
138 DiskGB int64 `json:"disk_gb,omitempty" jsonschema:"default 10"` 175 DiskGB int64 `json:"disk_gb,omitempty" jsonschema:"default 10"`
139 CloudInit string `json:"cloud_init,omitempty" jsonschema:"optional user cloud-init"` 176 CloudInit string `json:"cloud_init,omitempty" jsonschema:"optional user cloud-init"`
177 Network string `json:"network,omitempty" jsonschema:"named host network to give the guest a SECOND NIC on, addressed by that network's own DHCP; omit for the private NAT underlay every guest has anyway. A host serves only the names it advertises: naming another refuses the create and names the ones it does serve"`
140 Wait *bool `json:"wait,omitempty" jsonschema:"wait for ready+cloud-init (default true)"` 178 Wait *bool `json:"wait,omitempty" jsonschema:"wait for ready+cloud-init (default true)"`
141 } 179 }
142 180
143 type VMCreateOut struct { 181 type VMCreateOut struct {
144 ID string `json:"id"` 182 ID string `json:"id"`
145 Name string `json:"name"` 183 Name string `json:"name"`
146 IP string `json:"ip,omitempty"` 184 // IP is the guest's address on its host's private fabric — the one the ssh
185 // command reaches it at, and never routable off that host.
186 IP string `json:"ip,omitempty"`
187 // Network is the named host network the guest was created on, echoed back
188 // because reaching this field at all means admission accepted the name.
189 Network string `json:"network,omitempty"`
190 // NetworkIP is the address that network's own DHCP granted the second NIC —
191 // what the rest of that network knows this guest by. Empty when no network
192 // was asked for, and empty at ready when the site's DHCP has not answered
193 // yet: readiness is boot, and the lease arrives when it arrives. Read it
194 // back with vm_info.
195 NetworkIP string `json:"network_ip,omitempty"`
147 SSHCommand string `json:"ssh_command,omitempty"` 196 SSHCommand string `json:"ssh_command,omitempty"`
148 // CloudInit warns about a non-clean-but-usable boot: set to a degraded 197 // CloudInit warns about a non-clean-but-usable boot: set to a degraded
149 // message when cloud-init finished with recoverable errors (exit 2), omitted 198 // message when cloud-init finished with recoverable errors (exit 2), omitted
@@ -158,6 +207,7 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error
158 VCPUs: in.VCPUs, 207 VCPUs: in.VCPUs,
159 MemMB: in.MemMB, 208 MemMB: in.MemMB,
160 DiskGB: in.DiskGB, 209 DiskGB: in.DiskGB,
210 Network: in.Network,
161 } 211 }
162 if req.Name == "" { 212 if req.Name == "" {
163 req.Name = "claude-" + random.Hex(3) 213 req.Name = "claude-" + random.Hex(3)
@@ -166,6 +216,14 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error
166 // the control plane owns those defaults (types.Default*), so a zero here 216 // the control plane owns those defaults (types.Default*), so a zero here
167 // inherits the one named value rather than pinning a second, drift-prone set 217 // inherits the one named value rather than pinning a second, drift-prone set
168 // on this side. 218 // on this side.
219 //
220 // A caller who named a host gets that host, network or not: the control
221 // plane owns whether that pairing is legal and refuses it in one message
222 // naming what the host does advertise. Re-deciding it here would put a
223 // second, drifting copy of the rule in front of the first. Only the DEFAULT
224 // placement — where nothing has been named and this side is choosing —
225 // takes the network into account, because choosing a host that cannot serve
226 // it would manufacture that refusal out of a free choice.
169 if in.Host != "" { 227 if in.Host != "" {
170 hostID, err := t.resolveHost(ctx, in.Host) 228 hostID, err := t.resolveHost(ctx, in.Host)
171 if err != nil { 229 if err != nil {
@@ -173,7 +231,7 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error
173 } 231 }
174 req.HostID = hostID 232 req.HostID = hostID
175 } else { 233 } else {
176 h, err := t.API.FirstEligibleHost(ctx) 234 h, err := t.API.FirstEligibleHost(ctx, in.Network)
177 if err != nil { 235 if err != nil {
178 return VMCreateOut{}, err 236 return VMCreateOut{}, err
179 } 237 }
@@ -184,7 +242,9 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error
184 if err != nil { 242 if err != nil {
185 return VMCreateOut{}, fmt.Errorf("create vm: %w", err) 243 return VMCreateOut{}, fmt.Errorf("create vm: %w", err)
186 } 244 }
187 out := VMCreateOut{ID: created.ID, Name: created.Name} 245 // The network is echoed from the accepted request rather than waited for:
246 // it is frozen at create, and getting this far means admission took it.
247 out := VMCreateOut{ID: created.ID, Name: created.Name, Network: req.Network}
188 if in.Wait != nil && !*in.Wait { 248 if in.Wait != nil && !*in.Wait {
189 return out, nil 249 return out, nil
190 } 250 }
@@ -192,12 +252,23 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error
192 // One shared deadline bounds the whole wait (ready + cloud-init) by 252 // One shared deadline bounds the whole wait (ready + cloud-init) by
193 // WaitTimeout, rather than letting each phase burn a full budget. 253 // WaitTimeout, rather than letting each phase burn a full budget.
194 deadline := time.Now().Add(t.waitTimeout()) 254 deadline := time.Now().Add(t.waitTimeout())
195 ip, err := t.waitReady(ctx, created.ID, created.Name, deadline) 255 ready, err := t.waitReady(ctx, created.ID, created.Name, deadline)
196 if err != nil { 256 if err != nil {
197 // Spec: report state, never auto-destroy — the VM may just be slow. 257 // Spec: report state, never auto-destroy — the VM may just be slow.
198 return out, err 258 return out, err
199 } 259 }
260 ip := ready.AssignedIP
200 out.IP = ip 261 out.IP = ip
262 // Once there is a server record, it outranks the echo above: the echo is
263 // what was ASKED for, and would quietly lie the day the server normalizes a
264 // stored name. The echo still stands alone on the no-wait path, where no
265 // record has been read.
266 if ready.Network != "" {
267 out.Network = ready.Network
268 }
269 // Whatever the named network has granted BY NOW, which may be nothing: the
270 // wait above never held for it.
271 out.NetworkIP = ready.NetworkIP
201 out.SSHCommand = t.sshCommand(ctx, created.Name) 272 out.SSHCommand = t.sshCommand(ctx, created.Name)
202 // "ready" means cloud-hypervisor is up and the IP is ALLOCATED — NOT that the 273 // "ready" means cloud-hypervisor is up and the IP is ALLOCATED — NOT that the
203 // guest has booted Linux, brought up its NIC, and started sshd. The first SSH 274 // guest has booted Linux, brought up its NIC, and started sshd. The first SSH
@@ -256,11 +327,18 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error
256 } 327 }
257 328
258 // waitReady polls until the VM reaches "ready" with an IP, or the shared 329 // waitReady polls until the VM reaches "ready" with an IP, or the shared
259 // deadline expires. Transient ListVMs failures (control-plane restart/blip/ 330 // deadline expires, and hands back the listing that satisfied it. Transient
260 // 5xx) do NOT abort the wait — they are stashed and polling continues, per the 331 // ListVMs failures (control-plane restart/blip/5xx) do NOT abort the wait —
261 // tool's "do not assume failure" contract. Only real cancellation (ctx.Done) 332 // they are stashed and polling continues, per the tool's "do not assume
262 // aborts immediately. 333 // failure" contract. Only real cancellation (ctx.Done) aborts immediately.
263 func (t *Tools) waitReady(ctx context.Context, id, name string, deadline time.Time) (string, error) { 334 //
335 // The IP waited for is assigned_ip, the host-fabric address, and only that one.
336 // A guest on a named host network gets its second address from that network's
337 // own DHCP server, which is not eitri's and owes eitri no schedule; holding
338 // readiness open for it would make a working guest look like a failed create
339 // because a site's DHCP was slow. So network_ip travels back as whatever it is
340 // at ready, empty included.
341 func (t *Tools) waitReady(ctx context.Context, id, name string, deadline time.Time) (client.VM, error) {
264 last, detail := "", "" 342 last, detail := "", ""
265 sawListing := false 343 sawListing := false
266 var lastErr error 344 var lastErr error
@@ -276,21 +354,21 @@ func (t *Tools) waitReady(ctx context.Context, id, name string, deadline time.Ti
276 } 354 }
277 last, detail = vm.Lifecycle, vm.StatusDetail 355 last, detail = vm.Lifecycle, vm.StatusDetail
278 if vm.Lifecycle == "ready" && vm.AssignedIP != "" { 356 if vm.Lifecycle == "ready" && vm.AssignedIP != "" {
279 return vm.AssignedIP, nil 357 return vm, nil
280 } 358 }
281 } 359 }
282 } 360 }
283 reportProgress(ctx, fmt.Sprintf("creating vm %s: %s", name, lifecycleOrUnknown(last, detail))) 361 reportProgress(ctx, fmt.Sprintf("creating vm %s: %s", name, lifecycleOrUnknown(last, detail)))
284 select { 362 select {
285 case <-ctx.Done(): 363 case <-ctx.Done():
286 return "", ctx.Err() 364 return client.VM{}, ctx.Err()
287 case <-time.After(t.pollEvery()): 365 case <-time.After(t.pollEvery()):
288 } 366 }
289 } 367 }
290 if sawListing { 368 if sawListing {
291 return "", fmt.Errorf("vm %s (%s) not ready after %s (last lifecycle %q); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), last) 369 return client.VM{}, fmt.Errorf("vm %s (%s) not ready after %s (last lifecycle %q); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), last)
292 } 370 }
293 return "", fmt.Errorf("vm %s (%s) not ready after %s (control-plane never listed successfully; last error: %v); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), lastErr) 371 return client.VM{}, fmt.Errorf("vm %s (%s) not ready after %s (control-plane never listed successfully; last error: %v); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), lastErr)
294 } 372 }
295 373
296 // lifecycleOrUnknown words a lifecycle for a progress message, covering the 374 // lifecycleOrUnknown words a lifecycle for a progress message, covering the
@@ -349,6 +427,37 @@ func describeHosts(hosts []client.Host) string {
349 return strings.Join(parts, ", ") 427 return strings.Join(parts, ", ")
350 } 428 }
351 429
430 // describeHostNetworks renders what each host a VM could actually be placed on
431 // advertises, as "onyx: lan, dmz; mewtwo: none".
432 //
433 // It lists exactly the hosts eligibleHost would consider, by asking the same
434 // predicate rather than by keeping a matching copy of it. Both exclusions earn
435 // their place: an offline host would be listed as serving nothing, but
436 // advertised networks are registry state emptied when a host stops reporting,
437 // so "none" for a machine that is down is a claim nobody asked it; and a host
438 // whose agent predates certified host keys would be recommended here only to
439 // refuse the create for an unrelated reason, sending the caller to configure a
440 // bridge when the remedy is an upgrade.
441 func describeHostNetworks(hosts []client.Host) string {
442 parts := make([]string, 0, len(hosts))
443 for _, h := range hosts {
444 if !placeable(h) {
445 continue
446 }
447 names := "none"
448 if len(h.HostNetworks) > 0 {
449 names = strings.Join(h.HostNetworks, ", ")
450 }
451 parts = append(parts, fmt.Sprintf("%s: %s", h.Name, names))
452 }
453 if len(parts) == 0 {
454 // Defensive: eligibleHost only asks after a host has passed both checks,
455 // so there is always at least one entry on that path.
456 return "no host is eligible to place on"
457 }
458 return strings.Join(parts, "; ")
459 }
460
352 func (t *Tools) sshCommand(ctx context.Context, name string) string { 461 func (t *Tools) sshCommand(ctx context.Context, name string) string {
353 if t.Gate == "" { 462 if t.Gate == "" {
354 // No gate: reach the VM directly by name/IP (a different deployment 463 // No gate: reach the VM directly by name/IP (a different deployment
internal/mcpserver/tools_test.go
Old New
@@ -17,6 +17,7 @@ import (
17 "golang.org/x/crypto/ssh" 17 "golang.org/x/crypto/ssh"
18 18
19 "github.com/a73x/eitri/internal/server/api/client" 19 "github.com/a73x/eitri/internal/server/api/client"
20 "github.com/a73x/eitri/internal/server/release"
20 ) 21 )
21 22
22 type fakeToolsAPI struct { 23 type fakeToolsAPI struct {
@@ -30,6 +31,11 @@ type fakeToolsAPI struct {
30 // number of leading ListVMs calls that fail (control-plane blip); after 31 // number of leading ListVMs calls that fail (control-plane blip); after
31 // they are exhausted the phases sequence takes over. 32 // they are exhausted the phases sequence takes over.
32 listErrs int 33 listErrs int
34 // what the scripted VM reports about its named network once ready: the
35 // network it was created on, and the address that network's DHCP granted
36 // (empty for a lease that has not landed).
37 network string
38 networkIP string
33 // exposures the fake serves per VM id, and the exposure ids revoked. 39 // exposures the fake serves per VM id, and the exposure ids revoked.
34 exposures map[string][]client.Exposure 40 exposures map[string][]client.Exposure
35 revoked []string 41 revoked []string
@@ -46,14 +52,20 @@ type fakeToolsAPI struct {
46 } 52 }
47 53
48 // fleet is the fake's host list, defaulted so tests that don't care about 54 // fleet is the fake's host list, defaulted so tests that don't care about
49 // placement get a single online host. 55 // placement get a single online host. Its agent is current, because placement
56 // rules an older one out and these tests are not about that.
50 func (f *fakeToolsAPI) fleet() []client.Host { 57 func (f *fakeToolsAPI) fleet() []client.Host {
51 if f.hosts != nil { 58 if f.hosts != nil {
52 return f.hosts 59 return f.hosts
53 } 60 }
54 return []client.Host{{ID: "h1", Name: "mewtwo", Online: true}} 61 return []client.Host{{ID: "h1", Name: "mewtwo", Online: true, AgentVersion: currentAgent}}
55 } 62 }
56 63
64 // currentAgent is any version placement accepts: the floor is exactly the
65 // question CertifiesGuestHostKeys asks, so naming it here keeps the fixture
66 // from pinning a release literal of its own.
67 const currentAgent = release.FirstCertifiedHostKeys
68
57 func (f *fakeToolsAPI) ListVMs(ctx context.Context) ([]client.VM, error) { 69 func (f *fakeToolsAPI) ListVMs(ctx context.Context) ([]client.VM, error) {
58 if f.listErrs > 0 { 70 if f.listErrs > 0 {
59 f.listErrs-- 71 f.listErrs--
@@ -65,9 +77,10 @@ func (f *fakeToolsAPI) ListVMs(ctx context.Context) ([]client.VM, error) {
65 i = len(f.phases) - 1 77 i = len(f.phases) - 1
66 } 78 }
67 f.calls++ 79 f.calls++
68 vm := client.VM{ID: "new1", Name: "claude-abc", Lifecycle: f.phases[i]} 80 vm := client.VM{ID: "new1", Name: "claude-abc", Lifecycle: f.phases[i], Network: f.network}
69 if f.phases[i] == "ready" { 81 if f.phases[i] == "ready" {
70 vm.AssignedIP = "10.77.1.9" 82 vm.AssignedIP = "10.77.1.9"
83 vm.NetworkIP = f.networkIP
71 } 84 }
72 return append(append([]client.VM{}, f.vms...), vm), nil 85 return append(append([]client.VM{}, f.vms...), vm), nil
73 } 86 }
@@ -84,13 +97,11 @@ func (f *fakeToolsAPI) DeleteVM(ctx context.Context, id string) error {
84 func (f *fakeToolsAPI) ListHosts(ctx context.Context) ([]client.Host, error) { 97 func (f *fakeToolsAPI) ListHosts(ctx context.Context) ([]client.Host, error) {
85 return f.fleet(), nil 98 return f.fleet(), nil
86 } 99 }
87 func (f *fakeToolsAPI) FirstEligibleHost(ctx context.Context) (client.Host, error) { 100
88 for _, h := range f.fleet() { 101 // FirstEligibleHost runs the real placement rule over the fake's fleet, so a
89 if h.Online { 102 // test cannot pass against a kinder stand-in than the one in production.
90 return h, nil 103 func (f *fakeToolsAPI) FirstEligibleHost(ctx context.Context, network string) (client.Host, error) {
91 } 104 return eligibleHost(f.fleet(), network)
92 }
93 return client.Host{}, fmt.Errorf("no online hosts")
94 } 105 }
95 106
96 // CreateExposure mirrors the control plane: host port 0 is allocated from the 107 // CreateExposure mirrors the control plane: host port 0 is allocated from the
@@ -183,12 +194,32 @@ func (f *fakeRunner) ConnectName(ctx context.Context, vmName string) (string, er
183 func newTestTools(api *fakeToolsAPI, r *fakeRunner) *Tools { 194 func newTestTools(api *fakeToolsAPI, r *fakeRunner) *Tools {
184 return &Tools{ 195 return &Tools{
185 API: api, Runner: r, 196 API: api, Runner: r,
186 Gate: "localhost:2223", 197 Gate: "localhost:2223",
187 VMUser: "ubuntu", 198 VMUser: "ubuntu",
188 PollEvery: time.Millisecond, // fast tests 199 // Fast tests, and a budget that ENDS. At the 10-minute default a test
200 // whose VM never reaches the state it is waiting for does not fail —
201 // it hangs, and CI pays ten minutes to learn what the timeout error
202 // says in seconds. The two tests that assert on exhausting the budget
203 // shorten it further themselves.
204 PollEvery: time.Millisecond,
205 WaitTimeout: 5 * time.Second,
189 } 206 }
190 } 207 }
191 208
209 // Every test above hands Tools a short budget, so the shipped defaults are
210 // exercised nowhere else — and they are the ones production runs on: a poll
211 // interval and a ceiling that a real create, downloading an image over a slow
212 // link, spends minutes inside.
213 func TestUnconfiguredToolsUseTheShippedWaitBudget(t *testing.T) {
214 var zero Tools
215 assert.Equal(t, 2*time.Second, zero.pollEvery())
216 assert.Equal(t, 10*time.Minute, zero.waitTimeout())
217
218 set := Tools{PollEvery: time.Millisecond, WaitTimeout: time.Second}
219 assert.Equal(t, time.Millisecond, set.pollEvery(), "a configured interval wins over the default")
220 assert.Equal(t, time.Second, set.waitTimeout())
221 }
222
192 func TestCreateWaitsForReadyAndCloudInit(t *testing.T) { 223 func TestCreateWaitsForReadyAndCloudInit(t *testing.T) {
193 api := &fakeToolsAPI{phases: []string{"creating", "creating", "ready"}} 224 api := &fakeToolsAPI{phases: []string{"creating", "creating", "ready"}}
194 run := &fakeRunner{out: ExecResult{ExitCode: 0}} 225 run := &fakeRunner{out: ExecResult{ExitCode: 0}}
@@ -332,12 +363,13 @@ func TestCreateNoWaitReturnsImmediately(t *testing.T) {
332 } 363 }
333 364
334 // mixedFleet is a two-host fleet with a Mac second and a decommissioned-looking 365 // mixedFleet is a two-host fleet with a Mac second and a decommissioned-looking
335 // host offline, the shape placement has to get right. 366 // host offline, the shape placement has to get right. Only the Linux host
367 // advertises a named network — a Mac cannot serve one at all.
336 func mixedFleet() []client.Host { 368 func mixedFleet() []client.Host {
337 return []client.Host{ 369 return []client.Host{
338 {ID: "h1", Name: "onyx", Online: true}, 370 {ID: "h1", Name: "onyx", Online: true, AgentVersion: currentAgent, HostNetworks: []string{"lan", "dmz"}},
339 {ID: "h2", Name: "Squirtle.local", Online: true}, 371 {ID: "h2", Name: "Squirtle.local", Online: true, AgentVersion: currentAgent},
340 {ID: "h3", Name: "charmander", Online: false}, 372 {ID: "h3", Name: "charmander", Online: false, AgentVersion: currentAgent},
341 } 373 }
342 } 374 }
343 375
@@ -402,6 +434,129 @@ func TestCreateWithoutAHostTakesTheFirstOnlineOne(t *testing.T) {
402 assert.Equal(t, "h2", api.created[0].HostID) 434 assert.Equal(t, "h2", api.created[0].HostID)
403 } 435 }
404 436
437 // The network the caller names has to reach the API as a network — it is the
438 // whole request, and nothing downstream can infer it.
439 func TestCreateAsksForTheNamedNetwork(t *testing.T) {
440 api := &fakeToolsAPI{hosts: mixedFleet()}
441 tl := newTestTools(api, &fakeRunner{})
442 no := false
443
444 out, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan", Wait: &no})
445 require.NoError(t, err)
446 require.Len(t, api.created, 1)
447 assert.Equal(t, "lan", api.created[0].Network)
448 assert.Equal(t, "lan", out.Network, "the accepted network is reported back without waiting for anything")
449 }
450
451 // A create that names no network asks for none: an empty field must stay empty
452 // on the wire rather than becoming some default underlay name.
453 func TestCreateWithoutANetworkAsksForNone(t *testing.T) {
454 api := &fakeToolsAPI{hosts: mixedFleet()}
455 tl := newTestTools(api, &fakeRunner{})
456 no := false
457
458 out, err := tl.VMCreate(t.Context(), VMCreateIn{Wait: &no})
459 require.NoError(t, err)
460 require.Len(t, api.created, 1)
461 assert.Empty(t, api.created[0].Network)
462 assert.Empty(t, out.Network)
463 assert.Empty(t, out.NetworkIP)
464 }
465
466 // Both addresses come back: the host-fabric one the ssh command uses, and the
467 // one the named network's own DHCP granted.
468 func TestCreateReportsBothAddresses(t *testing.T) {
469 api := &fakeToolsAPI{hosts: mixedFleet(), phases: []string{"creating", "ready"}, network: "lan", networkIP: "192.168.0.232"}
470 tl := newTestTools(api, &fakeRunner{out: ExecResult{ExitCode: 0}})
471
472 out, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan"})
473 require.NoError(t, err)
474 assert.Equal(t, "10.77.1.9", out.IP, "the host-fabric address is still the one ssh reaches it at")
475 assert.Equal(t, "lan", out.Network)
476 assert.Equal(t, "192.168.0.232", out.NetworkIP)
477 }
478
479 // Two sources can name the network: what was asked for, and what the server
480 // recorded. Once the record exists it wins — the echo is only what was asked,
481 // and it would quietly lie the day the server normalizes a stored name.
482 func TestCreateReportsTheServersRecordOfTheNetworkOverTheAsk(t *testing.T) {
483 // The host is named, so placement does not filter on the asked-for spelling
484 // and the request reaches the server exactly as written.
485 api := &fakeToolsAPI{hosts: mixedFleet(), phases: []string{"ready"}, network: "lan", networkIP: "192.168.0.232"}
486 tl := newTestTools(api, &fakeRunner{out: ExecResult{ExitCode: 0}})
487
488 out, err := tl.VMCreate(t.Context(), VMCreateIn{Host: "onyx", Network: "LAN"})
489 require.NoError(t, err)
490 require.Len(t, api.created, 1)
491 assert.Equal(t, "LAN", api.created[0].Network, "the ask travels unaltered")
492 assert.Equal(t, "lan", out.Network, "but the answer is the server's record")
493 }
494
495 // Readiness is the guest booting. A site's DHCP server owes eitri no schedule,
496 // so a lease that has not landed by then must not hold the create open or fail
497 // it — the VM is up and usable, and network_ip is simply not known yet.
498 func TestCreateSucceedsWithTheNetworkLeaseStillOutstanding(t *testing.T) {
499 api := &fakeToolsAPI{hosts: mixedFleet(), phases: []string{"creating", "ready"}, network: "lan"} // networkIP never arrives
500 tl := newTestTools(api, &fakeRunner{out: ExecResult{ExitCode: 0}})
501
502 out, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan"})
503 require.NoError(t, err)
504 assert.Equal(t, "10.77.1.9", out.IP)
505 assert.Equal(t, "lan", out.Network)
506 assert.Empty(t, out.NetworkIP, "an unanswered lease is an empty field, not a failed create")
507 }
508
509 // Default placement is a free choice, so it is made among hosts that can
510 // actually serve what was asked for: picking the first eligible host and
511 // letting the control plane refuse it would manufacture a failure.
512 func TestCreateWithoutAHostPlacesOnOneServingTheNetwork(t *testing.T) {
513 hosts := mixedFleet()
514 hosts[0].HostNetworks = nil // onyx, first and otherwise eligible, serves none
515 hosts[1].HostNetworks = []string{"lan"} // the second one does
516 api := &fakeToolsAPI{hosts: hosts}
517 tl := newTestTools(api, &fakeRunner{})
518 no := false
519
520 _, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan", Wait: &no})
521 require.NoError(t, err)
522 require.Len(t, api.created, 1)
523 assert.Equal(t, "h2", api.created[0].HostID)
524 }
525
526 // With nothing in the fleet serving it, the refusal names what IS served —
527 // the one place an MCP caller learns the fleet's network names without
528 // provoking a refusal from the control plane.
529 func TestCreateWithNoHostForTheNetworkNamesWhatIsAdvertised(t *testing.T) {
530 api := &fakeToolsAPI{hosts: mixedFleet()}
531 tl := newTestTools(api, &fakeRunner{})
532 no := false
533
534 _, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "storage", Wait: &no})
535 require.Error(t, err)
536 assert.ErrorContains(t, err, `no eligible host advertises network "storage"`)
537 assert.ErrorContains(t, err, "onyx: lan, dmz", "the error names what the fleet DOES advertise")
538 assert.ErrorContains(t, err, "Squirtle.local: none")
539 assert.NotContains(t, err.Error(), "charmander", "an offline host has not been asked what it serves")
540 assert.ErrorContains(t, err, "--host-network storage=<bridge>", "with the remedy in the operator's own vocabulary")
541 assert.ErrorContains(t, err, "NAT underlay")
542 assert.Empty(t, api.created, "nothing is created on a host that cannot serve the request")
543 }
544
545 // A caller who names both a host and a network gets that host: the control
546 // plane owns the pairing and refuses it in one message. Re-deciding it here
547 // would put a second copy of the rule in front of the first.
548 func TestCreateOnANamedHostLeavesTheNetworkToTheControlPlane(t *testing.T) {
549 api := &fakeToolsAPI{hosts: mixedFleet()}
550 tl := newTestTools(api, &fakeRunner{})
551 no := false
552
553 _, err := tl.VMCreate(t.Context(), VMCreateIn{Host: "Squirtle.local", Network: "lan", Wait: &no})
554 require.NoError(t, err)
555 require.Len(t, api.created, 1)
556 assert.Equal(t, "h2", api.created[0].HostID)
557 assert.Equal(t, "lan", api.created[0].Network, "the request travels as asked, for the server to judge")
558 }
559
405 func TestCreateWaitTimeoutDoesNotDestroy(t *testing.T) { 560 func TestCreateWaitTimeoutDoesNotDestroy(t *testing.T) {
406 api := &fakeToolsAPI{phases: []string{"creating"}} // never ready 561 api := &fakeToolsAPI{phases: []string{"creating"}} // never ready
407 run := &fakeRunner{} 562 run := &fakeRunner{}
@@ -468,6 +623,30 @@ func TestReadFileNotReadyVMRejectedWithoutCallingRunner(t *testing.T) {
468 assert.Empty(t, run.reads, "runner must not be called on a non-ready VM") 623 assert.Empty(t, run.reads, "runner must not be called on a non-ready VM")
469 } 624 }
470 625
626 // vm_list and vm_info hand back the API's own VM, so a guest's second address
627 // is already in both. This pins that: flattening either into a hand-written
628 // view is exactly how a field goes missing from the surface that is supposed to
629 // be the programmatic one.
630 func TestListAndInfoCarryBothAddresses(t *testing.T) {
631 api := &fakeToolsAPI{vms: []client.VM{{
632 ID: "abc123", Name: "web-1", Lifecycle: "ready",
633 AssignedIP: "10.77.1.5", Network: "lan", NetworkIP: "192.168.0.232",
634 }}}
635 tl := newTestTools(api, &fakeRunner{})
636
637 listed, err := tl.VMList(t.Context(), VMListIn{})
638 require.NoError(t, err)
639 require.Len(t, listed.VMs, 1)
640 assert.Equal(t, "lan", listed.VMs[0].Network)
641 assert.Equal(t, "192.168.0.232", listed.VMs[0].NetworkIP)
642 assert.Equal(t, "10.77.1.5", listed.VMs[0].AssignedIP)
643
644 info, err := tl.VMInfo(t.Context(), VMInfoIn{VM: "web-1"})
645 require.NoError(t, err)
646 assert.Equal(t, "lan", info.VM.Network)
647 assert.Equal(t, "192.168.0.232", info.VM.NetworkIP, "vm_info is where a caller reads a lease that landed late")
648 }
649
471 // TestVMInfoHintNamespacesConnectName pins that vm_info's ssh_command hint dials 650 // TestVMInfoHintNamespacesConnectName pins that vm_info's ssh_command hint dials
472 // the gate by the <tenant>.<name> connect name (the gate rejects a bare name), 651 // the gate by the <tenant>.<name> connect name (the gate rejects a bare name),
473 // and that the gateless deployment still emits the plain `ssh <user>@<name>`. 652 // and that the gateless deployment still emits the plain `ssh <user>@<name>`.
internal/names/names.go
Old New
@@ -1,6 +1,10 @@
1 // Package names validates the DNS-label shape shared across planes: a VM's 1 // Package names validates the DNS-label shape shared across planes: a VM's
2 // name doubles as its guest hostname, so it must be a valid RFC-1123 label. 2 // name doubles as its guest hostname, so it must be a valid RFC-1123 label.
3 // This package is a dependency-free leaf. 3 // It also validates the distinct, narrower grammar of a named network's name
4 // (see IsNetworkName), and holds the one cross-plane fact about which host OS
5 // can serve a named network at all (see OSServesNamedNetworks). This package
6 // is a dependency-free leaf, which is exactly why both facts live here rather
7 // than beside either plane.
4 package names 8 package names
5 9
6 import "regexp" 10 import "regexp"
@@ -22,6 +26,44 @@ var sha256Hex = regexp.MustCompile(`^[a-f0-9]{64}$`)
22 // path or trusting the value, so it lives in this dependency-free leaf. 26 // path or trusting the value, so it lives in this dependency-free leaf.
23 func IsSHA256Hex(s string) bool { return sha256Hex.MatchString(s) } 27 func IsSHA256Hex(s string) bool { return sha256Hex.MatchString(s) }
24 28
29 // IsNetworkName reports whether s is a legal named-network name: 1–32 chars
30 // of [a-z0-9-], no leading/trailing hyphen, and not the reserved word "nat"
31 // (the anonymous default underlay's name in prose; a network named "nat"
32 // would make every conversation about it ambiguous). One home for the
33 // grammar — the API validates with it and the agent flag parser refuses
34 // with it, so the two cannot drift. The byte-wise loop below is sound for
35 // multi-byte UTF-8 input too: every legal character here is single-byte
36 // ASCII, so any byte of a multi-byte rune falls outside the allowed ranges
37 // and rejects.
38 func IsNetworkName(s string) bool {
39 if s == "" || s == "nat" || len(s) > 32 {
40 return false
41 }
42 if s[0] == '-' || s[len(s)-1] == '-' {
43 return false
44 }
45 for i := 0; i < len(s); i++ {
46 switch c := s[i]; {
47 case c >= 'a' && c <= 'z', c >= '0' && c <= '9', c == '-':
48 // allowed
49 default:
50 return false
51 }
52 }
53 return true
54 }
55
56 // OSServesNamedNetworks reports whether an agent running on goos (the value
57 // runtime.GOOS reports) can serve a named network at all: only a Linux host
58 // owns a bridge to attach a named network's taps to. This is the one home
59 // for that fact — the agent's flag parser refuses --host-network at startup
60 // with it, and the server words a create's network refusal from it using
61 // store.Host.OS (set once, from this same runtime.GOOS, at the host's
62 // enrollment — see internal/server/api/networks.go for why that source is
63 // trustworthy without a live report), so a third OS gaining support is a
64 // one-line change made once, here, rather than found and fixed twice.
65 func OSServesNamedNetworks(goos string) bool { return goos == "linux" }
66
25 // ConnectName returns a VM's connect name, "<tenant>.<name>" — the form the SSH 67 // ConnectName returns a VM's connect name, "<tenant>.<name>" — the form the SSH
26 // gate resolves and the VM's host certificate is issued for. It splits back 68 // gate resolves and the VM's host certificate is issued for. It splits back
27 // unambiguously on the first dot only because tenant ids and VM names are both 69 // unambiguously on the first dot only because tenant ids and VM names are both
internal/names/names_test.go
Old New
@@ -0,0 +1,87 @@
1 package names
2
3 import (
4 "os"
5 "path/filepath"
6 "strings"
7 "testing"
8 )
9
10 // TestIsNetworkName exercises the network-name grammar's boundaries directly
11 // against the implementation: the length limits, the leading/trailing-hyphen
12 // rule, the reserved "nat" word, and the lowercase-alnum-and-hyphen charset.
13 func TestIsNetworkName(t *testing.T) {
14 valid := []string{"lan", "lab-2", "a", strings.Repeat("x", 32)}
15 invalid := []string{"", "nat", "LAN", "l_n", "-lan", "lan-", strings.Repeat("x", 33), "br0.7"}
16 for _, s := range valid {
17 if !IsNetworkName(s) {
18 t.Errorf("IsNetworkName(%q) = false, want true", s)
19 }
20 }
21 for _, s := range invalid {
22 if IsNetworkName(s) {
23 t.Errorf("IsNetworkName(%q) = true, want false", s)
24 }
25 }
26 }
27
28 // TestDocsMatchNetworkNameGrammar is the docs-match tripwire for the network-
29 // name grammar. IsNetworkName is the one home for the rule (see its doc
30 // comment), but the agent's flag parser (internal/agent/run/cli.go) and
31 // docs/quickstart.md both restate it in prose for a human reader, in a form
32 // that cannot import the Go function. This test reads the quickstart prose
33 // and fails if its load-bearing fragments go missing, and separately proves
34 // the stated bounds still match IsNetworkName's behavior — so a change to
35 // the grammar cannot silently leave the docs describing the old rule.
36 func TestDocsMatchNetworkNameGrammar(t *testing.T) {
37 quickstart := readRepoFile(t, "docs/quickstart.md")
38 for _, want := range []string{
39 "1–32 of `[a-z0-9-]`",
40 "no leading",
41 "or trailing hyphen",
42 "`nat` is reserved",
43 } {
44 if !strings.Contains(quickstart, want) {
45 t.Errorf("docs/quickstart.md is missing %q — the network-name grammar prose drifted from IsNetworkName", want)
46 }
47 }
48
49 // Behavioral half: the bounds the prose claims (1-32 chars, no
50 // leading/trailing hyphen, "nat" reserved) must match what the code
51 // actually accepts. TestIsNetworkName already covers this in depth;
52 // these three checks pin the specific numbers the doc sentence quotes.
53 if !IsNetworkName(strings.Repeat("x", 32)) {
54 t.Error("IsNetworkName rejects a 32-char name — docs/quickstart.md's \"1–32\" would be wrong")
55 }
56 if IsNetworkName(strings.Repeat("x", 33)) {
57 t.Error("IsNetworkName accepts a 33-char name — docs/quickstart.md's \"1–32\" would be wrong")
58 }
59 if IsNetworkName("nat") {
60 t.Error("IsNetworkName accepts \"nat\" — docs/quickstart.md's \"nat is reserved\" would be wrong")
61 }
62 }
63
64 // TestOSServesNamedNetworks pins the one platform this fact is true for today
65 // and confirms it is a real comparison, not a stub that always answers true.
66 func TestOSServesNamedNetworks(t *testing.T) {
67 if !OSServesNamedNetworks("linux") {
68 t.Error("OSServesNamedNetworks(\"linux\") = false, want true")
69 }
70 for _, goos := range []string{"darwin", "windows", ""} {
71 if OSServesNamedNetworks(goos) {
72 t.Errorf("OSServesNamedNetworks(%q) = true, want false", goos)
73 }
74 }
75 }
76
77 // readRepoFile reads a repo-root-relative path, locating the root by walking
78 // up from this test file's own location so the result does not depend on the
79 // working directory the test is run from.
80 func readRepoFile(t *testing.T, rel string) string {
81 t.Helper()
82 b, err := os.ReadFile(filepath.Join("..", "..", rel))
83 if err != nil {
84 t.Fatalf("reading %s: %v", rel, err)
85 }
86 return string(b)
87 }
internal/pb/sync.pb.go
Old New
@@ -245,6 +245,11 @@ type Hello struct {
245 Capacity *Capacity `protobuf:"bytes,8,opt,name=capacity,proto3" json:"capacity,omitempty"` 245 Capacity *Capacity `protobuf:"bytes,8,opt,name=capacity,proto3" json:"capacity,omitempty"`
246 Credential string `protobuf:"bytes,9,opt,name=credential,proto3" json:"credential,omitempty"` // Bearer host credential, verified in first frame 246 Credential string `protobuf:"bytes,9,opt,name=credential,proto3" json:"credential,omitempty"` // Bearer host credential, verified in first frame
247 Facts *HostFacts `protobuf:"bytes,10,opt,name=facts,proto3" json:"facts,omitempty"` // best-effort static host identity; refreshed each Hello 247 Facts *HostFacts `protobuf:"bytes,10,opt,name=facts,proto3" json:"facts,omitempty"` // best-effort static host identity; refreshed each Hello
248 // The named guest networks this host is configured to serve
249 // (--host-network name=bridge), names only. In Hello rather than the
250 // report because the set changes only with agent configuration, i.e. a
251 // restart. Empty for a Mac, and for a Linux host with none configured.
252 HostNetworks []string `protobuf:"bytes,11,rep,name=host_networks,json=hostNetworks,proto3" json:"host_networks,omitempty"`
248 unknownFields protoimpl.UnknownFields 253 unknownFields protoimpl.UnknownFields
249 sizeCache protoimpl.SizeCache 254 sizeCache protoimpl.SizeCache
250 } 255 }
@@ -342,6 +347,13 @@ func (x *Hello) GetFacts() *HostFacts {
342 return nil 347 return nil
343 } 348 }
344 349
350 func (x *Hello) GetHostNetworks() []string {
351 if x != nil {
352 return x.HostNetworks
353 }
354 return nil
355 }
356
345 type Capacity struct { 357 type Capacity struct {
346 state protoimpl.MessageState `protogen:"open.v1"` 358 state protoimpl.MessageState `protogen:"open.v1"`
347 Vcpus int64 `protobuf:"varint,1,opt,name=vcpus,proto3" json:"vcpus,omitempty"` 359 Vcpus int64 `protobuf:"varint,1,opt,name=vcpus,proto3" json:"vcpus,omitempty"`
@@ -603,7 +615,7 @@ type ActualVM struct {
603 VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` 615 VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"`
604 Power string `protobuf:"bytes,2,opt,name=power,proto3" json:"power,omitempty"` // "running"|"stopped" 616 Power string `protobuf:"bytes,2,opt,name=power,proto3" json:"power,omitempty"` // "running"|"stopped"
605 Phase string `protobuf:"bytes,3,opt,name=phase,proto3" json:"phase,omitempty"` // "creating"|"ready"|"failed"|"quarantined" 617 Phase string `protobuf:"bytes,3,opt,name=phase,proto3" json:"phase,omitempty"` // "creating"|"ready"|"failed"|"quarantined"
606 Ip string `protobuf:"bytes,4,opt,name=ip,proto3" json:"ip,omitempty"` // the address this guest has, however its host came by it 618 Ip string `protobuf:"bytes,4,opt,name=ip,proto3" json:"ip,omitempty"` // the guest's address on its host's NAT underlay — every guest has one, from boot
607 LastError string `protobuf:"bytes,5,opt,name=last_error,json=lastError,proto3" json:"last_error,omitempty"` 619 LastError string `protobuf:"bytes,5,opt,name=last_error,json=lastError,proto3" json:"last_error,omitempty"`
608 // The guest's ed25519 HOST public key. It is generated on the host, and the 620 // The guest's ed25519 HOST public key. It is generated on the host, and the
609 // private half never leaves it — this is the only half that travels. Sent on 621 // private half never leaves it — this is the only half that travels. Sent on
@@ -618,7 +630,16 @@ type ActualVM struct {
618 // Free text for display, never parsed. Empty is the normal state of a settled 630 // Free text for display, never parsed. Empty is the normal state of a settled
619 // VM and means only "nothing to add": an agent that predates this field sends 631 // VM and means only "nothing to add": an agent that predates this field sends
620 // it never, and a console reading it reads exactly what it read before. 632 // it never, and a console reading it reads exactly what it read before.
621 StatusDetail string `protobuf:"bytes,7,opt,name=status_detail,json=statusDetail,proto3" json:"status_detail,omitempty"` 633 StatusDetail string `protobuf:"bytes,7,opt,name=status_detail,json=statusDetail,proto3" json:"status_detail,omitempty"`
634 // The address the site's DHCP server granted this guest on its SECOND NIC,
635 // the one attached to the named host network its spec asked for (see
636 // VMDesired.network). Empty for the guests that have no such NIC — the
637 // majority — and for one DHCP round-trip after a networked guest boots,
638 // because the host learns it by watching the exchange rather than granting
639 // it. Never a substitute for ip: that one is known before the guest is even
640 // running, which is why the gate path uses it and this field never appears
641 // there.
642 NetworkIp string `protobuf:"bytes,8,opt,name=network_ip,json=networkIp,proto3" json:"network_ip,omitempty"`
622 unknownFields protoimpl.UnknownFields 643 unknownFields protoimpl.UnknownFields
623 sizeCache protoimpl.SizeCache 644 sizeCache protoimpl.SizeCache
624 } 645 }
@@ -702,6 +723,13 @@ func (x *ActualVM) GetStatusDetail() string {
702 return "" 723 return ""
703 } 724 }
704 725
726 func (x *ActualVM) GetNetworkIp() string {
727 if x != nil {
728 return x.NetworkIp
729 }
730 return ""
731 }
732
705 type QuarantinedVM struct { 733 type QuarantinedVM struct {
706 state protoimpl.MessageState `protogen:"open.v1"` 734 state protoimpl.MessageState `protogen:"open.v1"`
707 VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` 735 VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"`
@@ -936,8 +964,14 @@ type VMDesired struct {
936 // the agent could not tell "the gate is off, boot uncertified" from "your 964 // the agent could not tell "the gate is off, boot uncertified" from "your
937 // certificate has not come back yet" — both are an empty ssh_host_cert. 965 // certificate has not come back yet" — both are an empty ssh_host_cert.
938 HostCertRequired bool `protobuf:"varint,19,opt,name=host_cert_required,json=hostCertRequired,proto3" json:"host_cert_required,omitempty"` 966 HostCertRequired bool `protobuf:"varint,19,opt,name=host_cert_required,json=hostCertRequired,proto3" json:"host_cert_required,omitempty"`
939 unknownFields protoimpl.UnknownFields 967 // The named host network this guest attaches to, or "" for the host's
940 sizeCache protoimpl.SizeCache 968 // NAT'd private bridge (the default, and the only behavior agents before
969 // this field know). A name is only ever placed on a host that advertised
970 // it (see Hello.host_networks); an agent that has the name but no longer
971 // the configuration fails the VM legibly rather than silently NAT-ing it.
972 Network string `protobuf:"bytes,20,opt,name=network,proto3" json:"network,omitempty"`
973 unknownFields protoimpl.UnknownFields
974 sizeCache protoimpl.SizeCache
941 } 975 }
942 976
943 func (x *VMDesired) Reset() { 977 func (x *VMDesired) Reset() {
@@ -1075,6 +1109,13 @@ func (x *VMDesired) GetHostCertRequired() bool {
1075 return false 1109 return false
1076 } 1110 }
1077 1111
1112 func (x *VMDesired) GetNetwork() string {
1113 if x != nil {
1114 return x.Network
1115 }
1116 return ""
1117 }
1118
1078 type DesiredStateSnapshot struct { 1119 type DesiredStateSnapshot struct {
1079 state protoimpl.MessageState `protogen:"open.v1"` 1120 state protoimpl.MessageState `protogen:"open.v1"`
1080 Epoch uint64 `protobuf:"varint,1,opt,name=epoch,proto3" json:"epoch,omitempty"` // agents refuse epoch < highest seen 1121 Epoch uint64 `protobuf:"varint,1,opt,name=epoch,proto3" json:"epoch,omitempty"` // agents refuse epoch < highest seen
@@ -1671,7 +1712,7 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" +
1671 "\bsnapshot\x18\x01 \x01(\v2\x1e.eitri.v1.DesiredStateSnapshotH\x00R\bsnapshot\x12:\n" + 1712 "\bsnapshot\x18\x01 \x01(\v2\x1e.eitri.v1.DesiredStateSnapshotH\x00R\bsnapshot\x12:\n" +
1672 "\fconsole_open\x18\x02 \x01(\v2\x15.eitri.v1.ConsoleOpenH\x00R\vconsoleOpen\x12.\n" + 1713 "\fconsole_open\x18\x02 \x01(\v2\x15.eitri.v1.ConsoleOpenH\x00R\vconsoleOpen\x12.\n" +
1673 "\btcp_open\x18\x03 \x01(\v2\x11.eitri.v1.TCPOpenH\x00R\atcpOpenB\x05\n" + 1714 "\btcp_open\x18\x03 \x01(\v2\x11.eitri.v1.TCPOpenH\x00R\atcpOpenB\x05\n" +
1674 "\x03msg\"\xb8\x02\n" + 1715 "\x03msg\"\xdd\x02\n" +
1675 "\x05Hello\x12\x17\n" + 1716 "\x05Hello\x12\x17\n" +
1676 "\ahost_id\x18\x01 \x01(\tR\x06hostId\x12\x1a\n" + 1717 "\ahost_id\x18\x01 \x01(\tR\x06hostId\x12\x1a\n" +
1677 "\bhostname\x18\x02 \x01(\tR\bhostname\x12\x0e\n" + 1718 "\bhostname\x18\x02 \x01(\tR\bhostname\x12\x0e\n" +
@@ -1684,7 +1725,8 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" +
1684 "credential\x18\t \x01(\tR\n" + 1725 "credential\x18\t \x01(\tR\n" +
1685 "credential\x12)\n" + 1726 "credential\x12)\n" +
1686 "\x05facts\x18\n" + 1727 "\x05facts\x18\n" +
1687 " \x01(\v2\x13.eitri.v1.HostFactsR\x05factsJ\x04\b\x06\x10\aR\vbridge_cidr\"P\n" + 1728 " \x01(\v2\x13.eitri.v1.HostFactsR\x05facts\x12#\n" +
1729 "\rhost_networks\x18\v \x03(\tR\fhostNetworksJ\x04\b\x06\x10\aR\vbridge_cidr\"P\n" +
1688 "\bCapacity\x12\x14\n" + 1730 "\bCapacity\x12\x14\n" +
1689 "\x05vcpus\x18\x01 \x01(\x03R\x05vcpus\x12\x15\n" + 1731 "\x05vcpus\x18\x01 \x01(\x03R\x05vcpus\x12\x15\n" +
1690 "\x06mem_mb\x18\x02 \x01(\x03R\x05memMb\x12\x17\n" + 1732 "\x06mem_mb\x18\x02 \x01(\x03R\x05memMb\x12\x17\n" +
@@ -1708,7 +1750,7 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" +
1708 "\fdisk_used_gb\x18\a \x01(\x03R\n" + 1750 "\fdisk_used_gb\x18\a \x01(\x03R\n" +
1709 "diskUsedGb\x12 \n" + 1751 "diskUsedGb\x12 \n" +
1710 "\fdisk_free_gb\x18\b \x01(\x03R\n" + 1752 "\fdisk_free_gb\x18\b \x01(\x03R\n" +
1711 "diskFreeGb\"\xc7\x01\n" + 1753 "diskFreeGb\"\xe6\x01\n" +
1712 "\bActualVM\x12\x13\n" + 1754 "\bActualVM\x12\x13\n" +
1713 "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x14\n" + 1755 "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x14\n" +
1714 "\x05power\x18\x02 \x01(\tR\x05power\x12\x14\n" + 1756 "\x05power\x18\x02 \x01(\tR\x05power\x12\x14\n" +
@@ -1717,7 +1759,9 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" +
1717 "\n" + 1759 "\n" +
1718 "last_error\x18\x05 \x01(\tR\tlastError\x12&\n" + 1760 "last_error\x18\x05 \x01(\tR\tlastError\x12&\n" +
1719 "\x0fssh_host_pubkey\x18\x06 \x01(\tR\rsshHostPubkey\x12#\n" + 1761 "\x0fssh_host_pubkey\x18\x06 \x01(\tR\rsshHostPubkey\x12#\n" +
1720 "\rstatus_detail\x18\a \x01(\tR\fstatusDetail\"\x81\x01\n" + 1762 "\rstatus_detail\x18\a \x01(\tR\fstatusDetail\x12\x1d\n" +
1763 "\n" +
1764 "network_ip\x18\b \x01(\tR\tnetworkIp\"\x81\x01\n" +
1721 "\rQuarantinedVM\x12\x13\n" + 1765 "\rQuarantinedVM\x12\x13\n" +
1722 "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" + 1766 "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" +
1723 "\x04name\x18\x02 \x01(\tR\x04name\x12\x1f\n" + 1767 "\x04name\x18\x02 \x01(\tR\x04name\x12\x1f\n" +
@@ -1736,7 +1780,7 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" +
1736 "guest_cidr\x18\b \x01(\tR\tguestCidr\x126\n" + 1780 "guest_cidr\x18\b \x01(\tR\tguestCidr\x126\n" +
1737 "\texposures\x18\t \x03(\v2\x18.eitri.v1.ExposureActualR\texposures\x12(\n" + 1781 "\texposures\x18\t \x03(\v2\x18.eitri.v1.ExposureActualR\texposures\x12(\n" +
1738 "\x10host_uplink_addr\x18\n" + 1782 "\x10host_uplink_addr\x18\n" +
1739 " \x01(\tR\x0ehostUplinkAddr\"\xa2\x04\n" + 1783 " \x01(\tR\x0ehostUplinkAddr\"\xbc\x04\n" +
1740 "\tVMDesired\x12\x13\n" + 1784 "\tVMDesired\x12\x13\n" +
1741 "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" + 1785 "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" +
1742 "\x04name\x18\x02 \x01(\tR\x04name\x12\x1b\n" + 1786 "\x04name\x18\x02 \x01(\tR\x04name\x12\x1b\n" +
@@ -1759,7 +1803,8 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" +
1759 "\x12ssh_authorized_key\x18\f \x01(\tR\x10sshAuthorizedKey\x12\"\n" + 1803 "\x12ssh_authorized_key\x18\f \x01(\tR\x10sshAuthorizedKey\x12\"\n" +
1760 "\rssh_host_cert\x18\x11 \x01(\tR\vsshHostCert\x12<\n" + 1804 "\rssh_host_cert\x18\x11 \x01(\tR\vsshHostCert\x12<\n" +
1761 "\x1bssh_user_ca_authorized_keys\x18\x12 \x03(\tR\x17sshUserCaAuthorizedKeys\x12,\n" + 1805 "\x1bssh_user_ca_authorized_keys\x18\x12 \x03(\tR\x17sshUserCaAuthorizedKeys\x12,\n" +
1762 "\x12host_cert_required\x18\x13 \x01(\bR\x10hostCertRequiredJ\x04\b\r\x10\x0eJ\x04\b\x0e\x10\x0fJ\x04\b\x0f\x10\x10J\x04\b\x10\x10\x11R\x10ssh_host_key_pem\"\xc9\x01\n" + 1806 "\x12host_cert_required\x18\x13 \x01(\bR\x10hostCertRequired\x12\x18\n" +
1807 "\anetwork\x18\x14 \x01(\tR\anetworkJ\x04\b\r\x10\x0eJ\x04\b\x0e\x10\x0fJ\x04\b\x0f\x10\x10J\x04\b\x10\x10\x11R\x10ssh_host_key_pem\"\xc9\x01\n" +
1763 "\x14DesiredStateSnapshot\x12\x14\n" + 1808 "\x14DesiredStateSnapshot\x12\x14\n" +
1764 "\x05epoch\x18\x01 \x01(\x04R\x05epoch\x12%\n" + 1809 "\x05epoch\x18\x01 \x01(\x04R\x05epoch\x12%\n" +
1765 "\x03vms\x18\x02 \x03(\v2\x13.eitri.v1.VMDesiredR\x03vms\x12;\n" + 1810 "\x03vms\x18\x02 \x03(\v2\x13.eitri.v1.VMDesiredR\x03vms\x12;\n" +
internal/server/api/api.go
Old New
@@ -11,6 +11,7 @@ import (
11 "fmt" 11 "fmt"
12 "log/slog" 12 "log/slog"
13 "net/http" 13 "net/http"
14 "slices"
14 "strings" 15 "strings"
15 "time" 16 "time"
16 17
@@ -429,12 +430,24 @@ func toHostResponse(h store.Host, st registry.HostState, ok bool, alloc store.Al
429 CPUModel: h.CPUModel, 430 CPUModel: h.CPUModel,
430 Virt: h.Virt, 431 Virt: h.Virt,
431 UplinkAddr: h.UplinkAddr, 432 UplinkAddr: h.UplinkAddr,
433 // A host that has advertised nothing serves an empty list, never JSON
434 // null: the field is declared non-nullable, and "no networks" is the
435 // same answer whether the host said so or has not spoken at all.
436 HostNetworks: []string{},
432 } 437 }
433 if ok { 438 if ok {
434 hr.Online = st.Online 439 hr.Online = st.Online
435 hr.Stale = st.Stale 440 hr.Stale = st.Stale
436 hr.Sessions = st.Sessions 441 hr.Sessions = st.Sessions
437 hr.AgentVersion = st.AgentVersion 442 hr.AgentVersion = st.AgentVersion
443 // Registry-held like AgentVersion, and for the same reason it is gated
444 // on `ok`: only a host that has connected has said which networks it
445 // serves, and a host this server has not heard from advertises nothing
446 // rather than whatever it once did. The nil check keeps the empty list
447 // above: a connected host that named no networks still serves [].
448 if st.HostNetworks != nil {
449 hr.HostNetworks = st.HostNetworks
450 }
438 // LastSeen unset ⇒ connected but never reported: leave the age fields 451 // LastSeen unset ⇒ connected but never reported: leave the age fields
439 // null rather than emit a bogus "last seen at the zero time". 452 // null rather than emit a bogus "last seen at the zero time".
440 if !st.LastSeen.IsZero() { 453 if !st.LastSeen.IsZero() {
@@ -577,6 +590,8 @@ func toVMResponse(vm store.VM, actualPower, phase, statusDetail string, destroyA
577 Status: vm.Status, 590 Status: vm.Status,
578 LastError: vm.LastError, 591 LastError: vm.LastError,
579 AssignedIP: vm.AssignedIP, 592 AssignedIP: vm.AssignedIP,
593 Network: vm.Network,
594 NetworkIP: vm.NetworkIP,
580 CreatedAt: vm.CreatedAt, 595 CreatedAt: vm.CreatedAt,
581 Deleted: vm.DeletedAt != nil, 596 Deleted: vm.DeletedAt != nil,
582 ActualPower: actualPower, 597 ActualPower: actualPower,
@@ -740,6 +755,13 @@ func validateCreateVM(req *types.CreateVMRequest) (string, int) {
740 if req.VCPUs < 1 || req.MemMB < 1 || req.DiskGB < 1 { 755 if req.VCPUs < 1 || req.MemMB < 1 || req.DiskGB < 1 {
741 return "vcpus, mem_mb and disk_gb must each be >= 1", http.StatusBadRequest 756 return "vcpus, mem_mb and disk_gb must each be >= 1", http.StatusBadRequest
742 } 757 }
758 // A network is optional (empty is the NAT underlay), but a name that cannot
759 // be a network name is malformed here rather than a refusal later — the
760 // grammar is names.IsNetworkName, the same one the agent's --host-network
761 // parser refuses with, so a name the API accepts is one a host could serve.
762 if req.Network != "" && !names.IsNetworkName(req.Network) {
763 return "invalid network", http.StatusBadRequest
764 }
743 return "", 0 765 return "", 0
744 } 766 }
745 767
@@ -882,6 +904,27 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) {
882 } 904 }
883 } 905 }
884 906
907 // Named-network precondition, the fourth refusal of this shape: the VM
908 // asked for a network, and only the agent's own configuration can say
909 // whether this host serves it. Advertisement arrives in the Hello, so —
910 // unlike the offline-tolerant refusals above — silence refuses: an agent
911 // that predates the field advertises nothing and would silently NAT the
912 // guest, which is the one outcome the spec forbids. A VM that asked for
913 // the LAN either gets it or does not exist.
914 //
915 // hostHasSpoken already folds in Online (see above): the server trusts no
916 // Hello-derived fact, including a network name the host may have just
917 // advertised, until the host is Online. A freshly-connected host can
918 // therefore see this refusal fire for one report cadence even though it
919 // did advertise the network — self-healing once the next report lands,
920 // and uniform with the sibling refusals this one sits beside.
921 if req.Network != "" {
922 if !hostHasSpoken || !slices.Contains(hostState.HostNetworks, req.Network) {
923 http.Error(w, noNetworkRefusal(host.Name, req.HostID, req.Network, host.OS, hostState.HostNetworks, hostHasSpoken), http.StatusConflict)
924 return
925 }
926 }
927
885 // Install the SSH key into user-supplied cloud-init. When only one of the 928 // Install the SSH key into user-supplied cloud-init. When only one of the
886 // two is set the seed builder handles it (verbatim user-data, or the 929 // two is set the seed builder handles it (verbatim user-data, or the
887 // generated default template); it's the BOTH case that used to silently 930 // generated default template); it's the BOTH case that used to silently
@@ -937,6 +980,7 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) {
937 MemMB: req.MemMB, 980 MemMB: req.MemMB,
938 DiskGB: req.DiskGB, 981 DiskGB: req.DiskGB,
939 PowerState: req.PowerState, 982 PowerState: req.PowerState,
983 Network: req.Network,
940 } 984 }
941 985
942 // The row carries no host key. A guest's host key is generated by the host 986 // The row carries no host key. A guest's host key is generated by the host
internal/server/api/api_test.go
Old New
@@ -80,7 +80,7 @@ func TestResponseJSONKeysAreSnakeCase(t *testing.T) {
80 for _, k := range []string{ 80 for _, k := range []string{
81 "id", "host_id", "name", "image_url", "vcpus", "mem_mb", "disk_gb", 81 "id", "host_id", "name", "image_url", "vcpus", "mem_mb", "disk_gb",
82 "power_state", "status", "status_detail", "last_error", "assigned_ip", 82 "power_state", "status", "status_detail", "last_error", "assigned_ip",
83 "created_at", "deleted", "actual_power", "phase", 83 "network", "network_ip", "created_at", "deleted", "actual_power", "phase",
84 "destroy_at", "lifecycle", "trusted_cas", "injected_key", 84 "destroy_at", "lifecycle", "trusted_cas", "injected_key",
85 } { 85 } {
86 assert.Contains(t, v, k, "vm response must contain key %q", k) 86 assert.Contains(t, v, k, "vm response must contain key %q", k)
@@ -397,12 +397,20 @@ func enroll(t *testing.T, ts *httptest.Server) map[string]string {
397 // host in the fleet for a moment after the server restarts. 397 // host in the fleet for a moment after the server restarts.
398 func enrollSilent(t *testing.T, ts *httptest.Server) map[string]string { 398 func enrollSilent(t *testing.T, ts *httptest.Server) map[string]string {
399 t.Helper() 399 t.Helper()
400 return enrollSilentOS(t, ts, "linux")
401 }
402
403 // enrollSilentOS is enrollSilent with the host's reported OS as a parameter,
404 // for tests that need a host whose OS itself is the refusal — a Mac cannot
405 // serve a named network no matter what it later advertises.
406 func enrollSilentOS(t *testing.T, ts *httptest.Server, os string) map[string]string {
407 t.Helper()
400 resp := do(t, "POST", ts.URL+"/api/v1/enroll-tokens", testPAT, nil) 408 resp := do(t, "POST", ts.URL+"/api/v1/enroll-tokens", testPAT, nil)
401 require.Equal(t, 201, resp.StatusCode) 409 require.Equal(t, 201, resp.StatusCode)
402 var tok map[string]string 410 var tok map[string]string
403 json.NewDecoder(resp.Body).Decode(&tok) 411 json.NewDecoder(resp.Body).Decode(&tok)
404 resp = do(t, "POST", ts.URL+"/api/v1/enroll", "", map[string]string{ 412 resp = do(t, "POST", ts.URL+"/api/v1/enroll", "", map[string]string{
405 "token": tok["token"], "name": "host-a", "os": "linux", "arch": "amd64", "provisioner": "cloudhv"}) 413 "token": tok["token"], "name": "host-a", "os": os, "arch": "amd64", "provisioner": "cloudhv"})
406 require.Equal(t, 201, resp.StatusCode) 414 require.Equal(t, 201, resp.StatusCode)
407 var out map[string]string 415 var out map[string]string
408 json.NewDecoder(resp.Body).Decode(&out) 416 json.NewDecoder(resp.Body).Decode(&out)
internal/server/api/network_api_test.go
Old New
@@ -0,0 +1,235 @@
1 package api
2
3 import (
4 "encoding/json"
5 "net/http"
6 "net/http/httptest"
7 "testing"
8
9 "github.com/stretchr/testify/assert"
10 "github.com/stretchr/testify/require"
11 )
12
13 // agentAdvertisesNetworks is the Hello of an agent started with --host-network:
14 // the names it will serve, held in the registry beside its version. Followed by
15 // the first report, because only a host that is online has told this server
16 // anything — the same predicate the capacity refusal is judged on.
17 func agentAdvertisesNetworks(hostID string, networks ...string) {
18 testReg.SetHostNetworks(hostID, networks)
19 agentReports(hostID)
20 }
21
22 // createVMOnNetwork posts a create asking for a named network, so a test reads
23 // as the placement it is really making.
24 func createVMOnNetwork(t *testing.T, ts *httptest.Server, hostID, name, network string) *http.Response {
25 t.Helper()
26 return do(t, "POST", ts.URL+"/api/v1/vms", testPAT, map[string]any{
27 "host_id": hostID, "name": name, "network": network,
28 })
29 }
30
31 // TestCreateVMBridgedRefusedWhenHostSilent is where this refusal parts company
32 // with its siblings. Capacity and the certified-host-key check tolerate a host
33 // that has not spoken, because desired state is allowed to run ahead of a host
34 // that is merely offline. A network cannot: advertisement arrives only in the
35 // Hello, so silence is indistinguishable from an agent that predates the field
36 // — and that agent would put the guest on NAT without a word, which is the one
37 // outcome this design forbids.
38 func TestCreateVMBridgedRefusedWhenHostSilent(t *testing.T) {
39 ts, st, _ := testServer(t)
40 out := enroll(t, ts) // a Hello and nothing since: online is false
41
42 resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan")
43 require.Equal(t, 409, resp.StatusCode)
44 msg := bodyOf(t, resp)
45 assert.Contains(t, msg, `network "lan"`, "the refusal must name the network asked for")
46 assert.Contains(t, msg, `host "host-a" (`+out["host_id"]+")", "and the host it was asked of")
47 assert.Contains(t, msg, "it is not currently reporting, so its networks cannot be confirmed",
48 "a silent host's networks are unknown, not absent")
49 assert.Contains(t, msg, "create without a network", "the refusal must name a way out")
50
51 vms, err := st.ListVMs()
52 require.NoError(t, err)
53 assert.Empty(t, vms, "a refused create must leave no row behind")
54 }
55
56 // TestCreateVMBridgedRefusedWhenNotAdvertised: the host is right there and
57 // serving, and what it serves is not this. Both shapes of "no" have to be
58 // actionable — an agent configured with no networks at all needs the flag, one
59 // serving other names needs to be told which.
60 func TestCreateVMBridgedRefusedWhenNotAdvertised(t *testing.T) {
61 t.Run("advertises nothing", func(t *testing.T) {
62 ts, st, _ := testServer(t)
63 out := enroll(t, ts)
64 agentAdvertisesNetworks(out["host_id"])
65
66 resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan")
67 require.Equal(t, 409, resp.StatusCode)
68 msg := bodyOf(t, resp)
69 assert.Contains(t, msg, `network "lan"`)
70 assert.Contains(t, msg, "configured with no named networks")
71 assert.Contains(t, msg, "--host-network lan=<bridge>", "the remedy is the flag that would serve it")
72
73 vms, err := st.ListVMs()
74 require.NoError(t, err)
75 assert.Empty(t, vms)
76 })
77
78 t.Run("advertises another name", func(t *testing.T) {
79 ts, _, _ := testServer(t)
80 out := enroll(t, ts)
81 agentAdvertisesNetworks(out["host_id"], "lab", "dmz")
82
83 resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan")
84 require.Equal(t, 409, resp.StatusCode)
85 msg := bodyOf(t, resp)
86 assert.Contains(t, msg, `network "lan"`)
87 assert.Contains(t, msg, "it advertises lab, dmz", "what the host does serve is half the answer")
88 assert.Contains(t, msg, "--host-network lan=<bridge>")
89 })
90 }
91
92 // TestCreateVMBridgedRefusedOnIncapableOS is the refusal for a host whose OS
93 // cannot serve a named network at all, distinct from a host that merely isn't
94 // advertising one: no --host-network remedy exists for a Mac, so the message
95 // must not offer it, and must instead point at what does exist — a Linux
96 // host. This fires even before the host has spoken (see noNetworkRefusal's
97 // os parameter), because an OS, once enrolled, does not change out from under
98 // a host the way its advertised networks do.
99 func TestCreateVMBridgedRefusedOnIncapableOS(t *testing.T) {
100 ts, st, _ := testServer(t)
101 out := enrollSilentOS(t, ts, "darwin")
102
103 resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan")
104 require.Equal(t, 409, resp.StatusCode)
105 msg := bodyOf(t, resp)
106 assert.Contains(t, msg, `network "lan"`)
107 assert.Contains(t, msg, "darwin", "the refusal must name the incapable OS")
108 assert.Contains(t, msg, "Linux host", "the remedy is a different host, not a flag")
109 assert.NotContains(t, msg, "--host-network", "no flag can make a Mac serve a named network")
110
111 vms, err := st.ListVMs()
112 require.NoError(t, err)
113 assert.Empty(t, vms, "a refused create must leave no row behind")
114 }
115
116 // TestCreateVMBridgedNotRefusedOnIncapableOSWhenOSUnknown pins the empty-os
117 // fallthrough: an enroll request that omitted the OS field is not the same
118 // claim as a host known to be incapable, so it must get the ordinary
119 // not-currently-reporting text, not the OS-incapable one.
120 func TestCreateVMBridgedNotRefusedOnIncapableOSWhenOSUnknown(t *testing.T) {
121 ts, _, _ := testServer(t)
122 out := enrollSilentOS(t, ts, "")
123
124 resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan")
125 require.Equal(t, 409, resp.StatusCode)
126 msg := bodyOf(t, resp)
127 assert.Contains(t, msg, "it is not currently reporting", "unknown OS must fall through to the silent-host text")
128 assert.NotContains(t, msg, "cannot serve named networks", "an unset OS is not a known-incapable one")
129 }
130
131 // TestCreateVMBridgedAcceptedWhenAdvertised is the accepting half: a name the
132 // host advertises is placed, and the name lands on the row — the record the
133 // snapshot reads to tell the agent which underlay to build.
134 func TestCreateVMBridgedAcceptedWhenAdvertised(t *testing.T) {
135 ts, st, _ := testServer(t)
136 out := enroll(t, ts)
137 agentAdvertisesNetworks(out["host_id"], "lan")
138
139 resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan")
140 require.Equal(t, 201, resp.StatusCode)
141 var created map[string]string
142 require.NoError(t, json.NewDecoder(resp.Body).Decode(&created))
143
144 vm, err := st.GetVM(created["id"])
145 require.NoError(t, err)
146 assert.Equal(t, "lan", vm.Network)
147 }
148
149 // TestCreateVMWithoutANetworkIsNeverJudged: the NAT default is what nearly
150 // every create asks for, and admission must not have made it conditional on a
151 // host saying anything. A silent host still takes a NAT create.
152 func TestCreateVMWithoutANetworkIsNeverJudged(t *testing.T) {
153 ts, st, _ := testServer(t)
154 out := enrollSilent(t, ts)
155
156 resp := do(t, "POST", ts.URL+"/api/v1/vms", testPAT, map[string]any{
157 "host_id": out["host_id"], "name": "natted",
158 })
159 require.Equal(t, 201, resp.StatusCode)
160 var created map[string]string
161 require.NoError(t, json.NewDecoder(resp.Body).Decode(&created))
162
163 vm, err := st.GetVM(created["id"])
164 require.NoError(t, err)
165 assert.Equal(t, "", vm.Network, "an unnamed network is the NAT underlay")
166 }
167
168 // TestCreateVMInvalidNetworkName: a name that cannot be a network name is a
169 // malformed request, judged before any host is consulted — the grammar lives in
170 // names.IsNetworkName, which the agent's flag parser refuses with too.
171 func TestCreateVMInvalidNetworkName(t *testing.T) {
172 ts, _, _ := testServer(t)
173 out := enroll(t, ts)
174 agentAdvertisesNetworks(out["host_id"], "lan")
175
176 for _, bad := range []string{"NAT!", "nat", "-lan", "lan_1"} {
177 resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", bad)
178 require.Equal(t, 400, resp.StatusCode, "network %q must be refused as malformed", bad)
179 assert.Contains(t, bodyOf(t, resp), "invalid network")
180 }
181 }
182
183 // TestHostViewServesAdvertisedNetworks: the console cannot offer a network the
184 // operator did not configure, so what a host advertises has to reach the host
185 // view — and a host that has said nothing must serve an empty set rather than
186 // the last thing it said.
187 func TestHostViewServesAdvertisedNetworks(t *testing.T) {
188 ts, _, _ := testServer(t)
189 out := enroll(t, ts)
190 agentAdvertisesNetworks(out["host_id"], "lan", "lab")
191
192 resp := do(t, "GET", ts.URL+"/api/v1/hosts", testPAT, nil)
193 require.Equal(t, 200, resp.StatusCode)
194 var hosts []struct {
195 ID string `json:"id"`
196 HostNetworks []string `json:"host_networks"`
197 }
198 require.NoError(t, json.NewDecoder(resp.Body).Decode(&hosts))
199 require.Len(t, hosts, 1)
200 assert.Equal(t, out["host_id"], hosts[0].ID)
201 assert.Equal(t, []string{"lan", "lab"}, hosts[0].HostNetworks)
202 }
203
204 // TestHostViewServesEmptyNetworksNotNull asserts on the response bytes rather
205 // than a decoded struct because the consumer is TypeScript: the contract is the
206 // literal JSON the browser receives, so that is where it has to be checked.
207 // host_networks is declared non-nullable and the console indexes it without a
208 // guard, so a host with nothing to advertise has to serve [] — the empty answer
209 // — and never null. Both silences produce it: a host that has never connected,
210 // and a connected host started without a single --host-network.
211 func TestHostViewServesEmptyNetworksNotNull(t *testing.T) {
212 for _, tc := range []struct {
213 name string
214 enrolls func(t *testing.T, ts *httptest.Server)
215 }{
216 {"never connected", func(t *testing.T, ts *httptest.Server) {
217 enrollSilent(t, ts)
218 }},
219 {"connected, advertising nothing", func(t *testing.T, ts *httptest.Server) {
220 out := enroll(t, ts)
221 agentAdvertisesNetworks(out["host_id"])
222 }},
223 } {
224 t.Run(tc.name, func(t *testing.T) {
225 ts, _, _ := testServer(t)
226 tc.enrolls(t, ts)
227
228 resp := do(t, "GET", ts.URL+"/api/v1/hosts", testPAT, nil)
229 require.Equal(t, 200, resp.StatusCode)
230 body := bodyOf(t, resp)
231 assert.Contains(t, body, `"host_networks":[]`, "the empty set is a list, not a null")
232 assert.NotContains(t, body, `"host_networks":null`)
233 })
234 }
235 }
internal/server/api/networks.go
Old New
@@ -0,0 +1,51 @@
1 package api
2
3 import (
4 "fmt"
5 "strings"
6
7 "github.com/a73x/eitri/internal/names"
8 )
9
10 // noNetworkRefusal explains a create naming a network its host is not
11 // advertising, with the remedy in the operator's vocabulary. Which remedy
12 // depends on what the host said: its OS cannot serve one at all (name a
13 // Linux host instead), nothing at all (wait for it), no networks (configure
14 // one), or other names (pick one, or configure this one). Every branch ends
15 // at the same escape hatch — the NAT underlay is always there.
16 //
17 // os is store.Host.OS, an enrollment-time fact: it is written once, at
18 // RedeemEnrollmentToken, from the joining agent's runtime.GOOS, and
19 // UpdateHostFacts (the per-Hello refresh) deliberately never touches it — so
20 // it needs no spoken/hostHasSpoken gate the way advertised networks do, a
21 // kernel does not change OS between Hellos. An empty os is not a host that
22 // has gone quiet; it is an enroll request that omitted the field (handleEnroll
23 // passes req.OS through unvalidated). That falls through to the existing
24 // branches rather than being told it cannot serve anything: unknown is not
25 // the same claim as known-incapable, and only the latter is worth
26 // interrupting the operator over.
27 //
28 // The comparison stays un-normalised on purpose: os is trusted to be a
29 // literal runtime.GOOS string, because the only thing that ever writes it is
30 // the agent's own enroll request, and every real agent sends exactly what
31 // runtime.GOOS gives it. A malformed value here (say "Linux", capitalized)
32 // would be a different failure than the ones this function is for — an
33 // enroller lying about its platform — and OSServesNamedNetworks would refuse
34 // it exactly like an unsupported OS: permanently, with no flag able to fix
35 // it. That is the right failure mode for a value nothing legitimate can
36 // produce, so it gets no special-cased remedy of its own.
37 func noNetworkRefusal(hostName, hostID, network, os string, advertised []string, spoken bool) string {
38 var b strings.Builder
39 fmt.Fprintf(&b, "host %q (%s) is not advertising network %q", hostName, hostID, network)
40 switch {
41 case os != "" && !names.OSServesNamedNetworks(os):
42 fmt.Fprintf(&b, ": its OS (%s) cannot serve named networks. Place bridged guests on a Linux host, or create without a network for the NAT underlay.", os)
43 case !spoken:
44 b.WriteString(": it is not currently reporting, so its networks cannot be confirmed. Wait for the host to come online, or create without a network for the NAT underlay.")
45 case len(advertised) == 0:
46 b.WriteString(": its agent is configured with no named networks. Restart the agent with --host-network " + network + "=<bridge>, or create without a network for the NAT underlay.")
47 default:
48 fmt.Fprintf(&b, ": it advertises %s. Pick one of those, restart the agent with --host-network %s=<bridge>, or create without a network for the NAT underlay.", strings.Join(advertised, ", "), network)
49 }
50 return b.String()
51 }
internal/server/api/testdata/create-vm-request.golden.json
Old New
@@ -8,5 +8,6 @@
8 "power_state": "running", 8 "power_state": "running",
9 "vcpus": 4, 9 "vcpus": 4,
10 "mem_mb": 4096, 10 "mem_mb": 4096,
11 "disk_gb": 20 11 "disk_gb": 20,
12 "network": "lan"
12 } 13 }
internal/server/api/testdata/host.golden.json
Old New
@@ -35,6 +35,10 @@
35 "cpu_model": "AMD Ryzen 9 7950X", 35 "cpu_model": "AMD Ryzen 9 7950X",
36 "virt": "kvm", 36 "virt": "kvm",
37 "uplink_addr": "192.168.0.190", 37 "uplink_addr": "192.168.0.190",
38 "host_networks": [
39 "lan",
40 "lab"
41 ],
38 "metrics": { 42 "metrics": {
39 "uptime_s": 86400, 43 "uptime_s": 86400,
40 "mem_used_mb": 12000, 44 "mem_used_mb": 12000,
internal/server/api/testdata/snapshot.golden.json
Old New
@@ -37,6 +37,10 @@
37 "cpu_model": "AMD Ryzen 9 7950X", 37 "cpu_model": "AMD Ryzen 9 7950X",
38 "virt": "kvm", 38 "virt": "kvm",
39 "uplink_addr": "192.168.0.190", 39 "uplink_addr": "192.168.0.190",
40 "host_networks": [
41 "lan",
42 "lab"
43 ],
40 "metrics": { 44 "metrics": {
41 "uptime_s": 86400, 45 "uptime_s": 86400,
42 "mem_used_mb": 12000, 46 "mem_used_mb": 12000,
@@ -62,6 +66,8 @@
62 "status": "ready", 66 "status": "ready",
63 "last_error": "boot timeout", 67 "last_error": "boot timeout",
64 "assigned_ip": "10.77.1.2", 68 "assigned_ip": "10.77.1.2",
69 "network": "lan",
70 "network_ip": "192.168.0.42",
65 "created_at": "2026-07-27T12:01:00Z", 71 "created_at": "2026-07-27T12:01:00Z",
66 "deleted": true, 72 "deleted": true,
67 "actual_power": "stopped", 73 "actual_power": "stopped",
internal/server/api/testdata/vm-trusted-cas-empty.golden.json
Old New
@@ -10,6 +10,8 @@
10 "status": "ready", 10 "status": "ready",
11 "last_error": "boot timeout", 11 "last_error": "boot timeout",
12 "assigned_ip": "10.77.1.2", 12 "assigned_ip": "10.77.1.2",
13 "network": "lan",
14 "network_ip": "192.168.0.42",
13 "created_at": "2026-07-27T12:01:00Z", 15 "created_at": "2026-07-27T12:01:00Z",
14 "deleted": true, 16 "deleted": true,
15 "actual_power": "stopped", 17 "actual_power": "stopped",
internal/server/api/testdata/vm-trusted-cas-null.golden.json
Old New
@@ -10,6 +10,8 @@
10 "status": "ready", 10 "status": "ready",
11 "last_error": "boot timeout", 11 "last_error": "boot timeout",
12 "assigned_ip": "10.77.1.2", 12 "assigned_ip": "10.77.1.2",
13 "network": "lan",
14 "network_ip": "192.168.0.42",
13 "created_at": "2026-07-27T12:01:00Z", 15 "created_at": "2026-07-27T12:01:00Z",
14 "deleted": true, 16 "deleted": true,
15 "actual_power": "stopped", 17 "actual_power": "stopped",
internal/server/api/testdata/vm.golden.json
Old New
@@ -10,6 +10,8 @@
10 "status": "ready", 10 "status": "ready",
11 "last_error": "boot timeout", 11 "last_error": "boot timeout",
12 "assigned_ip": "10.77.1.2", 12 "assigned_ip": "10.77.1.2",
13 "network": "lan",
14 "network_ip": "192.168.0.42",
13 "created_at": "2026-07-27T12:01:00Z", 15 "created_at": "2026-07-27T12:01:00Z",
14 "deleted": true, 16 "deleted": true,
15 "actual_power": "stopped", 17 "actual_power": "stopped",
internal/server/api/types/types.go
Old New
@@ -94,6 +94,11 @@ type Host struct {
94 // the address to dial for a published guest port. Empty until the host has 94 // the address to dial for a published guest port. Empty until the host has
95 // said. 95 // said.
96 UplinkAddr string `json:"uplink_addr"` 96 UplinkAddr string `json:"uplink_addr"`
97 // HostNetworks are the named guest networks this host's agent advertised
98 // at connect (--host-network). Registry state: empty when the host has
99 // not spoken since server start. A create naming one of these gains a
100 // second NIC on that network; a create naming anything else is refused.
101 HostNetworks []string `json:"host_networks"`
97 // Metrics is live MEASURED utilization, present only when online. Distinct 102 // Metrics is live MEASURED utilization, present only when online. Distinct
98 // from Allocated (control-plane bookkeeping / VM quotas). 103 // from Allocated (control-plane bookkeeping / VM quotas).
99 Metrics *Metrics `json:"metrics"` 104 Metrics *Metrics `json:"metrics"`
@@ -104,17 +109,28 @@ type Host struct {
104 // cloud_init, ssh_authorized_key — are deliberately excluded. Every field is 109 // cloud_init, ssh_authorized_key — are deliberately excluded. Every field is
105 // spelled out — no struct embedding. 110 // spelled out — no struct embedding.
106 type VM struct { 111 type VM struct {
107 ID string `json:"id"` 112 ID string `json:"id"`
108 HostID string `json:"host_id"` 113 HostID string `json:"host_id"`
109 Name string `json:"name"` 114 Name string `json:"name"`
110 ImageURL string `json:"image_url"` 115 ImageURL string `json:"image_url"`
111 VCPUs int64 `json:"vcpus"` 116 VCPUs int64 `json:"vcpus"`
112 MemMB int64 `json:"mem_mb"` 117 MemMB int64 `json:"mem_mb"`
113 DiskGB int64 `json:"disk_gb"` 118 DiskGB int64 `json:"disk_gb"`
114 PowerState string `json:"power_state"` 119 PowerState string `json:"power_state"`
115 Status string `json:"status"` 120 Status string `json:"status"`
116 LastError string `json:"last_error"` 121 LastError string `json:"last_error"`
117 AssignedIP string `json:"assigned_ip"` 122 // AssignedIP is the guest's address on its host's private fabric — every
123 // guest has one, from boot: the gate splices to it, exposures publish it,
124 // and its siblings on that host reach it there. Never routable off-host.
125 AssignedIP string `json:"assigned_ip"`
126 // Network is the named host network this guest attaches to with a SECOND
127 // NIC, frozen at create; "" means it has only the private one.
128 Network string `json:"network"`
129 // NetworkIP is the address that named network's own DHCP server granted
130 // that second NIC, as the host snooped it — the address the rest of that
131 // network knows this guest by. Empty until the network answers, and
132 // always empty for a guest that asked for none.
133 NetworkIP string `json:"network_ip"`
118 CreatedAt time.Time `json:"created_at"` 134 CreatedAt time.Time `json:"created_at"`
119 Deleted bool `json:"deleted"` 135 Deleted bool `json:"deleted"`
120 ActualPower string `json:"actual_power"` 136 ActualPower string `json:"actual_power"`
@@ -254,6 +270,12 @@ type CreateVMRequest struct {
254 VCPUs int64 `json:"vcpus"` 270 VCPUs int64 `json:"vcpus"`
255 MemMB int64 `json:"mem_mb"` 271 MemMB int64 `json:"mem_mb"`
256 DiskGB int64 `json:"disk_gb"` 272 DiskGB int64 `json:"disk_gb"`
273 // Network names a host network to attach this guest to — one of the names
274 // its host advertises in host_networks. Empty (the default) is the NAT
275 // underlay. A name the chosen host is not advertising is a 409: there is
276 // no silent fallback, a VM that asked for the LAN either gets it or is
277 // never created.
278 Network string `json:"network"`
257 } 279 }
258 280
259 // Default VM sizes the control plane applies when a create request leaves a 281 // Default VM sizes the control plane applies when a create request leaves a
internal/server/api/wire_golden_test.go
Old New
@@ -77,6 +77,7 @@ func TestWireGolden(t *testing.T) {
77 CPUModel: "AMD Ryzen 9 7950X", 77 CPUModel: "AMD Ryzen 9 7950X",
78 Virt: "kvm", 78 Virt: "kvm",
79 UplinkAddr: "192.168.0.190", 79 UplinkAddr: "192.168.0.190",
80 HostNetworks: []string{"lan", "lab"},
80 Metrics: &types.Metrics{ 81 Metrics: &types.Metrics{
81 UptimeS: 86400, 82 UptimeS: 86400,
82 MemUsedMB: 12000, 83 MemUsedMB: 12000,
@@ -102,6 +103,8 @@ func TestWireGolden(t *testing.T) {
102 Status: "ready", 103 Status: "ready",
103 LastError: "boot timeout", 104 LastError: "boot timeout",
104 AssignedIP: "10.77.1.2", 105 AssignedIP: "10.77.1.2",
106 Network: "lan",
107 NetworkIP: "192.168.0.42",
105 CreatedAt: base.Add(time.Minute), 108 CreatedAt: base.Add(time.Minute),
106 Deleted: true, 109 Deleted: true,
107 ActualPower: "stopped", 110 ActualPower: "stopped",
@@ -179,6 +182,7 @@ func TestWireGolden(t *testing.T) {
179 VCPUs: 4, 182 VCPUs: 4,
180 MemMB: 4096, 183 MemMB: 4096,
181 DiskGB: 20, 184 DiskGB: 20,
185 Network: "lan",
182 }) 186 })
183 187
184 goldenCheck(t, "patch-vm-request", types.PatchVMRequest{ 188 goldenCheck(t, "patch-vm-request", types.PatchVMRequest{
internal/server/registry/registry.go
Old New
@@ -90,6 +90,13 @@ type HostState struct {
90 // (like Sessions, owned by the connect path, preserved across reports). 90 // (like Sessions, owned by the connect path, preserved across reports).
91 // Empty until a version-reporting agent connects. 91 // Empty until a version-reporting agent connects.
92 AgentVersion string 92 AgentVersion string
93 // HostNetworks are the named guest networks this host's agent was
94 // configured with (--host-network), as advertised in each Hello. Owned by
95 // the connect path exactly like AgentVersion — no report carries them, so
96 // UpdateReport preserves them — and empty for a host that has not spoken
97 // since server start, for a Mac agent, and for any agent that predates the
98 // field. Create-time admission reads this set.
99 HostNetworks []string
93 // The following are derived on each Get from LastSeen and the clock; they 100 // The following are derived on each Get from LastSeen and the clock; they
94 // are not stored. 101 // are not stored.
95 Online bool 102 Online bool
@@ -118,7 +125,9 @@ func (r *Registry) UpdateReport(hostID string, rep Report) {
118 // whole HostState, and Sessions is owned by RecordConnect, not the report. 125 // whole HostState, and Sessions is owned by RecordConnect, not the report.
119 sessions := r.m[hostID].Sessions 126 sessions := r.m[hostID].Sessions
120 agentVersion := r.m[hostID].AgentVersion 127 agentVersion := r.m[hostID].AgentVersion
121 r.m[hostID] = HostState{Report: rep, LastSeen: r.now(), Sessions: sessions, AgentVersion: agentVersion} 128 hostNetworks := r.m[hostID].HostNetworks
129 r.m[hostID] = HostState{Report: rep, LastSeen: r.now(), Sessions: sessions,
130 AgentVersion: agentVersion, HostNetworks: hostNetworks}
122 } 131 }
123 132
124 // RecordConnect increments the host's session counter, marking one agent 133 // RecordConnect increments the host's session counter, marking one agent
@@ -142,6 +151,23 @@ func (r *Registry) SetAgentVersion(hostID, v string) {
142 r.m[hostID] = st 151 r.m[hostID] = st
143 } 152 }
144 153
154 // SetHostNetworks records the named guest networks a host advertised in its
155 // Hello. Connect-path-owned like AgentVersion: reports never carry it, so
156 // UpdateReport preserves it. Each Hello replaces the set — an agent restarted
157 // without its --host-network flags advertises none, and the fleet must see
158 // that rather than the networks it used to serve.
159 //
160 // Stores networks by reference, same convention as the Report slices: the
161 // caller must not retain or mutate the slice after passing it in. Get clones
162 // on the way out, so a reader can never observe or corrupt this copy.
163 func (r *Registry) SetHostNetworks(hostID string, networks []string) {
164 r.mu.Lock()
165 defer r.mu.Unlock()
166 st := r.m[hostID]
167 st.HostNetworks = networks
168 r.m[hostID] = st
169 }
170
145 func (r *Registry) Get(hostID string) (HostState, bool) { 171 func (r *Registry) Get(hostID string) (HostState, bool) {
146 r.mu.RLock() 172 r.mu.RLock()
147 defer r.mu.RUnlock() 173 defer r.mu.RUnlock()
@@ -166,5 +192,6 @@ func (r *Registry) Get(hostID string) (HostState, bool) {
166 quarantined[i].VMSpecJSON = bytes.Clone(quarantined[i].VMSpecJSON) 192 quarantined[i].VMSpecJSON = bytes.Clone(quarantined[i].VMSpecJSON)
167 } 193 }
168 st.Report.Quarantined = quarantined 194 st.Report.Quarantined = quarantined
195 st.HostNetworks = slices.Clone(st.HostNetworks)
169 return st, true 196 return st, true
170 } 197 }
internal/server/registry/registry_test.go
Old New
@@ -132,6 +132,27 @@ func TestAgentVersionSurvivesReports(t *testing.T) {
132 } 132 }
133 } 133 }
134 134
135 // TestHostNetworksSurviveReports: the advertised network names arrive in the
136 // Hello and never again, so the report path — which replaces the whole state —
137 // must carry them forward. Create-time admission reads them on every create; a
138 // report that blanked them would refuse a bridged VM on the host that serves it.
139 func TestHostNetworksSurviveReports(t *testing.T) {
140 now := time.Now()
141 r := New(func() time.Time { return now })
142 r.SetHostNetworks("h1", []string{"lan"})
143 r.UpdateReport("h1", Report{})
144 st, ok := r.Get("h1")
145 require.True(t, ok)
146 assert.Equal(t, []string{"lan"}, st.HostNetworks, "a report must preserve what the Hello advertised")
147
148 // A reconnect re-advertises: an agent restarted without its --host-network
149 // flags advertises none, and the registry must say so rather than hold the
150 // networks it used to serve.
151 r.SetHostNetworks("h1", nil)
152 st, _ = r.Get("h1")
153 assert.Empty(t, st.HostNetworks, "a Hello with no networks retracts the old set")
154 }
155
135 // TestGetClonesExposureStatuses pins that a caller cannot corrupt registry 156 // TestGetClonesExposureStatuses pins that a caller cannot corrupt registry
136 // state through the slice it is handed — the same rule the VM rows follow. 157 // state through the slice it is handed — the same rule the VM rows follow.
137 func TestGetClonesExposureStatuses(t *testing.T) { 158 func TestGetClonesExposureStatuses(t *testing.T) {
internal/server/store/store.go
Old New
@@ -79,7 +79,18 @@ type VM struct {
79 ID, HostID, Name, ImageURL, ImageSHA256, CloudInit string 79 ID, HostID, Name, ImageURL, ImageSHA256, CloudInit string
80 VCPUs, MemMB, DiskGB int64 80 VCPUs, MemMB, DiskGB int64
81 PowerState, Status, LastError, AssignedIP string 81 PowerState, Status, LastError, AssignedIP string
82 SSHAuthorizedKey string 82 // Network is the named host network this VM attaches to, as recorded at
83 // create — the same name a host advertised in its Hello. '' is the NAT
84 // underlay: every VM created before this column existed, and any VM
85 // created without naming one, backfills to it.
86 Network string
87 // NetworkIP is the address the site's DHCP server granted this guest on
88 // its named-network NIC, as its host snooped it. AssignedIP is the other
89 // address the same guest has, on the host's private fabric — every VM has
90 // that one, and only a VM with a Network can have this one. '' means not
91 // discovered (yet, or ever: a guest configured static never asks).
92 NetworkIP string
93 SSHAuthorizedKey string
83 // SSHHostPubKey is the public half of the guest's SSH host key, as its host 94 // SSHHostPubKey is the public half of the guest's SSH host key, as its host
84 // reported it (authorized_keys form). The private half lives on that host 95 // reported it (authorized_keys form). The private half lives on that host
85 // and the control plane never sees it. SSHHostCert is the certificate the 96 // and the control plane never sees it. SSHHostCert is the certificate the
@@ -105,7 +116,7 @@ type VM struct {
105 // makes nil unambiguously "unrecorded" rather than "trusts nothing", which 116 // makes nil unambiguously "unrecorded" rather than "trusts nothing", which
106 // is the distinction every reader of this field depends on. 117 // is the distinction every reader of this field depends on.
107 TrustedCAs []TrustedCA 118 TrustedCAs []TrustedCA
108 // The VM's reachable address is AssignedIP (the agent-reported bridge IP). 119 // The address the gate and every exposure target is AssignedIP.
109 CreatedAt time.Time 120 CreatedAt time.Time
110 DeletedAt *time.Time 121 DeletedAt *time.Time
111 // Tenant is the owning tenant, always derived from the host's — never client-set. 122 // Tenant is the owning tenant, always derived from the host's — never client-set.
@@ -371,6 +382,17 @@ func Open(path, cidrPool string) (*Store, error) {
371 // proto field 9 — the same discipline for the same reason: retire the 382 // proto field 9 — the same discipline for the same reason: retire the
372 // compatibility only once no v0.0.5 can still be out there. 383 // compatibility only once no v0.0.5 can still be out there.
373 {"vms", "persistent", "INTEGER NOT NULL DEFAULT 1"}, 384 {"vms", "persistent", "INTEGER NOT NULL DEFAULT 1"},
385 // The named host network this VM attaches to, as recorded at create.
386 // '' is the NAT underlay — every row that predates this column
387 // backfills to it, which is exactly what an unset network means.
388 {"vms", "network", "TEXT NOT NULL DEFAULT ''"},
389 // The address the site's DHCP server granted a guest on its named
390 // network, as its host snooped it. Agent-reported like assigned_ip and
391 // separate from it: assigned_ip is the private-fabric address every
392 // guest has from boot, this one exists only for a guest with a second
393 // NIC and only once that network has answered. '' is "not discovered",
394 // which is also what every row predating the column honestly says.
395 {"vms", "network_ip", "TEXT NOT NULL DEFAULT ''"},
374 // The address a host presents on the network it reaches the fleet 396 // The address a host presents on the network it reaches the fleet
375 // over. Reported every tick like the guest subnet, and stored for the 397 // over. Reported every tick like the guest subnet, and stored for the
376 // same reason: the console renders `host:port` for every exposure, 398 // same reason: the console renders `host:port` for every exposure,
@@ -842,12 +864,12 @@ func (s *Store) CreateVM(vm VM) error {
842 _, err = tx.Exec( 864 _, err = tx.Exec(
843 `INSERT INTO vms(id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key, 865 `INSERT INTO vms(id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key,
844 injected_key_type, injected_key_fp, injected_key_comment, trusted_cas, 866 injected_key_type, injected_key_fp, injected_key_comment, trusted_cas,
845 vcpus, mem_mb, disk_gb, power_state, created_at) 867 vcpus, mem_mb, disk_gb, power_state, network, created_at)
846 VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, 868 VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`,
847 vm.ID, vm.HostID, vm.Name, vm.Tenant, vm.ImageURL, vm.ImageSHA256, 869 vm.ID, vm.HostID, vm.Name, vm.Tenant, vm.ImageURL, vm.ImageSHA256,
848 vm.CloudInit, vm.SSHAuthorizedKey, 870 vm.CloudInit, vm.SSHAuthorizedKey,
849 vm.InjectedKeyType, vm.InjectedKeyFP, vm.InjectedKeyComment, trustedCAs, 871 vm.InjectedKeyType, vm.InjectedKeyFP, vm.InjectedKeyComment, trustedCAs,
850 vm.VCPUs, vm.MemMB, vm.DiskGB, vm.PowerState, 872 vm.VCPUs, vm.MemMB, vm.DiskGB, vm.PowerState, vm.Network,
851 now.Format(time.RFC3339), 873 now.Format(time.RFC3339),
852 ) 874 )
853 if err != nil { 875 if err != nil {
@@ -1387,6 +1409,55 @@ func (s *Store) RecordVMStatus(id, status, lastErr, ip string) (string, error) {
1387 return ip, nil 1409 return ip, nil
1388 } 1410 }
1389 1411
1412 // RecordVMNetworkIP persists the address a guest's named-network NIC was
1413 // granted, as its host snooped it from the site's DHCP server.
1414 //
1415 // Separate from RecordVMStatus because it is a separate fact with a separate
1416 // clock: assigned_ip exists the moment the reservation is made, while this one
1417 // arrives whenever the site's server answers — which can be long before the VM
1418 // is ready, the only phase RecordVMStatus writes in.
1419 //
1420 // The CASE is the assigned_ip rule, for the assigned_ip reason: an empty report
1421 // is "not discovered", never "no longer has one", so it keeps what was stored
1422 // rather than erasing it. An unusable address (link-local — what a guest shows
1423 // when nothing answered — loopback, multicast, unparseable) is dropped the same
1424 // way, so a NIC that never got a real lease reports nothing rather than noise.
1425 //
1426 // hostID is part of the WHERE clause rather than a check made before it, the
1427 // same idiom RecordVMHostKey uses and for the same reason: a host may only
1428 // ever speak for the VMs it holds, and a predicate leaves no read-then-write
1429 // window in which the VM could move. A host that reports a lease for a VM
1430 // that is not its own changes no rows and gets sql.ErrNoRows.
1431 //
1432 // network is also in the WHERE: a VM that never asked for a named network can
1433 // never grow a LAN address, however an agent misbehaves — a forged report, or
1434 // one from an agent whose view of the VM has skewed from the server's own.
1435 // Refusing the write makes that pairing unrepresentable in the row rather
1436 // than merely unrendered by the clients that know to hide it. This means
1437 // sql.ErrNoRows now covers three predicates — the row is gone, the wrong host
1438 // reported it, or the VM has no network — and the caller cannot tell which:
1439 // a pre-check that classified the failure would reopen the read-then-write
1440 // window this whole idiom exists to close. The resulting slog.Warn recurs at
1441 // sync cadence by design (the write-through cache only commits on success, so
1442 // it is never rate-limited away); it says only that something is wrong, and
1443 // the VM id in the log line is what lets an operator resolve it in one query.
1444 func (s *Store) RecordVMNetworkIP(id, hostID, ip string) error {
1445 if ip != "" && !usableAddress(ip) {
1446 ip = ""
1447 }
1448 res, err := s.db.Exec(
1449 `UPDATE vms SET network_ip=CASE WHEN ?='' THEN network_ip ELSE ? END WHERE id=? AND host_id=? AND network != ''`,
1450 ip, ip, id, hostID,
1451 )
1452 if err != nil {
1453 return err
1454 }
1455 if n, _ := res.RowsAffected(); n == 0 {
1456 return sql.ErrNoRows
1457 }
1458 return nil
1459 }
1460
1390 // RecordVMHostKey stores the public host key a host generated for one of its 1461 // RecordVMHostKey stores the public host key a host generated for one of its
1391 // guests, together with the certificate the control plane signed for it. 1462 // guests, together with the certificate the control plane signed for it.
1392 // 1463 //
@@ -1417,7 +1488,7 @@ func (s *Store) RecordVMHostKey(vmID, hostID, pubkey, cert string) error {
1417 const vmColumns = `id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key, 1488 const vmColumns = `id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key,
1418 ssh_host_pubkey, ssh_host_cert, 1489 ssh_host_pubkey, ssh_host_cert,
1419 injected_key_type, injected_key_fp, injected_key_comment, trusted_cas, 1490 injected_key_type, injected_key_fp, injected_key_comment, trusted_cas,
1420 vcpus, mem_mb, disk_gb, power_state, status, last_error, assigned_ip, 1491 vcpus, mem_mb, disk_gb, power_state, network, status, last_error, assigned_ip, network_ip,
1421 created_at, deleted_at` 1492 created_at, deleted_at`
1422 1493
1423 // scanVM's column order must match vmColumns exactly — it is positional, not 1494 // scanVM's column order must match vmColumns exactly — it is positional, not
@@ -1432,7 +1503,7 @@ func scanVM(rows *sql.Rows) (VM, error) {
1432 &vm.CloudInit, &vm.SSHAuthorizedKey, &vm.SSHHostPubKey, &vm.SSHHostCert, 1503 &vm.CloudInit, &vm.SSHAuthorizedKey, &vm.SSHHostPubKey, &vm.SSHHostCert,
1433 &vm.InjectedKeyType, &vm.InjectedKeyFP, &vm.InjectedKeyComment, &trustedCAs, 1504 &vm.InjectedKeyType, &vm.InjectedKeyFP, &vm.InjectedKeyComment, &trustedCAs,
1434 &vm.VCPUs, &vm.MemMB, &vm.DiskGB, 1505 &vm.VCPUs, &vm.MemMB, &vm.DiskGB,
1435 &vm.PowerState, &vm.Status, &vm.LastError, &vm.AssignedIP, 1506 &vm.PowerState, &vm.Network, &vm.Status, &vm.LastError, &vm.AssignedIP, &vm.NetworkIP,
1436 &createdAt, &deletedAt, 1507 &createdAt, &deletedAt,
1437 ) 1508 )
1438 if err != nil { 1509 if err != nil {
internal/server/store/store_test.go
Old New
@@ -88,6 +88,150 @@ func TestGetVM(t *testing.T) {
88 assert.NotNil(t, got.DeletedAt) 88 assert.NotNil(t, got.DeletedAt)
89 } 89 }
90 90
91 // TestVMNetworkRoundTrip pins that the empty string is the NAT default an
92 // unset create stores, and a named network survives the row whole.
93 func TestVMNetworkRoundTrip(t *testing.T) {
94 s := newStore(t)
95 h := enrollHost(t, s)
96
97 require.NoError(t, s.CreateVM(VM{
98 ID: "vm-lan", HostID: h.ID, Name: "on-lan",
99 ImageURL: "http://img", ImageSHA256: "abc",
100 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running",
101 Network: "lan",
102 }))
103 got, err := s.GetVM("vm-lan")
104 require.NoError(t, err)
105 assert.Equal(t, "lan", got.Network)
106
107 // A VM created without naming a network gets the NAT underlay, spelled ''.
108 require.NoError(t, s.CreateVM(VM{
109 ID: "vm-nat", HostID: h.ID, Name: "on-nat",
110 ImageURL: "http://img", ImageSHA256: "abc",
111 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running",
112 }))
113 got, err = s.GetVM("vm-nat")
114 require.NoError(t, err)
115 assert.Equal(t, "", got.Network)
116 }
117
118 // TestRecordVMNetworkIPKeepsBothAddresses pins that the two addresses a
119 // networked guest has are two independent columns: the named network's lease
120 // lands in network_ip and never touches the private-fabric address the gate
121 // and every exposure aim at.
122 func TestRecordVMNetworkIPKeepsBothAddresses(t *testing.T) {
123 s := newStore(t)
124 h := enrollHost(t, s)
125 require.NoError(t, s.CreateVM(VM{
126 ID: "vm-lan", HostID: h.ID, Name: "on-lan", ImageURL: "u", ImageSHA256: "s",
127 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan",
128 }))
129
130 // A fresh row has the reservation and nothing from the site's DHCP yet.
131 _, err := s.RecordVMStatus("vm-lan", "ready", "", "10.77.1.2")
132 require.NoError(t, err)
133 got, err := s.GetVM("vm-lan")
134 require.NoError(t, err)
135 assert.Equal(t, "10.77.1.2", got.AssignedIP)
136 assert.Empty(t, got.NetworkIP, "nothing has answered on the named NIC yet")
137
138 require.NoError(t, s.RecordVMNetworkIP("vm-lan", h.ID, "192.168.0.42"))
139 got, err = s.GetVM("vm-lan")
140 require.NoError(t, err)
141 assert.Equal(t, "192.168.0.42", got.NetworkIP)
142 assert.Equal(t, "10.77.1.2", got.AssignedIP, "the private address is a separate fact")
143
144 // And the status write does not disturb the discovered one either.
145 _, err = s.RecordVMStatus("vm-lan", "failed", "boom", "10.77.1.2")
146 require.NoError(t, err)
147 got, err = s.GetVM("vm-lan")
148 require.NoError(t, err)
149 assert.Equal(t, "192.168.0.42", got.NetworkIP)
150 }
151
152 // TestRecordVMNetworkIPKeepsPriorOnNothingUsable pins the empty-never-clears
153 // rule assigned_ip already follows: a report with no address, or one naming
154 // nobody, leaves the last known address standing. A guest whose lease renews
155 // to a different address is still followed.
156 func TestRecordVMNetworkIPKeepsPriorOnNothingUsable(t *testing.T) {
157 const prior = "192.168.0.42"
158 for _, tc := range []struct {
159 name string
160 ip string
161 want string
162 }{
163 {"a new lease on the site's network", "192.168.0.77", "192.168.0.77"},
164 {"none discovered", "", prior},
165 {"link-local: the site's DHCP never answered", "169.254.11.2", prior},
166 {"loopback", "127.0.0.1", prior},
167 {"unspecified", "0.0.0.0", prior},
168 {"multicast", "224.0.0.1", prior},
169 {"unparseable", "not-an-ip", prior},
170 } {
171 t.Run(tc.name, func(t *testing.T) {
172 s := newStore(t)
173 h := enrollHost(t, s)
174 require.NoError(t, s.CreateVM(VM{
175 ID: "vm1", HostID: h.ID, Name: "a", ImageURL: "u", ImageSHA256: "s",
176 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan",
177 }))
178 require.NoError(t, s.RecordVMNetworkIP("vm1", h.ID, prior))
179
180 require.NoError(t, s.RecordVMNetworkIP("vm1", h.ID, tc.ip))
181 vm, err := s.GetVM("vm1")
182 require.NoError(t, err)
183 assert.Equal(t, tc.want, vm.NetworkIP)
184 })
185 }
186 }
187
188 // TestRecordVMNetworkIPRefusesNetworklessVM pins that a VM which never named
189 // a network cannot grow a network_ip: the pairing is impossible, so the write
190 // itself refuses it rather than leaving it for a client to hide. The same
191 // sql.ErrNoRows a foreign host or a missing row gets is what a report against
192 // the NAT underlay gets too.
193 func TestRecordVMNetworkIPRefusesNetworklessVM(t *testing.T) {
194 s := newStore(t)
195 h := enrollHost(t, s)
196 require.NoError(t, s.CreateVM(VM{
197 ID: "vm-nat", HostID: h.ID, Name: "on-nat", ImageURL: "u", ImageSHA256: "s",
198 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running",
199 }))
200
201 err := s.RecordVMNetworkIP("vm-nat", h.ID, "192.168.0.42")
202 require.ErrorIs(t, err, sql.ErrNoRows)
203 got, err := s.GetVM("vm-nat")
204 require.NoError(t, err)
205 assert.Empty(t, got.NetworkIP, "a VM with no named network stores no discovered address")
206 }
207
208 // TestRecordVMNetworkIPUnknownVM pins that a report for a row that is gone is
209 // an error, not a silent no-op — the same answer RecordVMStatus gives.
210 func TestRecordVMNetworkIPUnknownVM(t *testing.T) {
211 s := newStore(t)
212 h := enrollHost(t, s)
213 assert.ErrorIs(t, s.RecordVMNetworkIP("nope", h.ID, "192.168.0.42"), sql.ErrNoRows)
214 }
215
216 // TestRecordVMNetworkIPRefusesAnotherHostsVM pins the same idiom
217 // RecordVMHostKey uses: host_id is a predicate on the UPDATE, not a check made
218 // before it, so a lease reported for a VM that belongs to a different host
219 // changes no rows and gets sql.ErrNoRows rather than silently landing.
220 func TestRecordVMNetworkIPRefusesAnotherHostsVM(t *testing.T) {
221 s := newStore(t)
222 h := enrollHost(t, s)
223 require.NoError(t, s.CreateVM(VM{
224 ID: "vm1", HostID: h.ID, Name: "on-lan", ImageURL: "u", ImageSHA256: "s",
225 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan",
226 }))
227
228 err := s.RecordVMNetworkIP("vm1", "some-other-host", "192.168.0.42")
229 require.ErrorIs(t, err, sql.ErrNoRows)
230 got, err := s.GetVM("vm1")
231 require.NoError(t, err)
232 assert.Empty(t, got.NetworkIP, "a claim from a host that does not hold this VM writes nothing")
233 }
234
91 func TestRecordVMHostKeyCertifiesOnlyItsOwnHostsVM(t *testing.T) { 235 func TestRecordVMHostKeyCertifiesOnlyItsOwnHostsVM(t *testing.T) {
92 s := newStore(t) 236 s := newStore(t)
93 h := enrollHost(t, s) 237 h := enrollHost(t, s)
internal/server/syncsvc/network_test.go
Old New
@@ -0,0 +1,179 @@
1 package syncsvc
2
3 import (
4 "testing"
5 "time"
6
7 "github.com/a73x/eitri/internal/pb"
8 "github.com/a73x/eitri/internal/server/store"
9 "github.com/stretchr/testify/assert"
10 "github.com/stretchr/testify/require"
11 )
12
13 // TestHelloAdvertisesHostNetworks is the whole basis of named-network
14 // admission: a create is only allowed onto a host that says it serves the
15 // network, and the only place a host ever says so is its Hello. If this stops
16 // reaching the registry, every bridged create on the fleet is refused.
17 func TestHelloAdvertisesHostNetworks(t *testing.T) {
18 f := setup(t)
19 _, err := dialHello(t, f.addr, f.fp, &pb.Hello{
20 HostId: f.host.ID, Provisioner: "cloudhv", Credential: f.cred,
21 HostNetworks: []string{"lan", "lab"},
22 })
23 require.NoError(t, err)
24
25 require.Eventually(t, func() bool {
26 st, ok := f.reg.Get(f.host.ID)
27 return ok && len(st.HostNetworks) == 2
28 }, 2*time.Second, 20*time.Millisecond, "the Hello's networks must reach the registry")
29 st, _ := f.reg.Get(f.host.ID)
30 assert.Equal(t, []string{"lan", "lab"}, st.HostNetworks)
31 }
32
33 // TestSnapshotCarriesTheVMsNetwork: the row records which underlay a guest was
34 // admitted onto, and the snapshot is the only way that reaches the agent that
35 // builds it. A NAT guest carries the empty name — the agent's default path —
36 // and neither one is inferred anywhere downstream.
37 func TestSnapshotCarriesTheVMsNetwork(t *testing.T) {
38 f := setup(t)
39 require.NoError(t, f.st.CreateVM(store.VM{
40 ID: "vm1", HostID: f.host.ID, Name: "bridged", ImageURL: "u", ImageSHA256: "s",
41 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan",
42 }))
43 require.NoError(t, f.st.CreateVM(store.VM{
44 ID: "vm2", HostID: f.host.ID, Name: "natted", ImageURL: "u", ImageSHA256: "s",
45 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running",
46 }))
47
48 snap, err := f.svc.buildSnapshot(f.host.ID)
49 require.NoError(t, err)
50 got := map[string]string{}
51 for _, v := range snap.GetVms() {
52 got[v.GetVmId()] = v.GetNetwork()
53 }
54 assert.Equal(t, map[string]string{"vm1": "lan", "vm2": ""}, got)
55 }
56
57 // networkedVM creates one VM on the named network `lan` for the report tests.
58 func networkedVM(t *testing.T, f *fixture, id string) {
59 t.Helper()
60 require.NoError(t, f.st.CreateVM(store.VM{
61 ID: id, HostID: f.host.ID, Name: id, ImageURL: "u", ImageSHA256: "s",
62 VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan",
63 }))
64 }
65
66 // TestApplyReportRecordsBothAddresses pins the inbound leg of the second NIC:
67 // the address the site's DHCP granted is durable beside — never instead of —
68 // the private-fabric address the gate and every exposure aim at.
69 func TestApplyReportRecordsBothAddresses(t *testing.T) {
70 f := setup(t)
71 networkedVM(t, f, "vm1")
72
73 f.svc.applyReport(f.host.ID, &pb.ActualStateReport{
74 Vms: []*pb.ActualVM{{
75 VmId: "vm1", Power: "running", Phase: "ready",
76 Ip: "10.77.1.2", NetworkIp: "192.168.0.42",
77 }},
78 })
79
80 vm, err := f.st.GetVM("vm1")
81 require.NoError(t, err)
82 assert.Equal(t, "10.77.1.2", vm.AssignedIP)
83 assert.Equal(t, "192.168.0.42", vm.NetworkIP)
84 }
85
86 // TestApplyReportRecordsTheLeaseBeforeTheGuestIsReady: the site's DHCP answers
87 // on its own clock, which is not the lifecycle's. A lease that lands while the
88 // guest is still booting is recorded then — the status write's ready/failed
89 // filter must not hold the address hostage.
90 func TestApplyReportRecordsTheLeaseBeforeTheGuestIsReady(t *testing.T) {
91 f := setup(t)
92 networkedVM(t, f, "vm1")
93
94 f.svc.applyReport(f.host.ID, &pb.ActualStateReport{
95 Vms: []*pb.ActualVM{{VmId: "vm1", Power: "running", Phase: "creating", NetworkIp: "192.168.0.42"}},
96 })
97
98 vm, err := f.st.GetVM("vm1")
99 require.NoError(t, err)
100 assert.Equal(t, "192.168.0.42", vm.NetworkIP)
101 assert.Equal(t, "pending", vm.Status, "the lifecycle write still waits for ready or failed")
102 }
103
104 // TestApplyReportKeepsTheLastKnownLease: an agent restarted before its guest
105 // renews reports no address on the named NIC, and silence is not "the guest
106 // left the network". The stored address stands until a new one replaces it.
107 func TestApplyReportKeepsTheLastKnownLease(t *testing.T) {
108 f := setup(t)
109 networkedVM(t, f, "vm1")
110 ready := func(netIP string) *pb.ActualStateReport {
111 return &pb.ActualStateReport{Vms: []*pb.ActualVM{{
112 VmId: "vm1", Power: "running", Phase: "ready", Ip: "10.77.1.2", NetworkIp: netIP,
113 }}}
114 }
115
116 f.svc.applyReport(f.host.ID, ready("192.168.0.42"))
117 f.svc.applyReport(f.host.ID, ready(""))
118 vm, err := f.st.GetVM("vm1")
119 require.NoError(t, err)
120 assert.Equal(t, "192.168.0.42", vm.NetworkIP, "an empty report is 'not known', not 'gone'")
121
122 // A renewal onto a different address is followed, not stuck.
123 f.svc.applyReport(f.host.ID, ready("192.168.0.77"))
124 vm, err = f.st.GetVM("vm1")
125 require.NoError(t, err)
126 assert.Equal(t, "192.168.0.77", vm.NetworkIP)
127
128 // An unusable address (what a guest shows when nothing answered) is one
129 // the store drops and keeps the prior value for — the cache still commits
130 // the raw report, so this proves the divergence between what netIPTrack
131 // remembers and what the row holds is harmless: the next GOOD address
132 // still reads as a change against the cache and lands.
133 f.svc.applyReport(f.host.ID, ready("169.254.11.2"))
134 vm, err = f.st.GetVM("vm1")
135 require.NoError(t, err)
136 assert.Equal(t, "192.168.0.77", vm.NetworkIP, "an unusable address is dropped, not stored")
137
138 f.svc.applyReport(f.host.ID, ready("192.168.0.99"))
139 vm, err = f.st.GetVM("vm1")
140 require.NoError(t, err)
141 assert.Equal(t, "192.168.0.99", vm.NetworkIP, "a good address after junk still lands")
142 }
143
144 // TestApplyReportSkipsUnchangedLeases proves the per-VM dedup: a bridged guest
145 // re-reports the same address every tick forever, and the store runs on one
146 // connection. Only a change may cost a write.
147 func TestApplyReportSkipsUnchangedLeases(t *testing.T) {
148 f := setup(t)
149 networkedVM(t, f, "vm1")
150 rep := &pb.ActualStateReport{Vms: []*pb.ActualVM{{
151 VmId: "vm1", Power: "running", Phase: "ready", Ip: "10.77.1.2", NetworkIp: "192.168.0.42",
152 }}}
153
154 f.svc.applyReport(f.host.ID, rep)
155 f.svc.applyReport(f.host.ID, rep)
156 f.svc.applyReport(f.host.ID, rep)
157
158 f.svc.netIPTrack.mu.Lock()
159 defer f.svc.netIPTrack.mu.Unlock()
160 assert.Equal(t, map[string]string{"vm1": "192.168.0.42"}, f.svc.netIPTrack.last,
161 "the address is remembered once, so the repeats write nothing")
162 }
163
164 // TestApplyReportForgetsAReapedVMsLease bounds the cache: a VM whose row is
165 // hard-deleted must not leave its address remembered forever.
166 func TestApplyReportForgetsAReapedVMsLease(t *testing.T) {
167 f := setup(t)
168 networkedVM(t, f, "vm1")
169
170 f.svc.applyReport(f.host.ID, &pb.ActualStateReport{
171 Vms: []*pb.ActualVM{{VmId: "vm1", Power: "running", Phase: "ready", NetworkIp: "192.168.0.42"}},
172 })
173 require.NoError(t, f.st.TombstoneVM("vm1"))
174 f.svc.applyReport(f.host.ID, &pb.ActualStateReport{Destroyed: []string{"vm1"}})
175
176 f.svc.netIPTrack.mu.Lock()
177 defer f.svc.netIPTrack.mu.Unlock()
178 assert.Empty(t, f.svc.netIPTrack.last, "a reaped VM's address is dead weight")
179 }
internal/server/syncsvc/syncsvc.go
Old New
@@ -59,6 +59,10 @@ type Service struct {
59 // the same reason netTrack does: every host reports every tick forever, and 59 // the same reason netTrack does: every host reports every tick forever, and
60 // the store runs on a single connection. 60 // the store runs on a single connection.
61 uplinkTrack *netTracker 61 uplinkTrack *netTracker
62 // netIPTrack remembers the named-network address each VM last had WRITTEN.
63 // Keyed by vmID like certTrack, and for the same reason: a bridged guest
64 // re-reports the address its lease renewed to, every tick, forever.
65 netIPTrack *netTracker
62 // maxCredAge, when non-zero, rejects credentials whose issued-at is older. 66 // maxCredAge, when non-zero, rejects credentials whose issued-at is older.
63 // Zero disables the age check (default: expiry without an auto-renewal 67 // Zero disables the age check (default: expiry without an auto-renewal
64 // channel would force periodic re-enrolls; per-host generation revocation 68 // channel would force periodic re-enrolls; per-host generation revocation
@@ -106,7 +110,8 @@ func newWithWriteTimeout(st *store.Store, reg *registry.Registry, h *hub.Hub, se
106 } 110 }
107 return &Service{st: st, reg: reg, hub: h, secret: secret, maxCredAge: maxCredAge, writeTimeout: writeTimeout, 111 return &Service{st: st, reg: reg, hub: h, secret: secret, maxCredAge: maxCredAge, writeTimeout: writeTimeout,
108 conns: map[string]quic.Connection{}, recorder: st, tracker: newStatusTracker(), netTrack: newNetTracker(), 112 conns: map[string]quic.Connection{}, recorder: st, tracker: newStatusTracker(), netTrack: newNetTracker(),
109 uplinkTrack: newNetTracker(), certTrack: newNetTracker(), offers: map[string]offer{}, now: time.Now} 113 uplinkTrack: newNetTracker(), certTrack: newNetTracker(), netIPTrack: newNetTracker(),
114 offers: map[string]offer{}, now: time.Now}
110 } 115 }
111 116
112 // Serve accepts QUIC connections until ctx is cancelled. 117 // Serve accepts QUIC connections until ctx is cancelled.
@@ -205,6 +210,7 @@ func (s *Service) handleConn(ctx context.Context, conn quic.Connection) {
205 } 210 }
206 s.reg.RecordConnect(hostID) 211 s.reg.RecordConnect(hostID)
207 s.reg.SetAgentVersion(hostID, h.GetFacts().GetAgentVersion()) 212 s.reg.SetAgentVersion(hostID, h.GetFacts().GetAgentVersion())
213 s.reg.SetHostNetworks(hostID, h.GetHostNetworks())
208 s.clearOfferIfDone(hostID, h.GetFacts().GetAgentVersion()) 214 s.clearOfferIfDone(hostID, h.GetFacts().GetAgentVersion())
209 slog.Info("agent connected", "host", hostID, "provisioner", h.GetProvisioner(), "last_seen_epoch", h.GetLastSeenEpoch()) 215 slog.Info("agent connected", "host", hostID, "provisioner", h.GetProvisioner(), "last_seen_epoch", h.GetLastSeenEpoch())
210 216
@@ -380,6 +386,11 @@ func (s *Service) buildSnapshot(hostID string) (*pb.DesiredStateSnapshot, error)
380 SshAuthorizedKey: v.SSHAuthorizedKey, 386 SshAuthorizedKey: v.SSHAuthorizedKey,
381 SshUserCaAuthorizedKeys: cas, 387 SshUserCaAuthorizedKeys: cas,
382 SshHostCert: v.SSHHostCert, 388 SshHostCert: v.SSHHostCert,
389 // Which underlay this guest attaches to: a named host network, or
390 // "" for the NAT default. Admission has already proven the host
391 // advertises the name, so the agent is only being told what it
392 // said it could serve.
393 Network: v.Network,
383 // With a CA in hand the fleet issues host certificates, so a guest 394 // With a CA in hand the fleet issues host certificates, so a guest
384 // must present one. The host generates the key, reports the public 395 // must present one. The host generates the key, reports the public
385 // half, and holds the guest at the gate until the certificate for 396 // half, and holds the guest at the gate until the certificate for
@@ -523,11 +534,33 @@ func (s *Service) applyReport(hostID string, rep *pb.ActualStateReport) {
523 // the cache is updated only after a successful write, so a rejected write 534 // the cache is updated only after a successful write, so a rejected write
524 // never suppresses the next retry. 535 // never suppresses the next retry.
525 for _, v := range rep.GetVms() { 536 for _, v := range rep.GetVms() {
537 vmID := v.GetVmId()
538 // The address the site's own DHCP server granted this guest on its
539 // named network. Recorded in every phase, not just the two below: the
540 // lease can land while the guest is still booting, and it is a fact
541 // about the guest's second NIC rather than about its lifecycle. Empty
542 // is "not discovered", never "gone" — the rule assigned_ip and
543 // guest_cidr both follow, enforced in the UPDATE itself.
544 //
545 // The cache below commits the RAW reported netIP, while the store may
546 // still drop it as unusable and keep the prior value — unlike
547 // statusTracker, this tracker never learns what the store actually
548 // wrote. That divergence is benign: a later good address still reads
549 // as a change and lands, and a repeat of the same unusable address is
550 // just a no-op skip, which is no different from what the store would
551 // have done with it anyway.
552 if netIP := v.GetNetworkIp(); netIP != "" {
553 if err := s.netIPTrack.writeThrough(vmID, netIP, func() error {
554 return s.st.RecordVMNetworkIP(vmID, hostID, netIP)
555 }); err != nil {
556 slog.Warn("record vm network ip", "vm", vmID, "host", hostID, "ip", netIP, "err", err)
557 }
558 }
559
526 phase := v.GetPhase() 560 phase := v.GetPhase()
527 if phase != "ready" && phase != "failed" { 561 if phase != "ready" && phase != "failed" {
528 continue 562 continue
529 } 563 }
530 vmID := v.GetVmId()
531 // Pass the RAW reported ip to RecordVMStatus to preserve its 564 // Pass the RAW reported ip to RecordVMStatus to preserve its
532 // empty-ip-keeps-prior UPDATE semantics, and cache the address it 565 // empty-ip-keeps-prior UPDATE semantics, and cache the address it
533 // reports back rather than the one we sent it. 566 // reports back rather than the one we sent it.
@@ -593,6 +626,7 @@ func (s *Service) applyReport(hostID string, rep *pb.ActualStateReport) {
593 // memory-bounding thing regardless of whether the delete succeeds). 626 // memory-bounding thing regardless of whether the delete succeeds).
594 s.tracker.forget(id) 627 s.tracker.forget(id)
595 s.certTrack.forget(id) 628 s.certTrack.forget(id)
629 s.netIPTrack.forget(id)
596 if err := s.st.HardDeleteVM(id); err != nil { 630 if err := s.st.HardDeleteVM(id); err != nil {
597 slog.Warn("HardDeleteVM failed", "vm", id, "host", hostID, "err", err) 631 slog.Warn("HardDeleteVM failed", "vm", id, "host", hostID, "err", err)
598 } else { 632 } else {
internal/server/syncsvc/syncsvc_test.go
Old New
@@ -120,6 +120,13 @@ func dial(t *testing.T, addr, fp, hostID, cred string) (*testConn, error) {
120 // host change backend. 120 // host change backend.
121 func dialAs(t *testing.T, addr, fp, hostID, cred, provisioner string) (*testConn, error) { 121 func dialAs(t *testing.T, addr, fp, hostID, cred, provisioner string) (*testConn, error) {
122 t.Helper() 122 t.Helper()
123 return dialHello(t, addr, fp, &pb.Hello{HostId: hostID, Provisioner: provisioner, Credential: cred})
124 }
125
126 // dialHello is the connect handshake with the Hello spelled out, for the tests
127 // that care what an agent says about itself on the way in.
128 func dialHello(t *testing.T, addr, fp string, h *pb.Hello) (*testConn, error) {
129 t.Helper()
123 ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) 130 ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
124 defer cancel() 131 defer cancel()
125 conn, err := quic.DialAddr(ctx, addr, transport.ClientTLS(fp), 132 conn, err := quic.DialAddr(ctx, addr, transport.ClientTLS(fp),
@@ -131,8 +138,7 @@ func dialAs(t *testing.T, addr, fp, hostID, cred, provisioner string) (*testConn
131 if err != nil { 138 if err != nil {
132 return nil, err 139 return nil, err
133 } 140 }
134 hello := &pb.AgentMessage{Msg: &pb.AgentMessage_Hello{Hello: &pb.Hello{ 141 hello := &pb.AgentMessage{Msg: &pb.AgentMessage_Hello{Hello: h}}
135 HostId: hostID, Provisioner: provisioner, Credential: cred}}}
136 if err := transport.WriteMsg(up, hello); err != nil { 142 if err := transport.WriteMsg(up, hello); err != nil {
137 return nil, err 143 return nil, err
138 } 144 }
internal/transport/fieldnumbers_test.go
Old New
@@ -55,6 +55,7 @@ var wireSchema = map[string]map[string]protoreflect.FieldNumber{
55 "capacity": 8, 55 "capacity": 8,
56 "credential": 9, 56 "credential": 9,
57 "facts": 10, 57 "facts": 10,
58 "host_networks": 11,
58 }, 59 },
59 "Capacity": { 60 "Capacity": {
60 "vcpus": 1, 61 "vcpus": 1,
@@ -88,6 +89,7 @@ var wireSchema = map[string]map[string]protoreflect.FieldNumber{
88 "last_error": 5, 89 "last_error": 5,
89 "ssh_host_pubkey": 6, 90 "ssh_host_pubkey": 6,
90 "status_detail": 7, 91 "status_detail": 7,
92 "network_ip": 8,
91 }, 93 },
92 "QuarantinedVM": { 94 "QuarantinedVM": {
93 "vm_id": 1, 95 "vm_id": 1,
@@ -123,6 +125,7 @@ var wireSchema = map[string]map[string]protoreflect.FieldNumber{
123 "ssh_host_cert": 17, 125 "ssh_host_cert": 17,
124 "ssh_user_ca_authorized_keys": 18, 126 "ssh_user_ca_authorized_keys": 18,
125 "host_cert_required": 19, 127 "host_cert_required": 19,
128 "network": 20,
126 }, 129 },
127 "DesiredStateSnapshot": { 130 "DesiredStateSnapshot": {
128 "epoch": 1, 131 "epoch": 1,
proto/eitri/v1/sync.proto
Old New
@@ -32,6 +32,11 @@ message Hello {
32 Capacity capacity = 8; 32 Capacity capacity = 8;
33 string credential = 9; // Bearer host credential, verified in first frame 33 string credential = 9; // Bearer host credential, verified in first frame
34 HostFacts facts = 10; // best-effort static host identity; refreshed each Hello 34 HostFacts facts = 10; // best-effort static host identity; refreshed each Hello
35 // The named guest networks this host is configured to serve
36 // (--host-network name=bridge), names only. In Hello rather than the
37 // report because the set changes only with agent configuration, i.e. a
38 // restart. Empty for a Mac, and for a Linux host with none configured.
39 repeated string host_networks = 11;
35 } 40 }
36 41
37 message Capacity { 42 message Capacity {
@@ -69,7 +74,7 @@ message ActualVM {
69 string vm_id = 1; 74 string vm_id = 1;
70 string power = 2; // "running"|"stopped" 75 string power = 2; // "running"|"stopped"
71 string phase = 3; // "creating"|"ready"|"failed"|"quarantined" 76 string phase = 3; // "creating"|"ready"|"failed"|"quarantined"
72 string ip = 4; // the address this guest has, however its host came by it 77 string ip = 4; // the guest's address on its host's NAT underlay — every guest has one, from boot
73 string last_error = 5; 78 string last_error = 5;
74 // The guest's ed25519 HOST public key. It is generated on the host, and the 79 // The guest's ed25519 HOST public key. It is generated on the host, and the
75 // private half never leaves it — this is the only half that travels. Sent on 80 // private half never leaves it — this is the only half that travels. Sent on
@@ -85,6 +90,15 @@ message ActualVM {
85 // VM and means only "nothing to add": an agent that predates this field sends 90 // VM and means only "nothing to add": an agent that predates this field sends
86 // it never, and a console reading it reads exactly what it read before. 91 // it never, and a console reading it reads exactly what it read before.
87 string status_detail = 7; 92 string status_detail = 7;
93 // The address the site's DHCP server granted this guest on its SECOND NIC,
94 // the one attached to the named host network its spec asked for (see
95 // VMDesired.network). Empty for the guests that have no such NIC — the
96 // majority — and for one DHCP round-trip after a networked guest boots,
97 // because the host learns it by watching the exchange rather than granting
98 // it. Never a substitute for ip: that one is known before the guest is even
99 // running, which is why the gate path uses it and this field never appears
100 // there.
101 string network_ip = 8;
88 } 102 }
89 103
90 message QuarantinedVM { 104 message QuarantinedVM {
@@ -164,6 +178,12 @@ message VMDesired {
164 // the agent could not tell "the gate is off, boot uncertified" from "your 178 // the agent could not tell "the gate is off, boot uncertified" from "your
165 // certificate has not come back yet" — both are an empty ssh_host_cert. 179 // certificate has not come back yet" — both are an empty ssh_host_cert.
166 bool host_cert_required = 19; 180 bool host_cert_required = 19;
181 // The named host network this guest attaches to, or "" for the host's
182 // NAT'd private bridge (the default, and the only behavior agents before
183 // this field know). A name is only ever placed on a host that advertised
184 // it (see Hello.host_networks); an agent that has the name but no longer
185 // the configuration fails the VM legibly rather than silently NAT-ing it.
186 string network = 20;
167 } 187 }
168 188
169 message DesiredStateSnapshot { 189 message DesiredStateSnapshot {
scripts/coverage.sh
Old New
@@ -27,12 +27,22 @@ declare -A FLOOR=(
27 [internal/agent/ipalloc]=90 27 [internal/agent/ipalloc]=90
28 [internal/agent/hostinfo]=99 28 [internal/agent/hostinfo]=99
29 [internal/agent/imagecache]=80 29 [internal/agent/imagecache]=80
30 [internal/agent/netenv]=85 30 [internal/agent/netenv]=88
31 # netsnoop's parser, its packet filter and the direction check that decides
32 # whether a lease is real are all covered here; what is left is opening and
33 # binding the AF_PACKET socket, which needs CAP_NET_RAW and a live tap and is
34 # proven on real hardware, so the package number stays in the 60s.
35 [internal/agent/netsnoop]=63
31 [internal/agent/cloudhv]=86 36 [internal/agent/cloudhv]=86
32 [internal/agent/hyperlog]=92 37 [internal/agent/hyperlog]=92
33 [internal/agent/pidfile]=100 38 [internal/agent/pidfile]=100
34 [internal/agent/vfkit]=88 39 [internal/agent/vfkit]=88
35 [internal/agent/syncclient]=82 40 # syncclient has a load-sensitive timing test: 82.3% measured standalone, but
41 # 81.8–83.1% under the all-package parallel run, where a loaded machine can
42 # cost it a branch. The floor sits under the bottom of that range rather than
43 # under the standalone number — a floor that only holds on an idle machine
44 # fails CI at random (collab issue 4cb268b3).
45 [internal/agent/syncclient]=81
36 # exposeproxy runs real proxy goroutines; its coverage wobbles run to run 46 # exposeproxy runs real proxy goroutines; its coverage wobbles run to run
37 # (measured 92.5–93.8%), so this floor carries a margin the others don't need. 47 # (measured 92.5–93.8%), so this floor carries a margin the others don't need.
38 [internal/agent/exposeproxy]=91 48 [internal/agent/exposeproxy]=91
@@ -56,7 +66,7 @@ declare -A FLOOR=(
56 [internal/server/release]=93 66 [internal/server/release]=93
57 [internal/agent/selfupdate]=70 67 [internal/agent/selfupdate]=70
58 [internal/agent/bootstrap]=73 68 [internal/agent/bootstrap]=73
59 [internal/agent/run]=40 69 [internal/agent/run]=48
60 [internal/server/hosttoken]=100 70 [internal/server/hosttoken]=100
61 [internal/server/hub]=94 71 [internal/server/hub]=94
62 [internal/server/syncsvc]=85 72 [internal/server/syncsvc]=85
@@ -66,21 +76,21 @@ declare -A FLOOR=(
66 [internal/site]=84 76 [internal/site]=84
67 [internal/smoke]=56 77 [internal/smoke]=56
68 [internal/cli]=71 78 [internal/cli]=71
69 [internal/mcpserver]=73 79 [internal/mcpserver]=76
70 [internal/oidcprovider]=79 80 [internal/oidcprovider]=79
71 [internal/server/config]=100 81 [internal/server/config]=100
72 [internal/cloudinit]=77 82 [internal/cloudinit]=77
73 [internal/covsnap]=77 83 [internal/covsnap]=77
74 [internal/joinblob]=96 84 [internal/joinblob]=96
75 [internal/gateclient]=60 85 [internal/gateclient]=60
76 # internal/names and internal/random have no tests of their own: they are 86 [internal/names]=74
77 # exercised only through the packages that call them, so they report 0.0% 87 # internal/random has no tests of its own: it is exercised only through the
78 # here. A real floor would be a lie. The 0 floor keeps them COUNTED — so the 88 # packages that call it, so it reports 0.0% here. A real floor would be a
79 # renamed/removed-package guard still accounts for them — while asserting 89 # lie. The 0 floor keeps it COUNTED — so the renamed/removed-package guard
80 # nothing about a number nothing measures. Give a package a real floor only 90 # still accounts for it — while asserting nothing about a number nothing
81 # once it has tests of its own. (internal/pb is generated and stays unlisted; 91 # measures. Give it a real floor once it has tests of its own. (internal/pb
82 # internal/guest has no tests AND no coverable line to gate yet.) 92 # is generated and stays unlisted; internal/guest has no tests AND no
83 [internal/names]=0 93 # coverable line to gate yet.)
84 [internal/random]=0 94 [internal/random]=0
85 ) 95 )
86 96
web/src/lib/api-types.ts
Old New
@@ -1526,6 +1526,7 @@ export interface components {
1526 image_url?: string; 1526 image_url?: string;
1527 mem_mb?: number; 1527 mem_mb?: number;
1528 name?: string; 1528 name?: string;
1529 network?: string;
1529 power_state?: string; 1530 power_state?: string;
1530 ssh_authorized_key?: string; 1531 ssh_authorized_key?: string;
1531 vcpus?: number; 1532 vcpus?: number;
@@ -1598,6 +1599,7 @@ export interface components {
1598 cpu_model: string; 1599 cpu_model: string;
1599 /** Format: date-time */ 1600 /** Format: date-time */
1600 enrolled_at: string; 1601 enrolled_at: string;
1602 host_networks: string[];
1601 id: string; 1603 id: string;
1602 kernel: string; 1604 kernel: string;
1603 /** Format: date-time */ 1605 /** Format: date-time */
@@ -1700,6 +1702,8 @@ export interface components {
1700 lifecycle: string; 1702 lifecycle: string;
1701 mem_mb: number; 1703 mem_mb: number;
1702 name: string; 1704 name: string;
1705 network: string;
1706 network_ip: string;
1703 phase: string; 1707 phase: string;
1704 power_state: string; 1708 power_state: string;
1705 status: string; 1709 status: string;
web/src/lib/fleet.svelte.ts
Old New
@@ -511,6 +511,45 @@ export function vmIP(vm: VM): string {
511 return vm.assigned_ip || '—'; 511 return vm.assigned_ip || '—';
512 } 512 }
513 513
514 /** VM_IP_HINT is the note beside a VM's address. One sentence for every guest,
515 * because every guest has this NIC and it says the same thing on all of them:
516 * an address on the host's private bridge, NAT'd out, reachable off that host
517 * only through the gate or a published port. A named network does not change
518 * this address — it adds a second one, which vmNetworkAddr reports. */
519 export const VM_IP_HINT = '(host bridge, NAT—not reachable off-host)';
520
521 /** vmNetworkAddr is the address a guest's named network granted its second
522 * NIC, or '' when there is nothing to show: a guest that asked for no
523 * network, or one whose network has not answered yet. Used by vmNetworkValue
524 * to extend the Network row's value once the address is known, and by the
525 * fleet-table search box, which matches on it directly.
526 *
527 * Both fields are checked because they can disagree: a VM created before
528 * the freeze, or one a stale row otherwise carries a leftover network_ip
529 * for, must not read as networked once vm.network is empty — an address
530 * with no network to hang it on is not a fact worth showing. */
531 export function vmNetworkAddr(vm: VM): string {
532 return vm.network && vm.network_ip ? vm.network_ip : '';
533 }
534
535 /** vmNetworkAddrHint names the network that did the addressing, because the
536 * address alone does not say which of the host's networks it came from. */
537 export function vmNetworkAddrHint(vm: VM): string {
538 return `(on ${vm.network}, addressed by that network's DHCP)`;
539 }
540
541 /** vmNetworkValue is the Network row's whole value: the operator's own name
542 * for the network alone, until that network's DHCP has answered — then the
543 * name and the address it granted, joined the way the console joins a
544 * primary value onto a secondary one (host status's "· offline", the
545 * platform labels, os·kernel, sessionSummary). One row, one fact, and no
546 * word in it that eitri coined; '' when the guest asked for no network. */
547 export function vmNetworkValue(vm: VM): string {
548 if (!vm.network) return '';
549 const addr = vmNetworkAddr(vm);
550 return addr ? `${vm.network} · ${addr}` : vm.network;
551 }
552
514 /** vmStatus is the single lifecycle status folded from the orthogonal state axes. 553 /** vmStatus is the single lifecycle status folded from the orthogonal state axes.
515 * 554 *
516 * The server owns this derivation (deriveLifecycle in internal/server/api) 555 * The server owns this derivation (deriveLifecycle in internal/server/api)
@@ -531,6 +570,19 @@ export function hostStatusLabel(h: Host): string {
531 return `${h.status}${h.online ? '' : ' · offline'}`; 570 return `${h.status}${h.online ? '' : ' · offline'}`;
532 } 571 }
533 572
573 /** hostNetworkOptions is the named networks a create may ask a host for: the
574 * last set that host advertised, whether or not it is online this instant.
575 *
576 * Online is deliberately not consulted. Gating on it empties the list for
577 * every SSE tick a host spends dark, and an emptied list is how a create for
578 * a named network turns into a NAT VM without anyone saying so; a create
579 * against a host that really is gone is refused by the server, which is the
580 * right place for that refusal. A host not in the fleet (or none selected)
581 * offers nothing. */
582 export function hostNetworkOptions(hosts: Host[], hostID: string | undefined): string[] {
583 return hosts.find((h) => h.id === hostID)?.host_networks ?? [];
584 }
585
534 /** shortFingerprint abbreviates an OpenSSH SHA256 fingerprint for a table cell, 586 /** shortFingerprint abbreviates an OpenSSH SHA256 fingerprint for a table cell,
535 * keeping the SHA256: prefix (so it still reads as a fingerprint and not a 587 * keeping the SHA256: prefix (so it still reads as a fingerprint and not a
536 * hash of some other kind) and enough of the digest to tell two CAs apart by 588 * hash of some other kind) and enough of the digest to tell two CAs apart by
web/src/lib/fleet.test.ts
Old New
@@ -4,6 +4,7 @@ import {
4 fleet, 4 fleet,
5 formatUptime, 5 formatUptime,
6 hostBundle, 6 hostBundle,
7 hostNetworkOptions,
7 hostStatusLabel, 8 hostStatusLabel,
8 joinCommands, 9 joinCommands,
9 sessionSummary, 10 sessionSummary,
@@ -11,6 +12,9 @@ import {
11 upgradeStuck, 12 upgradeStuck,
12 upgradeStuckNote, 13 upgradeStuckNote,
13 vmDetail, 14 vmDetail,
15 vmNetworkAddr,
16 vmNetworkAddrHint,
17 vmNetworkValue,
14 vmTrustStale, 18 vmTrustStale,
15 UPGRADE_STUCK_S, 19 UPGRADE_STUCK_S,
16 type Exposure, 20 type Exposure,
@@ -46,6 +50,8 @@ function vmTrusting(trusted_cas: TrustedCA[] | null): VM {
46 lifecycle: 'ready', 50 lifecycle: 'ready',
47 mem_mb: 1024, 51 mem_mb: 1024,
48 name: 'guest', 52 name: 'guest',
53 network: '',
54 network_ip: '',
49 phase: 'ready', 55 phase: 'ready',
50 power_state: 'running', 56 power_state: 'running',
51 status: 'ready', 57 status: 'ready',
@@ -152,6 +158,7 @@ function hostUpgrading(
152 capacity: { vcpus: 8, mem_mb: 16384, disk_gb: 256 }, 158 capacity: { vcpus: 8, mem_mb: 16384, disk_gb: 256 },
153 cpu_model: 'AMD Ryzen 9 7950X', 159 cpu_model: 'AMD Ryzen 9 7950X',
154 enrolled_at: '2026-08-01T09:00:00Z', 160 enrolled_at: '2026-08-01T09:00:00Z',
161 host_networks: [],
155 id: 'host-1', 162 id: 'host-1',
156 kernel: '6.15.4-arch1-1', 163 kernel: '6.15.4-arch1-1',
157 last_seen: '2026-08-11T09:00:00Z', 164 last_seen: '2026-08-11T09:00:00Z',
@@ -282,6 +289,34 @@ describe('hostStatusLabel', () => {
282 }); 289 });
283 }); 290 });
284 291
292 describe('hostNetworkOptions', () => {
293 /** advertising is one host in the fleet, named and serving `networks`. */
294 function advertising(id: string, networks: string[], online = true): Host {
295 return { ...hostUpgrading('v0.0.6', null, online), id, host_networks: networks };
296 }
297
298 test('the selected host offers the networks it advertised', () => {
299 const hosts = [advertising('host-1', ['lan', 'lab']), advertising('host-2', ['dmz'])];
300 expect(hostNetworkOptions(hosts, 'host-2')).toEqual(['dmz']);
301 });
302
303 test('a host that has gone dark still offers them', () => {
304 // The regression this guards: a host flapping offline for one snapshot
305 // used to empty the list, which cleared the operator's choice and sent a
306 // create for a named network as a create for NAT.
307 expect(hostNetworkOptions([advertising('host-1', ['lan'], false)], 'host-1')).toEqual(['lan']);
308 });
309
310 test('a host advertising nothing offers nothing', () => {
311 expect(hostNetworkOptions([advertising('host-1', [])], 'host-1')).toEqual([]);
312 });
313
314 test('a host not in the fleet offers nothing', () => {
315 expect(hostNetworkOptions([advertising('host-1', ['lan'])], 'host-2')).toEqual([]);
316 expect(hostNetworkOptions([advertising('host-1', ['lan'])], undefined)).toEqual([]);
317 });
318 });
319
285 describe('capacityReading', () => { 320 describe('capacityReading', () => {
286 test('a half-full host has room and a calm level', () => { 321 test('a half-full host has room and a calm level', () => {
287 expect(capacityReading(2, 4)).toEqual({ pct: 50, free: 2, over: 0, level: 'ok' }); 322 expect(capacityReading(2, 4)).toEqual({ pct: 50, free: 2, over: 0, level: 'ok' });
@@ -436,6 +471,45 @@ describe('vmDetail', () => {
436 }); 471 });
437 }); 472 });
438 473
474 describe('the two addresses a guest can have', () => {
475 test('a guest with no named network has no discovered address', () => {
476 expect(vmNetworkAddr(vmTrusting(null))).toBe('');
477 });
478
479 test('a guest whose network has not answered yet has no discovered address', () => {
480 // The NIC exists; the address does not. A placeholder would present a
481 // fact nobody has.
482 expect(vmNetworkAddr({ ...vmTrusting(null), network: 'lan' })).toBe('');
483 });
484
485 test('a leftover network_ip with no network to hang it on has no discovered address', () => {
486 // A stale or pre-freeze row can carry an address with an empty
487 // network — that pairing is not a fact worth showing.
488 expect(vmNetworkAddr({ ...vmTrusting(null), network_ip: '192.168.0.42' })).toBe('');
489 });
490
491 test('a discovered address is shown with the network that granted it', () => {
492 const vm = { ...vmTrusting(null), network: 'lan', network_ip: '192.168.0.42' };
493 expect(vmNetworkAddr(vm)).toBe('192.168.0.42');
494 expect(vmNetworkAddrHint(vm)).toBe("(on lan, addressed by that network's DHCP)");
495 });
496 });
497
498 describe('vmNetworkValue', () => {
499 test('a guest with no named network has no value', () => {
500 expect(vmNetworkValue(vmTrusting(null))).toBe('');
501 });
502
503 test('a guest whose network has not answered yet shows the name alone', () => {
504 expect(vmNetworkValue({ ...vmTrusting(null), network: 'lan' })).toBe('lan');
505 });
506
507 test('a discovered address joins onto the name', () => {
508 const vm = { ...vmTrusting(null), network: 'lan', network_ip: '192.168.0.42' };
509 expect(vmNetworkValue(vm)).toBe('lan · 192.168.0.42');
510 });
511 });
512
439 describe('sessionSummary', () => { 513 describe('sessionSummary', () => {
440 test('a counted port says what it holds and what it has turned away', () => { 514 test('a counted port says what it holds and what it has turned away', () => {
441 expect(sessionSummary(exposure({ active: 7, refused: 12, dropped: 3 }))).toBe( 515 expect(sessionSummary(exposure({ active: 7, refused: 12, dropped: 3 }))).toBe(
web/src/routes/+page.svelte
Old New
@@ -12,6 +12,7 @@
12 restoreVM, 12 restoreVM,
13 vmPower, 13 vmPower,
14 vmIP, 14 vmIP,
15 vmNetworkValue,
15 vmPowerAction, 16 vmPowerAction,
16 vmTrustStale, 17 vmTrustStale,
17 deleteConfirm, 18 deleteConfirm,
@@ -21,6 +22,7 @@
21 refreshUserCAs, 22 refreshUserCAs,
22 capacityReading, 23 capacityReading,
23 hostStatusLabel, 24 hostStatusLabel,
25 hostNetworkOptions,
24 type CreateVMRequest, 26 type CreateVMRequest,
25 type VM 27 type VM
26 } from '$lib/fleet.svelte'; 28 } from '$lib/fleet.svelte';
@@ -84,7 +86,7 @@
84 const shownVMs = $derived( 86 const shownVMs = $derived(
85 fleet.vms.filter((v) => { 87 fleet.vms.filter((v) => {
86 if (!needle) return true; 88 if (!needle) return true;
87 return `${v.name} ${hostName(v.host_id)} ${vmStatus(v)} ${vmPower(v)} ${vmIP(v)}` 89 return `${v.name} ${hostName(v.host_id)} ${vmStatus(v)} ${vmPower(v)} ${vmIP(v)} ${vmNetworkValue(v)}`
88 .toLowerCase() 90 .toLowerCase()
89 .includes(needle); 91 .includes(needle);
90 }) 92 })
@@ -92,6 +94,28 @@
92 94
93 let form = $state<CreateVMRequest>({ host_id: '' }); 95 let form = $state<CreateVMRequest>({ host_id: '' });
94 96
97 // A network belongs to a host, so the choice on offer is the selected host's
98 // advertised set—the last set it advertised, online or not (hostNetworkOptions
99 // carries why online is no part of it).
100 const hostNetworks = $derived(hostNetworkOptions(fleet.hosts, form.host_id));
101 // A picked name belongs to the host it was picked on, so only a change of
102 // host clears it—not the name's absence from the list above. Clearing on
103 // absence would let a host flapping dark for one tick silently turn a
104 // named-network create into a NAT one, which is the outcome this design
105 // forbids: a VM that asked for a named network either gets it or does not
106 // exist. A name the newly
107 // chosen host does not serve therefore reaches the server, which refuses it
108 // and says which networks that host has. pickedOn only remembers which host
109 // answered for the current choice, and is deliberately not $state: this
110 // effect both reads and writes it, so a reactive latch would make the effect
111 // depend on its own write. Guarded on a real change, the shape Console
112 // resets a reused session with.
113 let pickedOn: string | undefined;
114 $effect(() => {
115 if (pickedOn !== undefined && pickedOn !== form.host_id) form.network = '';
116 pickedOn = form.host_id;
117 });
118
95 // noCA is the tenant with nothing registered, which is worth saying before 119 // noCA is the tenant with nothing registered, which is worth saying before
96 // the form is filled in rather than after it is submitted. Held off until 120 // the form is filled in rather than after it is submitted. Held off until
97 // the CAs have actually been fetched, so an empty list that only means 121 // the CAs have actually been fetched, so an empty list that only means
@@ -270,7 +294,18 @@
270 {/if} 294 {/if}
271 </td> 295 </td>
272 <td class="num">{vmCountByHost.get(h.id) ?? 0}</td> 296 <td class="num">{vmCountByHost.get(h.id) ?? 0}</td>
273 <td>{h.bridge_cidr || '—'}</td> 297 <td>
298 {h.bridge_cidr || '—'}
299 <!-- The named networks beside the NAT bridge every host has, so
300 that adding --host-network to a unit can be confirmed here
301 rather than inferred from what the create dialog offers. A
302 host serving none says nothing. -->
303 {#if h.host_networks.length > 0}
304 <span class="networks" title="advertised with --host-network"
305 >· {h.host_networks.join(', ')}</span
306 >
307 {/if}
308 </td>
274 <td class="num" class:over={capacityReading(h.allocated.vcpus, h.capacity.vcpus).over > 0} 309 <td class="num" class:over={capacityReading(h.allocated.vcpus, h.capacity.vcpus).over > 0}
275 >{h.allocated.vcpus}/{h.capacity.vcpus || '?'}</td 310 >{h.allocated.vcpus}/{h.capacity.vcpus || '?'}</td
276 > 311 >
@@ -378,6 +413,17 @@
378 {/each} 413 {/each}
379 </select> 414 </select>
380 </label> 415 </label>
416 {#if hostNetworks.length > 0}
417 <label>
418 Network
419 <select bind:value={form.network}>
420 <option value="">NAT (default)</option>
421 {#each hostNetworks as n}
422 <option value={n}>{n}</option>
423 {/each}
424 </select>
425 </label>
426 {/if}
381 <label> 427 <label>
382 Name (optional) 428 Name (optional)
383 <input bind:value={form.name} placeholder="auto: sandbox-xxxx" /> 429 <input bind:value={form.name} placeholder="auto: sandbox-xxxx" />
@@ -501,6 +547,12 @@
501 color: var(--bad); 547 color: var(--bad);
502 cursor: help; 548 cursor: help;
503 } 549 }
550 /* Faint, like every other secondary row fact: the bridge is the reading,
551 the names are what else this host can put a guest on. */
552 .networks {
553 color: var(--faint);
554 cursor: help;
555 }
504 .actions { 556 .actions {
505 display: flex; 557 display: flex;
506 gap: 0.5em; 558 gap: 0.5em;
web/src/routes/vms/[id]/+page.svelte
Old New
@@ -12,6 +12,10 @@
12 teardownApprox, 12 teardownApprox,
13 vmPower, 13 vmPower,
14 vmIP, 14 vmIP,
15 VM_IP_HINT,
16 vmNetworkValue,
17 vmNetworkAddr,
18 vmNetworkAddrHint,
15 vmDetail, 19 vmDetail,
16 sessionSummary, 20 sessionSummary,
17 vmPowerAction, 21 vmPowerAction,
@@ -248,7 +252,13 @@
248 </td> 252 </td>
249 </tr> 253 </tr>
250 <tr><th>Power</th><td>{vmPower(vm)} (desired: {vm.power_state})</td></tr> 254 <tr><th>Power</th><td>{vmPower(vm)} (desired: {vm.power_state})</td></tr>
251 <tr><th>IP</th><td>{vmIP(vm)} <span class="hint">(host bridge, NAT—not reachable off-host)</span></td></tr> 255 <!-- One row for the whole story: the network the operator named at
256 create, and — once that network's own DHCP has answered — the
257 second address it granted this guest, joined onto the same
258 value. Nothing here is a word eitri invented: the label matches
259 the API field, and the name in the value is the operator's own. -->
260 {#if vmNetworkValue(vm)}<tr><th>Network</th><td>{vmNetworkValue(vm)}{#if vmNetworkAddr(vm)} <span class="hint">{vmNetworkAddrHint(vm)}</span>{/if}</td></tr>{/if}
261 <tr><th>IP</th><td>{vmIP(vm)} <span class="hint">{VM_IP_HINT}</span></td></tr>
252 <tr><th>Resources</th><td>{vm.vcpus}c / {vm.mem_mb}MB / {vm.disk_gb}GB</td></tr> 262 <tr><th>Resources</th><td>{vm.vcpus}c / {vm.mem_mb}MB / {vm.disk_gb}GB</td></tr>
253 <tr><th>Image</th><td class="wrap">{vm.image_url}</td></tr> 263 <tr><th>Image</th><td class="wrap">{vm.image_url}</td></tr>
254 <tr> 264 <tr>