f52ac6a1
networks: a VM joins the operator's LAN by name, and keeps its NAT life
a73x 2026-08-14 16:49
Commit message
ROADMAP.md
| Old | New | ||
|---|---|---|---|
| @@ -8,9 +8,10 @@ What shipped in each version is in the | |||
| 8 | - Connectors: browser clients (claude.ai among them) sign in to the MCP | 8 | - Connectors: browser clients (claude.ai among them) sign in to the MCP |
| 9 | endpoint with OAuth rather than a pasted token—dynamic client registration | 9 | endpoint with OAuth rather than a pasted token—dynamic client registration |
| 10 | on eitri-oidc, so a connector enrols itself | 10 | on eitri-oidc, so a connector enrols itself |
| 11 | - Bridged guest networking: a Linux host puts its guests on the operator's | 11 | - Per-VM named networks: a host advertises the bridges its operator declared |
| 12 | own network—taps on a named bridge, the site's DHCP addresses the guests, | 12 | (`--host-network lan=br0`), a VM requests one by name, and admission matches |
| 13 | the agent discovers the result | 13 | or refuses—taps on the operator's bridge, the site's DHCP addresses the |
| 14 | guests, the agent discovers the result | ||
| 14 | - Public gateway: route inbound traffic to guest services through eitri, | 15 | - Public gateway: route inbound traffic to guest services through eitri, |
| 15 | building on published ports | 16 | building on published ports |
| 16 | - Hot-reload trusted CAs into running guests: a CA added to the tenant reaches | 17 | - Hot-reload trusted CAs into running guests: a CA added to the tenant reaches |
docs/assumptions.md
| Old | New | ||
|---|---|---|---|
| @@ -570,3 +570,80 @@ refusing at boot — a `--max-vcpus` cap (no report carries a core count), a | |||
| 570 | failed probe, and a cap set at exactly the machine's total. False the moment | 570 | failed probe, and a cap set at exactly the machine's total. False the moment |
| 571 | the wire says which a number is; the refusal should then judge the declared | 571 | the wire says which a number is; the refusal should then judge the declared |
| 572 | capacity directly and this arithmetic goes away. | 572 | capacity directly and this arithmetic goes away. |
| 573 | |||
| 574 | ### A networked guest is flat on the LAN, and still on its host's private fabric | ||
| 575 | |||
| 576 | A guest on a named network is a machine on that network: every port, every | ||
| 577 | device, no gate in front of that NIC. That is the feature — the stance the | ||
| 578 | exposure proxy already takes for one published port ("the LAN address is the | ||
| 579 | feature"), here unbounded — and it underpins putting guests on a named network | ||
| 580 | with nothing of eitri's in front of them. NAT-only stays the default; flatness | ||
| 581 | is asked for by name, per VM. **True by construction**, and false as an | ||
| 582 | expectation the moment a tenant assumes eitri-enforced isolation between | ||
| 583 | guests sharing a LAN — that is VLAN filtering at the site's switch, | ||
| 584 | deliberately out of scope. | ||
| 585 | |||
| 586 | The second half is the price of the NIC being additive: the same guest also | ||
| 587 | sits on `eitri0` beside every other guest on its host, because that is where | ||
| 588 | its reservation, its gate splice and its egress live. VM-to-VM on the host is | ||
| 589 | exactly why the NIC is additive rather than a replacement — and it means | ||
| 590 | compromising a LAN-facing guest is a foothold on the host's private L2, not | ||
| 591 | only on the network that guest was exposed to. Stated rather than mitigated: | ||
| 592 | the alternative was a guest that trades one attachment for the other, which is | ||
| 593 | two networking models to ship and reason about instead of one. | ||
| 594 | |||
| 595 | ### A named network's LAN serves DHCP and tolerates new MACs | ||
| 596 | |||
| 597 | A guest's named NIC asks the site's DHCP server for an address and shows up at | ||
| 598 | the switch as one more MAC. Underpins allocating nothing for that NIC: the | ||
| 599 | site's DHCP addresses it, the agent discovers the result. **Falsified at the | ||
| 600 | switch**: 802.1X or MAC ACLs refuse the guest with the switch's diagnostics, | ||
| 601 | not eitri's. The soft half is a guest configured static — it never DHCPs, so | ||
| 602 | it is reachable there but unreported and `network_ip` stays empty, a | ||
| 603 | documented limitation rather than a fault. | ||
| 604 | |||
| 605 | Tolerating new MACs is not tolerating new servers, and eitri does not leave the | ||
| 606 | site to tell them apart. A bridge-family nft rule drops UDP source port 67 at | ||
| 607 | the ingress of every named tap, so a guest of eitri's cannot answer a lease — | ||
| 608 | not to the operator's real machines, not to a sibling guest across the bridge, | ||
| 609 | not to the switch beyond it. The rule is programmed only on hosts that serve | ||
| 610 | named networks, and a kernel that cannot take it fails agent startup rather | ||
| 611 | than bridging unfiltered. DHCPv6 (source port 547) and router advertisements | ||
| 612 | are deliberately unfiltered, and that is the design rather than a gap in it: a | ||
| 613 | bridged guest is meant to be a full peer, an operator may run a router guest on | ||
| 614 | purpose, and neither v6 path can poison what eitri records, since the snoop | ||
| 615 | reads IPv4 ACKs only. **True by construction** for v4; false as an expectation | ||
| 616 | the moment a tenant reads it as eitri policing the LAN's v6 too. | ||
| 617 | |||
| 618 | The blast radius of both halves is bounded by the NAT NIC being unconditional. | ||
| 619 | Whatever the site's DHCP does or refuses to do, `assigned_ip` is allocated | ||
| 620 | before the guest boots, so the gate, the console and every published port | ||
| 621 | still reach it: a network that never answers costs the guest its LAN identity, | ||
| 622 | not its existence. | ||
| 623 | |||
| 624 | One downgrade residue rides with this. Admission keeps networked VMs off agents | ||
| 625 | that never advertised the network — a pre-field agent advertises nothing and | ||
| 626 | is refused — but a host that advertised and was then downgraded to a pre-field | ||
| 627 | binary ignores the wire field and gives those guests the NAT NIC alone — no | ||
| 628 | second NIC, no LAN address, the one outcome admission exists to forbid. Closing | ||
| 629 | it takes a release floor at snapshot time, the `FirstDatagramExposures` pattern; | ||
| 630 | accepted for now because downgrading an advertising host is an operator's own | ||
| 631 | deliberate act, and rare. | ||
| 632 | |||
| 633 | ### The guest honours the route metrics the seed asks for | ||
| 634 | |||
| 635 | A two-NIC guest is handed `dhcp4-overrides.route-metric` in its network-config | ||
| 636 | v2 seed — 100 on the NAT NIC, 200 on the named one — and that is the whole | ||
| 637 | mechanism putting the default route on the NAT NIC. Underpins the egress | ||
| 638 | promise: outbound leaves through the host's masquerade whatever the operator's | ||
| 639 | network is doing, and either attachment can be down without taking the other | ||
| 640 | with it. **True on the Ubuntu cloud image**, where cloud-init renders v2 to | ||
| 641 | netplan and the metrics arrive verbatim. **Unverified off netplan**: | ||
| 642 | cloud-init's other renderers, sysconfig and NetworkManager, have historically | ||
| 643 | ignored `dhcp4-overrides.route-metric` when translating v2, which leaves both | ||
| 644 | NICs at the distro's default metrics and the default route undetermined — a | ||
| 645 | guest that egresses over the LAN, or that swaps between the two on a lease | ||
| 646 | renewal. | ||
| 647 | Falsified by an operator pointing the default image at a non-netplan distro, | ||
| 648 | and quietly: both NICs still address, the guest still boots, and nothing in | ||
| 649 | eitri reads a route table to notice. | ||
docs/decisions.md
| Old | New | ||
|---|---|---|---|
| @@ -150,3 +150,54 @@ an MCP server by self-hosting the plane, and a public front would target this | |||
| 150 | same endpoint rather than being a second thing. Reversed by a genuine air-gapped | 150 | same endpoint rather than being a second thing. Reversed by a genuine air-gapped |
| 151 | need: guests to drive with no plane to reach. | 151 | need: guests to drive with no plane to reach. |
| 152 | Details in [mcp.md](mcp.md). | 152 | Details in [mcp.md](mcp.md). |
| 153 | |||
| 154 | ### Per-VM named networks | ||
| 155 | |||
| 156 | A host advertises the bridge-backed networks its operator declared | ||
| 157 | (`--host-network name=bridge`); a VM requests one by name; admission places it | ||
| 158 | on a host advertising that name or refuses at create. Instead of a per-host | ||
| 159 | bridged mode, which could not express the actual fleet — one host serving a | ||
| 160 | LAN pet and NAT'd cattle at once. And never a silent fallback: an agent from | ||
| 161 | before the field advertises nothing and would quietly NAT the guest, so | ||
| 162 | silence refuses — a VM that asked for the LAN either gets it or does not | ||
| 163 | exist. | ||
| 164 | |||
| 165 | The operator owns the bridge; eitri uses it and never creates it. The | ||
| 166 | precedent is unanimous: runtime-created network objects are how everyone | ||
| 167 | builds NAT bridges and macvlan attachments — docker0, virbr0, eitri's own | ||
| 168 | eitri0, Docker's macvlan — while a bridge that enslaves the uplink is | ||
| 169 | persistent host configuration everywhere one exists — Proxmox's vmbr0, | ||
| 170 | libvirt's bridge mode. eitri copies the only precedent that exists. And the | ||
| 171 | address is discovered, never allocated: the agent snoops the guest's own tap | ||
| 172 | for the DHCP ACK keyed to its deterministic MAC — the site's DHCP addresses | ||
| 173 | the guests, the agent discovers the result. | ||
| 174 | |||
| 175 | ### A named network adds a NIC, it does not replace one | ||
| 176 | |||
| 177 | Every guest keeps its NAT attachment: `eth0` on `eitri0`, its reservation | ||
| 178 | allocated before boot, the gate's splice target, its egress and its way to the | ||
| 179 | other guests on that host. A named network is a SECOND NIC on the operator's | ||
| 180 | bridge, addressed by the site's own DHCP, and the VM reports both addresses — | ||
| 181 | `assigned_ip` from boot, `network_ip` once the network answers. The default | ||
| 182 | route stays on the NAT NIC (metric 100 against 200, as the default Ubuntu | ||
| 183 | image's netplan renders them), so egress is deterministic and either side can | ||
| 184 | be down without taking the other with it. | ||
| 185 | |||
| 186 | This is multipass's shape, which is the shape the daily driver already runs: | ||
| 187 | NAT primary with the default route, LAN secondary, and which address a service | ||
| 188 | advertises left to that service. The replace-shaped version — LAN *instead of* | ||
| 189 | NAT — was built first and rejected: it shipped two networking models, with a | ||
| 190 | guest's reachability, its gate path and its VM-to-VM story all forking on | ||
| 191 | which one it was, and it opened a discovery window on the gate path that the | ||
| 192 | NAT-only guest never had. One model costs a stated blast radius instead (see | ||
| 193 | [assumptions.md](assumptions.md) — a LAN-facing guest is also on the host's | ||
| 194 | private L2), and that trade was taken deliberately. | ||
| 195 | |||
| 196 | ### A named network is called by the operator's name for it | ||
| 197 | |||
| 198 | The API and wire already agreed with themselves — `network`, `network_ip`, | ||
| 199 | `--host-network` — but the console had coined a second word, "LAN", for the | ||
| 200 | same idea, on a second row for it. Both are gone: one Network row now carries | ||
| 201 | the operator's own name as its value, joined to the discovered address once | ||
| 202 | the network's DHCP answers (`lan · 192.168.0.37`). eitri displays the | ||
| 203 | operator's vocabulary rather than inventing its own. | ||
docs/faq.md
| Old | New | ||
|---|---|---|---|
| @@ -3,14 +3,21 @@ | |||
| 3 | ## How do VMs get network access? | 3 | ## How do VMs get network access? |
| 4 | 4 | ||
| 5 | Outbound works out of the box: guests are NAT'd through their host and reach | 5 | Outbound works out of the box: guests are NAT'd through their host and reach |
| 6 | the internet like any process on it. Inbound is deliberately minimal: guests | 6 | the internet like any process on it. That NIC is unconditional—the guest's |
| 7 | live on private per-host bridges, and the only built-in way in is the SSH | 7 | egress, its way to the other guests on that host, and where the SSH |
| 8 | [jump gate](ssh-access.md). | 8 | [jump gate](ssh-access.md) meets it. |
| 9 | 9 | ||
| 10 | To serve traffic from a VM, or to reach one directly from another network, | 10 | Inbound has two answers. The gate works from anywhere: it authenticates every |
| 11 | install [Tailscale](https://tailscale.com) (or WireGuard) inside it—it is | 11 | connection against your tenant's CA and tunnels you in. The other is to put the |
| 12 | a normal Linux machine. A public gateway is on the | 12 | VM on your own network—declare a bridge on a Linux host, name it to the agent |
| 13 | [roadmap](../ROADMAP.md). | 13 | (`--host-network lan=br0`), pick that name at create. Your own DHCP server |
| 14 | addresses the guest's second NIC, and it is then a machine on the network: | ||
| 15 | every port, every device, nothing of eitri's in front of that NIC. The | ||
| 16 | quickstart's [Put a VM on your LAN](quickstart.md) is the recipe. | ||
| 17 | |||
| 18 | For reach beyond that LAN it is still a normal Linux machine, so install | ||
| 19 | [Tailscale](https://tailscale.com) (or WireGuard) inside it. A public gateway | ||
| 20 | is on the [roadmap](../ROADMAP.md). | ||
| 14 | 21 | ||
| 15 | ## Is an exposed port authenticated? | 22 | ## Is an exposed port authenticated? |
| 16 | 23 | ||
docs/mcp.md
| Old | New | ||
|---|---|---|---|
| @@ -15,9 +15,9 @@ self-hosted eitri is a self-hosted MCP server, with nothing extra to install. | |||
| 15 | 15 | ||
| 16 | | Tool | Purpose | | 16 | | Tool | Purpose | |
| 17 | |---|---| | 17 | |---|---| |
| 18 | | `vm_create` | Create a VM; by default waits for `lifecycle=ready` + IP, then for guest SSH and `cloud-init status --wait` to finish. Refused while the tenant has no registered SSH user CA—call `ca_upload` first, or the guest would trust nothing. | | 18 | | `vm_create` | Create a VM; by default waits for `lifecycle=ready` + IP, then for guest SSH and `cloud-init status --wait` to finish. Takes an optional `network`, the named host network to give the guest a second NIC on; a host serves only the names it advertises, and naming another refuses the create. Returns `network` and `network_ip`—the address that network's own DHCP granted, empty at ready when it has not answered yet. Refused while the tenant has no registered SSH user CA—call `ca_upload` first, or the guest would trust nothing. | |
| 19 | | `vm_list` | List all VMs on the fleet (id, name, lifecycle, IP, size). | | 19 | | `vm_list` | List all VMs on the fleet (id, name, lifecycle, IP, size, `network` and `network_ip`). | |
| 20 | | `vm_info` | Show one VM's state plus a ready-to-use `ssh` command. | | 20 | | `vm_info` | Show one VM's state, both addresses, plus a ready-to-use `ssh` command. Where to read `network_ip` back when it was still empty at create. | |
| 21 | | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. | | 21 | | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. | |
| 22 | | `vm_write_file` | Write content to a path in a VM over SFTP (parent dirs created). | | 22 | | `vm_write_file` | Write content to a path in a VM over SFTP (parent dirs created). | |
| 23 | | `vm_read_file` | Read a file from a VM over SFTP (capped at 1 MiB, truncation flagged). | | 23 | | `vm_read_file` | Read a file from a VM over SFTP (capped at 1 MiB, truncation flagged). | |
docs/openapi.json
| Old | New | ||
|---|---|---|---|
| @@ -140,6 +140,9 @@ | |||
| 140 | "name": { | 140 | "name": { |
| 141 | "type": "string" | 141 | "type": "string" |
| 142 | }, | 142 | }, |
| 143 | "network": { | ||
| 144 | "type": "string" | ||
| 145 | }, | ||
| 143 | "power_state": { | 146 | "power_state": { |
| 144 | "type": "string" | 147 | "type": "string" |
| 145 | }, | 148 | }, |
| @@ -400,6 +403,12 @@ | |||
| 400 | "format": "date-time", | 403 | "format": "date-time", |
| 401 | "type": "string" | 404 | "type": "string" |
| 402 | }, | 405 | }, |
| 406 | "host_networks": { | ||
| 407 | "items": { | ||
| 408 | "type": "string" | ||
| 409 | }, | ||
| 410 | "type": "array" | ||
| 411 | }, | ||
| 403 | "id": { | 412 | "id": { |
| 404 | "type": "string" | 413 | "type": "string" |
| 405 | }, | 414 | }, |
| @@ -485,6 +494,7 @@ | |||
| 485 | "capacity", | 494 | "capacity", |
| 486 | "cpu_model", | 495 | "cpu_model", |
| 487 | "enrolled_at", | 496 | "enrolled_at", |
| 497 | "host_networks", | ||
| 488 | "id", | 498 | "id", |
| 489 | "kernel", | 499 | "kernel", |
| 490 | "name", | 500 | "name", |
| @@ -798,6 +808,12 @@ | |||
| 798 | "name": { | 808 | "name": { |
| 799 | "type": "string" | 809 | "type": "string" |
| 800 | }, | 810 | }, |
| 811 | "network": { | ||
| 812 | "type": "string" | ||
| 813 | }, | ||
| 814 | "network_ip": { | ||
| 815 | "type": "string" | ||
| 816 | }, | ||
| 801 | "phase": { | 817 | "phase": { |
| 802 | "type": "string" | 818 | "type": "string" |
| 803 | }, | 819 | }, |
| @@ -837,6 +853,8 @@ | |||
| 837 | "lifecycle", | 853 | "lifecycle", |
| 838 | "mem_mb", | 854 | "mem_mb", |
| 839 | "name", | 855 | "name", |
| 856 | "network", | ||
| 857 | "network_ip", | ||
| 840 | "phase", | 858 | "phase", |
| 841 | "power_state", | 859 | "power_state", |
| 842 | "status", | 860 | "status", |
docs/quickstart.md
| Old | New | ||
|---|---|---|---|
| @@ -245,6 +245,58 @@ more reason to publish only onto a network you are content to serve. | |||
| 245 | Remove the row to take the port down. Deleting the VM takes its exposures with | 245 | Remove the row to take the port down. Deleting the VM takes its exposures with |
| 246 | it. | 246 | it. |
| 247 | 247 | ||
| 248 | ### Put a VM on your LAN | ||
| 249 | |||
| 250 | Every guest lives on a private bridge behind its host, NAT'd out. A VM can | ||
| 251 | *also* sit on your own network: declare a bridge on a Linux host, name it to | ||
| 252 | the agent, pick that name at create—a second NIC there, and it keeps the first. | ||
| 253 | |||
| 254 | The bridge is yours, declared in the host's own network config; eitri attaches | ||
| 255 | taps and never creates or addresses one. netplan shown, networkd and | ||
| 256 | NetworkManager have equivalents: | ||
| 257 | |||
| 258 | ```yaml | ||
| 259 | network: | ||
| 260 | ethernets: | ||
| 261 | enp4s0: {} | ||
| 262 | bridges: | ||
| 263 | br0: | ||
| 264 | interfaces: [enp4s0] | ||
| 265 | dhcp4: true # or the host's static address, verbatim | ||
| 266 | ``` | ||
| 267 | |||
| 268 | The host's address moves onto `br0` at boot, and so must anything bound to the | ||
| 269 | uplink by name—a PXE dnsmasq's `interface=enp4s0` becomes `interface=br0`. | ||
| 270 | |||
| 271 | Name the network to the agent in `/etc/default/eitri-agent`, then restart it: | ||
| 272 | |||
| 273 | ```sh | ||
| 274 | EITRI_AGENT_FLAGS="--host-network lan=br0" | ||
| 275 | ``` | ||
| 276 | |||
| 277 | The flag repeats. Names are 1–32 of `[a-z0-9-]`, no leading | ||
| 278 | or trailing hyphen; `nat` is reserved. Linux hosts only. An agent whose named | ||
| 279 | bridge is missing refuses to start, naming it. | ||
| 280 | |||
| 281 | Create the VM on that network—the console's dialog shows a **Network** select | ||
| 282 | for a host advertising any, and naming one its host does not is refused. | ||
| 283 | |||
| 284 | The VM page then shows both addresses. **IP** is the private one the gate, the | ||
| 285 | console and published ports aim at; **Network** is your own name for the | ||
| 286 | network, and gains the address your DHCP server granted the second NIC once | ||
| 287 | that lease lands. A guest configured static there never DHCPs—reachable all | ||
| 288 | the same, but **Network** shows the name alone. | ||
| 289 | |||
| 290 | The private NIC keeps the default route (route metric 100, against 200 on the | ||
| 291 | second NIC), so outbound leaves through the host's masquerade whatever your | ||
| 292 | network is doing—on the default Ubuntu image, where netplan honours those | ||
| 293 | metrics. Which address a service advertises is that service's own business. | ||
| 294 | |||
| 295 | A guest on your network is flat on it: every port, every device, no gate in | ||
| 296 | front of that NIC—attach what you are content to have live there, the rest | ||
| 297 | NAT-only. It is also still on the host's private bridge beside that host's | ||
| 298 | other guests, which is what lets them talk and what a compromise of it reaches. | ||
| 299 | |||
| 248 | ## Self-hosting | 300 | ## Self-hosting |
| 249 | 301 | ||
| 250 | Run the whole control plane yourself—everything below is your own box. | 302 | Run the whole control plane yourself—everything below is your own box. |
docs/shape.html
| Old | New | ||
|---|---|---|---|
| @@ -208,11 +208,18 @@ | |||
| 208 | "imports": [ | 208 | "imports": [ |
| 209 | "internal/agent/dhcp", | 209 | "internal/agent/dhcp", |
| 210 | "internal/agent/exec", | 210 | "internal/agent/exec", |
| 211 | "internal/agent/netsnoop", | ||
| 211 | "internal/agent/permanent", | 212 | "internal/agent/permanent", |
| 212 | "internal/agent/state" | 213 | "internal/agent/state" |
| 213 | ] | 214 | ] |
| 214 | }, | 215 | }, |
| 215 | { | 216 | { |
| 217 | "importPath": "internal/agent/netsnoop", | ||
| 218 | "plane": "data", | ||
| 219 | "synopsis": "Package netsnoop discovers the address a guest's named-network NIC was granted, by watching that NIC's own tap for its DHCP exchange.", | ||
| 220 | "imports": [] | ||
| 221 | }, | ||
| 222 | { | ||
| 216 | "importPath": "internal/agent/permanent", | 223 | "importPath": "internal/agent/permanent", |
| 217 | "plane": "data", | 224 | "plane": "data", |
| 218 | "synopsis": "Package permanent mints agent errors that no retry can fix.", | 225 | "synopsis": "Package permanent mints agent errors that no retry can fix.", |
| @@ -253,7 +260,8 @@ | |||
| 253 | "internal/agent/statelock", | 260 | "internal/agent/statelock", |
| 254 | "internal/agent/syncclient", | 261 | "internal/agent/syncclient", |
| 255 | "internal/covsnap", | 262 | "internal/covsnap", |
| 256 | "internal/joinblob" | 263 | "internal/joinblob", |
| 264 | "internal/names" | ||
| 257 | ] | 265 | ] |
| 258 | }, | 266 | }, |
| 259 | { | 267 | { |
docs/shape.json
| Old | New | ||
|---|---|---|---|
| @@ -157,11 +157,18 @@ | |||
| 157 | "imports": [ | 157 | "imports": [ |
| 158 | "internal/agent/dhcp", | 158 | "internal/agent/dhcp", |
| 159 | "internal/agent/exec", | 159 | "internal/agent/exec", |
| 160 | "internal/agent/netsnoop", | ||
| 160 | "internal/agent/permanent", | 161 | "internal/agent/permanent", |
| 161 | "internal/agent/state" | 162 | "internal/agent/state" |
| 162 | ] | 163 | ] |
| 163 | }, | 164 | }, |
| 164 | { | 165 | { |
| 166 | "importPath": "internal/agent/netsnoop", | ||
| 167 | "plane": "data", | ||
| 168 | "synopsis": "Package netsnoop discovers the address a guest's named-network NIC was granted, by watching that NIC's own tap for its DHCP exchange.", | ||
| 169 | "imports": [] | ||
| 170 | }, | ||
| 171 | { | ||
| 165 | "importPath": "internal/agent/permanent", | 172 | "importPath": "internal/agent/permanent", |
| 166 | "plane": "data", | 173 | "plane": "data", |
| 167 | "synopsis": "Package permanent mints agent errors that no retry can fix.", | 174 | "synopsis": "Package permanent mints agent errors that no retry can fix.", |
| @@ -202,7 +209,8 @@ | |||
| 202 | "internal/agent/statelock", | 209 | "internal/agent/statelock", |
| 203 | "internal/agent/syncclient", | 210 | "internal/agent/syncclient", |
| 204 | "internal/covsnap", | 211 | "internal/covsnap", |
| 205 | "internal/joinblob" | 212 | "internal/joinblob", |
| 213 | "internal/names" | ||
| 206 | ] | 214 | ] |
| 207 | }, | 215 | }, |
| 208 | { | 216 | { |
go.mod
| Old | New | ||
|---|---|---|---|
| @@ -14,6 +14,7 @@ require ( | |||
| 14 | github.com/stretchr/testify v1.11.1 | 14 | github.com/stretchr/testify v1.11.1 |
| 15 | github.com/yuin/goldmark v1.8.4 | 15 | github.com/yuin/goldmark v1.8.4 |
| 16 | golang.org/x/crypto v0.54.0 | 16 | golang.org/x/crypto v0.54.0 |
| 17 | golang.org/x/net v0.56.0 | ||
| 17 | golang.org/x/oauth2 v0.36.0 | 18 | golang.org/x/oauth2 v0.36.0 |
| 18 | golang.org/x/sync v0.20.0 | 19 | golang.org/x/sync v0.20.0 |
| 19 | golang.org/x/sys v0.47.0 | 20 | golang.org/x/sys v0.47.0 |
| @@ -55,7 +56,6 @@ require ( | |||
| 55 | go.uber.org/mock v0.4.0 // indirect | 56 | go.uber.org/mock v0.4.0 // indirect |
| 56 | golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect | 57 | golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect |
| 57 | golang.org/x/mod v0.33.0 // indirect | 58 | golang.org/x/mod v0.33.0 // indirect |
| 58 | golang.org/x/net v0.56.0 // indirect | ||
| 59 | golang.org/x/tools v0.42.0 // indirect | 59 | golang.org/x/tools v0.42.0 // indirect |
| 60 | modernc.org/libc v1.72.3 // indirect | 60 | modernc.org/libc v1.72.3 // indirect |
| 61 | modernc.org/mathutil v1.7.1 // indirect | 61 | modernc.org/mathutil v1.7.1 // indirect |
internal/agent/cloudhv/cloudhv.go
| Old | New | ||
|---|---|---|---|
| @@ -45,19 +45,30 @@ type PumpHooks interface { | |||
| 45 | // responder) is the composition root's job and never appears here: it happens | 45 | // responder) is the composition root's job and never appears here: it happens |
| 46 | // once at agent start, not once per guest. | 46 | // once at agent start, not once per guest. |
| 47 | type Network interface { | 47 | type Network interface { |
| 48 | // ReserveIP returns this VM's sticky address, recording its DHCP | 48 | // ReserveIP returns this VM's sticky address on the NAT underlay, |
| 49 | // reservation. | 49 | // recording its DHCP reservation. Every guest has one, whatever else its |
| 50 | // spec asks for. | ||
| 50 | ReserveIP(vmID string) (string, error) | 51 | ReserveIP(vmID string) (string, error) |
| 51 | // Address returns the address already reserved for vmID, or "" if none. | 52 | // Address returns the NAT address already reserved for vmID, or "" if none. |
| 52 | // It never allocates. | 53 | // It never allocates. |
| 53 | Address(vmID string) string | 54 | Address(vmID string) string |
| 54 | // CreateTap creates the VM's tap device, enslaves it to the bridge and | 55 | // NetworkAddress returns the address the site's own DHCP server granted |
| 55 | // pins ip as the guest's DHCP reservation. Idempotent. | 56 | // this VM's named-network NIC, or "" — for a guest with no such NIC, and |
| 56 | CreateTap(ctx context.Context, vmID, ip string) error | 57 | // for one whose guest has not finished asking yet. The host learns it by |
| 57 | // DeleteTap removes the reservation and the tap. Idempotent. | 58 | // watching, so unlike Address it is not known at boot. |
| 59 | NetworkAddress(vmID string) string | ||
| 60 | // CreateTap creates the VM's tap on the NAT bridge and pins ip as its | ||
| 61 | // reservation. When network is non-empty it ALSO creates a second tap on | ||
| 62 | // that named network's bridge — additional, never instead. Idempotent. | ||
| 63 | CreateTap(ctx context.Context, vmID, ip, network string) error | ||
| 64 | // DeleteTap removes the reservation and both taps. Idempotent. | ||
| 58 | DeleteTap(ctx context.Context, vmID string) error | 65 | DeleteTap(ctx context.Context, vmID string) error |
| 59 | // TapName is the device name that goes in cloud-hypervisor's --net argument. | 66 | // TapName is the device name of the NAT NIC's tap — the first --net |
| 67 | // argument, and therefore the guest's eth0. | ||
| 60 | TapName(vmID string) string | 68 | TapName(vmID string) string |
| 69 | // NetTapName is the device name of the named-network NIC's tap, used only | ||
| 70 | // by a VM whose spec asked for one. | ||
| 71 | NetTapName(vmID string) string | ||
| 61 | } | 72 | } |
| 62 | 73 | ||
| 63 | // Provisioner manages cloud-hypervisor processes for all VMs on this host. | 74 | // Provisioner manages cloud-hypervisor processes for all VMs on this host. |
| @@ -149,8 +160,17 @@ func (p *Provisioner) buildArgs(spec state.VMSpec) []string { | |||
| 149 | } | 160 | } |
| 150 | args = append(args, fmt.Sprintf("path=%s,image_type=raw", d.Path)) | 161 | args = append(args, fmt.Sprintf("path=%s,image_type=raw", d.Path)) |
| 151 | } | 162 | } |
| 163 | // NIC ORDER IS THE GUEST ABI. The NAT NIC is first and therefore eth0: it | ||
| 164 | // is the management fabric every guest has, the address the gate splices | ||
| 165 | // to, and what the seed's primary netplan stanza expects to find. A guest | ||
| 166 | // that also asked for a named network gets that NIC second — same MAC | ||
| 167 | // determinism, its own tap, its own DHCP server answering it. | ||
| 168 | args = append(args, "--net", fmt.Sprintf("tap=%s,mac=%s", tap, mac)) | ||
| 169 | if spec.Network != "" { | ||
| 170 | args = append(args, "--net", | ||
| 171 | fmt.Sprintf("tap=%s,mac=%s", p.net.NetTapName(vmID), state.NetMAC(vmID))) | ||
| 172 | } | ||
| 152 | return append(args, | 173 | return append(args, |
| 153 | "--net", fmt.Sprintf("tap=%s,mac=%s", tap, mac), | ||
| 154 | "--serial", fmt.Sprintf("socket=%s", p.st.SerialSocketPath(vmID)), | 174 | "--serial", fmt.Sprintf("socket=%s", p.st.SerialSocketPath(vmID)), |
| 155 | "--console", "off", | 175 | "--console", "off", |
| 156 | ) | 176 | ) |
| @@ -241,7 +261,7 @@ func (p *Provisioner) Boot(ctx context.Context, vmID string, spec state.VMSpec) | |||
| 241 | // fail instead yields an illegible cloud-hypervisor error for the same | 261 | // fail instead yields an illegible cloud-hypervisor error for the same |
| 242 | // root cause — and %w keeps a Permanent() marker (a tap name collision) | 262 | // root cause — and %w keeps a Permanent() marker (a tap name collision) |
| 243 | // unwrappable, so reconcile still terminal-fails it in one attempt. | 263 | // unwrappable, so reconcile still terminal-fails it in one attempt. |
| 244 | if err := p.attachNet(ctx, vmID); err != nil { | 264 | if err := p.attachNet(ctx, spec); err != nil { |
| 245 | return fmt.Errorf("attach network %s: %w", vmID, err) | 265 | return fmt.Errorf("attach network %s: %w", vmID, err) |
| 246 | } | 266 | } |
| 247 | 267 | ||
| @@ -303,12 +323,17 @@ func (p *Provisioner) Boot(ctx context.Context, vmID string, spec state.VMSpec) | |||
| 303 | // carries the DHCP reservation, so the address has to be known first. | 323 | // carries the DHCP reservation, so the address has to be known first. |
| 304 | // Idempotent: both halves tolerate a re-run, which is what makes a Boot retry | 324 | // Idempotent: both halves tolerate a re-run, which is what makes a Boot retry |
| 305 | // and a restart-after-host-reboot the same code path. | 325 | // and a restart-after-host-reboot the same code path. |
| 306 | func (p *Provisioner) attachNet(ctx context.Context, vmID string) error { | 326 | // |
| 307 | ip, err := p.net.ReserveIP(vmID) | 327 | // It takes the whole spec rather than the id because whether the VM gets a |
| 328 | // second NIC is the spec's to say: the network name the control plane sent | ||
| 329 | // rides here and nowhere else. The reservation is unconditional either way — | ||
| 330 | // every guest is on the NAT underlay. | ||
| 331 | func (p *Provisioner) attachNet(ctx context.Context, spec state.VMSpec) error { | ||
| 332 | ip, err := p.net.ReserveIP(spec.VMID) | ||
| 308 | if err != nil { | 333 | if err != nil { |
| 309 | return err | 334 | return err |
| 310 | } | 335 | } |
| 311 | return p.net.CreateTap(ctx, vmID, ip) | 336 | return p.net.CreateTap(ctx, spec.VMID, ip, spec.Network) |
| 312 | } | 337 | } |
| 313 | 338 | ||
| 314 | // ownedPID is the pid of the cloud-hypervisor process THIS agent started for | 339 | // ownedPID is the pid of the cloud-hypervisor process THIS agent started for |
| @@ -354,6 +379,12 @@ func (p *Provisioner) Running(vmID string) bool { | |||
| 354 | // backend whose host OS hands out addresses cannot answer that early. | 379 | // backend whose host OS hands out addresses cannot answer that early. |
| 355 | func (p *Provisioner) Address(vmID string) string { return p.net.Address(vmID) } | 380 | func (p *Provisioner) Address(vmID string) string { return p.net.Address(vmID) } |
| 356 | 381 | ||
| 382 | // NetworkAddress returns what the site's DHCP server granted this VM's | ||
| 383 | // named-network NIC, or "" when it has none or has not been heard from yet. | ||
| 384 | // Unlike Address it is genuinely polled: the host discovers it by watching the | ||
| 385 | // guest's own exchange, which cannot have happened before the guest boots. | ||
| 386 | func (p *Provisioner) NetworkAddress(vmID string) string { return p.net.NetworkAddress(vmID) } | ||
| 387 | |||
| 357 | // socketClient returns an *http.Client whose transport dials over the VM's | 388 | // socketClient returns an *http.Client whose transport dials over the VM's |
| 358 | // Unix socket. | 389 | // Unix socket. |
| 359 | func (p *Provisioner) socketClient(vmID string) *http.Client { | 390 | func (p *Provisioner) socketClient(vmID string) *http.Client { |
internal/agent/cloudhv/cloudhv_test.go
| Old | New | ||
|---|---|---|---|
| @@ -41,6 +41,29 @@ func TestBuildArgs(t *testing.T) { | |||
| 41 | assert.Contains(t, joined, st.DiskPath("vm1")) | 41 | assert.Contains(t, joined, st.DiskPath("vm1")) |
| 42 | assert.Contains(t, joined, st.SeedPath("vm1")) | 42 | assert.Contains(t, joined, st.SeedPath("vm1")) |
| 43 | assert.Contains(t, joined, "tap=eit-vm1,mac="+state.MAC("vm1")) | 43 | assert.Contains(t, joined, "tap=eit-vm1,mac="+state.MAC("vm1")) |
| 44 | assert.Equal(t, 1, strings.Count(joined, "--net"), "a guest with no named network has one NIC") | ||
| 45 | } | ||
| 46 | |||
| 47 | // TestBuildArgsGivesANetworkedGuestTwoNICsNATFirst pins the guest ABI: the NAT | ||
| 48 | // NIC's --net comes first, so it is eth0 in every guest whether or not the VM | ||
| 49 | // also sits on the operator's LAN. The second NIC is the named one, on its own | ||
| 50 | // tap, with its own deterministic MAC — the one the host's snoop listens for | ||
| 51 | // and the one the seed's second netplan stanza matches. | ||
| 52 | func TestBuildArgsGivesANetworkedGuestTwoNICsNATFirst(t *testing.T) { | ||
| 53 | st, err := state.Open(t.TempDir()) | ||
| 54 | require.NoError(t, err) | ||
| 55 | p := New(st, "ch", "fw", nil, newFakeNet()) | ||
| 56 | |||
| 57 | args := p.buildArgs(state.VMSpec{VMID: "vm1", Network: "lan", VCPUs: 1, MemMB: 512}) | ||
| 58 | joined := strings.Join(args, " ") | ||
| 59 | |||
| 60 | nat := "tap=eit-vm1,mac=" + state.MAC("vm1") | ||
| 61 | named := "tap=eil-vm1,mac=" + state.NetMAC("vm1") | ||
| 62 | assert.Contains(t, joined, nat) | ||
| 63 | assert.Contains(t, joined, named) | ||
| 64 | assert.Equal(t, 2, strings.Count(joined, "--net"), "two NICs, no more") | ||
| 65 | assert.Less(t, strings.Index(joined, nat), strings.Index(joined, named), | ||
| 66 | "NIC order is the guest ABI: eth0 is the NAT NIC") | ||
| 44 | } | 67 | } |
| 45 | 68 | ||
| 46 | // TestBuildArgsDeclaresRawImageType pins image_type=raw on BOTH disk entries. | 69 | // TestBuildArgsDeclaresRawImageType pins image_type=raw on BOTH disk entries. |
| @@ -464,10 +487,20 @@ type fakeNet struct { | |||
| 464 | reserveErr error | 487 | reserveErr error |
| 465 | tapErr error | 488 | tapErr error |
| 466 | delErr error | 489 | delErr error |
| 490 | // tapNetwork records the network CreateTap was passed, so a test can see | ||
| 491 | // that the spec's request reached the networking layer intact without a | ||
| 492 | // bridge on the test machine. discovered stands in for what a snoop heard. | ||
| 493 | tapNetwork map[string]string | ||
| 494 | tapIP map[string]string | ||
| 495 | discovered map[string]string | ||
| 467 | } | 496 | } |
| 468 | 497 | ||
| 469 | func newFakeNet() *fakeNet { | 498 | func newFakeNet() *fakeNet { |
| 470 | return &fakeNet{reserved: map[string]string{}, taps: map[string]bool{}} | 499 | return &fakeNet{ |
| 500 | reserved: map[string]string{}, taps: map[string]bool{}, | ||
| 501 | tapNetwork: map[string]string{}, tapIP: map[string]string{}, | ||
| 502 | discovered: map[string]string{}, | ||
| 503 | } | ||
| 471 | } | 504 | } |
| 472 | 505 | ||
| 473 | func (f *fakeNet) ReserveIP(vmID string) (string, error) { | 506 | func (f *fakeNet) ReserveIP(vmID string) (string, error) { |
| @@ -484,7 +517,11 @@ func (f *fakeNet) ReserveIP(vmID string) (string, error) { | |||
| 484 | 517 | ||
| 485 | func (f *fakeNet) Address(vmID string) string { return f.reserved[vmID] } | 518 | func (f *fakeNet) Address(vmID string) string { return f.reserved[vmID] } |
| 486 | 519 | ||
| 487 | func (f *fakeNet) CreateTap(_ context.Context, vmID, _ string) error { | 520 | func (f *fakeNet) NetworkAddress(vmID string) string { return f.discovered[vmID] } |
| 521 | |||
| 522 | func (f *fakeNet) CreateTap(_ context.Context, vmID, ip, network string) error { | ||
| 523 | f.tapNetwork[vmID] = network | ||
| 524 | f.tapIP[vmID] = ip | ||
| 488 | if f.tapErr != nil { | 525 | if f.tapErr != nil { |
| 489 | return f.tapErr | 526 | return f.tapErr |
| 490 | } | 527 | } |
| @@ -498,14 +535,19 @@ func (f *fakeNet) DeleteTap(_ context.Context, vmID string) error { | |||
| 498 | } | 535 | } |
| 499 | delete(f.reserved, vmID) | 536 | delete(f.reserved, vmID) |
| 500 | delete(f.taps, vmID) | 537 | delete(f.taps, vmID) |
| 538 | delete(f.discovered, vmID) | ||
| 501 | return nil | 539 | return nil |
| 502 | } | 540 | } |
| 503 | 541 | ||
| 504 | func (f *fakeNet) TapName(vmID string) string { | 542 | func (f *fakeNet) TapName(vmID string) string { return "eit-" + shortID(vmID) } |
| 543 | |||
| 544 | func (f *fakeNet) NetTapName(vmID string) string { return "eil-" + shortID(vmID) } | ||
| 545 | |||
| 546 | func shortID(vmID string) string { | ||
| 505 | if len(vmID) > 8 { | 547 | if len(vmID) > 8 { |
| 506 | vmID = vmID[:8] | 548 | return vmID[:8] |
| 507 | } | 549 | } |
| 508 | return "eit-" + vmID | 550 | return vmID |
| 509 | } | 551 | } |
| 510 | 552 | ||
| 511 | // TestBootAttachesTheNetworkBeforeLaunching pins the fold: the --net argument | 553 | // TestBootAttachesTheNetworkBeforeLaunching pins the fold: the --net argument |
| @@ -529,6 +571,70 @@ func TestBootAttachesTheNetworkBeforeLaunching(t *testing.T) { | |||
| 529 | assert.Equal(t, "10.77.1.2", p.Address(vmID), "Boot must reserve the VM's address") | 571 | assert.Equal(t, "10.77.1.2", p.Address(vmID), "Boot must reserve the VM's address") |
| 530 | } | 572 | } |
| 531 | 573 | ||
| 574 | // TestBootNetworkedVMReservesAndPassesTheNetwork pins what this package owes | ||
| 575 | // the networking layer for a guest that asked for a named network: the address | ||
| 576 | // is still reserved (the NAT NIC is unconditional), and the network name | ||
| 577 | // reaches CreateTap unaltered so the second NIC can be built. | ||
| 578 | func TestBootNetworkedVMReservesAndPassesTheNetwork(t *testing.T) { | ||
| 579 | st, err := state.Open(t.TempDir()) | ||
| 580 | require.NoError(t, err) | ||
| 581 | vmID := "vm-lan" | ||
| 582 | spec := state.VMSpec{VMID: vmID, Network: "lan", VCPUs: 1, MemMB: 128} | ||
| 583 | require.NoError(t, st.SaveVM(state.Record{Spec: spec})) | ||
| 584 | |||
| 585 | fakeCH := filepath.Join(t.TempDir(), "fake-ch") | ||
| 586 | require.NoError(t, os.WriteFile(fakeCH, []byte("#!/bin/sh\nexec sleep 60\n"), 0o755)) | ||
| 587 | |||
| 588 | fn := newFakeNet() | ||
| 589 | p := New(st, fakeCH, "fw", nil, fn) | ||
| 590 | require.NoError(t, p.Boot(context.Background(), vmID, spec)) | ||
| 591 | t.Cleanup(func() { _ = p.Destroy(context.Background(), vmID) }) | ||
| 592 | |||
| 593 | assert.Equal(t, "lan", fn.tapNetwork[vmID]) | ||
| 594 | assert.Equal(t, "10.77.1.2", fn.tapIP[vmID], | ||
| 595 | "a networked guest is on the NAT underlay too — it has an address before it boots") | ||
| 596 | assert.Equal(t, "10.77.1.2", p.Address(vmID)) | ||
| 597 | assert.Empty(t, p.NetworkAddress(vmID), "and nothing on the named NIC until its guest asks") | ||
| 598 | assert.True(t, fn.taps[vmID], "the tap is still Boot's to create") | ||
| 599 | } | ||
| 600 | |||
| 601 | // TestNetworkAddressIsPolledFromTheNetworkingLayer pins the second address's | ||
| 602 | // only route upward: whatever the host discovered, unaltered and not confused | ||
| 603 | // with the reservation. | ||
| 604 | func TestNetworkAddressIsPolledFromTheNetworkingLayer(t *testing.T) { | ||
| 605 | st, err := state.Open(t.TempDir()) | ||
| 606 | require.NoError(t, err) | ||
| 607 | fn := newFakeNet() | ||
| 608 | p := New(st, "ch", "fw", nil, fn) | ||
| 609 | |||
| 610 | assert.Empty(t, p.NetworkAddress("vm-lan")) | ||
| 611 | fn.discovered["vm-lan"] = "192.168.0.42" | ||
| 612 | assert.Equal(t, "192.168.0.42", p.NetworkAddress("vm-lan")) | ||
| 613 | assert.Empty(t, p.Address("vm-lan"), "which is never mistaken for the NAT address") | ||
| 614 | } | ||
| 615 | |||
| 616 | // TestBootNATVMPassesNoNetwork is the counterpart, and the regression guard for | ||
| 617 | // every host running today: an unnamed spec asks for no second NIC and gets the | ||
| 618 | // address this host allocated. | ||
| 619 | func TestBootNATVMPassesNoNetwork(t *testing.T) { | ||
| 620 | st, err := state.Open(t.TempDir()) | ||
| 621 | require.NoError(t, err) | ||
| 622 | vmID := "vm-nat" | ||
| 623 | spec := state.VMSpec{VMID: vmID, VCPUs: 1, MemMB: 128} | ||
| 624 | require.NoError(t, st.SaveVM(state.Record{Spec: spec})) | ||
| 625 | |||
| 626 | fakeCH := filepath.Join(t.TempDir(), "fake-ch") | ||
| 627 | require.NoError(t, os.WriteFile(fakeCH, []byte("#!/bin/sh\nexec sleep 60\n"), 0o755)) | ||
| 628 | |||
| 629 | fn := newFakeNet() | ||
| 630 | p := New(st, fakeCH, "fw", nil, fn) | ||
| 631 | require.NoError(t, p.Boot(context.Background(), vmID, spec)) | ||
| 632 | t.Cleanup(func() { _ = p.Destroy(context.Background(), vmID) }) | ||
| 633 | |||
| 634 | assert.Empty(t, fn.tapNetwork[vmID]) | ||
| 635 | assert.Equal(t, "10.77.1.2", fn.tapIP[vmID], "the allocated address is what the tap pins") | ||
| 636 | } | ||
| 637 | |||
| 532 | // TestBootKeepsTheAddressTheVMAlreadyHolds pins that a reboot is not a renumber: | 638 | // TestBootKeepsTheAddressTheVMAlreadyHolds pins that a reboot is not a renumber: |
| 533 | // the reservation the VM already holds is what Boot re-attaches it to. | 639 | // the reservation the VM already holds is what Boot re-attaches it to. |
| 534 | func TestBootKeepsTheAddressTheVMAlreadyHolds(t *testing.T) { | 640 | func TestBootKeepsTheAddressTheVMAlreadyHolds(t *testing.T) { |
internal/agent/netenv/named.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,282 @@ | |||
| 1 | package netenv | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "context" | ||
| 5 | "fmt" | ||
| 6 | "log/slog" | ||
| 7 | "net" | ||
| 8 | "os" | ||
| 9 | "path/filepath" | ||
| 10 | "strings" | ||
| 11 | |||
| 12 | "github.com/a73x/eitri/internal/agent/permanent" | ||
| 13 | "github.com/a73x/eitri/internal/agent/state" | ||
| 14 | ) | ||
| 15 | |||
| 16 | // sysfsIsBridge reports whether name is a bridge device: | ||
| 17 | // /sys/class/net/<name>/bridge is a directory that exists for exactly those. | ||
| 18 | // Bridge-ness is checked rather than assumed because attaching a tap to a link | ||
| 19 | // that is not a bridge fails later with a raw RTNETLINK error naming neither | ||
| 20 | // the network nor the flag that asked for it. | ||
| 21 | func sysfsIsBridge(name string) bool { return sysfsIsBridgeAt("/sys/class/net", name) } | ||
| 22 | |||
| 23 | // sysfsIsBridgeAt is sysfsIsBridge with an injectable sysfs root (for tests). | ||
| 24 | func sysfsIsBridgeAt(root, name string) bool { | ||
| 25 | fi, err := os.Stat(filepath.Join(root, name, "bridge")) | ||
| 26 | return err == nil && fi.IsDir() | ||
| 27 | } | ||
| 28 | |||
| 29 | // VerifyNetworks confirms every configured network's bridge exists and is a | ||
| 30 | // bridge, before any VM can be asked for. eitri never creates these — a missing | ||
| 31 | // one is the operator's netplan/networkd change not yet made, and the only | ||
| 32 | // honest response is a startup failure that names it. Coming up anyway would | ||
| 33 | // advertise a network this host cannot honor, and every VM placed on it would | ||
| 34 | // fail one at a time instead. | ||
| 35 | func (n *Net) VerifyNetworks() error { | ||
| 36 | for name, br := range n.networks { | ||
| 37 | if !n.isBridge(br) { | ||
| 38 | return fmt.Errorf("--host-network %s=%s: %s is not an existing bridge on this host; "+ | ||
| 39 | "declare it in the host's own network config first (eitri attaches to bridges, it does not create them)", | ||
| 40 | name, br, br) | ||
| 41 | } | ||
| 42 | } | ||
| 43 | return nil | ||
| 44 | } | ||
| 45 | |||
| 46 | // blockGuestDHCP installs the one rule a host that lends out an operator bridge | ||
| 47 | // owes it: a frame entering that bridge from an eitri guest tap with UDP source | ||
| 48 | // port 67 — DHCPv4 server talk — is dropped. Called from EnsureBridge, so it is | ||
| 49 | // reprogrammed on every agent start; a host with no configured network runs | ||
| 50 | // none of it and its ruleset stays exactly what it was. | ||
| 51 | // | ||
| 52 | // One rule covers a host serving any number of named networks, because it keys | ||
| 53 | // on the tap-name prefix and not on the bridge: every named tap this agent | ||
| 54 | // creates is eil-something, whichever operator bridge it was enslaved to. | ||
| 55 | // | ||
| 56 | // It buys two separate things. The first is the half of the address snoop that | ||
| 57 | // no socket can settle by itself: internal/agent/netsnoop accepts only frames | ||
| 58 | // the kernel marks outgoing, which proves a guest cannot mint its own address, | ||
| 59 | // but an ACK one guest unicasts at a neighbour's MAC is forwarded by the bridge | ||
| 60 | // and leaves the neighbour tap outgoing too, indistinguishable there from the | ||
| 61 | // site answering. Dropping it at ingress settles it before the question can be | ||
| 62 | // asked — the frame never reaches the bridge, so there is nothing to forward. | ||
| 63 | // The second is plain protection for the LAN: a guest handing out leases would | ||
| 64 | // otherwise address the operator's real machines, and eitri is what put it on | ||
| 65 | // their network. | ||
| 66 | // | ||
| 67 | // Ingress at the tap is the hook point precisely because it is upstream of | ||
| 68 | // every path the frame could take. A drop at prerouting happens before the | ||
| 69 | // bridge picks a port, so the forged lease reaches neither a sibling guest nor | ||
| 70 | // the physical uplink and the switch beyond it. | ||
| 71 | // | ||
| 72 | // DHCPv6 (source port 547) and IPv6 router advertisements are deliberately not | ||
| 73 | // filtered, and the omission is the design rather than a gap in it: a bridged | ||
| 74 | // guest is meant to be a full peer on the operator's LAN, an operator may run a | ||
| 75 | // router guest on purpose, and neither v6 path can poison anything eitri | ||
| 76 | // records, since the snoop reads IPv4 ACKs only. | ||
| 77 | func (n *Net) blockGuestDHCP(ctx context.Context) error { | ||
| 78 | if len(n.networks) == 0 { | ||
| 79 | return nil | ||
| 80 | } | ||
| 81 | // Same idempotency as the NAT chain: add table/add chain are no-ops when the | ||
| 82 | // object is already there, and the flush before the rule keeps restarts from | ||
| 83 | // stacking copies of it. Two invocations are two transactions, though, so a | ||
| 84 | // restart on a host with live taps leaves the chain empty for the moment | ||
| 85 | // between them; the honest fix is one nft -f transaction, which the NAT chain | ||
| 86 | // needs identically and which belongs to its own change rather than this one. | ||
| 87 | steps := [][]string{ | ||
| 88 | {"nft", "add", "table", "bridge", "eitri"}, | ||
| 89 | // The priority is the number and not the name filter, which is -200 in the | ||
| 90 | // bridge family but 0 in the ip family the NAT chain above lives in: the | ||
| 91 | // number reads the same whichever table a reader has in mind. | ||
| 92 | {"nft", "add", "chain", "bridge", "eitri", "prerouting", | ||
| 93 | "{ type filter hook prerouting priority -200 ; }"}, | ||
| 94 | {"nft", "flush", "chain", "bridge", "eitri", "prerouting"}, | ||
| 95 | // The quotes around the pattern are nft syntax, not shell quoting that a | ||
| 96 | // runner without a shell would have eaten: nft re-lexes its arguments | ||
| 97 | // joined, and an unquoted trailing star is a parse error there. | ||
| 98 | {"nft", "add", "rule", "bridge", "eitri", "prerouting", | ||
| 99 | "iifname", fmt.Sprintf("%q", netTapPrefix+"*"), "udp", "sport", "67", "drop"}, | ||
| 100 | } | ||
| 101 | for _, s := range steps { | ||
| 102 | if _, err := n.run(ctx, s[0], s[1:]...); err != nil { | ||
| 103 | return fmt.Errorf("%s %s: %w", s[0], strings.Join(s[1:], " "), err) | ||
| 104 | } | ||
| 105 | } | ||
| 106 | return nil | ||
| 107 | } | ||
| 108 | |||
| 109 | // createNamedTap is CreateTap's second half for a guest that asked for a named | ||
| 110 | // network: a SECOND tap -> enslave to the operator's bridge -> up -> start the | ||
| 111 | // address snoop. No reservation, no masquerade, no forced DNS — the site's DHCP | ||
| 112 | // server owns all three on this NIC. The guest's first tap, on eitri0, was | ||
| 113 | // created before this ran and is untouched by it. | ||
| 114 | // | ||
| 115 | // The network is known to be configured: CreateTap refuses an unknown one | ||
| 116 | // before it creates anything at all. | ||
| 117 | func (n *Net) createNamedTap(ctx context.Context, vmID, network string) error { | ||
| 118 | bridge := n.networks[network] | ||
| 119 | tap := n.NetTapName(vmID) | ||
| 120 | if _, err := n.run(ctx, "ip", "link", "show", "dev", tap); err != nil { | ||
| 121 | if _, err := n.best(ctx, "ip", "tuntap", "add", "dev", tap, "mode", "tap"); err != nil { | ||
| 122 | return err | ||
| 123 | } | ||
| 124 | } else if !n.isTap(tap) { | ||
| 125 | return permanent.Errorf("link %s exists but is not a TAP device — name collision or stale interface; remove it or rename the VM", tap) | ||
| 126 | } | ||
| 127 | if _, err := n.best(ctx, "ip", "link", "set", tap, "master", bridge); err != nil { | ||
| 128 | return err | ||
| 129 | } | ||
| 130 | if _, err := n.best(ctx, "ip", "link", "set", tap, "up"); err != nil { | ||
| 131 | return err | ||
| 132 | } | ||
| 133 | n.mu.Lock() | ||
| 134 | n.attachLocked(vmID) | ||
| 135 | n.mu.Unlock() | ||
| 136 | // A snoop that will not start fails the create. The named NIC's address | ||
| 137 | // arrives only this way, so a guest booted without one is a guest whose LAN | ||
| 138 | // address nothing above the host can ever learn — visibly failing the | ||
| 139 | // create beats reporting a networked guest that is silently half-reported. | ||
| 140 | // The one refusal that is not a failure is the VM being deleted while this | ||
| 141 | // ran: that takes the attachment with it, and startSnoop leaves a guest on | ||
| 142 | // its way out unwatched rather than arming a goroutine for a dead tap. | ||
| 143 | //nolint:contextcheck // the snoop's lifetime is the tap's, not this call's: inheriting ctx would end discovery the moment CreateTap returned | ||
| 144 | return n.startSnoop(vmID, tap) | ||
| 145 | } | ||
| 146 | |||
| 147 | // attachLocked returns vmID's attachment, creating it if the VM has none. | ||
| 148 | // Called with n.mu held. | ||
| 149 | // | ||
| 150 | // It keeps any existing entry rather than replacing it, because both callers | ||
| 151 | // can arrive at a VM that is already attached: a Boot retry re-runs | ||
| 152 | // createNamedTap on a guest whose snoop is live, and neither that snoop nor the | ||
| 153 | // address it has already found may be dropped on the floor by the re-attach. | ||
| 154 | func (n *Net) attachLocked(vmID string) *attachment { | ||
| 155 | a, live := n.attached[vmID] | ||
| 156 | if !live { | ||
| 157 | a = &attachment{} | ||
| 158 | n.attached[vmID] = a | ||
| 159 | } | ||
| 160 | return a | ||
| 161 | } | ||
| 162 | |||
| 163 | // startSnoop begins DHCP-ACK discovery for vmID on tap, the VM's named-network | ||
| 164 | // one. It keys on state.NetMAC — the second NIC's address — so the guest's NAT | ||
| 165 | // lease, which crosses a different tap and this host granted itself, can never | ||
| 166 | // be mistaken for what the site's server said. Idempotent per VM: a second call | ||
| 167 | // while a snoop is live is a no-op, which is what makes it safe on a Boot retry | ||
| 168 | // and on the restart replay. A VM with no attachment has no NIC to watch, and | ||
| 169 | // gets no listener. | ||
| 170 | func (n *Net) startSnoop(vmID, tap string) error { | ||
| 171 | n.mu.Lock() | ||
| 172 | a, live := n.attached[vmID] | ||
| 173 | if !live || a.stopSnoop != nil { | ||
| 174 | n.mu.Unlock() | ||
| 175 | return nil | ||
| 176 | } | ||
| 177 | mac, err := net.ParseMAC(state.NetMAC(vmID)) | ||
| 178 | if err != nil { | ||
| 179 | n.mu.Unlock() | ||
| 180 | return err | ||
| 181 | } | ||
| 182 | ctx, cancel := context.WithCancel(context.Background()) | ||
| 183 | a.stopSnoop = cancel | ||
| 184 | n.mu.Unlock() | ||
| 185 | |||
| 186 | if err := n.listen(ctx, tap, mac, n.noteDiscovered(vmID)); err != nil { | ||
| 187 | // Leave nothing armed behind a failure: the cancel above would make the | ||
| 188 | // retry believe a snoop is already running and skip starting one. Only | ||
| 189 | // this attachment is disarmed — a VM deleted and re-created while the | ||
| 190 | // listener was refusing has a new entry, and that one's snoop is not | ||
| 191 | // this failure's to un-arm. | ||
| 192 | n.mu.Lock() | ||
| 193 | if cur, live := n.attached[vmID]; live && cur == a { | ||
| 194 | cur.stopSnoop = nil | ||
| 195 | } | ||
| 196 | n.mu.Unlock() | ||
| 197 | cancel() | ||
| 198 | return err | ||
| 199 | } | ||
| 200 | return nil | ||
| 201 | } | ||
| 202 | |||
| 203 | // NetworkAddress returns the address the site's DHCP server granted this VM's | ||
| 204 | // named-network NIC, or "" — for a VM that has no such NIC, and for a VM whose | ||
| 205 | // guest has not yet completed a DHCP exchange on it. Empty is "not yet known", | ||
| 206 | // never "unreachable": the VM's NAT address (Address) is what eitri itself | ||
| 207 | // reaches it by, and that one exists from boot. | ||
| 208 | func (n *Net) NetworkAddress(vmID string) string { | ||
| 209 | n.mu.Lock() | ||
| 210 | defer n.mu.Unlock() | ||
| 211 | if a, live := n.attached[vmID]; live { | ||
| 212 | return a.discovered | ||
| 213 | } | ||
| 214 | return "" | ||
| 215 | } | ||
| 216 | |||
| 217 | // noteDiscovered builds vmID's snoop callback. It records what the ACK granted, | ||
| 218 | // but only while the VM is still attached: the listener checks its context | ||
| 219 | // between reads, so a frame that arrived in the last poll window (~1s) can call | ||
| 220 | // back after DeleteTap already cancelled it, and a deleted VM must not come | ||
| 221 | // back holding an address. | ||
| 222 | // | ||
| 223 | // The callback closes over the id and looks the attachment up on every ACK, | ||
| 224 | // never over the entry itself: an entry DeleteTap dropped is a place nothing | ||
| 225 | // may still be writing to, and only the map can say whether it is still there. | ||
| 226 | func (n *Net) noteDiscovered(vmID string) func(ip string) { | ||
| 227 | return func(ip string) { | ||
| 228 | n.mu.Lock() | ||
| 229 | defer n.mu.Unlock() | ||
| 230 | if a, live := n.attached[vmID]; live { | ||
| 231 | a.discovered = ip | ||
| 232 | } | ||
| 233 | } | ||
| 234 | } | ||
| 235 | |||
| 236 | // AdoptNetwork rebuilds a networked VM's in-memory discovery state after an | ||
| 237 | // agent restart: the record's last known named-NIC address (so the fleet keeps | ||
| 238 | // reading it before the next DHCP renewal is snooped) and, when the tap is | ||
| 239 | // still up, a fresh snoop. It says nothing about the VM's NAT attachment — | ||
| 240 | // that half is AddReservation's, and the replay calls both for the same guest. | ||
| 241 | // | ||
| 242 | // It adopts the VM whether or not this agent still serves the network named in | ||
| 243 | // the record, and that is the kind answer rather than an oversight: an agent | ||
| 244 | // restarted without one of its --host-network flags is looking at a guest that | ||
| 245 | // is still running, still holding the tap the previous agent enslaved, still | ||
| 246 | // answering on the address the site gave it. Forgetting the attachment would | ||
| 247 | // take that address off the VM's report and disarm the snoop that keeps it | ||
| 248 | // current, punishing the guest for a change made on the host. | ||
| 249 | // | ||
| 250 | // What it will not do is stay quiet about it. The VM survives on borrowed | ||
| 251 | // configuration — CreateTap refuses the network permanently now, so the guest | ||
| 252 | // fails the next time it is booted — and the only moment an operator can be | ||
| 253 | // told before that happens is this one, so the drift is warned here. | ||
| 254 | func (n *Net) AdoptNetwork(vmID, network, ip string) { | ||
| 255 | n.mu.Lock() | ||
| 256 | a := n.attachLocked(vmID) | ||
| 257 | if ip != "" { | ||
| 258 | a.discovered = ip | ||
| 259 | } | ||
| 260 | n.mu.Unlock() | ||
| 261 | // networks is the agent's command line, fixed at construction, so it is read | ||
| 262 | // without the lock exactly as CreateTap reads it. | ||
| 263 | if _, served := n.networks[network]; !served { | ||
| 264 | slog.Warn("adopting a guest onto a named network this host no longer serves; it keeps its NIC and address, but its next reboot fails until --host-network names this network again", | ||
| 265 | "vm_id", vmID, "network", network) | ||
| 266 | } | ||
| 267 | tap := n.NetTapName(vmID) | ||
| 268 | if n.isTap(tap) { | ||
| 269 | // No tap means the VM is not running; Boot creates both it and the | ||
| 270 | // snoop. A snoop that refuses to start here is not worth failing the | ||
| 271 | // agent's startup over: the address above is already reported, and the | ||
| 272 | // VM's own converge is where a broken tap becomes a story. | ||
| 273 | // | ||
| 274 | // The cost is bounded but real: startSnoop leaves nothing armed after a | ||
| 275 | // failure, so a VM that is later booted re-arms — but a VM already | ||
| 276 | // running keeps no re-arm path until it reboots or the agent restarts, | ||
| 277 | // and reports the replayed address until then. That address is almost | ||
| 278 | // certainly still the guest's: the MAC is derived from the VM id and | ||
| 279 | // the LAN's lease table hands the same one back. | ||
| 280 | _ = n.startSnoop(vmID, tap) | ||
| 281 | } | ||
| 282 | } | ||
internal/agent/netenv/named_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,554 @@ | |||
| 1 | package netenv | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "bytes" | ||
| 5 | "context" | ||
| 6 | "errors" | ||
| 7 | "log/slog" | ||
| 8 | "net" | ||
| 9 | "os" | ||
| 10 | "strings" | ||
| 11 | "sync" | ||
| 12 | "testing" | ||
| 13 | |||
| 14 | "github.com/a73x/eitri/internal/agent/exec" | ||
| 15 | "github.com/stretchr/testify/assert" | ||
| 16 | "github.com/stretchr/testify/require" | ||
| 17 | ) | ||
| 18 | |||
| 19 | // snoopCall is one Listen the Net asked for. The context and the callback are | ||
| 20 | // kept so a test can play the part of the kernel: cancel like a teardown, or | ||
| 21 | // deliver an ACK like a guest. | ||
| 22 | type snoopCall struct { | ||
| 23 | ifname string | ||
| 24 | mac string | ||
| 25 | ctx context.Context | ||
| 26 | found func(ip string) | ||
| 27 | } | ||
| 28 | |||
| 29 | // fakeListen stands in for the AF_PACKET snoop. Production needs CAP_NET_RAW | ||
| 30 | // and a real tap; what this package owns is what it does with what the snoop | ||
| 31 | // hears, so the socket is injected out exactly like isTap and ifaces. | ||
| 32 | type fakeListen struct { | ||
| 33 | mu sync.Mutex | ||
| 34 | calls []snoopCall | ||
| 35 | err error | ||
| 36 | } | ||
| 37 | |||
| 38 | func (f *fakeListen) listen(ctx context.Context, ifname string, mac net.HardwareAddr, found func(ip string)) error { | ||
| 39 | f.mu.Lock() | ||
| 40 | defer f.mu.Unlock() | ||
| 41 | if f.err != nil { | ||
| 42 | return f.err | ||
| 43 | } | ||
| 44 | f.calls = append(f.calls, snoopCall{ifname: ifname, mac: mac.String(), ctx: ctx, found: found}) | ||
| 45 | return nil | ||
| 46 | } | ||
| 47 | |||
| 48 | func (f *fakeListen) last(t *testing.T) snoopCall { | ||
| 49 | t.Helper() | ||
| 50 | f.mu.Lock() | ||
| 51 | defer f.mu.Unlock() | ||
| 52 | require.NotEmpty(t, f.calls, "expected a snoop to have been started") | ||
| 53 | return f.calls[len(f.calls)-1] | ||
| 54 | } | ||
| 55 | |||
| 56 | func (f *fakeListen) count() int { | ||
| 57 | f.mu.Lock() | ||
| 58 | defer f.mu.Unlock() | ||
| 59 | return len(f.calls) | ||
| 60 | } | ||
| 61 | |||
| 62 | // bridgedNet builds a Net on a host that declares networks and whose bridges | ||
| 63 | // all exist, with the snoop faked out. | ||
| 64 | func bridgedNet(t *testing.T, run exec.Runner, networks map[string]string) (*Net, *fakeListen) { | ||
| 65 | t.Helper() | ||
| 66 | n, err := New(run, "10.77.1.0/24", networks) | ||
| 67 | require.NoError(t, err) | ||
| 68 | n.isBridge = func(string) bool { return true } | ||
| 69 | snoop := &fakeListen{} | ||
| 70 | n.listen = snoop.listen | ||
| 71 | return n, snoop | ||
| 72 | } | ||
| 73 | |||
| 74 | // TestVerifyNetworksRefusesMissingBridge pins the startup gate: eitri attaches | ||
| 75 | // to bridges the operator declared and creates none, so a name whose link is | ||
| 76 | // absent is a host that cannot honor what it is about to advertise. | ||
| 77 | func TestVerifyNetworksRefusesMissingBridge(t *testing.T) { | ||
| 78 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | ||
| 79 | n, err := New(noop, "10.77.1.0/24", map[string]string{"lan": "br0"}) | ||
| 80 | require.NoError(t, err) | ||
| 81 | n.isBridge = func(string) bool { return false } | ||
| 82 | |||
| 83 | err = n.VerifyNetworks() | ||
| 84 | require.Error(t, err) | ||
| 85 | assert.Contains(t, err.Error(), "lan", "the refusal must name the network") | ||
| 86 | assert.Contains(t, err.Error(), "br0", "and the link the operator has to declare") | ||
| 87 | |||
| 88 | n.isBridge = func(name string) bool { return name == "br0" } | ||
| 89 | assert.NoError(t, n.VerifyNetworks(), "a declared bridge passes") | ||
| 90 | } | ||
| 91 | |||
| 92 | // TestVerifyNetworksPassesWithNoNetworks pins that a host with no --host-network | ||
| 93 | // flags — every host today — starts as it always did. | ||
| 94 | func TestVerifyNetworksPassesWithNoNetworks(t *testing.T) { | ||
| 95 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | ||
| 96 | n, err := New(noop, "10.77.1.0/24", nil) | ||
| 97 | require.NoError(t, err) | ||
| 98 | assert.NoError(t, n.VerifyNetworks()) | ||
| 99 | } | ||
| 100 | |||
| 101 | // networkedNet builds a Net whose vm1 is about to get both taps: neither link | ||
| 102 | // exists yet, so both `ip link show` probes fail and both adds run. | ||
| 103 | func networkedNet(t *testing.T, networks map[string]string) (*Net, *fakeListen, *[]call) { | ||
| 104 | t.Helper() | ||
| 105 | run, calls := recorder(nil, map[string]error{ | ||
| 106 | "ip link show dev eit-vm1": errors.New(`Device "eit-vm1" does not exist.`), | ||
| 107 | "ip link show dev eil-vm1": errors.New(`Device "eil-vm1" does not exist.`), | ||
| 108 | }) | ||
| 109 | n, snoop := bridgedNet(t, run, networks) | ||
| 110 | return n, snoop, calls | ||
| 111 | } | ||
| 112 | |||
| 113 | // TestNamedNetworkAddsASecondNIC is the shape of the whole feature: the guest | ||
| 114 | // keeps everything a NAT-only guest has — its tap on eitri0, its reservation, | ||
| 115 | // its address — and GAINS a second tap on the operator's bridge, watched for | ||
| 116 | // the address the site's own DHCP server grants it. | ||
| 117 | func TestNamedNetworkAddsASecondNIC(t *testing.T) { | ||
| 118 | n, snoop, calls := networkedNet(t, map[string]string{"lan": "br0"}) | ||
| 119 | |||
| 120 | ip, err := n.ReserveIP("vm1") | ||
| 121 | require.NoError(t, err) | ||
| 122 | require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan")) | ||
| 123 | |||
| 124 | all := joinCalls(calls) | ||
| 125 | assert.Contains(t, all, "ip tuntap add dev eit-vm1 mode tap") | ||
| 126 | assert.Contains(t, all, "ip link set eit-vm1 master eitri0", "the NAT NIC is unconditional") | ||
| 127 | assert.Contains(t, all, "ip link set eit-vm1 up") | ||
| 128 | assert.Contains(t, all, "ip tuntap add dev eil-vm1 mode tap") | ||
| 129 | assert.Contains(t, all, "ip link set eil-vm1 master br0", "and the named NIC is additional") | ||
| 130 | assert.Contains(t, all, "ip link set eil-vm1 up") | ||
| 131 | |||
| 132 | mac, _ := net.ParseMAC(stateMAC("vm1")) | ||
| 133 | reserved, ok := n.dhcp.Lookup(mac) | ||
| 134 | require.True(t, ok, "a networked guest still holds a NAT reservation") | ||
| 135 | assert.Equal(t, ip, reserved.String()) | ||
| 136 | assert.Equal(t, ip, n.Address("vm1"), "known before the guest boots, as ever") | ||
| 137 | |||
| 138 | started := snoop.last(t) | ||
| 139 | assert.Equal(t, "eil-vm1", started.ifname, "the snoop watches the named NIC's tap, not the NAT one") | ||
| 140 | netMAC, _ := net.ParseMAC(stateNetMAC("vm1")) | ||
| 141 | assert.Equal(t, netMAC.String(), started.mac, | ||
| 142 | "and listens for the second NIC's MAC — the NAT lease crosses the other tap and is this host's own answer") | ||
| 143 | } | ||
| 144 | |||
| 145 | // TestNICOrderIsTheNATTapFirst pins the guest ABI at the layer that creates the | ||
| 146 | // devices: eth0 is the NAT NIC, so its tap is made and enslaved before the | ||
| 147 | // named one exists. A guest whose NICs arrive in the other order has its | ||
| 148 | // management fabric on eth1 and everything that assumes eth0 is wrong. | ||
| 149 | func TestNICOrderIsTheNATTapFirst(t *testing.T) { | ||
| 150 | n, _, calls := networkedNet(t, map[string]string{"lan": "br0"}) | ||
| 151 | require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan")) | ||
| 152 | |||
| 153 | all := joinCalls(calls) | ||
| 154 | assert.Less(t, strings.Index(all, "eit-vm1"), strings.Index(all, "eil-vm1"), | ||
| 155 | "the NAT tap is created first") | ||
| 156 | } | ||
| 157 | |||
| 158 | // TestUnknownNetworkIsPermanentAndCreatesNothing pins the one outcome the | ||
| 159 | // design forbids: an agent restarted without the flag fails the VM by name | ||
| 160 | // rather than quietly booting it with the NIC it asked for missing. It fails | ||
| 161 | // permanently — no retry on this agent can change the answer — and it fails | ||
| 162 | // before the first device is created, so the host is left as it was found. | ||
| 163 | func TestUnknownNetworkIsPermanentAndCreatesNothing(t *testing.T) { | ||
| 164 | run, calls := recorder(nil, nil) | ||
| 165 | n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"}) | ||
| 166 | |||
| 167 | err := n.CreateTap(context.Background(), "vm1", "10.77.1.2", "ghost") | ||
| 168 | require.Error(t, err) | ||
| 169 | assert.Contains(t, err.Error(), "ghost") | ||
| 170 | assert.Contains(t, err.Error(), "--host-network") | ||
| 171 | var p interface{ Permanent() bool } | ||
| 172 | assert.True(t, errors.As(err, &p) && p.Permanent(), "no retry on this agent can find the network") | ||
| 173 | assert.Empty(t, joinCalls(calls), "an unknown network creates nothing at all — not even the NAT tap") | ||
| 174 | } | ||
| 175 | |||
| 176 | // TestCreateTapFailsWhenSnoopWillNotStart pins the deliberate choice to fail the | ||
| 177 | // create: a named NIC's address arrives only through the snoop, so a guest | ||
| 178 | // booted without one is a guest whose LAN address nothing can ever learn. | ||
| 179 | func TestCreateTapFailsWhenSnoopWillNotStart(t *testing.T) { | ||
| 180 | n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"}) | ||
| 181 | snoop.err = errors.New("operation not permitted") | ||
| 182 | |||
| 183 | err := n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan") | ||
| 184 | require.Error(t, err) | ||
| 185 | assert.Contains(t, err.Error(), "operation not permitted") | ||
| 186 | |||
| 187 | // Nothing armed behind the failure, or the create-retry would believe a | ||
| 188 | // snoop is already running and never start one. | ||
| 189 | snoop.err = nil | ||
| 190 | require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan")) | ||
| 191 | assert.Equal(t, 1, snoop.count(), "the retry starts the snoop the first attempt could not") | ||
| 192 | } | ||
| 193 | |||
| 194 | // TestNetworkAddressComesFromDiscovery pins the poll-until-known contract for | ||
| 195 | // the named NIC — and, in the same breath, that it never leaks into the address | ||
| 196 | // the rest of eitri uses. Two NICs, two addresses, two questions. | ||
| 197 | func TestNetworkAddressComesFromDiscovery(t *testing.T) { | ||
| 198 | n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"}) | ||
| 199 | |||
| 200 | ip, err := n.ReserveIP("vm1") | ||
| 201 | require.NoError(t, err) | ||
| 202 | require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan")) | ||
| 203 | assert.Empty(t, n.NetworkAddress("vm1"), "no ACK seen yet — the honest answer is nothing") | ||
| 204 | assert.Equal(t, ip, n.Address("vm1"), "which never delays the address the gate splices to") | ||
| 205 | |||
| 206 | snoop.last(t).found("192.168.0.42") | ||
| 207 | assert.Equal(t, "192.168.0.42", n.NetworkAddress("vm1")) | ||
| 208 | assert.Equal(t, ip, n.Address("vm1"), "the LAN address must never be reported as the NAT one") | ||
| 209 | |||
| 210 | // A renewal that grants a different address is the guest's truth, not a | ||
| 211 | // conflict: the LAN moved it, and the report follows. | ||
| 212 | snoop.last(t).found("192.168.0.43") | ||
| 213 | assert.Equal(t, "192.168.0.43", n.NetworkAddress("vm1")) | ||
| 214 | } | ||
| 215 | |||
| 216 | // TestAdoptNetworkSeedsAddressAndRearmsSnoop pins the restart replay: the last | ||
| 217 | // known LAN address is readable again before the guest's next renewal is | ||
| 218 | // snooped, and the snoop is running again for when that renewal comes. | ||
| 219 | func TestAdoptNetworkSeedsAddressAndRearmsSnoop(t *testing.T) { | ||
| 220 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | ||
| 221 | n, snoop := bridgedNet(t, noop, map[string]string{"lan": "br0"}) | ||
| 222 | n.isTap = func(string) bool { return true } // the guest survived the restart | ||
| 223 | |||
| 224 | n.AdoptNetwork("vm1", "lan", "192.168.0.7") | ||
| 225 | |||
| 226 | assert.Equal(t, "192.168.0.7", n.NetworkAddress("vm1")) | ||
| 227 | assert.Equal(t, "eil-vm1", snoop.last(t).ifname, "the re-armed snoop watches the named NIC") | ||
| 228 | assert.Empty(t, n.Address("vm1"), | ||
| 229 | "the NAT half of the replay is AddReservation's, and this must not fake it") | ||
| 230 | } | ||
| 231 | |||
| 232 | // captureLogs redirects the default logger into a buffer for the duration of | ||
| 233 | // the test — a replay that adopts a guest onto a network this host no longer | ||
| 234 | // serves has no caller to tell, and reports it the only way it can. | ||
| 235 | func captureLogs(t *testing.T) *bytes.Buffer { | ||
| 236 | t.Helper() | ||
| 237 | var logs bytes.Buffer | ||
| 238 | prev := slog.Default() | ||
| 239 | slog.SetDefault(slog.New(slog.NewTextHandler(&logs, nil))) | ||
| 240 | t.Cleanup(func() { slog.SetDefault(prev) }) | ||
| 241 | return &logs | ||
| 242 | } | ||
| 243 | |||
| 244 | // TestAdoptNetworkWarnsWhenTheHostNoLongerServesTheNetwork pins the one moment | ||
| 245 | // an operator can be told about a dropped --host-network flag before it costs | ||
| 246 | // them a guest: the agent restarts, the running VM is adopted onto a network | ||
| 247 | // this host no longer has, and everything keeps working until the reboot that | ||
| 248 | // fails. The warning has to carry the consequence, because the state it | ||
| 249 | // describes looks entirely healthy from every other angle. | ||
| 250 | func TestAdoptNetworkWarnsWhenTheHostNoLongerServesTheNetwork(t *testing.T) { | ||
| 251 | logs := captureLogs(t) | ||
| 252 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | ||
| 253 | n, _ := bridgedNet(t, noop, map[string]string{"lan": "br0"}) | ||
| 254 | n.isTap = func(string) bool { return true } | ||
| 255 | |||
| 256 | n.AdoptNetwork("vm1", "office", "192.168.0.7") | ||
| 257 | |||
| 258 | warned := logs.String() | ||
| 259 | assert.Contains(t, warned, "level=WARN") | ||
| 260 | assert.Contains(t, warned, "vm_id=vm1", "the operator has to know which guest") | ||
| 261 | assert.Contains(t, warned, "network=office", "and which network went missing") | ||
| 262 | assert.Contains(t, warned, "--host-network", "the flag that would put it back") | ||
| 263 | assert.Contains(t, warned, "next reboot fails", "and what happens to this guest if they do not") | ||
| 264 | |||
| 265 | assert.Equal(t, "192.168.0.7", n.NetworkAddress("vm1"), | ||
| 266 | "the running guest keeps the address it has — the warning is instead of dropping it, not before") | ||
| 267 | } | ||
| 268 | |||
| 269 | // TestAdoptNetworkIsSilentForANetworkThisHostServes pins the other side: the | ||
| 270 | // ordinary restart, every guest adopted back onto a network the flags still | ||
| 271 | // name, says nothing. A warning that fires on the healthy path is one nobody | ||
| 272 | // reads on the unhealthy one. | ||
| 273 | func TestAdoptNetworkIsSilentForANetworkThisHostServes(t *testing.T) { | ||
| 274 | logs := captureLogs(t) | ||
| 275 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | ||
| 276 | n, _ := bridgedNet(t, noop, map[string]string{"lan": "br0"}) | ||
| 277 | n.isTap = func(string) bool { return true } | ||
| 278 | |||
| 279 | n.AdoptNetwork("vm1", "lan", "192.168.0.7") | ||
| 280 | |||
| 281 | assert.Empty(t, logs.String()) | ||
| 282 | } | ||
| 283 | |||
| 284 | // TestReAttachKeepsWhatTheAttachmentAlreadyFound pins that re-attaching a VM | ||
| 285 | // that is already attached — a Boot retry, or a replay racing one — adds the | ||
| 286 | // network to what is there rather than starting the guest's state over: the | ||
| 287 | // address the snoop already found survives, and so does the snoop. | ||
| 288 | func TestReAttachKeepsWhatTheAttachmentAlreadyFound(t *testing.T) { | ||
| 289 | n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"}) | ||
| 290 | require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan")) | ||
| 291 | snoop.last(t).found("192.168.0.42") | ||
| 292 | first := snoop.last(t) | ||
| 293 | |||
| 294 | n.isTap = func(string) bool { return true } | ||
| 295 | n.AdoptNetwork("vm1", "lan", "") | ||
| 296 | require.NoError(t, n.CreateTap(context.Background(), "vm1", "10.77.1.2", "lan")) | ||
| 297 | |||
| 298 | assert.Equal(t, "192.168.0.42", n.NetworkAddress("vm1"), "a re-attach is not a forgetting") | ||
| 299 | assert.Equal(t, 1, snoop.count(), "and the live snoop is still the only one") | ||
| 300 | require.NoError(t, n.DeleteTap(context.Background(), "vm1")) | ||
| 301 | assert.Error(t, first.ctx.Err(), "still the one teardown cancels") | ||
| 302 | } | ||
| 303 | |||
| 304 | // TestAdoptNetworkWithoutATapStartsNoSnoop pins the other half: a VM that is not | ||
| 305 | // running has no tap to watch, and Boot creates both. | ||
| 306 | func TestAdoptNetworkWithoutATapStartsNoSnoop(t *testing.T) { | ||
| 307 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | ||
| 308 | n, snoop := bridgedNet(t, noop, map[string]string{"lan": "br0"}) | ||
| 309 | n.isTap = func(string) bool { return false } | ||
| 310 | |||
| 311 | n.AdoptNetwork("vm1", "lan", "") | ||
| 312 | |||
| 313 | assert.Zero(t, snoop.count(), "nothing to snoop on a VM with no tap") | ||
| 314 | assert.Empty(t, n.NetworkAddress("vm1"), "and no address to claim on its behalf") | ||
| 315 | } | ||
| 316 | |||
| 317 | // TestDeleteTapTearsDownBothNICs pins the teardown of everything a networked | ||
| 318 | // guest held: both taps (a named one left behind holds a port on the operator's | ||
| 319 | // own bridge), the reservation, the snoop goroutine, and the discovered | ||
| 320 | // address. It also pins the race the listener concedes: it checks its context | ||
| 321 | // between reads, so an ACK received in the last poll window can call back after | ||
| 322 | // the cancel, and a deleted VM must not come back holding an address. | ||
| 323 | func TestDeleteTapTearsDownBothNICs(t *testing.T) { | ||
| 324 | n, snoop, calls := networkedNet(t, map[string]string{"lan": "br0"}) | ||
| 325 | ip, err := n.ReserveIP("vm1") | ||
| 326 | require.NoError(t, err) | ||
| 327 | require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan")) | ||
| 328 | started := snoop.last(t) | ||
| 329 | started.found("192.168.0.42") | ||
| 330 | |||
| 331 | require.NoError(t, n.DeleteTap(context.Background(), "vm1")) | ||
| 332 | |||
| 333 | all := joinCalls(calls) | ||
| 334 | assert.Contains(t, all, "ip link del eit-vm1") | ||
| 335 | assert.Contains(t, all, "ip link del eil-vm1", "the named NIC's tap goes too") | ||
| 336 | mac, _ := net.ParseMAC(stateMAC("vm1")) | ||
| 337 | _, reserved := n.dhcp.Lookup(mac) | ||
| 338 | assert.False(t, reserved, "the NAT reservation is released") | ||
| 339 | assert.Error(t, started.ctx.Err(), "the snoop's context must be cancelled") | ||
| 340 | assert.Empty(t, n.NetworkAddress("vm1"), "a deleted VM holds no address") | ||
| 341 | |||
| 342 | started.found("192.168.0.99") // the late frame from the last poll window | ||
| 343 | |||
| 344 | // Re-attach the id before reading, because that is the only way to see the | ||
| 345 | // write the guard prevents: with no entry for the VM, NetworkAddress answers | ||
| 346 | // "" whether or not the late ACK was recorded. A VM re-created under the same | ||
| 347 | // id would attach exactly like this, and it must not inherit the dead one's | ||
| 348 | // address. | ||
| 349 | n.AdoptNetwork("vm1", "lan", "") | ||
| 350 | assert.Empty(t, n.NetworkAddress("vm1"), "a late ACK must not resurrect a deleted VM") | ||
| 351 | } | ||
| 352 | |||
| 353 | // TestDeleteTapRemovesANamedTapThisAgentNeverSaw pins that the teardown asks | ||
| 354 | // the device, not the bookkeeping: the map of who has a named NIC is in-memory | ||
| 355 | // state, and a tap that outlives its guest holds a port on a bridge eitri does | ||
| 356 | // not own. | ||
| 357 | func TestDeleteTapRemovesANamedTapThisAgentNeverSaw(t *testing.T) { | ||
| 358 | run, calls := recorder(nil, nil) | ||
| 359 | n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"}) | ||
| 360 | |||
| 361 | require.NoError(t, n.DeleteTap(context.Background(), "vm1")) | ||
| 362 | |||
| 363 | assert.Contains(t, joinCalls(calls), "ip link del eil-vm1") | ||
| 364 | } | ||
| 365 | |||
| 366 | // TestDeleteTapRemovesTheNamedTapEvenWhenTheNATOneFails pins the ordering | ||
| 367 | // promise the comment makes: a named tap must never outlive its guest, so a | ||
| 368 | // NAT tap that refuses to go must not be allowed to keep the operator's bridge | ||
| 369 | // holding a port. Both are asked; both failures are reported. | ||
| 370 | func TestDeleteTapRemovesTheNamedTapEvenWhenTheNATOneFails(t *testing.T) { | ||
| 371 | run, calls := recorder(nil, map[string]error{ | ||
| 372 | "ip link del eit-vm1": errors.New("RTNETLINK answers: Operation not permitted"), | ||
| 373 | }) | ||
| 374 | n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"}) | ||
| 375 | |||
| 376 | err := n.DeleteTap(context.Background(), "vm1") | ||
| 377 | require.Error(t, err, "the NAT tap's failure is still reported") | ||
| 378 | assert.Contains(t, err.Error(), "Operation not permitted") | ||
| 379 | assert.Contains(t, joinCalls(calls), "ip link del eil-vm1", | ||
| 380 | "the named tap is asked regardless — it is on a bridge eitri does not own") | ||
| 381 | } | ||
| 382 | |||
| 383 | // TestCreateTapRetryStartsNoSecondSnoop pins the idempotency a Boot retry and | ||
| 384 | // the startup replay both lean on: a VM whose snoop is already running gets no | ||
| 385 | // second listener. Two goroutines on one tap would report every ACK twice and | ||
| 386 | // only one of them would answer the cancel — the other would outlive the guest. | ||
| 387 | func TestCreateTapRetryStartsNoSecondSnoop(t *testing.T) { | ||
| 388 | n, snoop, _ := networkedNet(t, map[string]string{"lan": "br0"}) | ||
| 389 | ip, err := n.ReserveIP("vm1") | ||
| 390 | require.NoError(t, err) | ||
| 391 | require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan")) | ||
| 392 | require.Equal(t, 1, snoop.count()) | ||
| 393 | first := snoop.last(t) | ||
| 394 | |||
| 395 | require.NoError(t, n.CreateTap(context.Background(), "vm1", ip, "lan")) | ||
| 396 | |||
| 397 | assert.Equal(t, 1, snoop.count(), "the live snoop is left alone") | ||
| 398 | require.NoError(t, n.DeleteTap(context.Background(), "vm1")) | ||
| 399 | assert.Error(t, first.ctx.Err(), "and it is still the one teardown cancels") | ||
| 400 | } | ||
| 401 | |||
| 402 | // TestCreateNamedTapRejectsANonTapDevice is TestCreateTapRejectsNonTapDevice's | ||
| 403 | // twin for the second NIC: a name collision on the operator's own bridge must | ||
| 404 | // fail here, legibly and permanently, rather than as an illegible | ||
| 405 | // cloud-hypervisor error after eitri has enslaved somebody else's device. | ||
| 406 | func TestCreateNamedTapRejectsANonTapDevice(t *testing.T) { | ||
| 407 | run, _ := recorder(nil, map[string]error{ | ||
| 408 | "ip link show dev eit-vm1": errors.New(`Device "eit-vm1" does not exist.`), | ||
| 409 | }) | ||
| 410 | n, snoop := bridgedNet(t, run, map[string]string{"lan": "br0"}) | ||
| 411 | n.isTap = func(name string) bool { return name != "eil-vm1" } // eil-vm1 exists and is something else | ||
| 412 | |||
| 413 | err := n.CreateTap(context.Background(), "vm1", "10.77.1.5", "lan") | ||
| 414 | |||
| 415 | require.Error(t, err) | ||
| 416 | assert.Contains(t, err.Error(), "not a TAP device") | ||
| 417 | assert.Contains(t, err.Error(), "eil-vm1") | ||
| 418 | var p interface{ Permanent() bool } | ||
| 419 | assert.True(t, errors.As(err, &p) && p.Permanent(), "a collision never self-resolves") | ||
| 420 | assert.Zero(t, snoop.count(), "and nothing is watched on a device that is not the guest's") | ||
| 421 | } | ||
| 422 | |||
| 423 | // TestNATOnlyCreateTapStartsNoSnoopOrSecondTap pins that the guest without a | ||
| 424 | // named network gained nothing: one tap, no listener, and the reservation | ||
| 425 | // pinned exactly as before. | ||
| 426 | func TestNATOnlyCreateTapStartsNoSnoopOrSecondTap(t *testing.T) { | ||
| 427 | errs := map[string]error{ | ||
| 428 | "ip link show dev eit-vm2": errors.New(`Device "eit-vm2" does not exist.`), | ||
| 429 | } | ||
| 430 | run, calls := recorder(nil, errs) | ||
| 431 | n, snoop := bridgedNet(t, run, map[string]string{"lan": "br0"}) | ||
| 432 | |||
| 433 | ip, err := n.ReserveIP("vm2") | ||
| 434 | require.NoError(t, err) | ||
| 435 | require.NoError(t, n.CreateTap(context.Background(), "vm2", ip, "")) | ||
| 436 | |||
| 437 | assert.Zero(t, snoop.count(), "a guest with one NIC has nothing to discover") | ||
| 438 | all := joinCalls(calls) | ||
| 439 | assert.Contains(t, all, "ip link set eit-vm2 master eitri0") | ||
| 440 | assert.NotContains(t, all, "eil-vm2", "and no second tap is created for it") | ||
| 441 | assert.Equal(t, ip, n.Address("vm2")) | ||
| 442 | assert.Empty(t, n.NetworkAddress("vm2")) | ||
| 443 | } | ||
| 444 | |||
| 445 | // bridgeAddrShow is the post-condition output EnsureBridge insists on before it | ||
| 446 | // gets as far as any nft command. | ||
| 447 | var bridgeAddrShow = map[string]string{ | ||
| 448 | "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0", | ||
| 449 | } | ||
| 450 | |||
| 451 | // nftCalls returns the nft commands a run issued, in order and without the | ||
| 452 | // program name — the whole ruleset this host asked the kernel for. | ||
| 453 | func nftCalls(calls *[]call) []string { | ||
| 454 | var out []string | ||
| 455 | for _, c := range *calls { | ||
| 456 | if c.name == "nft" { | ||
| 457 | out = append(out, c.args) | ||
| 458 | } | ||
| 459 | } | ||
| 460 | return out | ||
| 461 | } | ||
| 462 | |||
| 463 | // bridgeFamily narrows nftCalls to the bridge-family half. | ||
| 464 | func bridgeFamily(all []string) []string { | ||
| 465 | var out []string | ||
| 466 | for _, c := range all { | ||
| 467 | if strings.Contains(c, "bridge eitri") { | ||
| 468 | out = append(out, c) | ||
| 469 | } | ||
| 470 | } | ||
| 471 | return out | ||
| 472 | } | ||
| 473 | |||
| 474 | // TestEnsureBridgeDropsGuestDHCPOnNamedNetworkHost pins the rule that stops a | ||
| 475 | // guest being a DHCP server on the operator's LAN: without it a guest can both | ||
| 476 | // answer its neighbours (minting an address the snoop cannot tell from the | ||
| 477 | // site's) and lease to the operator's real machines. | ||
| 478 | func TestEnsureBridgeDropsGuestDHCPOnNamedNetworkHost(t *testing.T) { | ||
| 479 | run, calls := recorder(bridgeAddrShow, nil) | ||
| 480 | n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"}) | ||
| 481 | require.NoError(t, n.EnsureBridge(context.Background())) | ||
| 482 | |||
| 483 | assert.Equal(t, []string{ | ||
| 484 | "add table bridge eitri", | ||
| 485 | "add chain bridge eitri prerouting { type filter hook prerouting priority -200 ; }", | ||
| 486 | "flush chain bridge eitri prerouting", | ||
| 487 | `add rule bridge eitri prerouting iifname "` + netTapPrefix + `*" udp sport 67 drop`, | ||
| 488 | }, bridgeFamily(nftCalls(calls)), | ||
| 489 | "the flush precedes the rule, so restarts cannot stack copies of it") | ||
| 490 | assert.True(t, strings.HasPrefix(n.NetTapName("vm1"), netTapPrefix), | ||
| 491 | "and the pattern matches the taps this host actually creates") | ||
| 492 | } | ||
| 493 | |||
| 494 | // TestEnsureBridgeIssuesNoBridgeFamilyRuleWithoutNetworks pins the other side of | ||
| 495 | // it: a host with no --host-network flag — every host today — asks the kernel | ||
| 496 | // for exactly the ruleset it always did. | ||
| 497 | func TestEnsureBridgeIssuesNoBridgeFamilyRuleWithoutNetworks(t *testing.T) { | ||
| 498 | run, calls := recorder(bridgeAddrShow, nil) | ||
| 499 | n, err := New(run, "10.77.1.0/24", nil) | ||
| 500 | require.NoError(t, err) | ||
| 501 | require.NoError(t, n.EnsureBridge(context.Background())) | ||
| 502 | |||
| 503 | assert.Equal(t, []string{ | ||
| 504 | "add table ip eitri", | ||
| 505 | "add chain ip eitri postrouting { type nat hook postrouting priority srcnat ; }", | ||
| 506 | "flush chain ip eitri postrouting", | ||
| 507 | "add rule ip eitri postrouting ip saddr 10.77.1.0/24 masquerade", | ||
| 508 | }, nftCalls(calls), "a NAT-only host's ruleset is untouched by named networks") | ||
| 509 | assert.Empty(t, bridgeFamily(nftCalls(calls))) | ||
| 510 | } | ||
| 511 | |||
| 512 | // TestEnsureBridgeSurfacesGuestDHCPRuleFailure pins that a kernel which will not | ||
| 513 | // take the rule stops the agent naming the command, rather than starting a host | ||
| 514 | // that serves named networks with the block missing. | ||
| 515 | func TestEnsureBridgeSurfacesGuestDHCPRuleFailure(t *testing.T) { | ||
| 516 | rule := `nft add rule bridge eitri prerouting iifname "` + netTapPrefix + `*" udp sport 67 drop` | ||
| 517 | run, _ := recorder(bridgeAddrShow, map[string]error{ | ||
| 518 | rule: errors.New("Error: Could not process rule: Operation not supported"), | ||
| 519 | }) | ||
| 520 | n, _ := bridgedNet(t, run, map[string]string{"lan": "br0"}) | ||
| 521 | |||
| 522 | err := n.EnsureBridge(context.Background()) | ||
| 523 | require.Error(t, err) | ||
| 524 | assert.Contains(t, err.Error(), "udp sport 67 drop", "the refusal names the command") | ||
| 525 | assert.Contains(t, err.Error(), "Operation not supported", "and what the kernel said") | ||
| 526 | } | ||
| 527 | |||
| 528 | // TestNetTapNameIsTruncatedBelowIFNAMSIZ pins the second device name against | ||
| 529 | // the same limit as the first, and against ever colliding with it. | ||
| 530 | func TestNetTapNameIsTruncatedBelowIFNAMSIZ(t *testing.T) { | ||
| 531 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | ||
| 532 | n, err := New(noop, "10.77.1.0/24", nil) | ||
| 533 | require.NoError(t, err) | ||
| 534 | |||
| 535 | assert.Equal(t, "eil-vm1", n.NetTapName("vm1")) | ||
| 536 | assert.Equal(t, "eil-abcdefgh", n.NetTapName("abcdefghijklmnop")) | ||
| 537 | assert.LessOrEqual(t, len(n.NetTapName("abcdefghijklmnop")), 15, "IFNAMSIZ is 16 including NUL") | ||
| 538 | assert.NotEqual(t, n.TapName("vm1"), n.NetTapName("vm1"), "one guest, two devices, two names") | ||
| 539 | } | ||
| 540 | |||
| 541 | // TestSysfsIsBridge exercises the production probe against a fabricated /sys | ||
| 542 | // tree: a bridge has a bridge/ directory, an ordinary link does not. | ||
| 543 | func TestSysfsIsBridge(t *testing.T) { | ||
| 544 | root := t.TempDir() | ||
| 545 | require.NoError(t, os.MkdirAll(root+"/br0/bridge", 0o755)) | ||
| 546 | require.NoError(t, os.MkdirAll(root+"/eth0", 0o755)) | ||
| 547 | require.NoError(t, os.MkdirAll(root+"/notabr", 0o755)) | ||
| 548 | require.NoError(t, os.WriteFile(root+"/notabr/bridge", []byte("x"), 0o644)) | ||
| 549 | |||
| 550 | assert.True(t, sysfsIsBridgeAt(root, "br0")) | ||
| 551 | assert.False(t, sysfsIsBridgeAt(root, "eth0"), "an ordinary link is not a bridge") | ||
| 552 | assert.False(t, sysfsIsBridgeAt(root, "notabr"), "a file named bridge is not the directory") | ||
| 553 | assert.False(t, sysfsIsBridgeAt(root, "absent"), "a missing link is not a bridge") | ||
| 554 | } | ||
internal/agent/netenv/netenv.go
| Old | New | ||
|---|---|---|---|
| @@ -2,10 +2,18 @@ | |||
| 2 | // the host as .1 gateway, per-VM taps, and NAT for outbound internet. The | 2 | // the host as .1 gateway, per-VM taps, and NAT for outbound internet. The |
| 3 | // bridge is a pure masqueraded underlay; inbound admin reachability is via the | 3 | // bridge is a pure masqueraded underlay; inbound admin reachability is via the |
| 4 | // server's SSH-CA jump gate (internal/server/sshgate), not any guest overlay. | 4 | // server's SSH-CA jump gate (internal/server/sshgate), not any guest overlay. |
| 5 | // | ||
| 6 | // A VM whose spec names one of the host's configured networks gets a SECOND | ||
| 7 | // tap on top of all that, attached to a bridge the operator already owns (see | ||
| 8 | // named.go). None of the machinery above touches it — the site's own DHCP | ||
| 9 | // server addresses that NIC and its own gateway routes it — and none of it is | ||
| 10 | // withheld from the guest either: every VM keeps its NAT attachment, its | ||
| 11 | // reservation and its place on eitri0 beside its siblings. | ||
| 5 | package netenv | 12 | package netenv |
| 6 | 13 | ||
| 7 | import ( | 14 | import ( |
| 8 | "context" | 15 | "context" |
| 16 | "errors" | ||
| 9 | "fmt" | 17 | "fmt" |
| 10 | "net" | 18 | "net" |
| 11 | "net/netip" | 19 | "net/netip" |
| @@ -13,6 +21,7 @@ import ( | |||
| 13 | "path/filepath" | 21 | "path/filepath" |
| 14 | "strconv" | 22 | "strconv" |
| 15 | "strings" | 23 | "strings" |
| 24 | "sync" | ||
| 16 | "time" | 25 | "time" |
| 17 | 26 | ||
| 18 | "github.com/a73x/eitri/internal/agent/dhcp" | 27 | "github.com/a73x/eitri/internal/agent/dhcp" |
| @@ -37,6 +46,49 @@ type Net struct { | |||
| 37 | // the kernel, and the logic above it should be testable without one. | 46 | // the kernel, and the logic above it should be testable without one. |
| 38 | ifaces func() ([]Iface, error) | 47 | ifaces func() ([]Iface, error) |
| 39 | dhcp *dhcp.Server | 48 | dhcp *dhcp.Server |
| 49 | |||
| 50 | // networks maps a configured network name to its operator-owned bridge. | ||
| 51 | // Fixed at construction: it is the agent's command line, and changing it | ||
| 52 | // is a restart. | ||
| 53 | networks map[string]string | ||
| 54 | // isBridge reports whether a link is a bridge. Injectable like isTap; | ||
| 55 | // production reads /sys/class/net/<name>/bridge, a directory that exists | ||
| 56 | // exactly for bridge devices. | ||
| 57 | isBridge func(name string) bool | ||
| 58 | // listen watches a tap for the DHCP ACK addressed to a guest's MAC. | ||
| 59 | // Injectable like isTap, and for a second reason: the production snoop is | ||
| 60 | // a Linux packet socket, so the default is chosen per platform. | ||
| 61 | listen func(ctx context.Context, ifname string, mac net.HardwareAddr, found func(ip string)) error | ||
| 62 | |||
| 63 | mu sync.Mutex | ||
| 64 | // attached holds one entry per VM that has a second NIC on a named network, | ||
| 65 | // and holding an entry is what being attached means: it is created when the | ||
| 66 | // NIC is (CreateTap, or AdoptNetwork replaying one that survived a restart) | ||
| 67 | // and deleted once, whole, when the NIC goes (DeleteTap). Everything the | ||
| 68 | // attachment accumulates — the address the site granted it, the goroutine | ||
| 69 | // watching for the next one — lives and dies with that single entry, so | ||
| 70 | // there is no state left over to speak for a VM this map has forgotten. In | ||
| 71 | // memory, rebuilt at startup by the replay, exactly like the DHCP | ||
| 72 | // reservation table. | ||
| 73 | attached map[string]*attachment | ||
| 74 | } | ||
| 75 | |||
| 76 | // attachment is one VM's named-network NIC as this agent holds it. Which | ||
| 77 | // network that is does not appear here: the record the control plane placed is | ||
| 78 | // what says so, and the agent re-reads it on every replay, so a copy in memory | ||
| 79 | // would only be a second answer to a question already settled. What is here is | ||
| 80 | // what nothing else can answer — the address the site handed this NIC, and the | ||
| 81 | // goroutine listening for the next one. Both arrive after the attachment does, | ||
| 82 | // and both end with it, because the entry does. | ||
| 83 | type attachment struct { | ||
| 84 | // discovered is the last address the site's DHCP server granted this NIC. | ||
| 85 | // Never the NAT address: that one this host allocates, and it lives in the | ||
| 86 | // reservation table. Empty until an ACK is snooped or a restart replays one. | ||
| 87 | discovered string | ||
| 88 | // stopSnoop ends the goroutine watching this NIC's tap, and its presence is | ||
| 89 | // what says one is running: nil for a VM adopted with no tap to watch, and | ||
| 90 | // nil again after a snoop that would not start (so a retry re-arms). | ||
| 91 | stopSnoop context.CancelFunc | ||
| 40 | } | 92 | } |
| 41 | 93 | ||
| 42 | // guestDNS is handed to guests as their DHCP DNS servers (option 6). Public | 94 | // guestDNS is handed to guests as their DHCP DNS servers (option 6). Public |
| @@ -45,7 +97,11 @@ type Net struct { | |||
| 45 | var guestDNS = []net.IP{net.IPv4(1, 1, 1, 1), net.IPv4(9, 9, 9, 9)} | 97 | var guestDNS = []net.IP{net.IPv4(1, 1, 1, 1), net.IPv4(9, 9, 9, 9)} |
| 46 | 98 | ||
| 47 | // New constructs a Net. cidr must be a valid IPv4 prefix (e.g. "10.77.1.0/24"). | 99 | // New constructs a Net. cidr must be a valid IPv4 prefix (e.g. "10.77.1.0/24"). |
| 48 | func New(run exec.Runner, cidr string) (*Net, error) { | 100 | // networks maps a named guest network to the operator-owned bridge backing it; |
| 101 | // nil or empty means this host serves only the NAT underlay. The names are not | ||
| 102 | // checked against the host's links here — VerifyNetworks does that, so a | ||
| 103 | // caller decides whether a missing bridge stops the agent. | ||
| 104 | func New(run exec.Runner, cidr string, networks map[string]string) (*Net, error) { | ||
| 49 | p, err := netip.ParsePrefix(cidr) | 105 | p, err := netip.ParsePrefix(cidr) |
| 50 | if err != nil { | 106 | if err != nil { |
| 51 | return nil, err | 107 | return nil, err |
| @@ -53,7 +109,11 @@ func New(run exec.Runner, cidr string) (*Net, error) { | |||
| 53 | if !p.Addr().Is4() { | 109 | if !p.Addr().Is4() { |
| 54 | return nil, fmt.Errorf("bridge CIDR must be IPv4, got %s", cidr) | 110 | return nil, fmt.Errorf("bridge CIDR must be IPv4, got %s", cidr) |
| 55 | } | 111 | } |
| 56 | n := &Net{run: run, cidr: p, isTap: sysfsIsTap, ifaces: hostIfaces} | 112 | n := &Net{ |
| 113 | run: run, cidr: p, isTap: sysfsIsTap, ifaces: hostIfaces, | ||
| 114 | networks: networks, isBridge: sysfsIsBridge, listen: defaultListen, | ||
| 115 | attached: map[string]*attachment{}, | ||
| 116 | } | ||
| 57 | gw := net.ParseIP(n.Gateway()) | 117 | gw := net.ParseIP(n.Gateway()) |
| 58 | mask := net.CIDRMask(p.Bits(), 32) | 118 | mask := net.CIDRMask(p.Bits(), 32) |
| 59 | n.dhcp = dhcp.NewServer(Bridge, n.cidr.String(), gw, mask, guestDNS, 12*time.Hour) | 119 | n.dhcp = dhcp.NewServer(Bridge, n.cidr.String(), gw, mask, guestDNS, 12*time.Hour) |
| @@ -88,12 +148,30 @@ func (n *Net) Gateway() string { return n.cidr.Masked().Addr().Next().String() } | |||
| 88 | // IFNAMSIZ limit. It is a method rather than a package function because it is | 148 | // IFNAMSIZ limit. It is a method rather than a package function because it is |
| 89 | // part of what a host networking backend answers for — a backend with no tap | 149 | // part of what a host networking backend answers for — a backend with no tap |
| 90 | // device has no such name, and nothing above the platform line may assume one. | 150 | // device has no such name, and nothing above the platform line may assume one. |
| 91 | func (n *Net) TapName(vmID string) string { | 151 | func (n *Net) TapName(vmID string) string { return tapName("eit-", vmID) } |
| 92 | prefix := vmID | 152 | |
| 93 | if len(prefix) > 8 { | 153 | // netTapPrefix begins the name of every named-network tap, and is the one place |
| 94 | prefix = prefix[:8] | 154 | // that shape is written down. The nftables rule in named.go matches devices by |
| 155 | // it, so a rename that missed the rule would leave the bridge judging a name no | ||
| 156 | // tap has any more — hence one home, and both readers derived from it. | ||
| 157 | const netTapPrefix = "eil-" | ||
| 158 | |||
| 159 | // NetTapName returns the device name of the VM's SECOND tap, the one on a | ||
| 160 | // named host network: "eil-" plus the same 8 characters, so the two taps of | ||
| 161 | // one guest sort together and neither can be mistaken for the other in | ||
| 162 | // `ip link`. The "l" is just that marker — the second-NIC tap, distinct from | ||
| 163 | // "eit-" — not short for anything the console says; live devices already wear | ||
| 164 | // the prefix, so it stays even though the word it once stood for did not. | ||
| 165 | // Same 12-character budget as TapName. | ||
| 166 | func (n *Net) NetTapName(vmID string) string { return tapName(netTapPrefix, vmID) } | ||
| 167 | |||
| 168 | // tapName is the shared truncation both device names owe their IFNAMSIZ safety | ||
| 169 | // to: one rule, so a second NIC can never be the one that overruns it. | ||
| 170 | func tapName(prefix, vmID string) string { | ||
| 171 | if len(vmID) > 8 { | ||
| 172 | vmID = vmID[:8] | ||
| 95 | } | 173 | } |
| 96 | return "eit-" + prefix | 174 | return prefix + vmID |
| 97 | } | 175 | } |
| 98 | 176 | ||
| 99 | // ReserveIP returns a sticky IP for vmID, allocated from the bridge CIDR and | 177 | // ReserveIP returns a sticky IP for vmID, allocated from the bridge CIDR and |
| @@ -101,6 +179,11 @@ func (n *Net) TapName(vmID string) string { | |||
| 101 | // MAC). Idempotent: the same VM gets the same address across reboots and agent | 179 | // MAC). Idempotent: the same VM gets the same address across reboots and agent |
| 102 | // restarts. The table lives in memory, so an address survives an agent restart | 180 | // restarts. The table lives in memory, so an address survives an agent restart |
| 103 | // only by way of the startup replay that rebuilds it from durable records. | 181 | // only by way of the startup replay that rebuilds it from durable records. |
| 182 | // | ||
| 183 | // Every VM gets one, whatever else its spec asks for: the NAT NIC is the | ||
| 184 | // management fabric — the gate's splice target, the guests' way to each other, | ||
| 185 | // the egress — and a guest that also sits on the operator's LAN is a guest with | ||
| 186 | // two NICs, not a guest that gave this one up. | ||
| 104 | func (n *Net) ReserveIP(vmID string) (string, error) { | 187 | func (n *Net) ReserveIP(vmID string) (string, error) { |
| 105 | mac, err := net.ParseMAC(state.MAC(vmID)) | 188 | mac, err := net.ParseMAC(state.MAC(vmID)) |
| 106 | if err != nil { | 189 | if err != nil { |
| @@ -113,9 +196,13 @@ func (n *Net) ReserveIP(vmID string) (string, error) { | |||
| 113 | return ip.String(), nil | 196 | return ip.String(), nil |
| 114 | } | 197 | } |
| 115 | 198 | ||
| 116 | // Address returns the address currently reserved for vmID, or "" when the VM | 199 | // Address returns the address currently reserved for vmID on the NAT underlay, |
| 117 | // holds no reservation. Unlike ReserveIP it never allocates: it is the polled | 200 | // or "" when the VM holds no reservation. Unlike ReserveIP it never allocates: |
| 118 | // read of what this host believes the guest's address is. | 201 | // it is the polled read of what this host believes the guest's address is. |
| 202 | // | ||
| 203 | // It is unaffected by a named network. A guest's second NIC has its own address | ||
| 204 | // and its own question (NetworkAddress); this one is the address everything | ||
| 205 | // eitri does with a VM goes through, and it is known before the guest boots. | ||
| 119 | func (n *Net) Address(vmID string) string { | 206 | func (n *Net) Address(vmID string) string { |
| 120 | mac, err := net.ParseMAC(state.MAC(vmID)) | 207 | mac, err := net.ParseMAC(state.MAC(vmID)) |
| 121 | if err != nil { | 208 | if err != nil { |
| @@ -158,9 +245,13 @@ func (n *Net) best(ctx context.Context, name string, args ...string) (string, er | |||
| 158 | // forwarding, and installs an nftables NAT rule that masquerades VM outbound | 245 | // forwarding, and installs an nftables NAT rule that masquerades VM outbound |
| 159 | // traffic on every interface. | 246 | // traffic on every interface. |
| 160 | // | 247 | // |
| 248 | // On a host that serves named networks it also programs the bridge-family rule | ||
| 249 | // those networks owe the operator (blockGuestDHCP). A host with none issues no | ||
| 250 | // bridge-family command at all: its ruleset is what it always was. | ||
| 251 | // | ||
| 161 | // The function is safe to call on every agent restart: the nft chain is | 252 | // The function is safe to call on every agent restart: the nft chain is |
| 162 | // flushed before the masquerade rule is added, so rules never accumulate | 253 | // flushed before the masquerade rule is added, so rules never accumulate |
| 163 | // across restarts. | 254 | // across restarts. The same holds for the bridge-family chain. |
| 164 | func (n *Net) EnsureBridge(ctx context.Context) error { | 255 | func (n *Net) EnsureBridge(ctx context.Context) error { |
| 165 | gw := n.Gateway() | 256 | gw := n.Gateway() |
| 166 | bits := n.cidr.Bits() | 257 | bits := n.cidr.Bits() |
| @@ -224,7 +315,7 @@ func (n *Net) EnsureBridge(ctx context.Context) error { | |||
| 224 | return fmt.Errorf("nft add rule: %w", err) | 315 | return fmt.Errorf("nft add rule: %w", err) |
| 225 | } | 316 | } |
| 226 | 317 | ||
| 227 | return nil | 318 | return n.blockGuestDHCP(ctx) |
| 228 | } | 319 | } |
| 229 | 320 | ||
| 230 | // StartDHCP begins serving reserved leases on the bridge. Call after | 321 | // StartDHCP begins serving reserved leases on the bridge. Call after |
| @@ -256,7 +347,25 @@ func (n *Net) AddReservation(vmID, ip string) { | |||
| 256 | // allow-list can never chase across iproute2 versions. A create-retry hitting | 347 | // allow-list can never chase across iproute2 versions. A create-retry hitting |
| 257 | // the previous attempt's tap must not mask the retry's real error. Pinned by | 348 | // the previous attempt's tap must not mask the retry's real error. Pinned by |
| 258 | // TestCreateTapIdempotentWhenTapExists. | 349 | // TestCreateTapIdempotentWhenTapExists. |
| 259 | func (n *Net) CreateTap(ctx context.Context, vmID, ip string) error { | 350 | // |
| 351 | // network, when non-empty, adds a SECOND tap on the operator's own bridge (see | ||
| 352 | // createNamedTap) after this one — the guest keeps everything above and gains a | ||
| 353 | // NIC on the named network. It is validated before anything is created, so a | ||
| 354 | // network this agent does not serve leaves the host exactly as it found it. | ||
| 355 | func (n *Net) CreateTap(ctx context.Context, vmID, ip, network string) error { | ||
| 356 | if network != "" { | ||
| 357 | if _, ok := n.networks[network]; !ok { | ||
| 358 | // The control plane only places a network on a host that advertised | ||
| 359 | // it, so getting here means this agent restarted without the flag. | ||
| 360 | // Permanent, and refused before the NAT tap exists: converge must | ||
| 361 | // fail the VM legibly, never boot it with the NIC it asked for | ||
| 362 | // missing — a guest silently reachable on the private underlay | ||
| 363 | // alone is a guest nothing that was talking to it can find. | ||
| 364 | return permanent.Errorf("VM wants network %q but this agent is not configured with it; "+ | ||
| 365 | "restart the agent with --host-network %s=<bridge> (its NIC is never silently dropped)", | ||
| 366 | network, network) | ||
| 367 | } | ||
| 368 | } | ||
| 260 | tap := n.TapName(vmID) | 369 | tap := n.TapName(vmID) |
| 261 | if _, err := n.run(ctx, "ip", "link", "show", "dev", tap); err != nil { | 370 | if _, err := n.run(ctx, "ip", "link", "show", "dev", tap); err != nil { |
| 262 | if _, err := n.best(ctx, "ip", "tuntap", "add", "dev", tap, "mode", "tap"); err != nil { | 371 | if _, err := n.best(ctx, "ip", "tuntap", "add", "dev", tap, "mode", "tap"); err != nil { |
| @@ -275,22 +384,53 @@ func (n *Net) CreateTap(ctx context.Context, vmID, ip string) error { | |||
| 275 | return err | 384 | return err |
| 276 | } | 385 | } |
| 277 | n.AddReservation(vmID, ip) | 386 | n.AddReservation(vmID, ip) |
| 278 | return nil | 387 | if network == "" { |
| 388 | return nil | ||
| 389 | } | ||
| 390 | return n.createNamedTap(ctx, vmID, network) | ||
| 279 | } | 391 | } |
| 280 | 392 | ||
| 281 | // DeleteTap removes the VM's DHCP reservation and TAP device. Idempotent: | 393 | // DeleteTap removes the VM's DHCP reservation and BOTH of its TAP devices, plus |
| 282 | // a missing device is tolerated. | 394 | // the snoop and discovered address a named NIC leaves behind. Idempotent: a |
| 395 | // missing device is tolerated. | ||
| 396 | // | ||
| 397 | // The named tap is deleted whether or not this agent remembers the VM having | ||
| 398 | // one. The memory of it is in-memory state rebuilt from records at startup, and | ||
| 399 | // a tap outliving its guest holds a port on the operator's bridge — so the | ||
| 400 | // device, not the bookkeeping, is what gets asked. | ||
| 401 | // | ||
| 402 | // Both deletes are attempted and their errors joined, for the same reason: an | ||
| 403 | // early return on the NAT tap would leave the named one on the operator's | ||
| 404 | // bridge, alive, with nothing left that remembers to ask again. | ||
| 283 | func (n *Net) DeleteTap(ctx context.Context, vmID string) error { | 405 | func (n *Net) DeleteTap(ctx context.Context, vmID string) error { |
| 284 | tap := n.TapName(vmID) | 406 | n.mu.Lock() |
| 407 | var cancel context.CancelFunc | ||
| 408 | if a, live := n.attached[vmID]; live { | ||
| 409 | cancel = a.stopSnoop | ||
| 410 | } | ||
| 411 | delete(n.attached, vmID) | ||
| 412 | n.mu.Unlock() | ||
| 413 | if cancel != nil { | ||
| 414 | cancel() | ||
| 415 | } | ||
| 285 | if mac, err := net.ParseMAC(state.MAC(vmID)); err == nil { | 416 | if mac, err := net.ParseMAC(state.MAC(vmID)); err == nil { |
| 286 | n.dhcp.RemoveReservation(mac) | 417 | n.dhcp.RemoveReservation(mac) |
| 287 | } | 418 | } |
| 288 | out, err := n.run(ctx, "ip", "link", "del", tap) | 419 | return errors.Join( |
| 420 | n.deleteLink(ctx, n.TapName(vmID)), | ||
| 421 | n.deleteLink(ctx, n.NetTapName(vmID)), | ||
| 422 | ) | ||
| 423 | } | ||
| 424 | |||
| 425 | // deleteLink removes one device, tolerating its absence — the shape both taps | ||
| 426 | // are torn down in. | ||
| 427 | func (n *Net) deleteLink(ctx context.Context, dev string) error { | ||
| 428 | out, err := n.run(ctx, "ip", "link", "del", dev) | ||
| 289 | if err != nil { | 429 | if err != nil { |
| 290 | if tolerated(out, err, "Cannot find device") { | 430 | if tolerated(out, err, "Cannot find device") { |
| 291 | return nil | 431 | return nil |
| 292 | } | 432 | } |
| 293 | return fmt.Errorf("ip link del %s: %w", tap, err) | 433 | return fmt.Errorf("ip link del %s: %w", dev, err) |
| 294 | } | 434 | } |
| 295 | return nil | 435 | return nil |
| 296 | } | 436 | } |
internal/agent/netenv/netenv_test.go
| Old | New | ||
|---|---|---|---|
| @@ -17,6 +17,9 @@ import ( | |||
| 17 | // stateMAC re-exposes state.MAC for the test without a separate import block. | 17 | // stateMAC re-exposes state.MAC for the test without a separate import block. |
| 18 | func stateMAC(vmID string) string { return state.MAC(vmID) } | 18 | func stateMAC(vmID string) string { return state.MAC(vmID) } |
| 19 | 19 | ||
| 20 | // stateNetMAC re-exposes state.NetMAC — the second NIC's address — likewise. | ||
| 21 | func stateNetMAC(vmID string) string { return state.NetMAC(vmID) } | ||
| 22 | |||
| 20 | type call struct { | 23 | type call struct { |
| 21 | name string | 24 | name string |
| 22 | args string | 25 | args string |
| @@ -48,11 +51,8 @@ func TestEnsureBridgeSetsUpGatewayForwardingAndNAT(t *testing.T) { | |||
| 48 | errs := map[string]error{ | 51 | errs := map[string]error{ |
| 49 | "ip link show dev eitri0": errors.New("Device \"eitri0\" does not exist."), | 52 | "ip link show dev eitri0": errors.New("Device \"eitri0\" does not exist."), |
| 50 | } | 53 | } |
| 51 | out := map[string]string{ | 54 | run, calls := recorder(bridgeAddrShow, errs) |
| 52 | "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0", | 55 | n, err := New(run, "10.77.1.0/24", nil) |
| 53 | } | ||
| 54 | run, calls := recorder(out, errs) | ||
| 55 | n, err := New(run, "10.77.1.0/24") | ||
| 56 | require.NoError(t, err) | 56 | require.NoError(t, err) |
| 57 | require.NoError(t, n.EnsureBridge(context.Background())) | 57 | require.NoError(t, n.EnsureBridge(context.Background())) |
| 58 | 58 | ||
| @@ -82,7 +82,7 @@ func TestEnsureBridgeIdempotentOnRestart(t *testing.T) { | |||
| 82 | "ip addr add 10.77.1.1/24 dev eitri0": errors.New("Error: ipv4: Address already assigned."), | 82 | "ip addr add 10.77.1.1/24 dev eitri0": errors.New("Error: ipv4: Address already assigned."), |
| 83 | } | 83 | } |
| 84 | run, calls := recorder(out, errs) | 84 | run, calls := recorder(out, errs) |
| 85 | n, err := New(run, "10.77.1.0/24") | 85 | n, err := New(run, "10.77.1.0/24", nil) |
| 86 | require.NoError(t, err) | 86 | require.NoError(t, err) |
| 87 | require.NoError(t, n.EnsureBridge(context.Background()), | 87 | require.NoError(t, n.EnsureBridge(context.Background()), |
| 88 | "restart with existing bridge+address must succeed") | 88 | "restart with existing bridge+address must succeed") |
| @@ -99,8 +99,8 @@ func TestTapLifecycle(t *testing.T) { | |||
| 99 | "ip link show dev eit-abc123": errors.New(`Device "eit-abc123" does not exist.`), | 99 | "ip link show dev eit-abc123": errors.New(`Device "eit-abc123" does not exist.`), |
| 100 | } | 100 | } |
| 101 | run, calls := recorder(nil, errs) | 101 | run, calls := recorder(nil, errs) |
| 102 | n, _ := New(run, "10.77.1.0/24") | 102 | n, _ := New(run, "10.77.1.0/24", nil) |
| 103 | require.NoError(t, n.CreateTap(context.Background(), "abc123", "10.77.1.5")) | 103 | require.NoError(t, n.CreateTap(context.Background(), "abc123", "10.77.1.5", "")) |
| 104 | all := joinCalls(calls) | 104 | all := joinCalls(calls) |
| 105 | assert.Contains(t, all, "ip tuntap add dev eit-abc123 mode tap") | 105 | assert.Contains(t, all, "ip tuntap add dev eit-abc123 mode tap") |
| 106 | assert.Contains(t, all, "ip link set eit-abc123 master eitri0") | 106 | assert.Contains(t, all, "ip link set eit-abc123 master eitri0") |
| @@ -116,7 +116,7 @@ func TestEnsureBridgeFlushPrecedesRuleAdd(t *testing.T) { | |||
| 116 | "ip -o addr show dev eitri0": addrShowOut, | 116 | "ip -o addr show dev eitri0": addrShowOut, |
| 117 | } | 117 | } |
| 118 | run, calls := recorder(out, nil) | 118 | run, calls := recorder(out, nil) |
| 119 | n, err := New(run, "10.77.1.0/24") | 119 | n, err := New(run, "10.77.1.0/24", nil) |
| 120 | require.NoError(t, err) | 120 | require.NoError(t, err) |
| 121 | 121 | ||
| 122 | // Call EnsureBridge twice — simulating two agent restarts. | 122 | // Call EnsureBridge twice — simulating two agent restarts. |
| @@ -156,7 +156,7 @@ func TestEnsureBridgeGatewayConflictDetected(t *testing.T) { | |||
| 156 | "ip -o addr show dev eitri0": "2: eitri0 inet scope global eitri0", | 156 | "ip -o addr show dev eitri0": "2: eitri0 inet scope global eitri0", |
| 157 | } | 157 | } |
| 158 | run, _ := recorder(out, nil) | 158 | run, _ := recorder(out, nil) |
| 159 | n, err := New(run, "10.77.1.0/24") | 159 | n, err := New(run, "10.77.1.0/24", nil) |
| 160 | require.NoError(t, err) | 160 | require.NoError(t, err) |
| 161 | 161 | ||
| 162 | err = n.EnsureBridge(context.Background()) | 162 | err = n.EnsureBridge(context.Background()) |
| @@ -171,7 +171,7 @@ func TestEnsureBridgePlainMasquerade(t *testing.T) { | |||
| 171 | "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0", | 171 | "ip -o addr show dev eitri0": "2: eitri0 inet 10.77.1.1/24 brd 10.77.1.255 scope global eitri0", |
| 172 | } | 172 | } |
| 173 | run, calls := recorder(out, nil) | 173 | run, calls := recorder(out, nil) |
| 174 | n, err := New(run, "10.77.1.0/24") | 174 | n, err := New(run, "10.77.1.0/24", nil) |
| 175 | require.NoError(t, err) | 175 | require.NoError(t, err) |
| 176 | require.NoError(t, n.EnsureBridge(context.Background())) | 176 | require.NoError(t, n.EnsureBridge(context.Background())) |
| 177 | 177 | ||
| @@ -204,8 +204,8 @@ func TestCreateTapToleratesAlreadyExists(t *testing.T) { | |||
| 204 | "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`), | 204 | "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`), |
| 205 | }, | 205 | }, |
| 206 | ) | 206 | ) |
| 207 | n, _ := New(run, "10.77.1.0/24") | 207 | n, _ := New(run, "10.77.1.0/24", nil) |
| 208 | require.NoError(t, n.CreateTap(context.Background(), "x", "10.77.1.5"), | 208 | require.NoError(t, n.CreateTap(context.Background(), "x", "10.77.1.5", ""), |
| 209 | "already-exists must be tolerated for idempotency") | 209 | "already-exists must be tolerated for idempotency") |
| 210 | }) | 210 | }) |
| 211 | } | 211 | } |
| @@ -218,8 +218,8 @@ func TestCreateTapPropagatesRealError(t *testing.T) { | |||
| 218 | key: errors.New("Operation not permitted"), | 218 | key: errors.New("Operation not permitted"), |
| 219 | "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`), | 219 | "ip link show dev eit-x": errors.New(`Device "eit-x" does not exist.`), |
| 220 | }) | 220 | }) |
| 221 | n, _ := New(run, "10.77.1.0/24") | 221 | n, _ := New(run, "10.77.1.0/24", nil) |
| 222 | err := n.CreateTap(context.Background(), "x", "10.77.1.5") | 222 | err := n.CreateTap(context.Background(), "x", "10.77.1.5", "") |
| 223 | require.Error(t, err) | 223 | require.Error(t, err) |
| 224 | assert.Contains(t, err.Error(), "Operation not permitted") | 224 | assert.Contains(t, err.Error(), "Operation not permitted") |
| 225 | } | 225 | } |
| @@ -229,15 +229,15 @@ func TestCreateTapPropagatesRealError(t *testing.T) { | |||
| 229 | func TestDeleteTapToleratesMissingDevice(t *testing.T) { | 229 | func TestDeleteTapToleratesMissingDevice(t *testing.T) { |
| 230 | key := "ip link del eit-x" | 230 | key := "ip link del eit-x" |
| 231 | run, _ := recorder(nil, map[string]error{key: errors.New(`Cannot find device "eit-x"`)}) | 231 | run, _ := recorder(nil, map[string]error{key: errors.New(`Cannot find device "eit-x"`)}) |
| 232 | n, _ := New(run, "10.77.1.0/24") | 232 | n, _ := New(run, "10.77.1.0/24", nil) |
| 233 | require.NoError(t, n.DeleteTap(context.Background(), "x")) | 233 | require.NoError(t, n.DeleteTap(context.Background(), "x")) |
| 234 | 234 | ||
| 235 | run2, _ := recorder(map[string]string{key: "Cannot find device"}, map[string]error{key: errors.New("exit status 1")}) | 235 | run2, _ := recorder(map[string]string{key: "Cannot find device"}, map[string]error{key: errors.New("exit status 1")}) |
| 236 | n2, _ := New(run2, "10.77.1.0/24") | 236 | n2, _ := New(run2, "10.77.1.0/24", nil) |
| 237 | require.NoError(t, n2.DeleteTap(context.Background(), "x")) | 237 | require.NoError(t, n2.DeleteTap(context.Background(), "x")) |
| 238 | 238 | ||
| 239 | run3, _ := recorder(nil, map[string]error{key: errors.New("RTNETLINK answers: Operation not permitted")}) | 239 | run3, _ := recorder(nil, map[string]error{key: errors.New("RTNETLINK answers: Operation not permitted")}) |
| 240 | n3, _ := New(run3, "10.77.1.0/24") | 240 | n3, _ := New(run3, "10.77.1.0/24", nil) |
| 241 | require.Error(t, n3.DeleteTap(context.Background(), "x")) | 241 | require.Error(t, n3.DeleteTap(context.Background(), "x")) |
| 242 | } | 242 | } |
| 243 | 243 | ||
| @@ -258,10 +258,10 @@ func TestCreateTapIdempotentWhenTapExists(t *testing.T) { | |||
| 258 | "ip tuntap add dev eit-busy mode tap": errors.New("ioctl(TUNSETIFF): Device or resource busy"), | 258 | "ip tuntap add dev eit-busy mode tap": errors.New("ioctl(TUNSETIFF): Device or resource busy"), |
| 259 | } | 259 | } |
| 260 | run, calls := recorder(out, errs) | 260 | run, calls := recorder(out, errs) |
| 261 | n, err := New(run, "10.77.1.0/24") | 261 | n, err := New(run, "10.77.1.0/24", nil) |
| 262 | require.NoError(t, err) | 262 | require.NoError(t, err) |
| 263 | n.isTap = func(string) bool { return true } // the leftover IS a real tap | 263 | n.isTap = func(string) bool { return true } // the leftover IS a real tap |
| 264 | require.NoError(t, n.CreateTap(context.Background(), "busy", "10.77.1.5"), | 264 | require.NoError(t, n.CreateTap(context.Background(), "busy", "10.77.1.5", ""), |
| 265 | "CreateTap on an existing tap must succeed") | 265 | "CreateTap on an existing tap must succeed") |
| 266 | 266 | ||
| 267 | all := joinCalls(calls) | 267 | all := joinCalls(calls) |
| @@ -279,11 +279,11 @@ func TestCreateTapRejectsNonTapDevice(t *testing.T) { | |||
| 279 | "ip link show dev eit-clash": "5: eit-clash: <BROADCAST> state DOWN", // exists | 279 | "ip link show dev eit-clash": "5: eit-clash: <BROADCAST> state DOWN", // exists |
| 280 | } | 280 | } |
| 281 | run, calls := recorder(out, nil) | 281 | run, calls := recorder(out, nil) |
| 282 | n, err := New(run, "10.77.1.0/24") | 282 | n, err := New(run, "10.77.1.0/24", nil) |
| 283 | require.NoError(t, err) | 283 | require.NoError(t, err) |
| 284 | n.isTap = func(tap string) bool { return false } // not a tun/tap device | 284 | n.isTap = func(tap string) bool { return false } // not a tun/tap device |
| 285 | 285 | ||
| 286 | err = n.CreateTap(context.Background(), "clash", "10.77.1.5") | 286 | err = n.CreateTap(context.Background(), "clash", "10.77.1.5", "") |
| 287 | require.Error(t, err) | 287 | require.Error(t, err) |
| 288 | assert.Contains(t, err.Error(), "not a TAP device") | 288 | assert.Contains(t, err.Error(), "not a TAP device") |
| 289 | var p interface{ Permanent() bool } | 289 | var p interface{ Permanent() bool } |
| @@ -300,11 +300,11 @@ func TestCreateTapAcceptsExistingRealTap(t *testing.T) { | |||
| 300 | "ip link show dev eit-ok": "5: eit-ok: <NO-CARRIER,BROADCAST,MULTICAST,UP> master eitri0", | 300 | "ip link show dev eit-ok": "5: eit-ok: <NO-CARRIER,BROADCAST,MULTICAST,UP> master eitri0", |
| 301 | } | 301 | } |
| 302 | run, calls := recorder(out, nil) | 302 | run, calls := recorder(out, nil) |
| 303 | n, err := New(run, "10.77.1.0/24") | 303 | n, err := New(run, "10.77.1.0/24", nil) |
| 304 | require.NoError(t, err) | 304 | require.NoError(t, err) |
| 305 | n.isTap = func(tap string) bool { return true } | 305 | n.isTap = func(tap string) bool { return true } |
| 306 | 306 | ||
| 307 | require.NoError(t, n.CreateTap(context.Background(), "ok", "10.77.1.5")) | 307 | require.NoError(t, n.CreateTap(context.Background(), "ok", "10.77.1.5", "")) |
| 308 | all := joinCalls(calls) | 308 | all := joinCalls(calls) |
| 309 | assert.NotContains(t, all, "ip tuntap add") | 309 | assert.NotContains(t, all, "ip tuntap add") |
| 310 | assert.Contains(t, all, "ip link set eit-ok master eitri0") | 310 | assert.Contains(t, all, "ip link set eit-ok master eitri0") |
| @@ -342,10 +342,10 @@ func TestCreateTapAddsReservationAndTapCommands(t *testing.T) { | |||
| 342 | "ip link show dev eit-vm-abc12": errors.New("does not exist"), | 342 | "ip link show dev eit-vm-abc12": errors.New("does not exist"), |
| 343 | } | 343 | } |
| 344 | run, calls := recorder(nil, errs) | 344 | run, calls := recorder(nil, errs) |
| 345 | n, err := New(run, "10.77.1.0/24") | 345 | n, err := New(run, "10.77.1.0/24", nil) |
| 346 | require.NoError(t, err) | 346 | require.NoError(t, err) |
| 347 | 347 | ||
| 348 | require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7")) | 348 | require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7", "")) |
| 349 | 349 | ||
| 350 | all := joinCalls(calls) | 350 | all := joinCalls(calls) |
| 351 | assert.Contains(t, all, "ip tuntap add dev eit-vm-abc12 mode tap") | 351 | assert.Contains(t, all, "ip tuntap add dev eit-vm-abc12 mode tap") |
| @@ -359,7 +359,7 @@ func TestCreateTapAddsReservationAndTapCommands(t *testing.T) { | |||
| 359 | 359 | ||
| 360 | func TestReserveIPAllocatesAndRecordsReservation(t *testing.T) { | 360 | func TestReserveIPAllocatesAndRecordsReservation(t *testing.T) { |
| 361 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | 361 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } |
| 362 | n, err := New(noop, "10.77.1.0/24") | 362 | n, err := New(noop, "10.77.1.0/24", nil) |
| 363 | require.NoError(t, err) | 363 | require.NoError(t, err) |
| 364 | 364 | ||
| 365 | ip, err := n.ReserveIP("vm-alpha") | 365 | ip, err := n.ReserveIP("vm-alpha") |
| @@ -381,9 +381,9 @@ func TestDeleteTapRemovesReservationAndTap(t *testing.T) { | |||
| 381 | "ip link show dev eit-vm-abc12": errors.New("does not exist"), | 381 | "ip link show dev eit-vm-abc12": errors.New("does not exist"), |
| 382 | } | 382 | } |
| 383 | run, calls := recorder(nil, errs) | 383 | run, calls := recorder(nil, errs) |
| 384 | n, err := New(run, "10.77.1.0/24") | 384 | n, err := New(run, "10.77.1.0/24", nil) |
| 385 | require.NoError(t, err) | 385 | require.NoError(t, err) |
| 386 | require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7")) | 386 | require.NoError(t, n.CreateTap(context.Background(), "vm-abc12345", "10.77.1.7", "")) |
| 387 | 387 | ||
| 388 | require.NoError(t, n.DeleteTap(context.Background(), "vm-abc12345")) | 388 | require.NoError(t, n.DeleteTap(context.Background(), "vm-abc12345")) |
| 389 | 389 | ||
| @@ -395,7 +395,7 @@ func TestDeleteTapRemovesReservationAndTap(t *testing.T) { | |||
| 395 | 395 | ||
| 396 | func TestAddressReportsTheReservationWithoutAllocating(t *testing.T) { | 396 | func TestAddressReportsTheReservationWithoutAllocating(t *testing.T) { |
| 397 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | 397 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } |
| 398 | n, err := New(noop, "10.77.1.0/24") | 398 | n, err := New(noop, "10.77.1.0/24", nil) |
| 399 | require.NoError(t, err) | 399 | require.NoError(t, err) |
| 400 | 400 | ||
| 401 | assert.Empty(t, n.Address("vm-alpha"), "an unreserved VM has no address") | 401 | assert.Empty(t, n.Address("vm-alpha"), "an unreserved VM has no address") |
| @@ -415,7 +415,7 @@ func TestAddressReportsTheReservationWithoutAllocating(t *testing.T) { | |||
| 415 | 415 | ||
| 416 | func TestReserveIPKeepsAPreloadedAddress(t *testing.T) { | 416 | func TestReserveIPKeepsAPreloadedAddress(t *testing.T) { |
| 417 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | 417 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } |
| 418 | n, err := New(noop, "10.77.1.0/24") | 418 | n, err := New(noop, "10.77.1.0/24", nil) |
| 419 | require.NoError(t, err) | 419 | require.NoError(t, err) |
| 420 | 420 | ||
| 421 | // The reservation table is in-memory and preloaded at agent startup from | 421 | // The reservation table is in-memory and preloaded at agent startup from |
| @@ -437,7 +437,7 @@ func TestReserveIPKeepsAPreloadedAddress(t *testing.T) { | |||
| 437 | 437 | ||
| 438 | func TestTapNameIsTruncatedBelowIFNAMSIZ(t *testing.T) { | 438 | func TestTapNameIsTruncatedBelowIFNAMSIZ(t *testing.T) { |
| 439 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } | 439 | noop := func(_ context.Context, _ string, _ ...string) (string, error) { return "", nil } |
| 440 | n, err := New(noop, "10.77.1.0/24") | 440 | n, err := New(noop, "10.77.1.0/24", nil) |
| 441 | require.NoError(t, err) | 441 | require.NoError(t, err) |
| 442 | 442 | ||
| 443 | assert.Equal(t, "eit-vm1", n.TapName("vm1")) | 443 | assert.Equal(t, "eit-vm1", n.TapName("vm1")) |
internal/agent/netenv/snoop_linux.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,12 @@ | |||
| 1 | //go:build linux | ||
| 2 | |||
| 3 | package netenv | ||
| 4 | |||
| 5 | import "github.com/a73x/eitri/internal/agent/netsnoop" | ||
| 6 | |||
| 7 | // defaultListen is production address discovery for a guest's second NIC: a | ||
| 8 | // packet socket on that NIC's own tap, reading the DHCP ACK the site's server | ||
| 9 | // sends it. A variable behind a build tag rather than a direct call because | ||
| 10 | // the snoop needs AF_PACKET and this package is compiled on platforms that | ||
| 11 | // have none. | ||
| 12 | var defaultListen = netsnoop.Listen | ||
internal/agent/netenv/snoop_other.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,18 @@ | |||
| 1 | //go:build !linux | ||
| 2 | |||
| 3 | package netenv | ||
| 4 | |||
| 5 | import ( | ||
| 6 | "context" | ||
| 7 | "fmt" | ||
| 8 | "net" | ||
| 9 | ) | ||
| 10 | |||
| 11 | // defaultListen refuses where there is no packet socket to snoop with. Nothing | ||
| 12 | // reaches it — named networks come from a flag only a Linux host acts on, and | ||
| 13 | // their taps are created by the Linux backend — so this exists to keep the | ||
| 14 | // package buildable everywhere the agent is, and to fail loudly rather than | ||
| 15 | // silently report no address if that ever stops being true. | ||
| 16 | func defaultListen(_ context.Context, ifname string, _ net.HardwareAddr, _ func(ip string)) error { | ||
| 17 | return fmt.Errorf("snoop %s: named networks require linux", ifname) | ||
| 18 | } | ||
internal/agent/netenv/uplink_test.go
| Old | New | ||
|---|---|---|---|
| @@ -26,7 +26,7 @@ func ifaceFrom(t *testing.T, name string, cidrs ...string) Iface { | |||
| 26 | // The command runner is nil: this check reads interfaces and runs nothing. | 26 | // The command runner is nil: this check reads interfaces and runs nothing. |
| 27 | func newNetFor(t *testing.T, cidr string, ifaces ...Iface) *Net { | 27 | func newNetFor(t *testing.T, cidr string, ifaces ...Iface) *Net { |
| 28 | t.Helper() | 28 | t.Helper() |
| 29 | n, err := New(nil, cidr) | 29 | n, err := New(nil, cidr, nil) |
| 30 | if err != nil { | 30 | if err != nil { |
| 31 | t.Fatalf("New(%q): %v", cidr, err) | 31 | t.Fatalf("New(%q): %v", cidr, err) |
| 32 | } | 32 | } |
| @@ -105,7 +105,7 @@ func TestCheckUplinkCollisionSkipsAddressesItCannotCompare(t *testing.T) { | |||
| 105 | // could not be run would turn a guard against unreachability into a cause of | 105 | // could not be run would turn a guard against unreachability into a cause of |
| 106 | // it. | 106 | // it. |
| 107 | func TestCheckUplinkCollisionAllowsWhenItCannotLook(t *testing.T) { | 107 | func TestCheckUplinkCollisionAllowsWhenItCannotLook(t *testing.T) { |
| 108 | n, err := New(nil, "10.77.1.0/24") | 108 | n, err := New(nil, "10.77.1.0/24", nil) |
| 109 | if err != nil { | 109 | if err != nil { |
| 110 | t.Fatal(err) | 110 | t.Fatal(err) |
| 111 | } | 111 | } |
internal/agent/netsnoop/listen_linux.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,218 @@ | |||
| 1 | //go:build linux | ||
| 2 | |||
| 3 | package netsnoop | ||
| 4 | |||
| 5 | import ( | ||
| 6 | "bytes" | ||
| 7 | "context" | ||
| 8 | "errors" | ||
| 9 | "fmt" | ||
| 10 | "net" | ||
| 11 | "time" | ||
| 12 | |||
| 13 | "golang.org/x/net/bpf" | ||
| 14 | "golang.org/x/sys/unix" | ||
| 15 | ) | ||
| 16 | |||
| 17 | // pollInterval bounds how long the receive loop can sit in the kernel before | ||
| 18 | // it looks at ctx again — that is, how long after DeleteTap the goroutine and | ||
| 19 | // its socket survive. Short enough that teardown is not noticeable, long | ||
| 20 | // enough that an idle tap costs one wakeup a second. | ||
| 21 | const pollInterval = time.Second | ||
| 22 | |||
| 23 | // snaplen is how much of a frame the snoop keeps. The filter tells the kernel | ||
| 24 | // to copy this much and the receive buffer is cut to the same number, so the | ||
| 25 | // two cannot drift apart into a kernel that copies more than userspace reads. | ||
| 26 | // | ||
| 27 | // It is far more than a lease needs — the fixed DHCP header is 236 bytes and | ||
| 28 | // the options a site adds to it do not approach the rest — and everything that | ||
| 29 | // reaches the buffer has already passed the filter, so anything truncated here | ||
| 30 | // would be a truncated lease rather than traffic being skipped. Such a frame | ||
| 31 | // fails the DHCP decode and goes unreported, which is the trade this package | ||
| 32 | // makes everywhere: a missing address, never a wrong one. | ||
| 33 | const snaplen = 2048 | ||
| 34 | |||
| 35 | // dhcpReplyFilter is the shape of a lease as the kernel can recognise it: | ||
| 36 | // IPv4, not a fragment, UDP, from port 67 to port 68. Everything else on a | ||
| 37 | // busy tap is dropped before it ever reaches userspace, so a guest cannot make | ||
| 38 | // the snoop work by talking loudly. | ||
| 39 | // | ||
| 40 | // It is a classifier, not a guard. A guest can emit frames that match every | ||
| 41 | // line of it — that is exactly what forging a lease looks like — so the | ||
| 42 | // question of whose lease it is gets settled later, by direction, in lease. | ||
| 43 | // | ||
| 44 | // It must never be narrower than ParseACK: a frame the filter drops is one the | ||
| 45 | // parser never sees, so a filter that is stricter anywhere is a real lease the | ||
| 46 | // snoop silently never reports. TestFilterIsNoNarrowerThanParser holds the two | ||
| 47 | // to the same corpus. | ||
| 48 | // | ||
| 49 | // The IPv4 header is variable length, so the ports cannot be read at a fixed | ||
| 50 | // offset: LoadMemShift puts 4*IHL in X and the two LoadIndirect reads are | ||
| 51 | // relative to it. Every absolute offset here already includes the 14 bytes of | ||
| 52 | // Ethernet header the socket keeps. | ||
| 53 | var dhcpReplyFilter = []bpf.Instruction{ | ||
| 54 | bpf.LoadAbsolute{Off: 12, Size: 2}, // ethertype | ||
| 55 | bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 0x0800, SkipTrue: 10}, // not IPv4 | ||
| 56 | bpf.LoadAbsolute{Off: 14 + 9, Size: 1}, // IP protocol | ||
| 57 | bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 17, SkipTrue: 8}, // not UDP | ||
| 58 | bpf.LoadAbsolute{Off: 14 + 6, Size: 2}, // flags and fragment offset | ||
| 59 | bpf.JumpIf{Cond: bpf.JumpBitsSet, Val: 0x1fff, SkipTrue: 6}, // a later fragment has no ports | ||
| 60 | bpf.LoadMemShift{Off: 14}, // X = 4*IHL | ||
| 61 | bpf.LoadIndirect{Off: 14 + 0, Size: 2}, // UDP source port | ||
| 62 | bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 67, SkipTrue: 3}, // not from a DHCP server | ||
| 63 | bpf.LoadIndirect{Off: 14 + 2, Size: 2}, // UDP destination port | ||
| 64 | bpf.JumpIf{Cond: bpf.JumpNotEqual, Val: 68, SkipTrue: 1}, // not to a DHCP client | ||
| 65 | bpf.RetConstant{Val: snaplen}, // keep, as much of it as the buffer takes | ||
| 66 | bpf.RetConstant{Val: 0}, // drop | ||
| 67 | } | ||
| 68 | |||
| 69 | // rawDHCPReplyFilter is the same program in the form the kernel takes it. The | ||
| 70 | // program is a constant, so assembly can only fail if the constant above is | ||
| 71 | // malformed — a fact worth learning when the agent starts rather than when the | ||
| 72 | // first named-network VM is created. | ||
| 73 | var rawDHCPReplyFilter = mustAssemble(dhcpReplyFilter) | ||
| 74 | |||
| 75 | func mustAssemble(prog []bpf.Instruction) []unix.SockFilter { | ||
| 76 | raw, err := bpf.Assemble(prog) | ||
| 77 | if err != nil { | ||
| 78 | panic(fmt.Sprintf("netsnoop: DHCP reply filter does not assemble: %v", err)) | ||
| 79 | } | ||
| 80 | filter := make([]unix.SockFilter, len(raw)) | ||
| 81 | for i, ins := range raw { | ||
| 82 | filter[i] = unix.SockFilter{Code: ins.Op, Jt: ins.Jt, Jf: ins.Jf, K: ins.K} | ||
| 83 | } | ||
| 84 | return filter | ||
| 85 | } | ||
| 86 | |||
| 87 | // lease reports the address one frame grants to want, and is the judgement the | ||
| 88 | // receive loop is made of — kept out of the goroutine, which needs CAP_NET_RAW | ||
| 89 | // and a live tap to reach, so that the judgement itself can be tested. | ||
| 90 | // | ||
| 91 | // pkttype is what makes an answer attributable, and it is worth being exact | ||
| 92 | // about how far that reaches. A tap carries both directions and the kernel | ||
| 93 | // says which is which: a reply from a DHCP server arrives at the host and is | ||
| 94 | // written towards the guest, so the packet socket sees it on its way out — | ||
| 95 | // PACKET_OUTGOING. Anything the guest itself puts on the wire enters the host | ||
| 96 | // through the tap and is delivered as one of the inbound types instead. | ||
| 97 | // | ||
| 98 | // So this closes self-forgery, which is the attack a guest can mount alone: it | ||
| 99 | // cannot mint its own address by writing ACKs at its own NIC, however correct | ||
| 100 | // it makes the headers look. It does not close forgery aimed at a neighbour. | ||
| 101 | // The taps of one named network are enslaved to the same operator bridge, so | ||
| 102 | // an ACK a guest unicasts at another guest's MAC is forwarded by the bridge and | ||
| 103 | // leaves that guest's tap outgoing, indistinguishable from the site answering | ||
| 104 | // by the time it arrives here. Nothing this socket can see separates those two. | ||
| 105 | // Only the bridge can, by refusing UDP source port 67 from a guest tap outright | ||
| 106 | // so that no guest is ever a DHCP server; that rule lives in netenv, which owns | ||
| 107 | // the bridge (blockGuestDHCP), and not in this package. | ||
| 108 | func lease(pkttype uint8, frame []byte, want net.HardwareAddr) (ip string, ok bool) { | ||
| 109 | if pkttype != unix.PACKET_OUTGOING { | ||
| 110 | return "", false | ||
| 111 | } | ||
| 112 | mac, granted, ok := ParseACK(frame) | ||
| 113 | if !ok || !bytes.Equal(mac, want) { | ||
| 114 | return "", false | ||
| 115 | } | ||
| 116 | return granted.String(), true | ||
| 117 | } | ||
| 118 | |||
| 119 | // htons converts a uint16 to network byte order. AF_PACKET takes its protocol | ||
| 120 | // that way in both socket(2) and bind(2) — x/sys passes SockaddrLinklayer's | ||
| 121 | // Protocol straight through to the raw sockaddr without swapping it. | ||
| 122 | func htons(v uint16) uint16 { return v<<8 | v>>8 } | ||
| 123 | |||
| 124 | // Listen opens an AF_PACKET socket bound to ifname and calls found with every | ||
| 125 | // address a DHCP ACK grants to mac, until ctx ends. It returns once the socket | ||
| 126 | // is open and bound, so a caller learns about a bad interface immediately; | ||
| 127 | // errors after that point end the loop silently (the tap is being torn down, | ||
| 128 | // and the VM's converge owns that story). | ||
| 129 | // | ||
| 130 | // found runs on the receive path — it must not block, or it stalls the snoop | ||
| 131 | // — and it fires on every ACK, not only the first: a guest renewing its lease | ||
| 132 | // re-reports the same address, and one that is moved reports a new one, so the | ||
| 133 | // caller decides what a repeat means. | ||
| 134 | // | ||
| 135 | // One goroutine per named NIC, started by CreateTap and stopped by DeleteTap | ||
| 136 | // — the listener's lifetime is the tap's, mirroring serialpump's: started at | ||
| 137 | // create, stopped at delete. That is the only parallel; netsnoop has no | ||
| 138 | // Manager and no registry of its own to mirror. | ||
| 139 | func Listen(ctx context.Context, ifname string, mac net.HardwareAddr, found func(ip string)) error { | ||
| 140 | if len(mac) == 0 { | ||
| 141 | // A snoop with no MAC to match would run forever and report nothing; | ||
| 142 | // that is a wiring bug, not a runtime condition. | ||
| 143 | return fmt.Errorf("snoop %s: no MAC to watch for", ifname) | ||
| 144 | } | ||
| 145 | iface, err := net.InterfaceByName(ifname) | ||
| 146 | if err != nil { | ||
| 147 | return fmt.Errorf("snoop %s: %w", ifname, err) | ||
| 148 | } | ||
| 149 | // SOCK_RAW keeps the Ethernet header, which ParseACK and the filter both | ||
| 150 | // expect; SOCK_DGRAM would strip it. Opening with protocol 0 opens the | ||
| 151 | // socket to nothing: it is attached to no protocol and no interface, and | ||
| 152 | // the kernel queues nothing on it until the bind below. That is the only | ||
| 153 | // moment at which the filter can be installed with no gap behind it — | ||
| 154 | // filtering after the fact leaves a window in which unfiltered frames are | ||
| 155 | // already sitting in the receive queue, and a guest that floods its tap at | ||
| 156 | // boot fills exactly that window. | ||
| 157 | fd, err := unix.Socket(unix.AF_PACKET, unix.SOCK_RAW|unix.SOCK_CLOEXEC, 0) | ||
| 158 | if err != nil { | ||
| 159 | return fmt.Errorf("snoop %s: %w", ifname, err) | ||
| 160 | } | ||
| 161 | if err := unix.SetsockoptSockFprog(fd, unix.SOL_SOCKET, unix.SO_ATTACH_FILTER, &unix.SockFprog{ | ||
| 162 | Len: uint16(len(rawDHCPReplyFilter)), Filter: &rawDHCPReplyFilter[0], | ||
| 163 | }); err != nil { | ||
| 164 | unix.Close(fd) | ||
| 165 | return fmt.Errorf("snoop filter %s: %w", ifname, err) | ||
| 166 | } | ||
| 167 | // The bind narrows the socket to this VM's tap and, by naming a protocol at | ||
| 168 | // last, starts delivery. | ||
| 169 | if err := unix.Bind(fd, &unix.SockaddrLinklayer{ | ||
| 170 | Protocol: htons(unix.ETH_P_ALL), Ifindex: iface.Index, | ||
| 171 | }); err != nil { | ||
| 172 | unix.Close(fd) | ||
| 173 | return fmt.Errorf("snoop bind %s: %w", ifname, err) | ||
| 174 | } | ||
| 175 | // A receive timeout, rather than closing the fd from the cancel path, is | ||
| 176 | // what stops this loop: Linux does not reliably wake a thread blocked in | ||
| 177 | // recvfrom when another thread closes the descriptor, so a closing | ||
| 178 | // canceller can leave the reader parked forever on a descriptor number | ||
| 179 | // the kernel has already handed to someone else. With a timeout, the | ||
| 180 | // reading goroutine is the only owner of fd and the only closer of it, | ||
| 181 | // and cancellation is observed by looking rather than by being hit. | ||
| 182 | if err := unix.SetsockoptTimeval(fd, unix.SOL_SOCKET, unix.SO_RCVTIMEO, | ||
| 183 | &unix.Timeval{Sec: int64(pollInterval / time.Second)}); err != nil { | ||
| 184 | unix.Close(fd) | ||
| 185 | return fmt.Errorf("snoop timeout %s: %w", ifname, err) | ||
| 186 | } | ||
| 187 | go func() { | ||
| 188 | defer unix.Close(fd) | ||
| 189 | buf := make([]byte, snaplen) | ||
| 190 | for ctx.Err() == nil { | ||
| 191 | n, from, err := unix.Recvfrom(fd, buf, 0) | ||
| 192 | if err != nil { | ||
| 193 | // EAGAIN is the receive timeout expiring, EINTR a signal | ||
| 194 | // landing on this thread: both mean "nothing yet", not | ||
| 195 | // "stop". | ||
| 196 | if errors.Is(err, unix.EAGAIN) || errors.Is(err, unix.EINTR) { | ||
| 197 | continue | ||
| 198 | } | ||
| 199 | return | ||
| 200 | } | ||
| 201 | // An AF_PACKET read always comes with a link-layer address, and | ||
| 202 | // its packet type is what lease judges the frame by; a frame that | ||
| 203 | // arrived without one could not be judged at all. | ||
| 204 | ll, ok := from.(*unix.SockaddrLinklayer) | ||
| 205 | if !ok { | ||
| 206 | continue | ||
| 207 | } | ||
| 208 | // found is called from here rather than from inside lease so that | ||
| 209 | // it stays on the receive path in plain sight, where the contract | ||
| 210 | // documented above — it must not block, or the snoop stalls behind | ||
| 211 | // it — is a fact about this loop and not about a helper. | ||
| 212 | if ip, ok := lease(ll.Pkttype, buf[:n], mac); ok { | ||
| 213 | found(ip) | ||
| 214 | } | ||
| 215 | } | ||
| 216 | }() | ||
| 217 | return nil | ||
| 218 | } | ||
internal/agent/netsnoop/listen_linux_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,226 @@ | |||
| 1 | //go:build linux | ||
| 2 | |||
| 3 | package netsnoop | ||
| 4 | |||
| 5 | import ( | ||
| 6 | "context" | ||
| 7 | "errors" | ||
| 8 | "net" | ||
| 9 | "testing" | ||
| 10 | |||
| 11 | "github.com/insomniacslk/dhcp/dhcpv4" | ||
| 12 | "golang.org/x/net/bpf" | ||
| 13 | "golang.org/x/sys/unix" | ||
| 14 | ) | ||
| 15 | |||
| 16 | // The kernel wants the packet protocol in network byte order in both socket(2) | ||
| 17 | // and bind(2); getting the swap wrong binds to a protocol nothing matches and | ||
| 18 | // the snoop goes quiet instead of failing, so pin the value. | ||
| 19 | func TestHtons(t *testing.T) { | ||
| 20 | if got := htons(unix.ETH_P_ALL); got != 0x0300 { | ||
| 21 | t.Errorf("htons(ETH_P_ALL=%#x) = %#04x, want 0x0300", unix.ETH_P_ALL, got) | ||
| 22 | } | ||
| 23 | if got := htons(0x0800); got != 0x0008 { | ||
| 24 | t.Errorf("htons(ETH_P_IP) = %#04x, want 0x0008", got) | ||
| 25 | } | ||
| 26 | } | ||
| 27 | |||
| 28 | // Listen reports setup failures to its caller rather than swallowing them in | ||
| 29 | // the goroutine: a tap that cannot be watched is a VM that will never get an | ||
| 30 | // address, and CreateTap should hear about it. | ||
| 31 | func TestListenRejectsBadSetup(t *testing.T) { | ||
| 32 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 33 | if err := Listen(t.Context(), "eitri-no-such-tap", mac, func(string) {}); err == nil { | ||
| 34 | t.Error("Listen on a missing interface returned nil, want error") | ||
| 35 | } | ||
| 36 | if err := Listen(t.Context(), "lo", nil, func(string) {}); err == nil { | ||
| 37 | t.Error("Listen with no MAC returned nil, want error") | ||
| 38 | } | ||
| 39 | } | ||
| 40 | |||
| 41 | // Opening the socket needs CAP_NET_RAW, so this only verifies that setup | ||
| 42 | // under an already-cancelled context does not hang or panic. The receive | ||
| 43 | // loop itself is proven on real hardware, not here. | ||
| 44 | func TestListenOnAlreadyCancelledContext(t *testing.T) { | ||
| 45 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 46 | ctx, cancel := context.WithCancel(t.Context()) | ||
| 47 | cancel() | ||
| 48 | err := Listen(ctx, "lo", mac, func(string) {}) | ||
| 49 | if err != nil && !errors.Is(err, unix.EPERM) && !errors.Is(err, unix.EACCES) { | ||
| 50 | t.Fatalf("Listen on lo: %v", err) | ||
| 51 | } | ||
| 52 | } | ||
| 53 | |||
| 54 | // A real lease reaches the guest through the tap, so the packet socket sees it | ||
| 55 | // outgoing, and lease reports the address it grants. | ||
| 56 | func TestLeaseReportsOutgoingACK(t *testing.T) { | ||
| 57 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 58 | ack := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42")) | ||
| 59 | |||
| 60 | ip, ok := lease(unix.PACKET_OUTGOING, ack, mac) | ||
| 61 | if !ok || ip != "192.168.0.42" { | ||
| 62 | t.Fatalf("lease = %q %v, want 192.168.0.42 true", ip, ok) | ||
| 63 | } | ||
| 64 | } | ||
| 65 | |||
| 66 | // The forgery the direction check exists to close, and the reason it cannot be | ||
| 67 | // deleted: this is the byte-for-byte frame the test above accepts, put on the | ||
| 68 | // wire by the guest instead of by the site. Every check inside the frame | ||
| 69 | // passes — right ports, boot reply, ACK, the guest's own MAC — and only the | ||
| 70 | // packet type separates the two. A guest that got past this would have eitri | ||
| 71 | // publishing an address the guest invented for itself. | ||
| 72 | // | ||
| 73 | // What this does not prove, because it is not true, is that a guest cannot | ||
| 74 | // forge a lease for a NEIGHBOUR. Such a frame crosses the operator bridge and | ||
| 75 | // arrives outgoing on the neighbour's tap, which is the accepted case above. | ||
| 76 | // Only the bridge can refuse it; see the comment on lease. | ||
| 77 | func TestLeaseIgnoresGuestOriginatedACK(t *testing.T) { | ||
| 78 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 79 | ack := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "10.66.66.66")) | ||
| 80 | |||
| 81 | for name, pkttype := range map[string]uint8{ | ||
| 82 | "unicast to host": unix.PACKET_HOST, | ||
| 83 | "broadcast": unix.PACKET_BROADCAST, | ||
| 84 | "at a neighbour": unix.PACKET_OTHERHOST, | ||
| 85 | "multicast": unix.PACKET_MULTICAST, | ||
| 86 | } { | ||
| 87 | if ip, ok := lease(pkttype, ack, mac); ok { | ||
| 88 | t.Errorf("%s: lease reported %s from a guest-originated frame", name, ip) | ||
| 89 | } | ||
| 90 | } | ||
| 91 | } | ||
| 92 | |||
| 93 | // The tap carries the whole named network, so an outgoing ACK is routinely | ||
| 94 | // somebody else's; only this VM's MAC is this VM's address. | ||
| 95 | func TestLeaseIgnoresOtherFrames(t *testing.T) { | ||
| 96 | mine, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 97 | neighbour, _ := net.ParseMAC("52:54:00:11:22:33") | ||
| 98 | |||
| 99 | for name, b := range map[string][]byte{ | ||
| 100 | "a lease for another VM": frame(t, reply(t, neighbour, dhcpv4.MessageTypeAck, "192.168.0.43")), | ||
| 101 | "not a lease": frame(t, []byte{1, 2, 3, 4}), | ||
| 102 | "empty": {}, | ||
| 103 | } { | ||
| 104 | if ip, ok := lease(unix.PACKET_OUTGOING, b, mine); ok { | ||
| 105 | t.Errorf("%s: lease reported %s", name, ip) | ||
| 106 | } | ||
| 107 | } | ||
| 108 | } | ||
| 109 | |||
| 110 | // The kernel runs the filter before the parser sees anything, so a filter that | ||
| 111 | // is stricter than the parser anywhere is a real lease silently never | ||
| 112 | // reported — the worst failure this package has, because it looks exactly like | ||
| 113 | // a quiet network. Hold both halves to the one corpus of real leases. | ||
| 114 | // | ||
| 115 | // The filter cannot be exercised without CAP_NET_RAW, but the program is the | ||
| 116 | // same either way, so it runs here in the userspace VM. | ||
| 117 | func TestFilterIsNoNarrowerThanParser(t *testing.T) { | ||
| 118 | vm, err := bpf.NewVM(dhcpReplyFilter) | ||
| 119 | if err != nil { | ||
| 120 | t.Fatal(err) | ||
| 121 | } | ||
| 122 | for name, want := range grantedFrames(t) { | ||
| 123 | if _, _, ok := ParseACK(want.frame); !ok { | ||
| 124 | t.Errorf("%s: ParseACK rejects a frame the corpus calls a lease", name) | ||
| 125 | continue | ||
| 126 | } | ||
| 127 | n, err := vm.Run(want.frame) | ||
| 128 | if err != nil { | ||
| 129 | t.Fatalf("%s: %v", name, err) | ||
| 130 | } | ||
| 131 | if n == 0 { | ||
| 132 | t.Errorf("%s: filter drops a frame ParseACK accepts", name) | ||
| 133 | } | ||
| 134 | } | ||
| 135 | } | ||
| 136 | |||
| 137 | // What the filter is allowed to drop. It may be wider than the parser — it | ||
| 138 | // says nothing about opcodes — but everything it drops here the parser rejects | ||
| 139 | // too, so nothing is lost to the kernel that userspace would have wanted. | ||
| 140 | func TestFilterDropsWhatIsNotALease(t *testing.T) { | ||
| 141 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 142 | ack := reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42") | ||
| 143 | |||
| 144 | notIPv4 := frame(t, ack) | ||
| 145 | notIPv4[12], notIPv4[13] = 0x08, 0x06 // ARP | ||
| 146 | |||
| 147 | tcp := frame(t, ack) | ||
| 148 | tcp[14+9] = 6 | ||
| 149 | |||
| 150 | vm, err := bpf.NewVM(dhcpReplyFilter) | ||
| 151 | if err != nil { | ||
| 152 | t.Fatal(err) | ||
| 153 | } | ||
| 154 | for name, b := range map[string][]byte{ | ||
| 155 | "not IPv4": notIPv4, | ||
| 156 | "TCP": tcp, | ||
| 157 | "guest to server": framePorts(t, ack, 68, 67), | ||
| 158 | "unrelated UDP": framePorts(t, ack, 12345, 54321), | ||
| 159 | "runt": {0, 1, 2}, | ||
| 160 | } { | ||
| 161 | n, err := vm.Run(b) | ||
| 162 | if err != nil { | ||
| 163 | t.Fatalf("%s: %v", name, err) | ||
| 164 | } | ||
| 165 | if n != 0 { | ||
| 166 | t.Errorf("%s: filter returned %d, want it dropped", name, n) | ||
| 167 | } | ||
| 168 | if _, _, ok := ParseACK(b); ok { | ||
| 169 | t.Errorf("%s: filter drops it but ParseACK accepts it", name) | ||
| 170 | } | ||
| 171 | } | ||
| 172 | } | ||
| 173 | |||
| 174 | // The one place the kernel is deliberately stricter than the parser, and the | ||
| 175 | // reason the fragment line is in the filter at all. | ||
| 176 | // | ||
| 177 | // ParseACK reads the ports at 4*IHL without asking whether this is the first | ||
| 178 | // fragment. In a later fragment those bytes are not a UDP header, they are | ||
| 179 | // payload continuation — so a guest that fragments a packet such that the | ||
| 180 | // continuation happens to read as 67 to 68 followed by valid DHCP bytes gets | ||
| 181 | // an address believed by the parser alone. This frame is exactly that, and the | ||
| 182 | // assertion below records that the parser does fall for it: the filter is what | ||
| 183 | // stops it reaching userspace. | ||
| 184 | // | ||
| 185 | // A real ACK is a few hundred bytes and is never fragmented, so dropping later | ||
| 186 | // fragments costs no lease. Teaching ParseACK the same guard would be the | ||
| 187 | // belt to this braces, and is worth doing if the filter is ever bypassed. | ||
| 188 | func TestFilterDropsFragmentsThatFoolTheParser(t *testing.T) { | ||
| 189 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 190 | fragment := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "10.66.66.66")) | ||
| 191 | fragment[14+6], fragment[14+7] = 0x00, 0x25 // a later fragment, offset 37 | ||
| 192 | |||
| 193 | vm, err := bpf.NewVM(dhcpReplyFilter) | ||
| 194 | if err != nil { | ||
| 195 | t.Fatal(err) | ||
| 196 | } | ||
| 197 | n, err := vm.Run(fragment) | ||
| 198 | if err != nil { | ||
| 199 | t.Fatal(err) | ||
| 200 | } | ||
| 201 | if n != 0 { | ||
| 202 | t.Errorf("filter returned %d for a later fragment, want it dropped", n) | ||
| 203 | } | ||
| 204 | if _, _, ok := ParseACK(fragment); !ok { | ||
| 205 | t.Log("ParseACK now rejects later fragments too; the filter is no longer the only guard") | ||
| 206 | } | ||
| 207 | } | ||
| 208 | |||
| 209 | // The kernel takes the program as SockFilter, not as x/net instructions, and | ||
| 210 | // a truncated or reordered conversion would attach a filter that means | ||
| 211 | // something else entirely. | ||
| 212 | func TestRawFilterMatchesProgram(t *testing.T) { | ||
| 213 | raw, err := bpf.Assemble(dhcpReplyFilter) | ||
| 214 | if err != nil { | ||
| 215 | t.Fatal(err) | ||
| 216 | } | ||
| 217 | if len(rawDHCPReplyFilter) != len(raw) { | ||
| 218 | t.Fatalf("assembled filter has %d instructions, want %d", len(rawDHCPReplyFilter), len(raw)) | ||
| 219 | } | ||
| 220 | for i, ins := range raw { | ||
| 221 | got := rawDHCPReplyFilter[i] | ||
| 222 | if got.Code != ins.Op || got.Jt != ins.Jt || got.Jf != ins.Jf || got.K != ins.K { | ||
| 223 | t.Errorf("instruction %d = %+v, want %+v", i, got, ins) | ||
| 224 | } | ||
| 225 | } | ||
| 226 | } | ||
internal/agent/netsnoop/netsnoop.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,72 @@ | |||
| 1 | // Package netsnoop discovers the address a guest's named-network NIC was | ||
| 2 | // granted, by watching that NIC's own tap for its DHCP exchange. On a named | ||
| 3 | // network the site's own server assigns the address, so the agent is not the | ||
| 4 | // authority it is on the eitri bridge (internal/agent/dhcp) and cannot simply | ||
| 5 | // report what it handed out. With no in-guest presence — a boundary this | ||
| 6 | // project holds, the guest owns the guest — the tap is the only place the | ||
| 7 | // answer appears: it carries every frame that NIC sends or receives, so | ||
| 8 | // initial leases and unicast renewals are both visible without touching the | ||
| 9 | // guest or the site's server. | ||
| 10 | // | ||
| 11 | // The snoop reads plain untagged Ethernet frames. A guest that does its own | ||
| 12 | // 802.1Q tagging DHCPs inside the tag and is not discovered here; it keeps | ||
| 13 | // whatever address it negotiates, and eitri reports none — the guest owns the | ||
| 14 | // guest, and a wrong address would be worse than no address. | ||
| 15 | package netsnoop | ||
| 16 | |||
| 17 | import ( | ||
| 18 | "net" | ||
| 19 | "net/netip" | ||
| 20 | |||
| 21 | "github.com/insomniacslk/dhcp/dhcpv4" | ||
| 22 | ) | ||
| 23 | |||
| 24 | // ParseACK inspects one Ethernet frame and, when it is a DHCP ACK, returns | ||
| 25 | // the client MAC it is addressed to and the address it grants. ok=false for | ||
| 26 | // every other frame — this is called on all tap traffic, including whatever a | ||
| 27 | // hostile guest chooses to emit, so it must be cheap and unshockable: it | ||
| 28 | // indexes only after a length check and treats a parse failure as "not an | ||
| 29 | // ACK", never as an error worth reporting. | ||
| 30 | func ParseACK(frame []byte) (mac net.HardwareAddr, ip netip.Addr, ok bool) { | ||
| 31 | const ethHdr = 14 | ||
| 32 | if len(frame) < ethHdr+20+8 { | ||
| 33 | return nil, netip.Addr{}, false | ||
| 34 | } | ||
| 35 | if frame[12] != 0x08 || frame[13] != 0x00 { // not IPv4 | ||
| 36 | return nil, netip.Addr{}, false | ||
| 37 | } | ||
| 38 | ipb := frame[ethHdr:] | ||
| 39 | if ipb[0]>>4 != 4 { | ||
| 40 | return nil, netip.Addr{}, false | ||
| 41 | } | ||
| 42 | ihl := int(ipb[0]&0x0f) * 4 | ||
| 43 | if ihl < 20 || len(ipb) < ihl+8 || ipb[9] != 17 { // not UDP | ||
| 44 | return nil, netip.Addr{}, false | ||
| 45 | } | ||
| 46 | udp := ipb[ihl:] | ||
| 47 | srcPort := int(udp[0])<<8 | int(udp[1]) | ||
| 48 | dstPort := int(udp[2])<<8 | int(udp[3]) | ||
| 49 | if srcPort != 67 || dstPort != 68 { // not server->client DHCP | ||
| 50 | return nil, netip.Addr{}, false | ||
| 51 | } | ||
| 52 | // The opcode check sorts frames, it does not defend against them. A guest | ||
| 53 | // can write a boot reply on these ports as easily as a server can, so | ||
| 54 | // nothing read out of the frame says who sent it. What makes an address | ||
| 55 | // believable is the direction the frame was travelling on the tap, which | ||
| 56 | // the receive path settles before it calls here — see lease in | ||
| 57 | // listen_linux.go, which is also honest about what direction does not | ||
| 58 | // cover. | ||
| 59 | msg, err := dhcpv4.FromBytes(udp[8:]) | ||
| 60 | if err != nil || msg.OpCode != dhcpv4.OpcodeBootReply || | ||
| 61 | msg.MessageType() != dhcpv4.MessageTypeAck { | ||
| 62 | return nil, netip.Addr{}, false | ||
| 63 | } | ||
| 64 | // A wire yiaddr is four bytes, so To4 is non-nil in practice; the | ||
| 65 | // AddrFromSlice check is the belt to that braces. The live arm is | ||
| 66 | // IsUnspecified: a reply that grants 0.0.0.0 is not an address. | ||
| 67 | granted, okAddr := netip.AddrFromSlice(msg.YourIPAddr.To4()) | ||
| 68 | if !okAddr || granted.IsUnspecified() { | ||
| 69 | return nil, netip.Addr{}, false | ||
| 70 | } | ||
| 71 | return msg.ClientHWAddr, granted, true | ||
| 72 | } | ||
internal/agent/netsnoop/netsnoop_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,214 @@ | |||
| 1 | package netsnoop | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "bytes" | ||
| 5 | "net" | ||
| 6 | "testing" | ||
| 7 | |||
| 8 | "github.com/insomniacslk/dhcp/dhcpv4" | ||
| 9 | ) | ||
| 10 | |||
| 11 | // frame wraps a DHCP payload in Ethernet/IPv4/UDP headers the way it crosses | ||
| 12 | // a guest tap: server 67 -> client 68. | ||
| 13 | func frame(t *testing.T, payload []byte) []byte { | ||
| 14 | t.Helper() | ||
| 15 | return framePorts(t, payload, 67, 68) | ||
| 16 | } | ||
| 17 | |||
| 18 | // framePorts is frame with the UDP ports under the test's control, so the | ||
| 19 | // port guard can be exercised without hand-rolling a second builder. | ||
| 20 | func framePorts(t *testing.T, payload []byte, src, dst int) []byte { | ||
| 21 | t.Helper() | ||
| 22 | udp := make([]byte, 8+len(payload)) | ||
| 23 | udp[0], udp[1] = byte(src>>8), byte(src) | ||
| 24 | udp[2], udp[3] = byte(dst>>8), byte(dst) | ||
| 25 | udp[4], udp[5] = byte(len(udp)>>8), byte(len(udp)) | ||
| 26 | copy(udp[8:], payload) | ||
| 27 | ip := make([]byte, 20+len(udp)) | ||
| 28 | ip[0] = 0x45 // v4, ihl 5 | ||
| 29 | ip[2], ip[3] = byte(len(ip)>>8), byte(len(ip)) | ||
| 30 | ip[8] = 64 | ||
| 31 | ip[9] = 17 // UDP | ||
| 32 | copy(ip[20:], udp) | ||
| 33 | eth := make([]byte, 14+len(ip)) | ||
| 34 | eth[12], eth[13] = 0x08, 0x00 // IPv4 | ||
| 35 | copy(eth[14:], ip) | ||
| 36 | return eth | ||
| 37 | } | ||
| 38 | |||
| 39 | // newReply builds a server->client DHCP message. dhcpv4.New defaults the | ||
| 40 | // opcode to BootRequest (the client direction), so replies set it explicitly. | ||
| 41 | func newReply(t *testing.T, mods ...dhcpv4.Modifier) *dhcpv4.DHCPv4 { | ||
| 42 | t.Helper() | ||
| 43 | msg, err := dhcpv4.New(mods...) | ||
| 44 | if err != nil { | ||
| 45 | t.Fatal(err) | ||
| 46 | } | ||
| 47 | msg.OpCode = dhcpv4.OpcodeBootReply | ||
| 48 | return msg | ||
| 49 | } | ||
| 50 | |||
| 51 | // reply builds a server->client DHCP message of the given type for mac, | ||
| 52 | // granting yiaddr ("" leaves yiaddr unset). | ||
| 53 | func reply(t *testing.T, mac net.HardwareAddr, typ dhcpv4.MessageType, yiaddr string) []byte { | ||
| 54 | t.Helper() | ||
| 55 | mods := []dhcpv4.Modifier{ | ||
| 56 | dhcpv4.WithMessageType(typ), | ||
| 57 | dhcpv4.WithHwAddr(mac), | ||
| 58 | } | ||
| 59 | if yiaddr != "" { | ||
| 60 | mods = append(mods, dhcpv4.WithYourIP(net.ParseIP(yiaddr))) | ||
| 61 | } | ||
| 62 | return newReply(t, mods...).ToBytes() | ||
| 63 | } | ||
| 64 | |||
| 65 | // withIPOptions splices four bytes of IPv4 option (NOP, NOP, NOP, EOL) into a | ||
| 66 | // frame's IP header and fixes up IHL and total length, so that the UDP header | ||
| 67 | // starts at 4*IHL rather than at a fixed 20. It catches anyone reading the | ||
| 68 | // ports at a constant offset — in the parser or in the kernel filter. | ||
| 69 | func withIPOptions(t *testing.T, plain []byte) []byte { | ||
| 70 | t.Helper() | ||
| 71 | out := make([]byte, 0, len(plain)+4) | ||
| 72 | out = append(out, plain[:14+20]...) | ||
| 73 | out = append(out, 1, 1, 1, 0) | ||
| 74 | out = append(out, plain[14+20:]...) | ||
| 75 | out[14] = 0x46 // v4, ihl 6 | ||
| 76 | total := len(out) - 14 | ||
| 77 | out[16], out[17] = byte(total>>8), byte(total) | ||
| 78 | return out | ||
| 79 | } | ||
| 80 | |||
| 81 | // granted is one frame that carries a real lease, with what must be read out | ||
| 82 | // of it. | ||
| 83 | type granted struct { | ||
| 84 | frame []byte | ||
| 85 | mac net.HardwareAddr | ||
| 86 | ip string | ||
| 87 | } | ||
| 88 | |||
| 89 | // grantedFrames is the corpus of real leases: every shape a site's server | ||
| 90 | // grants one in that has to survive the receive path intact. Both halves of | ||
| 91 | // that path answer to it — ParseACK must report each of these, and the kernel | ||
| 92 | // filter must pass each of them — because the filter runs first, so a frame it | ||
| 93 | // drops is one the parser never sees. | ||
| 94 | func grantedFrames(t *testing.T) map[string]granted { | ||
| 95 | t.Helper() | ||
| 96 | first, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 97 | renewing, _ := net.ParseMAC("52:54:00:de:ad:01") | ||
| 98 | plain := frame(t, reply(t, first, dhcpv4.MessageTypeAck, "192.168.0.42")) | ||
| 99 | |||
| 100 | // A renewal is unicast to the guest and carries the options a real server | ||
| 101 | // sends a lease with, after the fixed header. Renewals are the only chance | ||
| 102 | // to notice a site server moving a guest, so they have to be read exactly | ||
| 103 | // like the broadcast first lease. | ||
| 104 | renewal := newReply(t, | ||
| 105 | dhcpv4.WithMessageType(dhcpv4.MessageTypeAck), | ||
| 106 | dhcpv4.WithHwAddr(renewing), | ||
| 107 | dhcpv4.WithYourIP(net.ParseIP("10.4.5.6")), | ||
| 108 | dhcpv4.WithNetmask(net.CIDRMask(24, 32)), | ||
| 109 | dhcpv4.WithRouter(net.ParseIP("10.4.5.1")), | ||
| 110 | dhcpv4.WithDNS(net.ParseIP("10.4.5.1")), | ||
| 111 | ) | ||
| 112 | |||
| 113 | return map[string]granted{ | ||
| 114 | "first lease": {plain, first, "192.168.0.42"}, | ||
| 115 | "renewal with options": {frame(t, renewal.ToBytes()), renewing, "10.4.5.6"}, | ||
| 116 | "IP header with options": {withIPOptions(t, plain), first, "192.168.0.42"}, | ||
| 117 | } | ||
| 118 | } | ||
| 119 | |||
| 120 | func TestParseACKAcceptsEveryGrantedLease(t *testing.T) { | ||
| 121 | for name, want := range grantedFrames(t) { | ||
| 122 | gotMAC, gotIP, ok := ParseACK(want.frame) | ||
| 123 | if !ok || !bytes.Equal(gotMAC, want.mac) || gotIP.String() != want.ip { | ||
| 124 | t.Errorf("%s: ParseACK = %v %v %v, want %v %s true", | ||
| 125 | name, gotMAC, gotIP, ok, want.mac, want.ip) | ||
| 126 | } | ||
| 127 | } | ||
| 128 | } | ||
| 129 | |||
| 130 | func TestParseACKRejects(t *testing.T) { | ||
| 131 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 132 | ack := reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42") | ||
| 133 | |||
| 134 | // A request the guest sends: client -> server, and a BootRequest (New's | ||
| 135 | // default opcode). | ||
| 136 | req, err := dhcpv4.New( | ||
| 137 | dhcpv4.WithMessageType(dhcpv4.MessageTypeRequest), | ||
| 138 | dhcpv4.WithHwAddr(mac), | ||
| 139 | ) | ||
| 140 | if err != nil { | ||
| 141 | t.Fatal(err) | ||
| 142 | } | ||
| 143 | |||
| 144 | truncatedIP := frame(t, ack)[:14+12] | ||
| 145 | badIHL := frame(t, ack) | ||
| 146 | badIHL[14] = 0x44 // v4, ihl 4 — shorter than the fixed header allows | ||
| 147 | |||
| 148 | notUDP := frame(t, ack) | ||
| 149 | notUDP[14+9] = 6 // TCP | ||
| 150 | |||
| 151 | // A guest that forges an ACK on the right ports still fails the opcode | ||
| 152 | // check: only a boot reply grants an address. | ||
| 153 | forged := newReply(t, | ||
| 154 | dhcpv4.WithMessageType(dhcpv4.MessageTypeAck), | ||
| 155 | dhcpv4.WithHwAddr(mac), | ||
| 156 | dhcpv4.WithYourIP(net.ParseIP("192.168.0.42")), | ||
| 157 | ) | ||
| 158 | forged.OpCode = dhcpv4.OpcodeBootRequest | ||
| 159 | |||
| 160 | for name, b := range map[string][]byte{ | ||
| 161 | "offer-not-ack": frame(t, reply(t, mac, dhcpv4.MessageTypeOffer, "192.168.0.42")), | ||
| 162 | "nak-not-ack": frame(t, reply(t, mac, dhcpv4.MessageTypeNak, "")), | ||
| 163 | "guest-request": framePorts(t, req.ToBytes(), 68, 67), | ||
| 164 | "opcode-request": frame(t, forged.ToBytes()), | ||
| 165 | "ack-wrong-dst": framePorts(t, ack, 67, 12345), | ||
| 166 | "ack-wrong-src": framePorts(t, ack, 12345, 68), | ||
| 167 | "zero-yiaddr": frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "")), | ||
| 168 | "truncated-ip": truncatedIP, | ||
| 169 | "bad-ihl": badIHL, | ||
| 170 | "not-udp": notUDP, | ||
| 171 | "garbage-dhcp": frame(t, []byte{1, 2, 3, 4}), | ||
| 172 | "short": {0, 1, 2}, | ||
| 173 | "not-ipv4": append(make([]byte, 14), 0xde), | ||
| 174 | "arp-ethertype": append([]byte{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x08, 0x06}, make([]byte, 60)...), | ||
| 175 | "ipv6-in-v4-eth": func() []byte { f := frame(t, ack); f[14] = 0x65; return f }(), | ||
| 176 | "empty": {}, | ||
| 177 | "nil": nil, | ||
| 178 | } { | ||
| 179 | if _, _, ok := ParseACK(b); ok { | ||
| 180 | t.Errorf("%s: ParseACK accepted, want reject", name) | ||
| 181 | } | ||
| 182 | } | ||
| 183 | } | ||
| 184 | |||
| 185 | // A DHCPv4 yiaddr is four bytes on the wire, so anything ParseACK decodes | ||
| 186 | // has a non-nil To4 and the To4==nil arm of the address guard is unreachable | ||
| 187 | // from a real frame. This pins that invariant: if the decoder ever started | ||
| 188 | // handing back a wider IP, the guard would become live and this test would be | ||
| 189 | // the record of why it exists. | ||
| 190 | func TestDecodedYourIPIsAlwaysV4(t *testing.T) { | ||
| 191 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 192 | msg, err := dhcpv4.FromBytes(reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42")) | ||
| 193 | if err != nil { | ||
| 194 | t.Fatal(err) | ||
| 195 | } | ||
| 196 | if msg.YourIPAddr.To4() == nil { | ||
| 197 | t.Fatalf("decoded yiaddr %v is not IPv4-shaped", msg.YourIPAddr) | ||
| 198 | } | ||
| 199 | } | ||
| 200 | |||
| 201 | // The snoop runs on every frame a busy tap carries, so it must survive | ||
| 202 | // arbitrary bytes without panicking — the only failure mode allowed is ok=false. | ||
| 203 | func TestParseACKFuzzResistant(t *testing.T) { | ||
| 204 | mac, _ := net.ParseMAC("52:54:00:aa:bb:cc") | ||
| 205 | full := frame(t, reply(t, mac, dhcpv4.MessageTypeAck, "192.168.0.42")) | ||
| 206 | for i := range full { | ||
| 207 | ParseACK(full[:i]) // every truncation of a real frame | ||
| 208 | } | ||
| 209 | for i := range full { | ||
| 210 | mutated := append([]byte(nil), full...) | ||
| 211 | mutated[i] ^= 0xff | ||
| 212 | ParseACK(mutated) | ||
| 213 | } | ||
| 214 | } | ||
internal/agent/reconcile/reconcile.go
| Old | New | ||
|---|---|---|---|
| @@ -109,6 +109,18 @@ type Provisioner interface { | |||
| 109 | // and Boot must not block inside the create slot waiting for that. Empty | 109 | // and Boot must not block inside the create slot waiting for that. Empty |
| 110 | // means "no answer", never "no address" — see noteAddress. | 110 | // means "no answer", never "no address" — see noteAddress. |
| 111 | Address(vmID string) string | 111 | Address(vmID string) string |
| 112 | |||
| 113 | // NetworkAddress returns the address the VM's SECOND NIC — the one on the | ||
| 114 | // named host network its spec asked for — was granted, or "" when it has no | ||
| 115 | // such NIC or the backend has not learned one yet. A backend that serves no | ||
| 116 | // named networks answers "" always, which is every Mac and every Linux host | ||
| 117 | // with no --host-network flag. | ||
| 118 | // | ||
| 119 | // Separate from Address rather than replacing it because the two are | ||
| 120 | // different facts with different timings: Address is allocated by the host | ||
| 121 | // and known at boot, this one is granted by someone else's DHCP server and | ||
| 122 | // discovered afterwards. Empty means "not yet known", never "unreachable". | ||
| 123 | NetworkAddress(vmID string) string | ||
| 112 | } | 124 | } |
| 113 | 125 | ||
| 114 | // Engine is the reconcile loop. All fields must be set before calling Step. | 126 | // Engine is the reconcile loop. All fields must be set before calling Step. |
| @@ -349,7 +361,7 @@ func (e *Engine) fenceReport(currentEpoch uint64) *pb.ActualStateReport { | |||
| 349 | if rec.LastError != "" { | 361 | if rec.LastError != "" { |
| 350 | phase = "failed" | 362 | phase = "failed" |
| 351 | } | 363 | } |
| 352 | res.report(rec.Spec.VMID, rec.IP, power, phase, rec.LastError) | 364 | res.report(rec.Spec.VMID, recAddrs(rec), power, phase, rec.LastError) |
| 353 | res.merge(rep) | 365 | res.merge(rep) |
| 354 | } | 366 | } |
| 355 | return rep | 367 | return rep |
| @@ -589,18 +601,27 @@ func (e *Engine) releaseCompute(vmID string) { | |||
| 589 | delete(e.committed, vmID) | 601 | delete(e.committed, vmID) |
| 590 | } | 602 | } |
| 591 | 603 | ||
| 592 | // noteAddress folds the backend's current answer for this VM's address into | 604 | // noteAddress folds the backend's current answers for this VM's addresses into |
| 593 | // rec, reporting whether it changed. An empty answer never clears a known | 605 | // rec, reporting whether either changed. An empty answer never clears a known |
| 594 | // address: "I don't know yet" is not "it has none", and blanking rec.IP would | 606 | // address: "I don't know yet" is not "it has none", and blanking rec.IP would |
| 595 | // break the guest SSH tunnel (syncclient refuses an empty address) for a VM | 607 | // break the guest SSH tunnel (syncclient refuses an empty address) for a VM |
| 596 | // that is perfectly reachable. | 608 | // that is perfectly reachable. |
| 609 | // | ||
| 610 | // Both NICs are asked on every poll, and the same rule covers both. The named | ||
| 611 | // NIC's address is the one that genuinely arrives late — the host discovers it | ||
| 612 | // by watching the guest's DHCP exchange — while rec.IP is known before the | ||
| 613 | // guest boots. | ||
| 597 | func (e *Engine) noteAddress(rec *state.Record) bool { | 614 | func (e *Engine) noteAddress(rec *state.Record) bool { |
| 598 | ip := e.Prov.Address(rec.Spec.VMID) | 615 | changed := false |
| 599 | if ip == "" || ip == rec.IP { | 616 | if ip := e.Prov.Address(rec.Spec.VMID); ip != "" && ip != rec.IP { |
| 600 | return false | 617 | rec.IP = ip |
| 618 | changed = true | ||
| 601 | } | 619 | } |
| 602 | rec.IP = ip | 620 | if netIP := e.Prov.NetworkAddress(rec.Spec.VMID); netIP != "" && netIP != rec.NetworkIP { |
| 603 | return true | 621 | rec.NetworkIP = netIP |
| 622 | changed = true | ||
| 623 | } | ||
| 624 | return changed | ||
| 604 | } | 625 | } |
| 605 | 626 | ||
| 606 | // SeedLedger rebuilds the compute ledger from persisted records at startup so | 627 | // SeedLedger rebuilds the compute ledger from persisted records at startup so |
| @@ -660,7 +681,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record, | |||
| 660 | // as cheap insurance for a caller that later threads a cancellable context | 681 | // as cheap insurance for a caller that later threads a cancellable context |
| 661 | // into a pass. Report and let the next tick do the work. | 682 | // into a pass. Report and let the next tick do the work. |
| 662 | if err := ctx.Err(); err != nil { | 683 | if err := ctx.Err(); err != nil { |
| 663 | res.report(d.VmId, rec.IP, "stopped", "creating", "reconcile aborted: "+err.Error()) | 684 | res.report(d.VmId, recAddrs(rec), "stopped", "creating", "reconcile aborted: "+err.Error()) |
| 664 | return | 685 | return |
| 665 | } | 686 | } |
| 666 | 687 | ||
| @@ -677,7 +698,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record, | |||
| 677 | 698 | ||
| 678 | // Terminal check: if we've hit MaxCreateAttempts, stop retrying. | 699 | // Terminal check: if we've hit MaxCreateAttempts, stop retrying. |
| 679 | if ok && rec.CreateAttempts >= e.MaxCreateAttempts { | 700 | if ok && rec.CreateAttempts >= e.MaxCreateAttempts { |
| 680 | res.report(d.VmId, rec.IP, "stopped", "failed", rec.LastError) | 701 | res.report(d.VmId, recAddrs(rec), "stopped", "failed", rec.LastError) |
| 681 | return | 702 | return |
| 682 | } | 703 | } |
| 683 | 704 | ||
| @@ -710,7 +731,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record, | |||
| 710 | // Reported, not yet certified. Publish the public key and wait: | 731 | // Reported, not yet certified. Publish the public key and wait: |
| 711 | // booting now would hand the guest a host key every client refuses, | 732 | // booting now would hand the guest a host key every client refuses, |
| 712 | // and converge never rebuilds a seed, so it would stay that way. | 733 | // and converge never rebuilds a seed, so it would stay that way. |
| 713 | res.report(d.VmId, rec.IP, "stopped", "creating", "awaiting host certificate") | 734 | res.report(d.VmId, recAddrs(rec), "stopped", "creating", "awaiting host certificate") |
| 714 | return | 735 | return |
| 715 | } | 736 | } |
| 716 | } | 737 | } |
| @@ -720,7 +741,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record, | |||
| 720 | // frees the next tick retries and boots. | 741 | // frees the next tick retries and boots. |
| 721 | rec.Spec = spec | 742 | rec.Spec = spec |
| 722 | if quotaMsg := e.admit(d.VmId, spec); quotaMsg != "" { | 743 | if quotaMsg := e.admit(d.VmId, spec); quotaMsg != "" { |
| 723 | res.report(d.VmId, rec.IP, "stopped", "failed", quotaMsg) | 744 | res.report(d.VmId, recAddrs(rec), "stopped", "failed", quotaMsg) |
| 724 | return | 745 | return |
| 725 | } | 746 | } |
| 726 | rec.CreateAttempts++ | 747 | rec.CreateAttempts++ |
| @@ -754,7 +775,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record, | |||
| 754 | } | 775 | } |
| 755 | var interim vmResult | 776 | var interim vmResult |
| 756 | interim.hostPubKey = res.hostPubKey | 777 | interim.hostPubKey = res.hostPubKey |
| 757 | interim.report(d.VmId, rec.IP, "stopped", "creating", "") | 778 | interim.report(d.VmId, recAddrs(rec), "stopped", "creating", "") |
| 758 | interim.vm.StatusDetail = detail | 779 | interim.vm.StatusDetail = detail |
| 759 | pub(interim) | 780 | pub(interim) |
| 760 | } | 781 | } |
| @@ -819,6 +840,12 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record, | |||
| 819 | // generated above and held on this host. Nothing else can supply one. | 840 | // generated above and held on this host. Nothing else can supply one. |
| 820 | SSHHostKeyPEM: hostKey.PrivatePEM, | 841 | SSHHostKeyPEM: hostKey.PrivatePEM, |
| 821 | SSHHostCert: d.SshHostCert, | 842 | SSHHostCert: d.SshHostCert, |
| 843 | // The guest's NICs, so its netplan matches each by the address the | ||
| 844 | // hypervisor gave it. The second one exists only for a VM that asked for | ||
| 845 | // a named network; both are derived from the VM id, so the seed and the | ||
| 846 | // host agree without either asking the hypervisor what it ended up with. | ||
| 847 | MAC: state.MAC(d.VmId), | ||
| 848 | NetworkMAC: netMACIfNetworked(rec.Spec), | ||
| 822 | }); err != nil { | 849 | }); err != nil { |
| 823 | e.failCreate(ctx, rec, err, res) | 850 | e.failCreate(ctx, rec, err, res) |
| 824 | return | 851 | return |
| @@ -848,7 +875,7 @@ func (e *Engine) create(ctx context.Context, d *pb.VMDesired, rec state.Record, | |||
| 848 | if d.PowerState != "running" { | 875 | if d.PowerState != "running" { |
| 849 | power = "stopped" | 876 | power = "stopped" |
| 850 | } | 877 | } |
| 851 | res.report(d.VmId, rec.IP, power, "ready", "") | 878 | res.report(d.VmId, recAddrs(rec), power, "ready", "") |
| 852 | } | 879 | } |
| 853 | 880 | ||
| 854 | // permanent reports whether err (anywhere in its chain) carries the | 881 | // permanent reports whether err (anywhere in its chain) carries the |
| @@ -884,7 +911,7 @@ func (e *Engine) failCreate(ctx context.Context, rec state.Record, err error, re | |||
| 884 | if rec.CreateAttempts >= e.MaxCreateAttempts { | 911 | if rec.CreateAttempts >= e.MaxCreateAttempts { |
| 885 | phase = "failed" | 912 | phase = "failed" |
| 886 | } | 913 | } |
| 887 | res.report(rec.Spec.VMID, rec.IP, "stopped", phase, rec.LastError) | 914 | res.report(rec.Spec.VMID, recAddrs(rec), "stopped", phase, rec.LastError) |
| 888 | } | 915 | } |
| 889 | 916 | ||
| 890 | // failConverge is the shared epilogue for the converge restart/boot paths: | 917 | // failConverge is the shared epilogue for the converge restart/boot paths: |
| @@ -902,7 +929,7 @@ func (e *Engine) failConverge(rec state.Record, err error, res *vmResult) { | |||
| 902 | rec.LastError += ": " + why | 929 | rec.LastError += ": " + why |
| 903 | } | 930 | } |
| 904 | _ = e.St.SaveVM(rec) | 931 | _ = e.St.SaveVM(rec) |
| 905 | res.report(rec.Spec.VMID, rec.IP, "stopped", "failed", rec.LastError) | 932 | res.report(rec.Spec.VMID, recAddrs(rec), "stopped", "failed", rec.LastError) |
| 906 | } | 933 | } |
| 907 | 934 | ||
| 908 | // converge drives an existing VM toward its desired power state, | 935 | // converge drives an existing VM toward its desired power state, |
| @@ -943,13 +970,13 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record | |||
| 943 | rec.StopRequested = false | 970 | rec.StopRequested = false |
| 944 | rec.LastError = "" // Fix 3: clear stale error on successful restart | 971 | rec.LastError = "" // Fix 3: clear stale error on successful restart |
| 945 | _ = e.St.SaveVM(rec) | 972 | _ = e.St.SaveVM(rec) |
| 946 | res.report(d.VmId, rec.IP, "running", "ready", "") | 973 | res.report(d.VmId, recAddrs(rec), "running", "ready", "") |
| 947 | } else { | 974 | } else { |
| 948 | // Lost + desired stopped: update boot ID, mark stop recorded. | 975 | // Lost + desired stopped: update boot ID, mark stop recorded. |
| 949 | rec.BootID = bootID | 976 | rec.BootID = bootID |
| 950 | rec.StopRequested = true | 977 | rec.StopRequested = true |
| 951 | _ = e.St.SaveVM(rec) | 978 | _ = e.St.SaveVM(rec) |
| 952 | res.report(d.VmId, rec.IP, "stopped", "ready", "") | 979 | res.report(d.VmId, recAddrs(rec), "stopped", "ready", "") |
| 953 | } | 980 | } |
| 954 | return | 981 | return |
| 955 | } | 982 | } |
| @@ -965,7 +992,7 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record | |||
| 965 | rec.StopRequested = false | 992 | rec.StopRequested = false |
| 966 | rec.LastError = "" // Fix 3: clear stale error on successful boot | 993 | rec.LastError = "" // Fix 3: clear stale error on successful boot |
| 967 | _ = e.St.SaveVM(rec) | 994 | _ = e.St.SaveVM(rec) |
| 968 | res.report(d.VmId, rec.IP, "running", "ready", "") | 995 | res.report(d.VmId, recAddrs(rec), "running", "ready", "") |
| 969 | } else if d.PowerState == "stopped" && running { | 996 | } else if d.PowerState == "stopped" && running { |
| 970 | // Stop the VM. Record stop BEFORE side effects so a crash between | 997 | // Stop the VM. Record stop BEFORE side effects so a crash between |
| 971 | // SaveVM and Shutdown is recoverable (the persisted StopRequested | 998 | // SaveVM and Shutdown is recoverable (the persisted StopRequested |
| @@ -977,11 +1004,11 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record | |||
| 977 | if err := e.St.SaveVM(rec); err != nil { | 1004 | if err := e.St.SaveVM(rec); err != nil { |
| 978 | // Cannot durably record the stop intent; skip Shutdown this cycle. | 1005 | // Cannot durably record the stop intent; skip Shutdown this cycle. |
| 979 | // The next reconcile will retry once the store recovers. | 1006 | // The next reconcile will retry once the store recovers. |
| 980 | res.report(d.VmId, rec.IP, "running", "failed", err.Error()) | 1007 | res.report(d.VmId, recAddrs(rec), "running", "failed", err.Error()) |
| 981 | return | 1008 | return |
| 982 | } | 1009 | } |
| 983 | _ = e.Prov.Shutdown(ctx, d.VmId) | 1010 | _ = e.Prov.Shutdown(ctx, d.VmId) |
| 984 | res.report(d.VmId, rec.IP, "stopped", "ready", "") | 1011 | res.report(d.VmId, recAddrs(rec), "stopped", "ready", "") |
| 985 | } else { | 1012 | } else { |
| 986 | // Already at desired state. | 1013 | // Already at desired state. |
| 987 | power := "stopped" | 1014 | power := "stopped" |
| @@ -990,7 +1017,7 @@ func (e *Engine) converge(ctx context.Context, d *pb.VMDesired, rec state.Record | |||
| 990 | } | 1017 | } |
| 991 | // Preserve last error in the report field but phase stays ready | 1018 | // Preserve last error in the report field but phase stays ready |
| 992 | // (the VM is converged; the error is informational history). | 1019 | // (the VM is converged; the error is informational history). |
| 993 | res.report(d.VmId, rec.IP, power, "ready", rec.LastError) | 1020 | res.report(d.VmId, recAddrs(rec), power, "ready", rec.LastError) |
| 994 | } | 1021 | } |
| 995 | } | 1022 | } |
| 996 | 1023 | ||
| @@ -1024,10 +1051,22 @@ type vmResult struct { | |||
| 1024 | hostPubKey string | 1051 | hostPubKey string |
| 1025 | } | 1052 | } |
| 1026 | 1053 | ||
| 1054 | // addrs is where a VM is: its address on its host's NAT underlay, which every | ||
| 1055 | // guest has from boot, and — for a guest with a second NIC on a named host | ||
| 1056 | // network — the address the site's own DHCP server granted that one. They | ||
| 1057 | // travel together because every report of one is a report of the other, and a | ||
| 1058 | // row that carried only the first would blank a networked guest's LAN address | ||
| 1059 | // on the fleet's side every time the host said anything about it. | ||
| 1060 | type addrs struct{ ip, networkIP string } | ||
| 1061 | |||
| 1062 | // recAddrs reads a record's pair. The record is the only place both are known | ||
| 1063 | // at once: the backend answers them one question at a time. | ||
| 1064 | func recAddrs(rec state.Record) addrs { return addrs{ip: rec.IP, networkIP: rec.NetworkIP} } | ||
| 1065 | |||
| 1027 | // report records this VM's actual row. A VM contributes at most one row, so a | 1066 | // report records this VM's actual row. A VM contributes at most one row, so a |
| 1028 | // later call in the same reconcile replaces an earlier one. | 1067 | // later call in the same reconcile replaces an earlier one. |
| 1029 | func (r *vmResult) report(vmID, ip, power, phase, lastError string) { | 1068 | func (r *vmResult) report(vmID string, at addrs, power, phase, lastError string) { |
| 1030 | r.vm = newActualVM(vmID, ip, power, phase, lastError) | 1069 | r.vm = newActualVM(vmID, at, power, phase, lastError) |
| 1031 | } | 1070 | } |
| 1032 | 1071 | ||
| 1033 | // clone returns a deep copy, so the caller's report owns its rows outright. | 1072 | // clone returns a deep copy, so the caller's report owns its rows outright. |
| @@ -1084,10 +1123,11 @@ func byteScale(n int64) (float64, string) { | |||
| 1084 | // newActualVM builds one ActualVM row from what a reconcile pass observed. | 1123 | // newActualVM builds one ActualVM row from what a reconcile pass observed. |
| 1085 | // The row's remaining field, ssh_host_pubkey, is stamped by merge — see | 1124 | // The row's remaining field, ssh_host_pubkey, is stamped by merge — see |
| 1086 | // vmResult.hostPubKey. Unset values are the proto zero-value "". | 1125 | // vmResult.hostPubKey. Unset values are the proto zero-value "". |
| 1087 | func newActualVM(vmID, ip, power, phase, lastError string) *pb.ActualVM { | 1126 | func newActualVM(vmID string, at addrs, power, phase, lastError string) *pb.ActualVM { |
| 1088 | return &pb.ActualVM{ | 1127 | return &pb.ActualVM{ |
| 1089 | VmId: vmID, | 1128 | VmId: vmID, |
| 1090 | Ip: ip, | 1129 | Ip: at.ip, |
| 1130 | NetworkIp: at.networkIP, | ||
| 1091 | Power: power, | 1131 | Power: power, |
| 1092 | Phase: phase, | 1132 | Phase: phase, |
| 1093 | LastError: lastError, | 1133 | LastError: lastError, |
| @@ -1105,6 +1145,16 @@ func joinCALines(lines []string) string { | |||
| 1105 | return b.String() | 1145 | return b.String() |
| 1106 | } | 1146 | } |
| 1107 | 1147 | ||
| 1148 | // netMACIfNetworked returns the second NIC's MAC for a VM that asked for a | ||
| 1149 | // named network, and "" for one that did not — which is what makes the seed | ||
| 1150 | // write the single-NIC network-config every guest running today already has. | ||
| 1151 | func netMACIfNetworked(spec state.VMSpec) string { | ||
| 1152 | if spec.Network == "" { | ||
| 1153 | return "" | ||
| 1154 | } | ||
| 1155 | return state.NetMAC(spec.VMID) | ||
| 1156 | } | ||
| 1157 | |||
| 1108 | // specFromDesired maps a pb.VMDesired to state.VMSpec. | 1158 | // specFromDesired maps a pb.VMDesired to state.VMSpec. |
| 1109 | func specFromDesired(d *pb.VMDesired) state.VMSpec { | 1159 | func specFromDesired(d *pb.VMDesired) state.VMSpec { |
| 1110 | return state.VMSpec{ | 1160 | return state.VMSpec{ |
| @@ -1114,6 +1164,7 @@ func specFromDesired(d *pb.VMDesired) state.VMSpec { | |||
| 1114 | ImageSHA256: d.ImageSha256, | 1164 | ImageSHA256: d.ImageSha256, |
| 1115 | CloudInit: d.CloudInit, | 1165 | CloudInit: d.CloudInit, |
| 1116 | SSHAuthorizedKey: d.SshAuthorizedKey, | 1166 | SSHAuthorizedKey: d.SshAuthorizedKey, |
| 1167 | Network: d.Network, | ||
| 1117 | VCPUs: d.Vcpus, | 1168 | VCPUs: d.Vcpus, |
| 1118 | MemMB: d.MemMb, | 1169 | MemMB: d.MemMb, |
| 1119 | DiskGB: d.DiskGb, | 1170 | DiskGB: d.DiskGb, |
internal/agent/reconcile/reconcile_test.go
| Old | New | ||
|---|---|---|---|
| @@ -43,6 +43,10 @@ type fakeProv struct { | |||
| 43 | 43 | ||
| 44 | cidr string | 44 | cidr string |
| 45 | addrs map[string]string // vmID -> ip (sticky, mirrors the DHCP table) | 45 | addrs map[string]string // vmID -> ip (sticky, mirrors the DHCP table) |
| 46 | // netAddrs is what a snoop on the named NIC heard, per VM. Never populated | ||
| 47 | // by Boot: this address comes from someone else's DHCP server, so a test | ||
| 48 | // that wants one plays the guest with answerNetworkAddress. | ||
| 49 | netAddrs map[string]string | ||
| 46 | 50 | ||
| 47 | // failReason is what the fake's "hypervisor" left behind; empty is a | 51 | // failReason is what the fake's "hypervisor" left behind; empty is a |
| 48 | // backend with nothing to add, which is the common case. | 52 | // backend with nothing to add, which is the common case. |
| @@ -56,9 +60,10 @@ type fakeProv struct { | |||
| 56 | 60 | ||
| 57 | func newFakeProv() *fakeProv { | 61 | func newFakeProv() *fakeProv { |
| 58 | return &fakeProv{ | 62 | return &fakeProv{ |
| 59 | running: map[string]bool{}, | 63 | running: map[string]bool{}, |
| 60 | addrs: map[string]string{}, | 64 | addrs: map[string]string{}, |
| 61 | cidr: "10.77.1.0/24", | 65 | netAddrs: map[string]string{}, |
| 66 | cidr: "10.77.1.0/24", | ||
| 62 | } | 67 | } |
| 63 | } | 68 | } |
| 64 | 69 | ||
| @@ -123,6 +128,21 @@ func (f *fakeProv) Address(id string) string { | |||
| 123 | return f.addrs[id] | 128 | return f.addrs[id] |
| 124 | } | 129 | } |
| 125 | 130 | ||
| 131 | func (f *fakeProv) NetworkAddress(id string) string { | ||
| 132 | f.mu.Lock() | ||
| 133 | defer f.mu.Unlock() | ||
| 134 | return f.netAddrs[id] | ||
| 135 | } | ||
| 136 | |||
| 137 | // answerNetworkAddress plays the site's own DHCP server: the guest's second NIC | ||
| 138 | // has now been granted an address, and the host's snoop heard it. Always late — | ||
| 139 | // there is no path by which this is known when Boot returns. | ||
| 140 | func (f *fakeProv) answerNetworkAddress(id, ip string) { | ||
| 141 | f.mu.Lock() | ||
| 142 | defer f.mu.Unlock() | ||
| 143 | f.netAddrs[id] = ip | ||
| 144 | } | ||
| 145 | |||
| 126 | // answerAddress starts answering for id: the guest has now booted and asked the | 146 | // answerAddress starts answering for id: the guest has now booted and asked the |
| 127 | // host's DHCP server, seconds after Boot returned. Only meaningful under | 147 | // host's DHCP server, seconds after Boot returned. Only meaningful under |
| 128 | // lateAddress. | 148 | // lateAddress. |
| @@ -826,6 +846,80 @@ func TestAddressDiscoveredAfterBootIsPersisted(t *testing.T) { | |||
| 826 | assert.Equal(t, "10.77.1.2", recs["vm1"].IP, "and persist it, not just report it") | 846 | assert.Equal(t, "10.77.1.2", recs["vm1"].IP, "and persist it, not just report it") |
| 827 | } | 847 | } |
| 828 | 848 | ||
| 849 | // TestNetworkedVMReportsBothAddresses pins the additive shape at the layer that | ||
| 850 | // reports it: the NAT address is there from the first tick, as it is for every | ||
| 851 | // guest, and the named NIC's address joins it — separately, later, without ever | ||
| 852 | // displacing it. The gate path uses the first, so it has no discovery window at | ||
| 853 | // all; the LAN address arrives when the site's DHCP server says so. | ||
| 854 | func TestNetworkedVMReportsBothAddresses(t *testing.T) { | ||
| 855 | f := setup(t) | ||
| 856 | lan := func(v *pb.VMDesired) { v.Network = "lan" } | ||
| 857 | |||
| 858 | rep := f.step(snap(1, vm("vm1", lan))) | ||
| 859 | av := findVM(rep, "vm1") | ||
| 860 | require.NotNil(t, av) | ||
| 861 | assert.Equal(t, "10.77.1.2", av.Ip, "the NAT NIC is unconditional and known at boot") | ||
| 862 | assert.Empty(t, av.NetworkIp, "the LAN address is not known until the guest asks for it") | ||
| 863 | |||
| 864 | f.prov.answerNetworkAddress("vm1", "192.168.0.42") // the site's server answered | ||
| 865 | |||
| 866 | rep = f.step(snap(2, vm("vm1", lan))) | ||
| 867 | av = findVM(rep, "vm1") | ||
| 868 | require.NotNil(t, av) | ||
| 869 | assert.Equal(t, "10.77.1.2", av.Ip, "which never moves") | ||
| 870 | assert.Equal(t, "192.168.0.42", av.NetworkIp) | ||
| 871 | |||
| 872 | recs, _ := f.st.LoadVMs() | ||
| 873 | require.Contains(t, recs, "vm1") | ||
| 874 | assert.Equal(t, "10.77.1.2", recs["vm1"].IP) | ||
| 875 | assert.Equal(t, "192.168.0.42", recs["vm1"].NetworkIP, "persisted, not just reported") | ||
| 876 | assert.Equal(t, "lan", recs["vm1"].Spec.Network, "and the spec that asked survives the restart") | ||
| 877 | |||
| 878 | // The empty guard covers this address too: a snoop that has heard nothing | ||
| 879 | // since the agent restarted must not erase the fleet's record of where the | ||
| 880 | // guest is on the LAN. | ||
| 881 | f.prov.answerNetworkAddress("vm1", "") | ||
| 882 | rep = f.step(snap(3, vm("vm1", lan))) | ||
| 883 | av = findVM(rep, "vm1") | ||
| 884 | require.NotNil(t, av) | ||
| 885 | assert.Equal(t, "192.168.0.42", av.NetworkIp, "an empty answer is 'not known', not 'gone'") | ||
| 886 | } | ||
| 887 | |||
| 888 | // TestNATOnlyVMReportsNoNetworkAddress is the guard for every guest running | ||
| 889 | // today: nothing about the second NIC leaks into a VM that never asked for one, | ||
| 890 | // even when the backend is asked about it every tick. | ||
| 891 | func TestNATOnlyVMReportsNoNetworkAddress(t *testing.T) { | ||
| 892 | f := setup(t) | ||
| 893 | |||
| 894 | rep := f.step(snap(1, vm("vm1"))) | ||
| 895 | av := findVM(rep, "vm1") | ||
| 896 | require.NotNil(t, av) | ||
| 897 | assert.Equal(t, "10.77.1.2", av.Ip) | ||
| 898 | assert.Empty(t, av.NetworkIp) | ||
| 899 | |||
| 900 | recs, _ := f.st.LoadVMs() | ||
| 901 | assert.Empty(t, recs["vm1"].NetworkIP) | ||
| 902 | } | ||
| 903 | |||
| 904 | // TestSeedCarriesBothNICsOnlyForANetworkedGuest pins what the guest is told | ||
| 905 | // about its own hardware: a networked guest's seed names both MACs so netplan | ||
| 906 | // can match each NIC on its own, and a NAT-only guest's names no second one at | ||
| 907 | // all — which is what keeps its network-config the single-stanza document every | ||
| 908 | // running guest already booted with. | ||
| 909 | func TestSeedCarriesBothNICsOnlyForANetworkedGuest(t *testing.T) { | ||
| 910 | f := setup(t) | ||
| 911 | var got seed.Params | ||
| 912 | f.eng.Seed = func(_ string, p seed.Params) error { got = p; return nil } | ||
| 913 | |||
| 914 | f.step(snap(1, vm("vm1", func(v *pb.VMDesired) { v.Network = "lan" }))) | ||
| 915 | assert.Equal(t, state.MAC("vm1"), got.MAC) | ||
| 916 | assert.Equal(t, state.NetMAC("vm1"), got.NetworkMAC) | ||
| 917 | |||
| 918 | f.step(snap(2, vm("vm2"))) | ||
| 919 | assert.Equal(t, state.MAC("vm2"), got.MAC) | ||
| 920 | assert.Empty(t, got.NetworkMAC, "a guest that asked for no network is told of no second NIC") | ||
| 921 | } | ||
| 922 | |||
| 829 | // TestEmptyAddressAnswerKeepsTheKnownOne pins noteAddress's empty guard: "I have | 923 | // TestEmptyAddressAnswerKeepsTheKnownOne pins noteAddress's empty guard: "I have |
| 830 | // no answer" is not "it has no address". Blanking rec.IP would cut the guest SSH | 924 | // no answer" is not "it has no address". Blanking rec.IP would cut the guest SSH |
| 831 | // tunnel to a VM that is reachable and running. | 925 | // tunnel to a VM that is reachable and running. |
internal/agent/run/cli.go
| Old | New | ||
|---|---|---|---|
| @@ -19,6 +19,7 @@ import ( | |||
| 19 | "os/exec" | 19 | "os/exec" |
| 20 | "os/signal" | 20 | "os/signal" |
| 21 | "runtime" | 21 | "runtime" |
| 22 | "strings" | ||
| 22 | "syscall" | 23 | "syscall" |
| 23 | "time" | 24 | "time" |
| 24 | 25 | ||
| @@ -33,6 +34,7 @@ import ( | |||
| 33 | "github.com/a73x/eitri/internal/agent/syncclient" | 34 | "github.com/a73x/eitri/internal/agent/syncclient" |
| 34 | "github.com/a73x/eitri/internal/covsnap" | 35 | "github.com/a73x/eitri/internal/covsnap" |
| 35 | "github.com/a73x/eitri/internal/joinblob" | 36 | "github.com/a73x/eitri/internal/joinblob" |
| 37 | "github.com/a73x/eitri/internal/names" | ||
| 36 | ) | 38 | ) |
| 37 | 39 | ||
| 38 | // hostRunner is the production one-shot command runner injected into the | 40 | // hostRunner is the production one-shot command runner injected into the |
| @@ -61,6 +63,10 @@ type Config struct { | |||
| 61 | // MaxVCPUs, MaxMemMB, MaxDiskGB cap the host resources this agent offers the | 63 | // MaxVCPUs, MaxMemMB, MaxDiskGB cap the host resources this agent offers the |
| 62 | // fleet (0 = unlimited): advertised to the server AND enforced at VM boot. | 64 | // fleet (0 = unlimited): advertised to the server AND enforced at VM boot. |
| 63 | MaxVCPUs, MaxMemMB, MaxDiskGB int64 | 65 | MaxVCPUs, MaxMemMB, MaxDiskGB int64 |
| 66 | // HostNetworks maps a named guest network to the bridge this host's | ||
| 67 | // operator already declared for it (Linux hosts only). eitri attaches taps | ||
| 68 | // to the bridge; it never creates or addresses it. | ||
| 69 | HostNetworks map[string]string | ||
| 64 | } | 70 | } |
| 65 | 71 | ||
| 66 | // RunCLI dispatches the eitri-agent command line (everything after the binary | 72 | // RunCLI dispatches the eitri-agent command line (everything after the binary |
| @@ -89,11 +95,24 @@ func RunCLI(args []string) error { | |||
| 89 | return serve(st, cfg) | 95 | return serve(st, cfg) |
| 90 | } | 96 | } |
| 91 | 97 | ||
| 92 | // parseConfig defines and parses the agent flags, returning the assembled | 98 | // parseConfig defines and parses the agent flags for THIS process's platform — |
| 99 | // a thin wrapper over parseConfigOn(runtime.GOOS, args), which exists so a | ||
| 100 | // test can drive the darwin refusal path without a GOOS-tagged build. | ||
| 101 | func parseConfig(args []string) (Config, []string, error) { | ||
| 102 | return parseConfigOn(runtime.GOOS, args) | ||
| 103 | } | ||
| 104 | |||
| 105 | // parseConfigOn defines and parses the agent flags, returning the assembled | ||
| 93 | // Config and any positional arguments (the "join <blob>" subcommand). Resource | 106 | // Config and any positional arguments (the "join <blob>" subcommand). Resource |
| 94 | // caps are validated here so a negative cap is rejected before any state is | 107 | // caps are validated here so a negative cap is rejected before any state is |
| 95 | // touched. | 108 | // touched. goos decides whether --host-network is refused (see |
| 96 | func parseConfig(args []string) (Config, []string, error) { | 109 | // validateHostNetworks); production always passes runtime.GOOS, and it is a |
| 110 | // parameter — not a read of runtime.GOOS inline — so this refusal is provable | ||
| 111 | // on every platform a test runs on, not only the one the flag actually parses | ||
| 112 | // under. goos governs ONLY that one refusal: defaultStateDir and every other | ||
| 113 | // per-platform choice below still come from the build-tagged wire_*.go for | ||
| 114 | // this binary's real GOOS — passing "darwin" here does not simulate a Mac. | ||
| 115 | func parseConfigOn(goos string, args []string) (Config, []string, error) { | ||
| 97 | fs := flag.NewFlagSet("eitri-agent", flag.ContinueOnError) | 116 | fs := flag.NewFlagSet("eitri-agent", flag.ContinueOnError) |
| 98 | // The default is per-platform: a Linux host keeps state where its | 117 | // The default is per-platform: a Linux host keeps state where its |
| 99 | // root-run unit can create it, a Mac in a dotdir the running account owns. | 118 | // root-run unit can create it, a Mac in a dotdir the running account owns. |
| @@ -124,10 +143,32 @@ func parseConfig(args []string) (Config, []string, error) { | |||
| 124 | maxVCPUs := fs.Int64("max-vcpus", 0, "cap the total vCPUs this host offers the fleet (0 = unlimited; reserves headroom, advertised + enforced at boot)") | 143 | maxVCPUs := fs.Int64("max-vcpus", 0, "cap the total vCPUs this host offers the fleet (0 = unlimited; reserves headroom, advertised + enforced at boot)") |
| 125 | maxMemMB := fs.Int64("max-mem-mb", 0, "cap the total memory (MB) this host offers the fleet (0 = unlimited)") | 144 | maxMemMB := fs.Int64("max-mem-mb", 0, "cap the total memory (MB) this host offers the fleet (0 = unlimited)") |
| 126 | maxDiskGB := fs.Int64("max-disk-gb", 0, "cap the total disk (GB) this host offers the fleet (0 = unlimited)") | 145 | maxDiskGB := fs.Int64("max-disk-gb", 0, "cap the total disk (GB) this host offers the fleet (0 = unlimited)") |
| 146 | hostNetworks := map[string]string{} | ||
| 147 | fs.Func("host-network", "a named guest network backed by a bridge this host's "+ | ||
| 148 | "operator already declared, as name=bridge (repeatable; Linux hosts only — "+ | ||
| 149 | "eitri attaches taps to the bridge, it never creates or addresses it)", | ||
| 150 | func(v string) error { | ||
| 151 | name, bridge, ok := strings.Cut(v, "=") | ||
| 152 | if !ok || bridge == "" { | ||
| 153 | return fmt.Errorf("want name=bridge, got %q", v) | ||
| 154 | } | ||
| 155 | if !names.IsNetworkName(name) { | ||
| 156 | return fmt.Errorf("invalid network name %q (1-32 of [a-z0-9-], no leading/trailing hyphen, \"nat\" reserved)", name) | ||
| 157 | } | ||
| 158 | if _, dup := hostNetworks[name]; dup { | ||
| 159 | return fmt.Errorf("network %q declared twice", name) | ||
| 160 | } | ||
| 161 | hostNetworks[name] = bridge | ||
| 162 | return nil | ||
| 163 | }) | ||
| 127 | if err := fs.Parse(args); err != nil { | 164 | if err := fs.Parse(args); err != nil { |
| 128 | return Config{}, nil, err | 165 | return Config{}, nil, err |
| 129 | } | 166 | } |
| 130 | 167 | ||
| 168 | if err := validateHostNetworks(names.OSServesNamedNetworks(goos), hostNetworks); err != nil { | ||
| 169 | return Config{}, nil, err | ||
| 170 | } | ||
| 171 | |||
| 131 | // Fixed order so the reported cap is deterministic when more than one is bad. | 172 | // Fixed order so the reported cap is deterministic when more than one is bad. |
| 132 | caps := []struct { | 173 | caps := []struct { |
| 133 | flag string | 174 | flag string |
| @@ -158,9 +199,27 @@ func parseConfig(args []string) (Config, []string, error) { | |||
| 158 | MaxVCPUs: *maxVCPUs, | 199 | MaxVCPUs: *maxVCPUs, |
| 159 | MaxMemMB: *maxMemMB, | 200 | MaxMemMB: *maxMemMB, |
| 160 | MaxDiskGB: *maxDiskGB, | 201 | MaxDiskGB: *maxDiskGB, |
| 202 | HostNetworks: hostNetworks, | ||
| 161 | }, fs.Args(), nil | 203 | }, fs.Args(), nil |
| 162 | } | 204 | } |
| 163 | 205 | ||
| 206 | // validateHostNetworks refuses --host-network on a host that cannot serve it. | ||
| 207 | // It takes the platform's support as a bool rather than a goos string so the | ||
| 208 | // one fact (names.OSServesNamedNetworks) has a single call site in this file, | ||
| 209 | // in parseConfigOn. | ||
| 210 | // | ||
| 211 | // Refusing here — at parse time, before state is opened or a bridge touched — | ||
| 212 | // matches this project's fail-loudly-at-boot precedent: a Mac that swallowed | ||
| 213 | // the flag would advertise no named networks anyway, and the operator's next | ||
| 214 | // signal would be a 409 telling them to add the exact flag they already | ||
| 215 | // passed. That is a dead end forever; this is a dead end at startup. | ||
| 216 | func validateHostNetworks(platformSupportsHostNetworks bool, hostNetworks map[string]string) error { | ||
| 217 | if len(hostNetworks) == 0 || platformSupportsHostNetworks { | ||
| 218 | return nil | ||
| 219 | } | ||
| 220 | return errors.New("--host-network: named networks are served by Linux hosts only; remove the flag (bridged guests need one of those)") | ||
| 221 | } | ||
| 222 | |||
| 164 | // join handles the "join <blob>" subcommand: decode the join blob, enroll, | 223 | // join handles the "join <blob>" subcommand: decode the join blob, enroll, |
| 165 | // and persist identity — pinning the server cert from the blob (the enroll | 224 | // and persist identity — pinning the server cert from the blob (the enroll |
| 166 | // response's fingerprint is ignored, so the blob is the sole trust root). | 225 | // response's fingerprint is ignored, so the blob is the sole trust root). |
| @@ -317,19 +376,20 @@ func serve(st *state.Store, cfg Config) error { | |||
| 317 | } | 376 | } |
| 318 | 377 | ||
| 319 | client := &syncclient.Client{ | 378 | client := &syncclient.Client{ |
| 320 | Engine: engine, | 379 | Engine: engine, |
| 321 | St: st, | 380 | St: st, |
| 322 | Identity: id, | 381 | Identity: id, |
| 323 | StateDir: cfg.StateDir, | 382 | StateDir: cfg.StateDir, |
| 324 | Provisioner: platformProvisioner, | 383 | Provisioner: platformProvisioner, |
| 325 | GuestCIDR: plat.GuestCIDR, | 384 | GuestCIDR: plat.GuestCIDR, |
| 326 | Runner: hostRunner, | 385 | HostNetworks: plat.HostNetworks, |
| 327 | Console: pumps, | 386 | Runner: hostRunner, |
| 328 | Exposures: proxy, | 387 | Console: pumps, |
| 329 | UplinkAddr: func() string { return hostinfo.UplinkAddr(uplinkVia) }, | 388 | Exposures: proxy, |
| 330 | MaxVCPUs: cfg.MaxVCPUs, | 389 | UplinkAddr: func() string { return hostinfo.UplinkAddr(uplinkVia) }, |
| 331 | MaxMemMB: cfg.MaxMemMB, | 390 | MaxVCPUs: cfg.MaxVCPUs, |
| 332 | MaxDiskGB: cfg.MaxDiskGB, | 391 | MaxMemMB: cfg.MaxMemMB, |
| 392 | MaxDiskGB: cfg.MaxDiskGB, | ||
| 333 | } | 393 | } |
| 334 | 394 | ||
| 335 | slog.Info("agent started", "host_id", id.HostID, "guest_cidr", plat.GuestCIDR()) | 395 | slog.Info("agent started", "host_id", id.HostID, "guest_cidr", plat.GuestCIDR()) |
internal/agent/run/cli_test.go
| Old | New | ||
|---|---|---|---|
| @@ -49,6 +49,7 @@ func TestParseConfigDefaults(t *testing.T) { | |||
| 49 | assert.Equal(t, int64(0), cfg.MaxVCPUs) | 49 | assert.Equal(t, int64(0), cfg.MaxVCPUs) |
| 50 | assert.Equal(t, int64(0), cfg.MaxMemMB) | 50 | assert.Equal(t, int64(0), cfg.MaxMemMB) |
| 51 | assert.Equal(t, int64(0), cfg.MaxDiskGB) | 51 | assert.Equal(t, int64(0), cfg.MaxDiskGB) |
| 52 | assert.Equal(t, map[string]string{}, cfg.HostNetworks) | ||
| 52 | } | 53 | } |
| 53 | 54 | ||
| 54 | // TestParseConfigOverrides confirms every flag threads through to the Config. | 55 | // TestParseConfigOverrides confirms every flag threads through to the Config. |
| @@ -108,6 +109,76 @@ func TestParseConfigBadFlag(t *testing.T) { | |||
| 108 | require.Error(t, err) | 109 | require.Error(t, err) |
| 109 | } | 110 | } |
| 110 | 111 | ||
| 112 | // TestParseHostNetworkFlag confirms --host-network is repeatable and each | ||
| 113 | // name=bridge pair lands in Config.HostNetworks. | ||
| 114 | func TestParseHostNetworkFlag(t *testing.T) { | ||
| 115 | cfg, _, err := parseConfig([]string{"--host-network", "lan=br0", "--host-network", "lab=br1"}) | ||
| 116 | require.NoError(t, err) | ||
| 117 | assert.Equal(t, map[string]string{"lan": "br0", "lab": "br1"}, cfg.HostNetworks) | ||
| 118 | } | ||
| 119 | |||
| 120 | // TestParseHostNetworkFlagRefusals rejects every malformed or ambiguous | ||
| 121 | // --host-network value: missing/empty bridge, the reserved "nat" name, a name | ||
| 122 | // that fails the shared grammar, and a duplicate declaration. | ||
| 123 | func TestParseHostNetworkFlagRefusals(t *testing.T) { | ||
| 124 | for _, args := range [][]string{ | ||
| 125 | {"--host-network", "lan"}, // no bridge | ||
| 126 | {"--host-network", "lan="}, // empty bridge | ||
| 127 | {"--host-network", "nat=br0"}, // reserved name | ||
| 128 | {"--host-network", "LAN=br0"}, // bad grammar | ||
| 129 | {"--host-network", "lan=br0", "--host-network", "lan=br1"}, // duplicate | ||
| 130 | } { | ||
| 131 | _, _, err := parseConfig(args) | ||
| 132 | assert.Error(t, err, "parseConfig(%v) accepted, want error", args) | ||
| 133 | } | ||
| 134 | } | ||
| 135 | |||
| 136 | // TestParseConfigOnRefusesHostNetworkOnUnsupportedGOOS proves the refusal is | ||
| 137 | // actually wired into the parse path, not just unit-tested in isolation: | ||
| 138 | // deleting the validateHostNetworks call in parseConfigOn makes this fail, | ||
| 139 | // where a test of validateHostNetworks alone would not notice. | ||
| 140 | func TestParseConfigOnRefusesHostNetworkOnUnsupportedGOOS(t *testing.T) { | ||
| 141 | _, _, err := parseConfigOn("darwin", []string{"--host-network", "lan=br0"}) | ||
| 142 | require.Error(t, err) | ||
| 143 | assert.Contains(t, err.Error(), "--host-network") | ||
| 144 | assert.Contains(t, err.Error(), "Linux") | ||
| 145 | } | ||
| 146 | |||
| 147 | // TestParseConfigOnAllowsUnsupportedGOOSWithoutTheFlag confirms a Mac agent | ||
| 148 | // with no --host-network at all still parses cleanly — the refusal is for the | ||
| 149 | // flag, not the platform. | ||
| 150 | func TestParseConfigOnAllowsUnsupportedGOOSWithoutTheFlag(t *testing.T) { | ||
| 151 | cfg, _, err := parseConfigOn("darwin", []string{"--max-vcpus=2"}) | ||
| 152 | require.NoError(t, err) | ||
| 153 | assert.Equal(t, int64(2), cfg.MaxVCPUs) | ||
| 154 | } | ||
| 155 | |||
| 156 | // TestValidateHostNetworksRefusesUnsupportedPlatform rejects --host-network on | ||
| 157 | // a platform that cannot serve it (Mac today), naming the remedy rather than | ||
| 158 | // leaving the operator to rediscover it from a later 409. The message says | ||
| 159 | // "Linux" once, not twice — the fact ("Linux hosts only") already carries the | ||
| 160 | // remedy's platform, so the remedy itself need not repeat it. | ||
| 161 | func TestValidateHostNetworksRefusesUnsupportedPlatform(t *testing.T) { | ||
| 162 | err := validateHostNetworks(false, map[string]string{"lan": "br0"}) | ||
| 163 | require.Error(t, err) | ||
| 164 | assert.Contains(t, err.Error(), "--host-network") | ||
| 165 | assert.Equal(t, 1, strings.Count(err.Error(), "Linux"), "message %q should name Linux once, not repeat it", err.Error()) | ||
| 166 | } | ||
| 167 | |||
| 168 | // TestValidateHostNetworksAllowsSupportedPlatform lets --host-network through | ||
| 169 | // on a platform that can serve it (Linux today). | ||
| 170 | func TestValidateHostNetworksAllowsSupportedPlatform(t *testing.T) { | ||
| 171 | err := validateHostNetworks(true, map[string]string{"lan": "br0"}) | ||
| 172 | require.NoError(t, err) | ||
| 173 | } | ||
| 174 | |||
| 175 | // TestValidateHostNetworksVacuousWithoutFlag is fine on any platform: an | ||
| 176 | // operator who never passed --host-network has nothing to be refused for. | ||
| 177 | func TestValidateHostNetworksVacuousWithoutFlag(t *testing.T) { | ||
| 178 | err := validateHostNetworks(false, map[string]string{}) | ||
| 179 | require.NoError(t, err) | ||
| 180 | } | ||
| 181 | |||
| 111 | // TestJoinEmptyBlob rejects a missing blob before any decode or network call; | 182 | // TestJoinEmptyBlob rejects a missing blob before any decode or network call; |
| 112 | // st is never touched, so nil is safe. | 183 | // st is never touched, so nil is safe. |
| 113 | func TestJoinEmptyBlob(t *testing.T) { | 184 | func TestJoinEmptyBlob(t *testing.T) { |
internal/agent/run/exposures_test.go
| Old | New | ||
|---|---|---|---|
| @@ -4,6 +4,7 @@ import ( | |||
| 4 | "testing" | 4 | "testing" |
| 5 | 5 | ||
| 6 | "github.com/a73x/eitri/internal/agent/state" | 6 | "github.com/a73x/eitri/internal/agent/state" |
| 7 | "github.com/stretchr/testify/assert" | ||
| 7 | "github.com/stretchr/testify/require" | 8 | "github.com/stretchr/testify/require" |
| 8 | ) | 9 | ) |
| 9 | 10 | ||
| @@ -30,3 +31,23 @@ func TestGuestAddrAnswersFromTheAgentsOwnRecords(t *testing.T) { | |||
| 30 | t.Errorf("guestAddr(vm2) = %q, want empty", got) | 31 | t.Errorf("guestAddr(vm2) = %q, want empty", got) |
| 31 | } | 32 | } |
| 32 | } | 33 | } |
| 34 | |||
| 35 | // TestGuestAddrIsTheNATAddressForANetworkedGuest pins which of a two-NIC | ||
| 36 | // guest's addresses this host reaches it by. Exposures and the SSH tunnel both | ||
| 37 | // run over the private bridge — that address exists before the guest boots and | ||
| 38 | // needs no site DHCP server to have answered — so a guest on the operator's LAN | ||
| 39 | // is published exactly like its NAT-only siblings. Answering with the LAN | ||
| 40 | // address instead would make a published port depend on someone else's network | ||
| 41 | // being up. | ||
| 42 | func TestGuestAddrIsTheNATAddressForANetworkedGuest(t *testing.T) { | ||
| 43 | st, err := state.Open(t.TempDir()) | ||
| 44 | require.NoError(t, err) | ||
| 45 | |||
| 46 | require.NoError(t, st.SaveVM(state.Record{ | ||
| 47 | Spec: state.VMSpec{VMID: "vm-lan", Network: "lan"}, | ||
| 48 | IP: "10.77.1.5", | ||
| 49 | NetworkIP: "192.168.0.42", | ||
| 50 | })) | ||
| 51 | |||
| 52 | assert.Equal(t, "10.77.1.5", guestAddr(st, "vm-lan")) | ||
| 53 | } | ||
internal/agent/run/reservations.go
| Old | New | ||
|---|---|---|---|
| @@ -10,8 +10,15 @@ import ( | |||
| 10 | // pinning one guest's address. Consumer-owned and minimal (arch R5) so a | 10 | // pinning one guest's address. Consumer-owned and minimal (arch R5) so a |
| 11 | // platform whose networking is nothing like a bridge and a DHCP responder can | 11 | // platform whose networking is nothing like a bridge and a DHCP responder can |
| 12 | // still be handed the surviving guests' addresses. | 12 | // still be handed the surviving guests' addresses. |
| 13 | // | ||
| 14 | // Two verbs because a guest can have two NICs and they rebuild differently: | ||
| 15 | // every guest re-pins the DHCP reservation this host serves its NAT NIC, and a | ||
| 16 | // guest that also has a named-network NIC re-seeds the last address seen there | ||
| 17 | // — never this host's to give — and re-arms the discovery that will confirm or | ||
| 18 | // replace it. A networked guest needs both calls, not one instead of the other. | ||
| 13 | type reservations interface { | 19 | type reservations interface { |
| 14 | AddReservation(vmID, ip string) | 20 | AddReservation(vmID, ip string) |
| 21 | AdoptNetwork(vmID, network, ip string) | ||
| 15 | } | 22 | } |
| 16 | 23 | ||
| 17 | // replayReservations rebuilds net's reservation table from durable records. A | 24 | // replayReservations rebuilds net's reservation table from durable records. A |
| @@ -33,5 +40,11 @@ func replayReservations(st *state.Store, net reservations) { | |||
| 33 | if rec.IP != "" { | 40 | if rec.IP != "" { |
| 34 | net.AddReservation(rec.Spec.VMID, rec.IP) | 41 | net.AddReservation(rec.Spec.VMID, rec.IP) |
| 35 | } | 42 | } |
| 43 | // A networked guest is adopted even with no recorded LAN address: the | ||
| 44 | // attachment itself is what has to be rebuilt, or nothing re-arms the | ||
| 45 | // discovery that reports the address the site's DHCP server granted it. | ||
| 46 | if rec.Spec.Network != "" { | ||
| 47 | net.AdoptNetwork(rec.Spec.VMID, rec.Spec.Network, rec.NetworkIP) | ||
| 48 | } | ||
| 36 | } | 49 | } |
| 37 | } | 50 | } |
internal/agent/run/reservations_test.go
| Old | New | ||
|---|---|---|---|
| @@ -12,16 +12,24 @@ import ( | |||
| 12 | "github.com/stretchr/testify/require" | 12 | "github.com/stretchr/testify/require" |
| 13 | ) | 13 | ) |
| 14 | 14 | ||
| 15 | // fakeReservations records what the replay pinned. The real table lives behind | 15 | // fakeReservations records what the replay pinned and what it adopted. The real |
| 16 | // a DHCP responder on a host bridge; the replay only has to decide what to pin. | 16 | // table lives behind a DHCP responder on a host bridge; the replay only has to |
| 17 | type fakeReservations struct{ pinned map[string]string } | 17 | // decide which of the two a surviving guest needs. |
| 18 | type fakeReservations struct { | ||
| 19 | pinned map[string]string | ||
| 20 | adopted map[string][2]string // vmID -> {network, ip} | ||
| 21 | } | ||
| 18 | 22 | ||
| 19 | func newFakeReservations() *fakeReservations { | 23 | func newFakeReservations() *fakeReservations { |
| 20 | return &fakeReservations{pinned: map[string]string{}} | 24 | return &fakeReservations{pinned: map[string]string{}, adopted: map[string][2]string{}} |
| 21 | } | 25 | } |
| 22 | 26 | ||
| 23 | func (f *fakeReservations) AddReservation(vmID, ip string) { f.pinned[vmID] = ip } | 27 | func (f *fakeReservations) AddReservation(vmID, ip string) { f.pinned[vmID] = ip } |
| 24 | 28 | ||
| 29 | func (f *fakeReservations) AdoptNetwork(vmID, network, ip string) { | ||
| 30 | f.adopted[vmID] = [2]string{network, ip} | ||
| 31 | } | ||
| 32 | |||
| 25 | // storeWithRecords opens a store in a temp dir and persists recs, returning the | 33 | // storeWithRecords opens a store in a temp dir and persists recs, returning the |
| 26 | // store and its directory. | 34 | // store and its directory. |
| 27 | func storeWithRecords(t *testing.T, recs ...state.Record) (*state.Store, string) { | 35 | func storeWithRecords(t *testing.T, recs ...state.Record) (*state.Store, string) { |
| @@ -77,6 +85,52 @@ func TestReplayReservationsSkipsRecordWithoutAddress(t *testing.T) { | |||
| 77 | assert.Equal(t, map[string]string{"vm-addressed": "10.77.1.7"}, table.pinned) | 85 | assert.Equal(t, map[string]string{"vm-addressed": "10.77.1.7"}, table.pinned) |
| 78 | } | 86 | } |
| 79 | 87 | ||
| 88 | // TestReplayReservationsRebuildsBothNICs pins the additive rebuild: a networked | ||
| 89 | // guest has two NICs and needs both halves back. Its NAT reservation is pinned | ||
| 90 | // like every other guest's — that is the address the gate splices to and the | ||
| 91 | // one it must not be renumbered out of — AND its named NIC is adopted with the | ||
| 92 | // last address the site's DHCP server was seen granting it, which this host | ||
| 93 | // pins nowhere because it was never this host's to hand out. | ||
| 94 | func TestReplayReservationsRebuildsBothNICs(t *testing.T) { | ||
| 95 | st, _ := storeWithRecords(t, | ||
| 96 | state.Record{Spec: state.VMSpec{VMID: "vm-nat"}, IP: "10.77.1.5"}, | ||
| 97 | state.Record{ | ||
| 98 | Spec: state.VMSpec{VMID: "vm-lan", Network: "lan"}, | ||
| 99 | IP: "10.77.1.6", | ||
| 100 | NetworkIP: "192.168.0.7", | ||
| 101 | }, | ||
| 102 | // Booted, but its guest has not been heard on the LAN yet: the | ||
| 103 | // attachment still has to be rebuilt, or nothing re-arms the discovery | ||
| 104 | // that will report the address when it comes. | ||
| 105 | state.Record{Spec: state.VMSpec{VMID: "vm-fresh", Network: "lab"}, IP: "10.77.1.7"}, | ||
| 106 | ) | ||
| 107 | |||
| 108 | table := newFakeReservations() | ||
| 109 | replayReservations(st, table) | ||
| 110 | |||
| 111 | assert.Equal(t, map[string]string{ | ||
| 112 | "vm-nat": "10.77.1.5", | ||
| 113 | "vm-lan": "10.77.1.6", | ||
| 114 | "vm-fresh": "10.77.1.7", | ||
| 115 | }, table.pinned, "every guest keeps its NAT reservation, networked or not") | ||
| 116 | assert.Equal(t, map[string][2]string{ | ||
| 117 | "vm-lan": {"lan", "192.168.0.7"}, | ||
| 118 | "vm-fresh": {"lab", ""}, | ||
| 119 | }, table.adopted, "and only a networked guest is adopted, with the LAN address alone") | ||
| 120 | } | ||
| 121 | |||
| 122 | // TestReplayAdoptsNoNetworkForANATOnlyGuest pins the other side: the guest | ||
| 123 | // every host runs today is rebuilt exactly as it was before named networks | ||
| 124 | // existed — pinned, and adopted by nothing. | ||
| 125 | func TestReplayAdoptsNoNetworkForANATOnlyGuest(t *testing.T) { | ||
| 126 | st, _ := storeWithRecords(t, state.Record{Spec: state.VMSpec{VMID: "vm-nat"}, IP: "10.77.1.5"}) | ||
| 127 | |||
| 128 | table := newFakeReservations() | ||
| 129 | replayReservations(st, table) | ||
| 130 | |||
| 131 | assert.Empty(t, table.adopted) | ||
| 132 | } | ||
| 133 | |||
| 80 | // TestReplayReservationsSurvivesStateLoadFailure pins the fail-closed | 134 | // TestReplayReservationsSurvivesStateLoadFailure pins the fail-closed |
| 81 | // behaviour: an unreadable state directory warns and returns, leaving the | 135 | // behaviour: an unreadable state directory warns and returns, leaving the |
| 82 | // table empty rather than aborting the agent's startup. | 136 | // table empty rather than aborting the agent's startup. |
internal/agent/run/wire_darwin.go
| Old | New | ||
|---|---|---|---|
| @@ -64,6 +64,11 @@ type platform struct { | |||
| 64 | // GuestCIDR reports the subnet this host's guests are on, asked once per | 64 | // GuestCIDR reports the subnet this host's guests are on, asked once per |
| 65 | // report rather than resolved at startup. See the sync client's field. | 65 | // report rather than resolved at startup. See the sync client's field. |
| 66 | GuestCIDR func() string | 66 | GuestCIDR func() string |
| 67 | // HostNetworks is always empty here, and the field exists only because | ||
| 68 | // serve() is platform-neutral. Attaching a guest to a named bridge is a | ||
| 69 | // Linux tap on a Linux bridge; vmnet gives a Mac neither, so this host | ||
| 70 | // advertises none and every create asking for one is refused above it. | ||
| 71 | HostNetworks []string | ||
| 67 | } | 72 | } |
| 68 | 73 | ||
| 69 | // newPlatform builds the macOS backend: vfkit over Apple's | 74 | // newPlatform builds the macOS backend: vfkit over Apple's |
internal/agent/run/wire_linux.go
| Old | New | ||
|---|---|---|---|
| @@ -5,6 +5,8 @@ package run | |||
| 5 | import ( | 5 | import ( |
| 6 | "context" | 6 | "context" |
| 7 | "fmt" | 7 | "fmt" |
| 8 | "maps" | ||
| 9 | "slices" | ||
| 8 | 10 | ||
| 9 | "github.com/a73x/eitri/internal/agent/bootstrap" | 11 | "github.com/a73x/eitri/internal/agent/bootstrap" |
| 10 | "github.com/a73x/eitri/internal/agent/cloudhv" | 12 | "github.com/a73x/eitri/internal/agent/cloudhv" |
| @@ -51,6 +53,11 @@ type platform struct { | |||
| 51 | // returns a constant; the poll shape costs it nothing and lets a platform | 53 | // returns a constant; the poll shape costs it nothing and lets a platform |
| 52 | // whose OS owns the network answer late, or not at all. | 54 | // whose OS owns the network answer late, or not at all. |
| 53 | GuestCIDR func() string | 55 | GuestCIDR func() string |
| 56 | // HostNetworks are the named guest networks this host serves, sorted. A | ||
| 57 | // value rather than a poll, unlike GuestCIDR: the set is the agent's | ||
| 58 | // command line, so it changes only with a restart — which is why it rides | ||
| 59 | // in Hello and not in every report. | ||
| 60 | HostNetworks []string | ||
| 54 | } | 61 | } |
| 55 | 62 | ||
| 56 | // proposeGuestCIDR is what this platform offers the fleet at enrollment: the | 63 | // proposeGuestCIDR is what this platform offers the fleet at enrollment: the |
| @@ -93,10 +100,16 @@ func newPlatform(ctx context.Context, cfg Config, st *state.Store) (platform, er | |||
| 93 | bridgeCIDR = lastResortGuestCIDR | 100 | bridgeCIDR = lastResortGuestCIDR |
| 94 | persistGuestCIDR(st, bridgeCIDR) | 101 | persistGuestCIDR(st, bridgeCIDR) |
| 95 | } | 102 | } |
| 96 | net, err := netenv.New(hostRunner, bridgeCIDR) | 103 | net, err := netenv.New(hostRunner, bridgeCIDR, cfg.HostNetworks) |
| 97 | if err != nil { | 104 | if err != nil { |
| 98 | return platform{}, fmt.Errorf("netenv init: %w", err) | 105 | return platform{}, fmt.Errorf("netenv init: %w", err) |
| 99 | } | 106 | } |
| 107 | // Before the NAT bridge and before any VM: a configured network whose | ||
| 108 | // bridge is missing is a host that cannot honor what it is about to | ||
| 109 | // advertise. Refuse to start, in the flag's own vocabulary. | ||
| 110 | if err := net.VerifyNetworks(); err != nil { | ||
| 111 | return platform{}, err | ||
| 112 | } | ||
| 100 | // Before anything is created: a subnet overlapping this host's own uplink | 113 | // Before anything is created: a subnet overlapping this host's own uplink |
| 101 | // would make its address local and stop it answering, and the only machine | 114 | // would make its address local and stop it answering, and the only machine |
| 102 | // positioned to fix that is the one that just lost the route. Refusing here | 115 | // positioned to fix that is the one that just lost the route. Refusing here |
| @@ -131,5 +144,8 @@ func newPlatform(ctx context.Context, cfg Config, st *state.Store) (platform, er | |||
| 131 | prov.Pumps = pumps | 144 | prov.Pumps = pumps |
| 132 | 145 | ||
| 133 | return platform{Prov: prov, Pumps: pumps, | 146 | return platform{Prov: prov, Pumps: pumps, |
| 134 | GuestCIDR: func() string { return bridgeCIDR }}, nil | 147 | GuestCIDR: func() string { return bridgeCIDR }, |
| 148 | // Sorted so every Hello from an unchanged host is byte-identical: map | ||
| 149 | // iteration order would make the advertisement churn for no reason. | ||
| 150 | HostNetworks: slices.Sorted(maps.Keys(cfg.HostNetworks))}, nil | ||
| 135 | } | 151 | } |
internal/agent/run/wire_linux_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,24 @@ | |||
| 1 | //go:build linux | ||
| 2 | |||
| 3 | package run | ||
| 4 | |||
| 5 | import ( | ||
| 6 | "testing" | ||
| 7 | |||
| 8 | "github.com/stretchr/testify/assert" | ||
| 9 | "github.com/stretchr/testify/require" | ||
| 10 | ) | ||
| 11 | |||
| 12 | // TestProposeGuestCIDRIsTheFlagOrNoOpinion pins what a Linux host tells the | ||
| 13 | // fleet at enrollment about the subnet its guests will sit on. Silence is an | ||
| 14 | // answer here — nil takes the control plane's suggestion, which is what a host | ||
| 15 | // started without --bridge-cidr wants — and it is only ever this platform's to | ||
| 16 | // give, because only a host that CREATES the guest network may name it. | ||
| 17 | func TestProposeGuestCIDRIsTheFlagOrNoOpinion(t *testing.T) { | ||
| 18 | assert.Nil(t, proposeGuestCIDR(Config{}), | ||
| 19 | "no flag is no opinion, not a fabricated subnet") | ||
| 20 | |||
| 21 | got := proposeGuestCIDR(Config{BridgeCIDR: "10.30.0.0/24"}) | ||
| 22 | require.NotNil(t, got) | ||
| 23 | assert.Equal(t, "10.30.0.0/24", *got) | ||
| 24 | } | ||
internal/agent/seed/seed.go
| Old | New | ||
|---|---|---|---|
| @@ -43,6 +43,15 @@ type Params struct { | |||
| 43 | // CA key. | 43 | // CA key. |
| 44 | SSHHostKeyPEM string | 44 | SSHHostKeyPEM string |
| 45 | SSHHostCert string | 45 | SSHHostCert string |
| 46 | // MAC and NetworkMAC are the guest's NICs, in the order the hypervisor | ||
| 47 | // attaches them: MAC is the NAT NIC every guest has, NetworkMAC the second | ||
| 48 | // one a guest on a named host network gets. NetworkMAC empty — the usual | ||
| 49 | // case — writes the single-NIC network-config verbatim as it has always | ||
| 50 | // been; non-empty requires MAC, because a two-NIC netplan matches both by | ||
| 51 | // hardware address and a stanza with nothing to match would claim the wrong | ||
| 52 | // link or none. | ||
| 53 | MAC string | ||
| 54 | NetworkMAC string | ||
| 46 | } | 55 | } |
| 47 | 56 | ||
| 48 | // validateParams checks that fields embedded into YAML do not contain newlines | 57 | // validateParams checks that fields embedded into YAML do not contain newlines |
| @@ -56,11 +65,19 @@ func validateParams(p Params) error { | |||
| 56 | {"Hostname", p.Hostname}, | 65 | {"Hostname", p.Hostname}, |
| 57 | {"InstanceID", p.InstanceID}, | 66 | {"InstanceID", p.InstanceID}, |
| 58 | {"SSHAuthorizedKey", p.SSHAuthorizedKey}, | 67 | {"SSHAuthorizedKey", p.SSHAuthorizedKey}, |
| 68 | {"MAC", p.MAC}, | ||
| 69 | {"NetworkMAC", p.NetworkMAC}, | ||
| 59 | } { | 70 | } { |
| 60 | if strings.ContainsAny(f.value, "\n\r") { | 71 | if strings.ContainsAny(f.value, "\n\r") { |
| 61 | return fmt.Errorf("seed: %s must not contain newline or carriage return", f.name) | 72 | return fmt.Errorf("seed: %s must not contain newline or carriage return", f.name) |
| 62 | } | 73 | } |
| 63 | } | 74 | } |
| 75 | // A second NIC with no first one to match is a netplan that claims the | ||
| 76 | // wrong link or no link at all — refuse to build it rather than ship a | ||
| 77 | // guest whose management NIC never comes up. | ||
| 78 | if p.NetworkMAC != "" && p.MAC == "" { | ||
| 79 | return fmt.Errorf("seed: NetworkMAC given without MAC — a two-NIC network-config matches both by hardware address") | ||
| 80 | } | ||
| 64 | return nil | 81 | return nil |
| 65 | } | 82 | } |
| 66 | 83 | ||
| @@ -286,8 +303,17 @@ func metaData(p Params) string { | |||
| 286 | // it under the identifier, so the address becomes unfindable, and worse, an | 303 | // it under the identifier, so the address becomes unfindable, and worse, an |
| 287 | // earlier MAC-keyed offer can linger and be read as current. Linux is unharmed | 304 | // earlier MAC-keyed offer can linger and be read as current. Linux is unharmed |
| 288 | // because our responder keys on the packet's hardware address either way. | 305 | // because our responder keys on the packet's hardware address either way. |
| 289 | func networkConfig() string { | 306 | // |
| 290 | return `network: | 307 | // A guest with a second NIC on a named host network gets a second stanza, and |
| 308 | // then BOTH are matched by MAC rather than by name: two NICs cannot share one | ||
| 309 | // name glob (netplan refuses a link claimed by two definitions), and the MACs | ||
| 310 | // are the only thing the host and the guest already agree on for each NIC | ||
| 311 | // independently. Route metrics decide the default route explicitly — 100 for | ||
| 312 | // the NAT NIC, 200 for the named one — so egress is deterministic and a LAN | ||
| 313 | // outage costs reachability, not the guest's outbound. | ||
| 314 | func networkConfig(p Params) string { | ||
| 315 | if p.NetworkMAC == "" { | ||
| 316 | return `network: | ||
| 291 | version: 2 | 317 | version: 2 |
| 292 | ethernets: | 318 | ethernets: |
| 293 | primary: | 319 | primary: |
| @@ -295,7 +321,28 @@ func networkConfig() string { | |||
| 295 | name: "en*" | 321 | name: "en*" |
| 296 | dhcp4: true | 322 | dhcp4: true |
| 297 | dhcp-identifier: mac | 323 | dhcp-identifier: mac |
| 324 | dhcp4-overrides: | ||
| 325 | route-metric: 100 | ||
| 298 | ` | 326 | ` |
| 327 | } | ||
| 328 | return fmt.Sprintf(`network: | ||
| 329 | version: 2 | ||
| 330 | ethernets: | ||
| 331 | primary: | ||
| 332 | match: | ||
| 333 | macaddress: "%s" | ||
| 334 | dhcp4: true | ||
| 335 | dhcp-identifier: mac | ||
| 336 | dhcp4-overrides: | ||
| 337 | route-metric: 100 | ||
| 338 | net1: | ||
| 339 | match: | ||
| 340 | macaddress: "%s" | ||
| 341 | dhcp4: true | ||
| 342 | dhcp-identifier: mac | ||
| 343 | dhcp4-overrides: | ||
| 344 | route-metric: 200 | ||
| 345 | `, p.MAC, p.NetworkMAC) | ||
| 299 | } | 346 | } |
| 300 | 347 | ||
| 301 | // Build creates a cloud-init NoCloud seed ISO at outPath. | 348 | // Build creates a cloud-init NoCloud seed ISO at outPath. |
| @@ -367,7 +414,7 @@ func Build(outPath string, p Params) error { | |||
| 367 | files := map[string]string{ | 414 | files := map[string]string{ |
| 368 | "/user-data": userData(p), | 415 | "/user-data": userData(p), |
| 369 | "/meta-data": metaData(p), | 416 | "/meta-data": metaData(p), |
| 370 | "/network-config": networkConfig(), | 417 | "/network-config": networkConfig(p), |
| 371 | } | 418 | } |
| 372 | // Vendor-data carries eitri's own config (disk-grow, and the SSH user-CA | 419 | // Vendor-data carries eitri's own config (disk-grow, and the SSH user-CA |
| 373 | // drop-in + host cert when set); it is always non-empty, so always written. | 420 | // drop-in + host cert when set); it is always non-empty, so always written. |
internal/agent/seed/seed_test.go
| Old | New | ||
|---|---|---|---|
| @@ -84,7 +84,7 @@ func TestUserDataCustomPassthrough(t *testing.T) { | |||
| 84 | } | 84 | } |
| 85 | 85 | ||
| 86 | func TestNetworkConfigUsesDHCP(t *testing.T) { | 86 | func TestNetworkConfigUsesDHCP(t *testing.T) { |
| 87 | got := networkConfig() | 87 | got := networkConfig(Params{}) |
| 88 | if !strings.Contains(got, "dhcp4: true") { | 88 | if !strings.Contains(got, "dhcp4: true") { |
| 89 | t.Fatalf("network-config should enable DHCP, got:\n%s", got) | 89 | t.Fatalf("network-config should enable DHCP, got:\n%s", got) |
| 90 | } | 90 | } |
| @@ -93,8 +93,62 @@ func TestNetworkConfigUsesDHCP(t *testing.T) { | |||
| 93 | } | 93 | } |
| 94 | } | 94 | } |
| 95 | 95 | ||
| 96 | // TestNetworkConfigSingleNICIsTheGuestEveryHostRunsToday pins the shape a guest | ||
| 97 | // with no named network gets, whole: one stanza, matched by name, DHCP, and an | ||
| 98 | // explicit default-route metric so the two-NIC guest's 100/200 ordering is a | ||
| 99 | // widening of a stated number rather than a guess about netplan's default. | ||
| 100 | func TestNetworkConfigSingleNICIsTheGuestEveryHostRunsToday(t *testing.T) { | ||
| 101 | assert.Equal(t, `network: | ||
| 102 | version: 2 | ||
| 103 | ethernets: | ||
| 104 | primary: | ||
| 105 | match: | ||
| 106 | name: "en*" | ||
| 107 | dhcp4: true | ||
| 108 | dhcp-identifier: mac | ||
| 109 | dhcp4-overrides: | ||
| 110 | route-metric: 100 | ||
| 111 | `, networkConfig(Params{MAC: "52:54:00:aa:bb:cc"}), | ||
| 112 | "a guest with no named network is matched by name, as it always was") | ||
| 113 | } | ||
| 114 | |||
| 115 | // TestNetworkConfigTwoNICsMatchByMACAndOrderTheDefaultRoute pins the widened | ||
| 116 | // shape: both NICs matched by their own hardware address (a name glob would | ||
| 117 | // claim both links and netplan refuses a link two definitions match), both on | ||
| 118 | // DHCP, and the NAT NIC carrying the default route at metric 100 against the | ||
| 119 | // named NIC's 200 — deterministic egress, with the LAN as fallback. | ||
| 120 | func TestNetworkConfigTwoNICsMatchByMACAndOrderTheDefaultRoute(t *testing.T) { | ||
| 121 | assert.Equal(t, `network: | ||
| 122 | version: 2 | ||
| 123 | ethernets: | ||
| 124 | primary: | ||
| 125 | match: | ||
| 126 | macaddress: "52:54:00:aa:bb:cc" | ||
| 127 | dhcp4: true | ||
| 128 | dhcp-identifier: mac | ||
| 129 | dhcp4-overrides: | ||
| 130 | route-metric: 100 | ||
| 131 | net1: | ||
| 132 | match: | ||
| 133 | macaddress: "52:54:00:dd:ee:ff" | ||
| 134 | dhcp4: true | ||
| 135 | dhcp-identifier: mac | ||
| 136 | dhcp4-overrides: | ||
| 137 | route-metric: 200 | ||
| 138 | `, networkConfig(Params{MAC: "52:54:00:aa:bb:cc", NetworkMAC: "52:54:00:dd:ee:ff"})) | ||
| 139 | } | ||
| 140 | |||
| 141 | // TestBuildRefusesASecondNICWithoutAFirst pins the refusal: a stanza matching | ||
| 142 | // nothing would leave the guest's management NIC unconfigured, which is worse | ||
| 143 | // than no seed at all. | ||
| 144 | func TestBuildRefusesASecondNICWithoutAFirst(t *testing.T) { | ||
| 145 | err := Build(t.TempDir()+"/seed.iso", Params{Hostname: "h", NetworkMAC: "52:54:00:dd:ee:ff"}) | ||
| 146 | require.Error(t, err) | ||
| 147 | assert.Contains(t, err.Error(), "NetworkMAC") | ||
| 148 | } | ||
| 149 | |||
| 96 | func TestNetworkConfigIdentifiesTheGuestByItsMAC(t *testing.T) { | 150 | func TestNetworkConfigIdentifiesTheGuestByItsMAC(t *testing.T) { |
| 97 | got := networkConfig() | 151 | got := networkConfig(Params{}) |
| 98 | 152 | ||
| 99 | // Without this, systemd-networkd sends a DUID-based client identifier and | 153 | // Without this, systemd-networkd sends a DUID-based client identifier and |
| 100 | // the DHCP server files the lease under that instead of the hardware | 154 | // the DHCP server files the lease under that instead of the hardware |
internal/agent/state/state.go
| Old | New | ||
|---|---|---|---|
| @@ -16,7 +16,14 @@ import ( | |||
| 16 | 16 | ||
| 17 | type VMSpec struct { | 17 | type VMSpec struct { |
| 18 | VMID, Name, ImageURL, ImageSHA256, CloudInit, SSHAuthorizedKey string | 18 | VMID, Name, ImageURL, ImageSHA256, CloudInit, SSHAuthorizedKey string |
| 19 | VCPUs, MemMB, DiskGB int64 | 19 | // Network names the host network this guest gets a SECOND NIC on; "" is a |
| 20 | // guest with the NAT NIC alone. Additive, never instead: every guest is on | ||
| 21 | // the host's private bridge whatever this says. Part of the spec, not the | ||
| 22 | // record: it is desired state the control plane sent, and it must survive | ||
| 23 | // an agent restart so the replay can rebuild the VM's in-memory discovery | ||
| 24 | // state. | ||
| 25 | Network string | ||
| 26 | VCPUs, MemMB, DiskGB int64 | ||
| 20 | } | 27 | } |
| 21 | 28 | ||
| 22 | // Disk is one block device attached to a VM, in attachment order. Index 0 is | 29 | // Disk is one block device attached to a VM, in attachment order. Index 0 is |
| @@ -28,8 +35,17 @@ type Disk struct { | |||
| 28 | } | 35 | } |
| 29 | 36 | ||
| 30 | type Record struct { | 37 | type Record struct { |
| 31 | Spec VMSpec | 38 | Spec VMSpec |
| 32 | IP string | 39 | // IP is the guest's address on the host's NAT underlay — allocated by this |
| 40 | // host, known before the guest boots, and what everything that has to reach | ||
| 41 | // the VM uses. | ||
| 42 | IP string | ||
| 43 | // NetworkIP is the address the site's DHCP server granted the guest on its | ||
| 44 | // named-network NIC, as this host snooped it. Empty for a guest with no | ||
| 45 | // such NIC, and until the first ACK crosses its tap. Durable so that an | ||
| 46 | // agent restart can report it again before the guest's next renewal is | ||
| 47 | // seen — the reservation table's twin for an address this host never gave. | ||
| 48 | NetworkIP string | ||
| 33 | BootID string // host boot ID at last start (lost-detection) | 49 | BootID string // host boot ID at last start (lost-detection) |
| 34 | StopRequested bool // set BEFORE stopping: stopped != lost | 50 | StopRequested bool // set BEFORE stopping: stopped != lost |
| 35 | QuarantinedAt *time.Time | 51 | QuarantinedAt *time.Time |
| @@ -106,6 +122,34 @@ func MAC(vmID string) string { | |||
| 106 | return fmt.Sprintf("52:54:00:%02x:%02x:%02x", h[0], h[1], h[2]) | 122 | return fmt.Sprintf("52:54:00:%02x:%02x:%02x", h[0], h[1], h[2]) |
| 107 | } | 123 | } |
| 108 | 124 | ||
| 125 | // NetMAC returns the deterministic MAC of the guest's SECOND NIC — the one | ||
| 126 | // attached to a named host network. Derived, like MAC(vmID), from a namespaced | ||
| 127 | // input ("net1:" + vmID) so a guest's two NICs can never collide and neither | ||
| 128 | // can two guests'. MAC(vmID) is left untouched by this: a VM's first NIC keeps | ||
| 129 | // the identity its DHCP reservation, its lease and its host's records were all | ||
| 130 | // filed under. | ||
| 131 | // | ||
| 132 | // It does not share MAC's OUI, on purpose. That NIC used to land on a private | ||
| 133 | // bridge this host alone spoke for; now it lands on the operator's real LAN, | ||
| 134 | // alongside whatever else is there — most of it, if it is QEMU or libvirt, | ||
| 135 | // answering under 52:54:00 too. The first octet here is 0x56: the | ||
| 136 | // locally-administered bit is set and the multicast bit is clear, so it is a | ||
| 137 | // legal unicast address, and because it differs from 0x52 no NetMAC can ever | ||
| 138 | // land in the same space as MAC's or as a neighbour's conventional guest MAC — | ||
| 139 | // not by any hash outcome, the two spaces are disjoint by construction. The | ||
| 140 | // remaining five octets carry SHA-256 of that input, instead of three, | ||
| 141 | // because a LAN is a bigger room than a private bridge: 40 bits of hash keeps | ||
| 142 | // a fleet's worth of guests comfortably below birthday-collision odds where | ||
| 143 | // 24 got uncomfortable at fleet scale. | ||
| 144 | // | ||
| 145 | // It is what the address snoop keys on, and what the guest's netplan matches | ||
| 146 | // the second stanza by, so both sides of a discovery agree without either | ||
| 147 | // asking the hypervisor what it ended up with. | ||
| 148 | func NetMAC(vmID string) string { | ||
| 149 | h := sha256.Sum256([]byte("net1:" + vmID)) | ||
| 150 | return fmt.Sprintf("56:%02x:%02x:%02x:%02x:%02x", h[0], h[1], h[2], h[3], h[4]) | ||
| 151 | } | ||
| 152 | |||
| 109 | // atomicWrite writes data to path using a tmp file + rename so that readers | 153 | // atomicWrite writes data to path using a tmp file + rename so that readers |
| 110 | // never see a partial write. | 154 | // never see a partial write. |
| 111 | func atomicWrite(path string, data []byte) error { | 155 | func atomicWrite(path string, data []byte) error { |
internal/agent/state/state_test.go
| Old | New | ||
|---|---|---|---|
| @@ -3,6 +3,7 @@ package state | |||
| 3 | import ( | 3 | import ( |
| 4 | "os" | 4 | "os" |
| 5 | "path/filepath" | 5 | "path/filepath" |
| 6 | "strconv" | ||
| 6 | "testing" | 7 | "testing" |
| 7 | "time" | 8 | "time" |
| 8 | 9 | ||
| @@ -92,6 +93,65 @@ func TestMACIsDeterministicAndLocallyAdministered(t *testing.T) { | |||
| 92 | } | 93 | } |
| 93 | } | 94 | } |
| 94 | 95 | ||
| 96 | // TestNetMACGoldenValue pins the exact byte layout for one known vmID: octet | ||
| 97 | // 0 is the fixed 0x56, the remaining five are SHA-256("net1:"+vmID)[:5]. A | ||
| 98 | // full revert to the old 3-byte QEMU-OUI form is already caught below by the | ||
| 99 | // ≠0x52 assertion; what only this golden catches is a narrowed hash width — | ||
| 100 | // 0x56 followed by just three hash bytes would pass every property assertion | ||
| 101 | // in this file and fail only here. | ||
| 102 | func TestNetMACGoldenValue(t *testing.T) { | ||
| 103 | assert.Equal(t, "56:78:a6:1f:4f:7e", NetMAC("vm-abc123")) | ||
| 104 | } | ||
| 105 | |||
| 106 | // TestNetMACIsTheOtherNICOfTheSameVM pins the second NIC's identity: same | ||
| 107 | // shape, same determinism, and never the first NIC's address — one guest with | ||
| 108 | // two NICs sharing a MAC is a guest whose bridge learns the wrong port, and a | ||
| 109 | // snoop keyed on it would report the NAT lease as the LAN one. It also pins | ||
| 110 | // the OUI split from MAC(): NetMAC lands on the operator's real LAN now, so | ||
| 111 | // its first octet must carry the locally-administered bit, leave the | ||
| 112 | // multicast bit clear, and differ from 0x52 — disjoint from both MAC's space | ||
| 113 | // and every conventional 52:54:00 QEMU/libvirt guest already on that LAN. | ||
| 114 | func TestNetMACIsTheOtherNICOfTheSameVM(t *testing.T) { | ||
| 115 | got := NetMAC("vm-abc123") | ||
| 116 | assert.Equal(t, got, NetMAC("vm-abc123"), "NetMAC must be deterministic") | ||
| 117 | assert.Len(t, got, 17) | ||
| 118 | |||
| 119 | first, err := strconv.ParseUint(got[:2], 16, 8) | ||
| 120 | require.NoError(t, err) | ||
| 121 | assert.NotZero(t, first&0x02, "locally-administered bit must be set") | ||
| 122 | assert.Zero(t, first&0x01, "multicast bit must be clear (valid unicast LAA)") | ||
| 123 | assert.NotEqual(t, byte(0x52), byte(first), "must not share MAC's OUI octet") | ||
| 124 | |||
| 125 | assert.NotEqual(t, MAC("vm-abc123"), got, "a VM's two NICs must never share a MAC") | ||
| 126 | assert.NotEqual(t, NetMAC("vm-xyz789"), got, "distinct VMs must not collide either") | ||
| 127 | // Not just for one id: every VM's two NICs must differ, or a guest whose id | ||
| 128 | // happened to hash into a collision would black-hole its own traffic. | ||
| 129 | for _, id := range []string{"a", "vm-1", "0ba1a1b6-6a71-4f6f-9e5f-2c0e3a3b1f42", ""} { | ||
| 130 | assert.NotEqual(t, MAC(id), NetMAC(id), "vmID %q", id) | ||
| 131 | } | ||
| 132 | } | ||
| 133 | |||
| 134 | // TestRecordCarriesBothAddresses pins that a networked guest's two addresses | ||
| 135 | // both survive an agent restart: the NAT one to re-pin its reservation, the | ||
| 136 | // named-network one to report while the next DHCP renewal is waited for. | ||
| 137 | func TestRecordCarriesBothAddresses(t *testing.T) { | ||
| 138 | dir := t.TempDir() | ||
| 139 | s, _ := Open(dir) | ||
| 140 | require.NoError(t, s.SaveVM(Record{ | ||
| 141 | Spec: VMSpec{VMID: "vm1", Network: "lan"}, | ||
| 142 | IP: "10.77.1.2", | ||
| 143 | NetworkIP: "192.168.0.42", | ||
| 144 | })) | ||
| 145 | |||
| 146 | s2, _ := Open(dir) | ||
| 147 | got, err := s2.LoadVMs() | ||
| 148 | require.NoError(t, err) | ||
| 149 | require.Contains(t, got, "vm1") | ||
| 150 | assert.Equal(t, "10.77.1.2", got["vm1"].IP) | ||
| 151 | assert.Equal(t, "192.168.0.42", got["vm1"].NetworkIP) | ||
| 152 | assert.Equal(t, "lan", got["vm1"].Spec.Network) | ||
| 153 | } | ||
| 154 | |||
| 95 | func TestSerialSocketPath(t *testing.T) { | 155 | func TestSerialSocketPath(t *testing.T) { |
| 96 | s := open(t) | 156 | s := open(t) |
| 97 | p := s.SerialSocketPath("vm1") | 157 | p := s.SerialSocketPath("vm1") |
internal/agent/syncclient/client.go
| Old | New | ||
|---|---|---|---|
| @@ -76,6 +76,12 @@ type Client struct { | |||
| 76 | // report. Nil, or an empty return, means "no answer" — never "no network". | 76 | // report. Nil, or an empty return, means "no answer" — never "no network". |
| 77 | GuestCIDR func() string | 77 | GuestCIDR func() string |
| 78 | 78 | ||
| 79 | // HostNetworks are the named guest networks this host is configured to | ||
| 80 | // serve (--host-network), advertised in Hello rather than in the report: | ||
| 81 | // the set changes only with the agent's command line, i.e. a restart. A | ||
| 82 | // host that advertises none takes no VM that asks for a network by name. | ||
| 83 | HostNetworks []string | ||
| 84 | |||
| 79 | // Runner executes host-introspection subprocesses (on Linux, | 85 | // Runner executes host-introspection subprocesses (on Linux, |
| 80 | // systemd-detect-virt via hostinfo). Injected so this data-plane package | 86 | // systemd-detect-virt via hostinfo). Injected so this data-plane package |
| 81 | // never imports os/exec (R6); nil is tolerated (virt reported as unknown). | 87 | // never imports os/exec (R6); nil is tolerated (virt reported as unknown). |
| @@ -336,8 +342,9 @@ func (c *Client) session(ctx context.Context) error { | |||
| 336 | HostId: c.Identity.HostID, Hostname: hostname, Os: runtime.GOOS, Arch: runtime.GOARCH, | 342 | HostId: c.Identity.HostID, Hostname: hostname, Os: runtime.GOOS, Arch: runtime.GOARCH, |
| 337 | Provisioner: c.Provisioner, | 343 | Provisioner: c.Provisioner, |
| 338 | LastSeenEpoch: c.St.Epoch(), Capacity: c.advertisedCapacity(stateDir), | 344 | LastSeenEpoch: c.St.Epoch(), Capacity: c.advertisedCapacity(stateDir), |
| 339 | Facts: helloFacts(ctx, c.Runner), | 345 | Facts: helloFacts(ctx, c.Runner), |
| 340 | Credential: c.Identity.Credential, | 346 | HostNetworks: c.HostNetworks, |
| 347 | Credential: c.Identity.Credential, | ||
| 341 | }}} | 348 | }}} |
| 342 | if err := transport.WriteMsg(up, hello); err != nil { | 349 | if err := transport.WriteMsg(up, hello); err != nil { |
| 343 | return classifyErr(err) | 350 | return classifyErr(err) |
internal/agent/syncclient/client_test.go
| Old | New | ||
|---|---|---|---|
| @@ -37,6 +37,7 @@ func (noopProv) Shutdown(context.Context, string) error { r | |||
| 37 | func (noopProv) Destroy(context.Context, string) error { return nil } | 37 | func (noopProv) Destroy(context.Context, string) error { return nil } |
| 38 | func (noopProv) Running(string) bool { return false } | 38 | func (noopProv) Running(string) bool { return false } |
| 39 | func (noopProv) Address(string) string { return "10.77.1.2" } | 39 | func (noopProv) Address(string) string { return "10.77.1.2" } |
| 40 | func (noopProv) NetworkAddress(string) string { return "" } | ||
| 40 | func (noopProv) FailureReason(string) string { return "" } | 41 | func (noopProv) FailureReason(string) string { return "" } |
| 41 | 42 | ||
| 42 | // testQUICIdle is the deliberately-short idle timeout the test listeners use so | 43 | // testQUICIdle is the deliberately-short idle timeout the test listeners use so |
| @@ -231,6 +232,51 @@ func newClient(t *testing.T, addr, fp, hostID, cred string) *Client { | |||
| 231 | } | 232 | } |
| 232 | } | 233 | } |
| 233 | 234 | ||
| 235 | // TestHelloAdvertisesHostNetworks pins the advertisement: the named networks | ||
| 236 | // this agent is configured to serve ride in the Hello, which is where the | ||
| 237 | // control plane's admission reads them. In the handshake and not in the report | ||
| 238 | // because the set is this agent's command line — it changes only with a | ||
| 239 | // restart, and a restart is a new session. | ||
| 240 | func TestHelloAdvertisesHostNetworks(t *testing.T) { | ||
| 241 | certPEM, keyPEM, fp := genTestCert(t) | ||
| 242 | lis := listenTestQUIC(t, "127.0.0.1:0", certPEM, keyPEM) | ||
| 243 | defer lis.Close() | ||
| 244 | |||
| 245 | hellos := make(chan *pb.Hello, 1) | ||
| 246 | acceptCtx, stopAccept := context.WithCancel(context.Background()) | ||
| 247 | defer stopAccept() | ||
| 248 | go func() { | ||
| 249 | conn, err := lis.Accept(acceptCtx) | ||
| 250 | if err != nil { | ||
| 251 | return | ||
| 252 | } | ||
| 253 | up, err := conn.AcceptStream(acceptCtx) | ||
| 254 | if err != nil { | ||
| 255 | return | ||
| 256 | } | ||
| 257 | var first pb.AgentMessage | ||
| 258 | if err := transport.ReadMsg(up, &first, transport.DefaultMaxFrame); err != nil { | ||
| 259 | return | ||
| 260 | } | ||
| 261 | hellos <- first.GetHello() | ||
| 262 | _ = conn.CloseWithError(0, "captured") | ||
| 263 | }() | ||
| 264 | |||
| 265 | c := newClient(t, lis.Addr().String(), fp, "host-nets", "host-nets.cred") | ||
| 266 | c.HostNetworks = []string{"lab", "lan"} | ||
| 267 | runCtx, stopRun := context.WithCancel(context.Background()) | ||
| 268 | done := make(chan struct{}) | ||
| 269 | go func() { defer close(done); c.Run(runCtx) }() | ||
| 270 | defer func() { stopRun(); <-done }() | ||
| 271 | |||
| 272 | select { | ||
| 273 | case h := <-hellos: | ||
| 274 | assert.Equal(t, []string{"lab", "lan"}, h.GetHostNetworks()) | ||
| 275 | case <-time.After(10 * time.Second): | ||
| 276 | t.Fatal("no Hello arrived") | ||
| 277 | } | ||
| 278 | } | ||
| 279 | |||
| 234 | func TestReconnectAfterDrop(t *testing.T) { | 280 | func TestReconnectAfterDrop(t *testing.T) { |
| 235 | h := newServerHarness(t) | 281 | h := newServerHarness(t) |
| 236 | hostID, cred := h.enroll() | 282 | hostID, cred := h.enroll() |
internal/agent/vfkit/leases.go
| Old | New | ||
|---|---|---|---|
| @@ -32,6 +32,14 @@ func (p *Provisioner) Address(vmID string) string { | |||
| 32 | return leaseAddress(string(raw), state.MAC(vmID)) | 32 | return leaseAddress(string(raw), state.MAC(vmID)) |
| 33 | } | 33 | } |
| 34 | 34 | ||
| 35 | // NetworkAddress is always "" on this backend: a Mac host advertises no named | ||
| 36 | // guest networks (bridged attachment needs the restricted | ||
| 37 | // com.apple.vm.networking entitlement), so no VM placed here can have a second | ||
| 38 | // NIC to report an address for. Answering "" is not a stub — it is the whole | ||
| 39 | // truth for this platform, and the create-time refusal upstream is what keeps | ||
| 40 | // it from ever being asked a different question. | ||
| 41 | func (p *Provisioner) NetworkAddress(string) string { return "" } | ||
| 42 | |||
| 35 | // leaseAddress finds the address most recently leased to mac in the contents | 43 | // leaseAddress finds the address most recently leased to mac in the contents |
| 36 | // of macOS's dhcpd_leases file, or "" if there is none. The file is a series | 44 | // of macOS's dhcpd_leases file, or "" if there is none. The file is a series |
| 37 | // of brace-delimited stanzas of key=value lines: | 45 | // of brace-delimited stanzas of key=value lines: |
internal/mcpserver/api_test.go
| Old | New | ||
|---|---|---|---|
| @@ -75,7 +75,7 @@ func TestFirstEligibleHost(t *testing.T) { | |||
| 75 | {"id": "h1", "name": "mewtwo", "online": true, "agent_version": "v0.0.5"}, | 75 | {"id": "h1", "name": "mewtwo", "online": true, "agent_version": "v0.0.5"}, |
| 76 | }) | 76 | }) |
| 77 | }) | 77 | }) |
| 78 | h, err := c.FirstEligibleHost(t.Context()) | 78 | h, err := c.FirstEligibleHost(t.Context(), "") |
| 79 | require.NoError(t, err) | 79 | require.NoError(t, err) |
| 80 | assert.Equal(t, "h1", h.ID) | 80 | assert.Equal(t, "h1", h.ID) |
| 81 | } | 81 | } |
| @@ -92,7 +92,7 @@ func TestFirstEligibleHostSkipsPreCSRAgents(t *testing.T) { | |||
| 92 | {"id": "h3", "name": "mewtwo", "online": true, "agent_version": "v0.0.4"}, | 92 | {"id": "h3", "name": "mewtwo", "online": true, "agent_version": "v0.0.4"}, |
| 93 | }) | 93 | }) |
| 94 | }) | 94 | }) |
| 95 | h, err := c.FirstEligibleHost(t.Context()) | 95 | h, err := c.FirstEligibleHost(t.Context(), "") |
| 96 | require.NoError(t, err) | 96 | require.NoError(t, err) |
| 97 | assert.Equal(t, "h3", h.ID) | 97 | assert.Equal(t, "h3", h.ID) |
| 98 | } | 98 | } |
| @@ -106,7 +106,7 @@ func TestFirstEligibleHostReportsWhyNoneAreEligible(t *testing.T) { | |||
| 106 | {"id": "h1", "name": "older", "online": true, "agent_version": "v0.0.1"}, | 106 | {"id": "h1", "name": "older", "online": true, "agent_version": "v0.0.1"}, |
| 107 | }) | 107 | }) |
| 108 | }) | 108 | }) |
| 109 | _, err := c.FirstEligibleHost(t.Context()) | 109 | _, err := c.FirstEligibleHost(t.Context(), "") |
| 110 | require.Error(t, err) | 110 | require.Error(t, err) |
| 111 | assert.Contains(t, err.Error(), "2 online host(s)") | 111 | assert.Contains(t, err.Error(), "2 online host(s)") |
| 112 | assert.Contains(t, err.Error(), "predate certified host keys (v0.0.4)") | 112 | assert.Contains(t, err.Error(), "predate certified host keys (v0.0.4)") |
| @@ -114,6 +114,60 @@ func TestFirstEligibleHostReportsWhyNoneAreEligible(t *testing.T) { | |||
| 114 | assert.NotContains(t, err.Error(), "no online hosts") | 114 | assert.NotContains(t, err.Error(), "no online hosts") |
| 115 | } | 115 | } |
| 116 | 116 | ||
| 117 | // Asking for a named network narrows placement to hosts that advertise it: the | ||
| 118 | // first eligible host is only the default when nothing further was asked for. | ||
| 119 | func TestFirstEligibleHostPrefersOneAdvertisingTheNetwork(t *testing.T) { | ||
| 120 | c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) { | ||
| 121 | json.NewEncoder(w).Encode([]map[string]any{ | ||
| 122 | {"id": "h1", "name": "onyx", "online": true, "agent_version": "v0.0.6"}, | ||
| 123 | {"id": "h2", "name": "mewtwo", "online": true, "agent_version": "v0.0.6", "host_networks": []string{"lan"}}, | ||
| 124 | }) | ||
| 125 | }) | ||
| 126 | h, err := c.FirstEligibleHost(t.Context(), "lan") | ||
| 127 | require.NoError(t, err) | ||
| 128 | assert.Equal(t, "h2", h.ID) | ||
| 129 | |||
| 130 | // Without the ask, the first eligible host still wins. | ||
| 131 | h, err = c.FirstEligibleHost(t.Context(), "") | ||
| 132 | require.NoError(t, err) | ||
| 133 | assert.Equal(t, "h1", h.ID) | ||
| 134 | } | ||
| 135 | |||
| 136 | // The refusal has to teach: an agent that cannot see the fleet learns the | ||
| 137 | // network names from here, and the operator learns the flag that adds one. | ||
| 138 | func TestFirstEligibleHostForAnUnservedNetworkNamesTheServedOnes(t *testing.T) { | ||
| 139 | c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) { | ||
| 140 | json.NewEncoder(w).Encode([]map[string]any{ | ||
| 141 | {"id": "h1", "name": "onyx", "online": true, "agent_version": "v0.0.6", "host_networks": []string{"lan"}}, | ||
| 142 | {"id": "h2", "name": "gone", "online": false, "agent_version": "v0.0.6", "host_networks": []string{"storage"}}, | ||
| 143 | {"id": "h3", "name": "ancient", "online": true, "agent_version": "v0.0.3", "host_networks": []string{"storage"}}, | ||
| 144 | }) | ||
| 145 | }) | ||
| 146 | _, err := c.FirstEligibleHost(t.Context(), "storage") | ||
| 147 | require.Error(t, err) | ||
| 148 | assert.Contains(t, err.Error(), `no eligible host advertises network "storage"`) | ||
| 149 | assert.Contains(t, err.Error(), "onyx: lan") | ||
| 150 | assert.NotContains(t, err.Error(), "gone", "an offline host's advertised networks are stale registry state, not an offer") | ||
| 151 | assert.NotContains(t, err.Error(), "ancient", | ||
| 152 | "a host that would refuse the create over its agent version is not an offer either — naming it sends the caller to configure a bridge when the remedy is an upgrade") | ||
| 153 | assert.Contains(t, err.Error(), "--host-network storage=<bridge>") | ||
| 154 | } | ||
| 155 | |||
| 156 | // An agent too old to certify host keys is ruled out before its networks are | ||
| 157 | // ever consulted, and the refusal is the upgrade one: the remedy is the agent, | ||
| 158 | // not the bridge. | ||
| 159 | func TestFirstEligibleHostReportsThePreCSRAgentEvenWhenANetworkWasAsked(t *testing.T) { | ||
| 160 | c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) { | ||
| 161 | json.NewEncoder(w).Encode([]map[string]any{ | ||
| 162 | {"id": "h1", "name": "old", "online": true, "agent_version": "v0.0.3", "host_networks": []string{"lan"}}, | ||
| 163 | }) | ||
| 164 | }) | ||
| 165 | _, err := c.FirstEligibleHost(t.Context(), "lan") | ||
| 166 | require.Error(t, err) | ||
| 167 | assert.Contains(t, err.Error(), "predate certified host keys") | ||
| 168 | assert.NotContains(t, err.Error(), "advertises network") | ||
| 169 | } | ||
| 170 | |||
| 117 | // A fleet with nothing up still says exactly that. | 171 | // A fleet with nothing up still says exactly that. |
| 118 | func TestFirstEligibleHostWithNothingOnline(t *testing.T) { | 172 | func TestFirstEligibleHostWithNothingOnline(t *testing.T) { |
| 119 | c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) { | 173 | c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) { |
| @@ -121,7 +175,7 @@ func TestFirstEligibleHostWithNothingOnline(t *testing.T) { | |||
| 121 | {"id": "h0", "name": "down", "online": false, "agent_version": "v0.0.5"}, | 175 | {"id": "h0", "name": "down", "online": false, "agent_version": "v0.0.5"}, |
| 122 | }) | 176 | }) |
| 123 | }) | 177 | }) |
| 124 | _, err := c.FirstEligibleHost(t.Context()) | 178 | _, err := c.FirstEligibleHost(t.Context(), "") |
| 125 | require.ErrorContains(t, err, "no online hosts") | 179 | require.ErrorContains(t, err, "no online hosts") |
| 126 | } | 180 | } |
| 127 | 181 | ||
internal/mcpserver/server.go
| Old | New | ||
|---|---|---|---|
| @@ -58,7 +58,13 @@ type DelegateCompleteIn struct { | |||
| 58 | func NewServer(t *Tools, opts Options) *mcp.Server { | 58 | func NewServer(t *Tools, opts Options) *mcp.Server { |
| 59 | s := mcp.NewServer(&mcp.Implementation{Name: "eitri", Version: "0.1.0"}, | 59 | s := mcp.NewServer(&mcp.Implementation{Name: "eitri", Version: "0.1.0"}, |
| 60 | &mcp.ServerOptions{SchemaCache: opts.SchemaCache}) | 60 | &mcp.ServerOptions{SchemaCache: opts.SchemaCache}) |
| 61 | register(s, "vm_create", "Create an eitri VM. Waits for ready+cloud-init by default.", t.VMCreate) | 61 | register(s, "vm_create", |
| 62 | "Create an eitri VM. Waits for ready+cloud-init by default. Name a network to also put the guest on one "+ | ||
| 63 | "of its host's named networks, where that network's own DHCP addresses it: the result reports that "+ | ||
| 64 | "address as network_ip, which may still be EMPTY when the tool returns — readiness is the guest booting, "+ | ||
| 65 | "and a lease from the site's DHCP server arrives when it arrives. Read it back with vm_info rather "+ | ||
| 66 | "than treating an empty one as a failed create.", | ||
| 67 | t.VMCreate) | ||
| 62 | register(s, "vm_list", "List all VMs on the eitri fleet.", t.VMList) | 68 | register(s, "vm_list", "List all VMs on the eitri fleet.", t.VMList) |
| 63 | register(s, "vm_info", "Show one VM's state and how to reach it.", t.VMInfo) | 69 | register(s, "vm_info", "Show one VM's state and how to reach it.", t.VMInfo) |
| 64 | register(s, "vm_exec", "Run a shell command in a VM over SSH; returns stdout/stderr/exit code.", t.VMExec) | 70 | register(s, "vm_exec", "Run a shell command in a VM over SSH; returns stdout/stderr/exit code.", t.VMExec) |
internal/mcpserver/server_test.go
| Old | New | ||
|---|---|---|---|
| @@ -2,6 +2,7 @@ package mcpserver | |||
| 2 | 2 | ||
| 3 | import ( | 3 | import ( |
| 4 | "context" | 4 | "context" |
| 5 | "encoding/json" | ||
| 5 | "errors" | 6 | "errors" |
| 6 | "testing" | 7 | "testing" |
| 7 | 8 | ||
| @@ -190,3 +191,35 @@ func TestCAUploadDescriptionNamesTheFrozenCASet(t *testing.T) { | |||
| 190 | assert.Contains(t, desc, "create new VMs after uploading") | 191 | assert.Contains(t, desc, "create new VMs after uploading") |
| 191 | assert.Contains(t, desc, "before vm_create") | 192 | assert.Contains(t, desc, "before vm_create") |
| 192 | } | 193 | } |
| 194 | |||
| 195 | // A guest on a named host network is addressed by a DHCP server eitri does not | ||
| 196 | // run, so vm_create can return ready with network_ip still empty. A model that | ||
| 197 | // has not been told reads that as a broken create and destroys a working VM, so | ||
| 198 | // the tool's own description says it, and says where to look instead. | ||
| 199 | func TestVMCreateDescriptionSaysTheNetworkAddressCanLagReady(t *testing.T) { | ||
| 200 | desc := describe(t, connect(t, NewServer(&Tools{}, Options{})), "vm_create") | ||
| 201 | require.NotEmpty(t, desc) | ||
| 202 | assert.Contains(t, desc, "network_ip") | ||
| 203 | assert.Contains(t, desc, "EMPTY") | ||
| 204 | assert.Contains(t, desc, "vm_info") | ||
| 205 | } | ||
| 206 | |||
| 207 | // The network field's own description has to carry the rule the control plane | ||
| 208 | // enforces: only names the host advertises, and no silent fallback to NAT for | ||
| 209 | // one it does not. | ||
| 210 | func TestVMCreateNetworkFieldDescribesWhatAHostWillAccept(t *testing.T) { | ||
| 211 | res, err := connect(t, NewServer(&Tools{}, Options{})).ListTools(t.Context(), nil) | ||
| 212 | require.NoError(t, err) | ||
| 213 | var schema string | ||
| 214 | for _, tool := range res.Tools { | ||
| 215 | if tool.Name == "vm_create" { | ||
| 216 | props, err := json.Marshal(tool.InputSchema) | ||
| 217 | require.NoError(t, err) | ||
| 218 | schema = string(props) | ||
| 219 | } | ||
| 220 | } | ||
| 221 | require.NotEmpty(t, schema) | ||
| 222 | assert.Contains(t, schema, "network") | ||
| 223 | assert.Contains(t, schema, "advertises") | ||
| 224 | assert.Contains(t, schema, "NAT underlay") | ||
| 225 | } | ||
internal/mcpserver/tools.go
| Old | New | ||
|---|---|---|---|
| @@ -6,6 +6,7 @@ import ( | |||
| 6 | "fmt" | 6 | "fmt" |
| 7 | "io/fs" | 7 | "io/fs" |
| 8 | "net" | 8 | "net" |
| 9 | "slices" | ||
| 9 | "strconv" | 10 | "strconv" |
| 10 | "strings" | 11 | "strings" |
| 11 | "time" | 12 | "time" |
| @@ -23,7 +24,7 @@ type api interface { | |||
| 23 | CreateVM(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) | 24 | CreateVM(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) |
| 24 | DeleteVM(ctx context.Context, id string) error | 25 | DeleteVM(ctx context.Context, id string) error |
| 25 | ListHosts(ctx context.Context) ([]client.Host, error) | 26 | ListHosts(ctx context.Context) ([]client.Host, error) |
| 26 | FirstEligibleHost(ctx context.Context) (client.Host, error) | 27 | FirstEligibleHost(ctx context.Context, network string) (client.Host, error) |
| 27 | CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) | 28 | CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) |
| 28 | ListExposures(ctx context.Context, vmID string) ([]client.Exposure, error) | 29 | ListExposures(ctx context.Context, vmID string) ([]client.Exposure, error) |
| 29 | DeleteExposure(ctx context.Context, id string) error | 30 | DeleteExposure(ctx context.Context, id string) error |
| @@ -64,31 +65,67 @@ func (a API) RegisterUserCA(ctx context.Context, caLine, label string) error { | |||
| 64 | } | 65 | } |
| 65 | 66 | ||
| 66 | // FirstEligibleHost returns the first host a VM can be placed on — the default | 67 | // FirstEligibleHost returns the first host a VM can be placed on — the default |
| 67 | // target when the caller doesn't name one. Eligible means online AND running an | 68 | // target when the caller doesn't name one. network, when non-empty, is the |
| 68 | // agent that certifies its guests' host keys: a VM created anywhere else is | 69 | // named host network the VM asked for, and narrows the field to hosts serving |
| 69 | // unreachable through the gate, and the control plane refuses the create, so | 70 | // it. |
| 70 | // such a host is no more a candidate than one that is down. Ordering is | 71 | func (a API) FirstEligibleHost(ctx context.Context, network string) (client.Host, error) { |
| 71 | // server-defined; callers must not assume stability across calls. | ||
| 72 | // | ||
| 73 | // When every online host is ruled out that way, the refusal says so — "no | ||
| 74 | // online hosts" would send the caller looking for hardware that is in fact up | ||
| 75 | // and one upgrade away from usable. | ||
| 76 | func (a API) FirstEligibleHost(ctx context.Context) (client.Host, error) { | ||
| 77 | hosts, err := a.ListHosts(ctx) | 72 | hosts, err := a.ListHosts(ctx) |
| 78 | if err != nil { | 73 | if err != nil { |
| 79 | return client.Host{}, err | 74 | return client.Host{}, err |
| 80 | } | 75 | } |
| 81 | precsr := 0 | 76 | return eligibleHost(hosts, network) |
| 77 | } | ||
| 78 | |||
| 79 | // placeable reports whether a VM can be put on this host at all: it is online | ||
| 80 | // AND runs an agent that certifies its guests' host keys. A VM created anywhere | ||
| 81 | // else is unreachable through the gate, and the control plane refuses the | ||
| 82 | // create, so such a host is no more a candidate than one that is down. | ||
| 83 | // | ||
| 84 | // It is one predicate because two callers must agree: the rule that picks a | ||
| 85 | // host and the message that lists the alternatives. A refusal naming a host | ||
| 86 | // placement would not have chosen sends the caller after the wrong remedy. | ||
| 87 | func placeable(h client.Host) bool { | ||
| 88 | return h.Online && release.CertifiesGuestHostKeys(h.AgentVersion) | ||
| 89 | } | ||
| 90 | |||
| 91 | // eligibleHost is the placement rule itself, over a fleet already read. It is a | ||
| 92 | // function of the listing alone so the tools' fake obeys the same rule the real | ||
| 93 | // client does, rather than a second, kinder one. | ||
| 94 | // | ||
| 95 | // A named network narrows the placeable hosts further — one that does not | ||
| 96 | // advertise it would refuse the create too. Ordering is server-defined; callers | ||
| 97 | // must not assume stability across calls. | ||
| 98 | // | ||
| 99 | // Each way of ruling every host out has its own refusal, because each has its | ||
| 100 | // own remedy. "no online hosts" would send a caller whose fleet is up and one | ||
| 101 | // upgrade away from usable looking for dead hardware; and it would tell a | ||
| 102 | // caller who asked for a network nothing about which hosts carry one. | ||
| 103 | func eligibleHost(hosts []client.Host, network string) (client.Host, error) { | ||
| 104 | precsr, unnetworked := 0, 0 | ||
| 82 | for _, h := range hosts { | 105 | for _, h := range hosts { |
| 83 | if !h.Online { | 106 | if !placeable(h) { |
| 107 | // Of the hosts that cannot take a VM, the ones that are UP are the | ||
| 108 | // ones an upgrade would fix; a host that is down needs nothing said | ||
| 109 | // about its agent. | ||
| 110 | if h.Online { | ||
| 111 | precsr++ | ||
| 112 | } | ||
| 84 | continue | 113 | continue |
| 85 | } | 114 | } |
| 86 | if !release.CertifiesGuestHostKeys(h.AgentVersion) { | 115 | if network != "" && !slices.Contains(h.HostNetworks, network) { |
| 87 | precsr++ | 116 | unnetworked++ |
| 88 | continue | 117 | continue |
| 89 | } | 118 | } |
| 90 | return h, nil | 119 | return h, nil |
| 91 | } | 120 | } |
| 121 | // The network refusal comes first when both apply: it answers the question | ||
| 122 | // the caller actually asked, and it names the fleet's networks, which is the | ||
| 123 | // one place an MCP caller can learn them without provoking a 409. | ||
| 124 | if unnetworked > 0 { | ||
| 125 | return client.Host{}, fmt.Errorf("no eligible host advertises network %q (online hosts advertise: %s); "+ | ||
| 126 | "name a host that does, restart an agent with --host-network %s=<bridge>, or omit network for the NAT underlay", | ||
| 127 | network, describeHostNetworks(hosts), network) | ||
| 128 | } | ||
| 92 | if precsr > 0 { | 129 | if precsr > 0 { |
| 93 | return client.Host{}, fmt.Errorf("no eligible hosts: %d online host(s) run agents that predate certified host keys (%s), "+ | 130 | return client.Host{}, fmt.Errorf("no eligible hosts: %d online host(s) run agents that predate certified host keys (%s), "+ |
| 94 | "so a VM created on them could not be verified; upgrade an agent (the console's upgrade button, or "+ | 131 | "so a VM created on them could not be verified; upgrade an agent (the console's upgrade button, or "+ |
| @@ -137,13 +174,25 @@ type VMCreateIn struct { | |||
| 137 | MemMB int64 `json:"mem_mb,omitempty" jsonschema:"default 2048"` | 174 | MemMB int64 `json:"mem_mb,omitempty" jsonschema:"default 2048"` |
| 138 | DiskGB int64 `json:"disk_gb,omitempty" jsonschema:"default 10"` | 175 | DiskGB int64 `json:"disk_gb,omitempty" jsonschema:"default 10"` |
| 139 | CloudInit string `json:"cloud_init,omitempty" jsonschema:"optional user cloud-init"` | 176 | CloudInit string `json:"cloud_init,omitempty" jsonschema:"optional user cloud-init"` |
| 177 | Network string `json:"network,omitempty" jsonschema:"named host network to give the guest a SECOND NIC on, addressed by that network's own DHCP; omit for the private NAT underlay every guest has anyway. A host serves only the names it advertises: naming another refuses the create and names the ones it does serve"` | ||
| 140 | Wait *bool `json:"wait,omitempty" jsonschema:"wait for ready+cloud-init (default true)"` | 178 | Wait *bool `json:"wait,omitempty" jsonschema:"wait for ready+cloud-init (default true)"` |
| 141 | } | 179 | } |
| 142 | 180 | ||
| 143 | type VMCreateOut struct { | 181 | type VMCreateOut struct { |
| 144 | ID string `json:"id"` | 182 | ID string `json:"id"` |
| 145 | Name string `json:"name"` | 183 | Name string `json:"name"` |
| 146 | IP string `json:"ip,omitempty"` | 184 | // IP is the guest's address on its host's private fabric — the one the ssh |
| 185 | // command reaches it at, and never routable off that host. | ||
| 186 | IP string `json:"ip,omitempty"` | ||
| 187 | // Network is the named host network the guest was created on, echoed back | ||
| 188 | // because reaching this field at all means admission accepted the name. | ||
| 189 | Network string `json:"network,omitempty"` | ||
| 190 | // NetworkIP is the address that network's own DHCP granted the second NIC — | ||
| 191 | // what the rest of that network knows this guest by. Empty when no network | ||
| 192 | // was asked for, and empty at ready when the site's DHCP has not answered | ||
| 193 | // yet: readiness is boot, and the lease arrives when it arrives. Read it | ||
| 194 | // back with vm_info. | ||
| 195 | NetworkIP string `json:"network_ip,omitempty"` | ||
| 147 | SSHCommand string `json:"ssh_command,omitempty"` | 196 | SSHCommand string `json:"ssh_command,omitempty"` |
| 148 | // CloudInit warns about a non-clean-but-usable boot: set to a degraded | 197 | // CloudInit warns about a non-clean-but-usable boot: set to a degraded |
| 149 | // message when cloud-init finished with recoverable errors (exit 2), omitted | 198 | // message when cloud-init finished with recoverable errors (exit 2), omitted |
| @@ -158,6 +207,7 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error | |||
| 158 | VCPUs: in.VCPUs, | 207 | VCPUs: in.VCPUs, |
| 159 | MemMB: in.MemMB, | 208 | MemMB: in.MemMB, |
| 160 | DiskGB: in.DiskGB, | 209 | DiskGB: in.DiskGB, |
| 210 | Network: in.Network, | ||
| 161 | } | 211 | } |
| 162 | if req.Name == "" { | 212 | if req.Name == "" { |
| 163 | req.Name = "claude-" + random.Hex(3) | 213 | req.Name = "claude-" + random.Hex(3) |
| @@ -166,6 +216,14 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error | |||
| 166 | // the control plane owns those defaults (types.Default*), so a zero here | 216 | // the control plane owns those defaults (types.Default*), so a zero here |
| 167 | // inherits the one named value rather than pinning a second, drift-prone set | 217 | // inherits the one named value rather than pinning a second, drift-prone set |
| 168 | // on this side. | 218 | // on this side. |
| 219 | // | ||
| 220 | // A caller who named a host gets that host, network or not: the control | ||
| 221 | // plane owns whether that pairing is legal and refuses it in one message | ||
| 222 | // naming what the host does advertise. Re-deciding it here would put a | ||
| 223 | // second, drifting copy of the rule in front of the first. Only the DEFAULT | ||
| 224 | // placement — where nothing has been named and this side is choosing — | ||
| 225 | // takes the network into account, because choosing a host that cannot serve | ||
| 226 | // it would manufacture that refusal out of a free choice. | ||
| 169 | if in.Host != "" { | 227 | if in.Host != "" { |
| 170 | hostID, err := t.resolveHost(ctx, in.Host) | 228 | hostID, err := t.resolveHost(ctx, in.Host) |
| 171 | if err != nil { | 229 | if err != nil { |
| @@ -173,7 +231,7 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error | |||
| 173 | } | 231 | } |
| 174 | req.HostID = hostID | 232 | req.HostID = hostID |
| 175 | } else { | 233 | } else { |
| 176 | h, err := t.API.FirstEligibleHost(ctx) | 234 | h, err := t.API.FirstEligibleHost(ctx, in.Network) |
| 177 | if err != nil { | 235 | if err != nil { |
| 178 | return VMCreateOut{}, err | 236 | return VMCreateOut{}, err |
| 179 | } | 237 | } |
| @@ -184,7 +242,9 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error | |||
| 184 | if err != nil { | 242 | if err != nil { |
| 185 | return VMCreateOut{}, fmt.Errorf("create vm: %w", err) | 243 | return VMCreateOut{}, fmt.Errorf("create vm: %w", err) |
| 186 | } | 244 | } |
| 187 | out := VMCreateOut{ID: created.ID, Name: created.Name} | 245 | // The network is echoed from the accepted request rather than waited for: |
| 246 | // it is frozen at create, and getting this far means admission took it. | ||
| 247 | out := VMCreateOut{ID: created.ID, Name: created.Name, Network: req.Network} | ||
| 188 | if in.Wait != nil && !*in.Wait { | 248 | if in.Wait != nil && !*in.Wait { |
| 189 | return out, nil | 249 | return out, nil |
| 190 | } | 250 | } |
| @@ -192,12 +252,23 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error | |||
| 192 | // One shared deadline bounds the whole wait (ready + cloud-init) by | 252 | // One shared deadline bounds the whole wait (ready + cloud-init) by |
| 193 | // WaitTimeout, rather than letting each phase burn a full budget. | 253 | // WaitTimeout, rather than letting each phase burn a full budget. |
| 194 | deadline := time.Now().Add(t.waitTimeout()) | 254 | deadline := time.Now().Add(t.waitTimeout()) |
| 195 | ip, err := t.waitReady(ctx, created.ID, created.Name, deadline) | 255 | ready, err := t.waitReady(ctx, created.ID, created.Name, deadline) |
| 196 | if err != nil { | 256 | if err != nil { |
| 197 | // Spec: report state, never auto-destroy — the VM may just be slow. | 257 | // Spec: report state, never auto-destroy — the VM may just be slow. |
| 198 | return out, err | 258 | return out, err |
| 199 | } | 259 | } |
| 260 | ip := ready.AssignedIP | ||
| 200 | out.IP = ip | 261 | out.IP = ip |
| 262 | // Once there is a server record, it outranks the echo above: the echo is | ||
| 263 | // what was ASKED for, and would quietly lie the day the server normalizes a | ||
| 264 | // stored name. The echo still stands alone on the no-wait path, where no | ||
| 265 | // record has been read. | ||
| 266 | if ready.Network != "" { | ||
| 267 | out.Network = ready.Network | ||
| 268 | } | ||
| 269 | // Whatever the named network has granted BY NOW, which may be nothing: the | ||
| 270 | // wait above never held for it. | ||
| 271 | out.NetworkIP = ready.NetworkIP | ||
| 201 | out.SSHCommand = t.sshCommand(ctx, created.Name) | 272 | out.SSHCommand = t.sshCommand(ctx, created.Name) |
| 202 | // "ready" means cloud-hypervisor is up and the IP is ALLOCATED — NOT that the | 273 | // "ready" means cloud-hypervisor is up and the IP is ALLOCATED — NOT that the |
| 203 | // guest has booted Linux, brought up its NIC, and started sshd. The first SSH | 274 | // guest has booted Linux, brought up its NIC, and started sshd. The first SSH |
| @@ -256,11 +327,18 @@ func (t *Tools) VMCreate(ctx context.Context, in VMCreateIn) (VMCreateOut, error | |||
| 256 | } | 327 | } |
| 257 | 328 | ||
| 258 | // waitReady polls until the VM reaches "ready" with an IP, or the shared | 329 | // waitReady polls until the VM reaches "ready" with an IP, or the shared |
| 259 | // deadline expires. Transient ListVMs failures (control-plane restart/blip/ | 330 | // deadline expires, and hands back the listing that satisfied it. Transient |
| 260 | // 5xx) do NOT abort the wait — they are stashed and polling continues, per the | 331 | // ListVMs failures (control-plane restart/blip/5xx) do NOT abort the wait — |
| 261 | // tool's "do not assume failure" contract. Only real cancellation (ctx.Done) | 332 | // they are stashed and polling continues, per the tool's "do not assume |
| 262 | // aborts immediately. | 333 | // failure" contract. Only real cancellation (ctx.Done) aborts immediately. |
| 263 | func (t *Tools) waitReady(ctx context.Context, id, name string, deadline time.Time) (string, error) { | 334 | // |
| 335 | // The IP waited for is assigned_ip, the host-fabric address, and only that one. | ||
| 336 | // A guest on a named host network gets its second address from that network's | ||
| 337 | // own DHCP server, which is not eitri's and owes eitri no schedule; holding | ||
| 338 | // readiness open for it would make a working guest look like a failed create | ||
| 339 | // because a site's DHCP was slow. So network_ip travels back as whatever it is | ||
| 340 | // at ready, empty included. | ||
| 341 | func (t *Tools) waitReady(ctx context.Context, id, name string, deadline time.Time) (client.VM, error) { | ||
| 264 | last, detail := "", "" | 342 | last, detail := "", "" |
| 265 | sawListing := false | 343 | sawListing := false |
| 266 | var lastErr error | 344 | var lastErr error |
| @@ -276,21 +354,21 @@ func (t *Tools) waitReady(ctx context.Context, id, name string, deadline time.Ti | |||
| 276 | } | 354 | } |
| 277 | last, detail = vm.Lifecycle, vm.StatusDetail | 355 | last, detail = vm.Lifecycle, vm.StatusDetail |
| 278 | if vm.Lifecycle == "ready" && vm.AssignedIP != "" { | 356 | if vm.Lifecycle == "ready" && vm.AssignedIP != "" { |
| 279 | return vm.AssignedIP, nil | 357 | return vm, nil |
| 280 | } | 358 | } |
| 281 | } | 359 | } |
| 282 | } | 360 | } |
| 283 | reportProgress(ctx, fmt.Sprintf("creating vm %s: %s", name, lifecycleOrUnknown(last, detail))) | 361 | reportProgress(ctx, fmt.Sprintf("creating vm %s: %s", name, lifecycleOrUnknown(last, detail))) |
| 284 | select { | 362 | select { |
| 285 | case <-ctx.Done(): | 363 | case <-ctx.Done(): |
| 286 | return "", ctx.Err() | 364 | return client.VM{}, ctx.Err() |
| 287 | case <-time.After(t.pollEvery()): | 365 | case <-time.After(t.pollEvery()): |
| 288 | } | 366 | } |
| 289 | } | 367 | } |
| 290 | if sawListing { | 368 | if sawListing { |
| 291 | return "", fmt.Errorf("vm %s (%s) not ready after %s (last lifecycle %q); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), last) | 369 | return client.VM{}, fmt.Errorf("vm %s (%s) not ready after %s (last lifecycle %q); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), last) |
| 292 | } | 370 | } |
| 293 | return "", fmt.Errorf("vm %s (%s) not ready after %s (control-plane never listed successfully; last error: %v); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), lastErr) | 371 | return client.VM{}, fmt.Errorf("vm %s (%s) not ready after %s (control-plane never listed successfully; last error: %v); it may still come up — check vm_info, do not assume failure", id, name, t.waitTimeout(), lastErr) |
| 294 | } | 372 | } |
| 295 | 373 | ||
| 296 | // lifecycleOrUnknown words a lifecycle for a progress message, covering the | 374 | // lifecycleOrUnknown words a lifecycle for a progress message, covering the |
| @@ -349,6 +427,37 @@ func describeHosts(hosts []client.Host) string { | |||
| 349 | return strings.Join(parts, ", ") | 427 | return strings.Join(parts, ", ") |
| 350 | } | 428 | } |
| 351 | 429 | ||
| 430 | // describeHostNetworks renders what each host a VM could actually be placed on | ||
| 431 | // advertises, as "onyx: lan, dmz; mewtwo: none". | ||
| 432 | // | ||
| 433 | // It lists exactly the hosts eligibleHost would consider, by asking the same | ||
| 434 | // predicate rather than by keeping a matching copy of it. Both exclusions earn | ||
| 435 | // their place: an offline host would be listed as serving nothing, but | ||
| 436 | // advertised networks are registry state emptied when a host stops reporting, | ||
| 437 | // so "none" for a machine that is down is a claim nobody asked it; and a host | ||
| 438 | // whose agent predates certified host keys would be recommended here only to | ||
| 439 | // refuse the create for an unrelated reason, sending the caller to configure a | ||
| 440 | // bridge when the remedy is an upgrade. | ||
| 441 | func describeHostNetworks(hosts []client.Host) string { | ||
| 442 | parts := make([]string, 0, len(hosts)) | ||
| 443 | for _, h := range hosts { | ||
| 444 | if !placeable(h) { | ||
| 445 | continue | ||
| 446 | } | ||
| 447 | names := "none" | ||
| 448 | if len(h.HostNetworks) > 0 { | ||
| 449 | names = strings.Join(h.HostNetworks, ", ") | ||
| 450 | } | ||
| 451 | parts = append(parts, fmt.Sprintf("%s: %s", h.Name, names)) | ||
| 452 | } | ||
| 453 | if len(parts) == 0 { | ||
| 454 | // Defensive: eligibleHost only asks after a host has passed both checks, | ||
| 455 | // so there is always at least one entry on that path. | ||
| 456 | return "no host is eligible to place on" | ||
| 457 | } | ||
| 458 | return strings.Join(parts, "; ") | ||
| 459 | } | ||
| 460 | |||
| 352 | func (t *Tools) sshCommand(ctx context.Context, name string) string { | 461 | func (t *Tools) sshCommand(ctx context.Context, name string) string { |
| 353 | if t.Gate == "" { | 462 | if t.Gate == "" { |
| 354 | // No gate: reach the VM directly by name/IP (a different deployment | 463 | // No gate: reach the VM directly by name/IP (a different deployment |
internal/mcpserver/tools_test.go
| Old | New | ||
|---|---|---|---|
| @@ -17,6 +17,7 @@ import ( | |||
| 17 | "golang.org/x/crypto/ssh" | 17 | "golang.org/x/crypto/ssh" |
| 18 | 18 | ||
| 19 | "github.com/a73x/eitri/internal/server/api/client" | 19 | "github.com/a73x/eitri/internal/server/api/client" |
| 20 | "github.com/a73x/eitri/internal/server/release" | ||
| 20 | ) | 21 | ) |
| 21 | 22 | ||
| 22 | type fakeToolsAPI struct { | 23 | type fakeToolsAPI struct { |
| @@ -30,6 +31,11 @@ type fakeToolsAPI struct { | |||
| 30 | // number of leading ListVMs calls that fail (control-plane blip); after | 31 | // number of leading ListVMs calls that fail (control-plane blip); after |
| 31 | // they are exhausted the phases sequence takes over. | 32 | // they are exhausted the phases sequence takes over. |
| 32 | listErrs int | 33 | listErrs int |
| 34 | // what the scripted VM reports about its named network once ready: the | ||
| 35 | // network it was created on, and the address that network's DHCP granted | ||
| 36 | // (empty for a lease that has not landed). | ||
| 37 | network string | ||
| 38 | networkIP string | ||
| 33 | // exposures the fake serves per VM id, and the exposure ids revoked. | 39 | // exposures the fake serves per VM id, and the exposure ids revoked. |
| 34 | exposures map[string][]client.Exposure | 40 | exposures map[string][]client.Exposure |
| 35 | revoked []string | 41 | revoked []string |
| @@ -46,14 +52,20 @@ type fakeToolsAPI struct { | |||
| 46 | } | 52 | } |
| 47 | 53 | ||
| 48 | // fleet is the fake's host list, defaulted so tests that don't care about | 54 | // fleet is the fake's host list, defaulted so tests that don't care about |
| 49 | // placement get a single online host. | 55 | // placement get a single online host. Its agent is current, because placement |
| 56 | // rules an older one out and these tests are not about that. | ||
| 50 | func (f *fakeToolsAPI) fleet() []client.Host { | 57 | func (f *fakeToolsAPI) fleet() []client.Host { |
| 51 | if f.hosts != nil { | 58 | if f.hosts != nil { |
| 52 | return f.hosts | 59 | return f.hosts |
| 53 | } | 60 | } |
| 54 | return []client.Host{{ID: "h1", Name: "mewtwo", Online: true}} | 61 | return []client.Host{{ID: "h1", Name: "mewtwo", Online: true, AgentVersion: currentAgent}} |
| 55 | } | 62 | } |
| 56 | 63 | ||
| 64 | // currentAgent is any version placement accepts: the floor is exactly the | ||
| 65 | // question CertifiesGuestHostKeys asks, so naming it here keeps the fixture | ||
| 66 | // from pinning a release literal of its own. | ||
| 67 | const currentAgent = release.FirstCertifiedHostKeys | ||
| 68 | |||
| 57 | func (f *fakeToolsAPI) ListVMs(ctx context.Context) ([]client.VM, error) { | 69 | func (f *fakeToolsAPI) ListVMs(ctx context.Context) ([]client.VM, error) { |
| 58 | if f.listErrs > 0 { | 70 | if f.listErrs > 0 { |
| 59 | f.listErrs-- | 71 | f.listErrs-- |
| @@ -65,9 +77,10 @@ func (f *fakeToolsAPI) ListVMs(ctx context.Context) ([]client.VM, error) { | |||
| 65 | i = len(f.phases) - 1 | 77 | i = len(f.phases) - 1 |
| 66 | } | 78 | } |
| 67 | f.calls++ | 79 | f.calls++ |
| 68 | vm := client.VM{ID: "new1", Name: "claude-abc", Lifecycle: f.phases[i]} | 80 | vm := client.VM{ID: "new1", Name: "claude-abc", Lifecycle: f.phases[i], Network: f.network} |
| 69 | if f.phases[i] == "ready" { | 81 | if f.phases[i] == "ready" { |
| 70 | vm.AssignedIP = "10.77.1.9" | 82 | vm.AssignedIP = "10.77.1.9" |
| 83 | vm.NetworkIP = f.networkIP | ||
| 71 | } | 84 | } |
| 72 | return append(append([]client.VM{}, f.vms...), vm), nil | 85 | return append(append([]client.VM{}, f.vms...), vm), nil |
| 73 | } | 86 | } |
| @@ -84,13 +97,11 @@ func (f *fakeToolsAPI) DeleteVM(ctx context.Context, id string) error { | |||
| 84 | func (f *fakeToolsAPI) ListHosts(ctx context.Context) ([]client.Host, error) { | 97 | func (f *fakeToolsAPI) ListHosts(ctx context.Context) ([]client.Host, error) { |
| 85 | return f.fleet(), nil | 98 | return f.fleet(), nil |
| 86 | } | 99 | } |
| 87 | func (f *fakeToolsAPI) FirstEligibleHost(ctx context.Context) (client.Host, error) { | 100 | |
| 88 | for _, h := range f.fleet() { | 101 | // FirstEligibleHost runs the real placement rule over the fake's fleet, so a |
| 89 | if h.Online { | 102 | // test cannot pass against a kinder stand-in than the one in production. |
| 90 | return h, nil | 103 | func (f *fakeToolsAPI) FirstEligibleHost(ctx context.Context, network string) (client.Host, error) { |
| 91 | } | 104 | return eligibleHost(f.fleet(), network) |
| 92 | } | ||
| 93 | return client.Host{}, fmt.Errorf("no online hosts") | ||
| 94 | } | 105 | } |
| 95 | 106 | ||
| 96 | // CreateExposure mirrors the control plane: host port 0 is allocated from the | 107 | // CreateExposure mirrors the control plane: host port 0 is allocated from the |
| @@ -183,12 +194,32 @@ func (f *fakeRunner) ConnectName(ctx context.Context, vmName string) (string, er | |||
| 183 | func newTestTools(api *fakeToolsAPI, r *fakeRunner) *Tools { | 194 | func newTestTools(api *fakeToolsAPI, r *fakeRunner) *Tools { |
| 184 | return &Tools{ | 195 | return &Tools{ |
| 185 | API: api, Runner: r, | 196 | API: api, Runner: r, |
| 186 | Gate: "localhost:2223", | 197 | Gate: "localhost:2223", |
| 187 | VMUser: "ubuntu", | 198 | VMUser: "ubuntu", |
| 188 | PollEvery: time.Millisecond, // fast tests | 199 | // Fast tests, and a budget that ENDS. At the 10-minute default a test |
| 200 | // whose VM never reaches the state it is waiting for does not fail — | ||
| 201 | // it hangs, and CI pays ten minutes to learn what the timeout error | ||
| 202 | // says in seconds. The two tests that assert on exhausting the budget | ||
| 203 | // shorten it further themselves. | ||
| 204 | PollEvery: time.Millisecond, | ||
| 205 | WaitTimeout: 5 * time.Second, | ||
| 189 | } | 206 | } |
| 190 | } | 207 | } |
| 191 | 208 | ||
| 209 | // Every test above hands Tools a short budget, so the shipped defaults are | ||
| 210 | // exercised nowhere else — and they are the ones production runs on: a poll | ||
| 211 | // interval and a ceiling that a real create, downloading an image over a slow | ||
| 212 | // link, spends minutes inside. | ||
| 213 | func TestUnconfiguredToolsUseTheShippedWaitBudget(t *testing.T) { | ||
| 214 | var zero Tools | ||
| 215 | assert.Equal(t, 2*time.Second, zero.pollEvery()) | ||
| 216 | assert.Equal(t, 10*time.Minute, zero.waitTimeout()) | ||
| 217 | |||
| 218 | set := Tools{PollEvery: time.Millisecond, WaitTimeout: time.Second} | ||
| 219 | assert.Equal(t, time.Millisecond, set.pollEvery(), "a configured interval wins over the default") | ||
| 220 | assert.Equal(t, time.Second, set.waitTimeout()) | ||
| 221 | } | ||
| 222 | |||
| 192 | func TestCreateWaitsForReadyAndCloudInit(t *testing.T) { | 223 | func TestCreateWaitsForReadyAndCloudInit(t *testing.T) { |
| 193 | api := &fakeToolsAPI{phases: []string{"creating", "creating", "ready"}} | 224 | api := &fakeToolsAPI{phases: []string{"creating", "creating", "ready"}} |
| 194 | run := &fakeRunner{out: ExecResult{ExitCode: 0}} | 225 | run := &fakeRunner{out: ExecResult{ExitCode: 0}} |
| @@ -332,12 +363,13 @@ func TestCreateNoWaitReturnsImmediately(t *testing.T) { | |||
| 332 | } | 363 | } |
| 333 | 364 | ||
| 334 | // mixedFleet is a two-host fleet with a Mac second and a decommissioned-looking | 365 | // mixedFleet is a two-host fleet with a Mac second and a decommissioned-looking |
| 335 | // host offline, the shape placement has to get right. | 366 | // host offline, the shape placement has to get right. Only the Linux host |
| 367 | // advertises a named network — a Mac cannot serve one at all. | ||
| 336 | func mixedFleet() []client.Host { | 368 | func mixedFleet() []client.Host { |
| 337 | return []client.Host{ | 369 | return []client.Host{ |
| 338 | {ID: "h1", Name: "onyx", Online: true}, | 370 | {ID: "h1", Name: "onyx", Online: true, AgentVersion: currentAgent, HostNetworks: []string{"lan", "dmz"}}, |
| 339 | {ID: "h2", Name: "Squirtle.local", Online: true}, | 371 | {ID: "h2", Name: "Squirtle.local", Online: true, AgentVersion: currentAgent}, |
| 340 | {ID: "h3", Name: "charmander", Online: false}, | 372 | {ID: "h3", Name: "charmander", Online: false, AgentVersion: currentAgent}, |
| 341 | } | 373 | } |
| 342 | } | 374 | } |
| 343 | 375 | ||
| @@ -402,6 +434,129 @@ func TestCreateWithoutAHostTakesTheFirstOnlineOne(t *testing.T) { | |||
| 402 | assert.Equal(t, "h2", api.created[0].HostID) | 434 | assert.Equal(t, "h2", api.created[0].HostID) |
| 403 | } | 435 | } |
| 404 | 436 | ||
| 437 | // The network the caller names has to reach the API as a network — it is the | ||
| 438 | // whole request, and nothing downstream can infer it. | ||
| 439 | func TestCreateAsksForTheNamedNetwork(t *testing.T) { | ||
| 440 | api := &fakeToolsAPI{hosts: mixedFleet()} | ||
| 441 | tl := newTestTools(api, &fakeRunner{}) | ||
| 442 | no := false | ||
| 443 | |||
| 444 | out, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan", Wait: &no}) | ||
| 445 | require.NoError(t, err) | ||
| 446 | require.Len(t, api.created, 1) | ||
| 447 | assert.Equal(t, "lan", api.created[0].Network) | ||
| 448 | assert.Equal(t, "lan", out.Network, "the accepted network is reported back without waiting for anything") | ||
| 449 | } | ||
| 450 | |||
| 451 | // A create that names no network asks for none: an empty field must stay empty | ||
| 452 | // on the wire rather than becoming some default underlay name. | ||
| 453 | func TestCreateWithoutANetworkAsksForNone(t *testing.T) { | ||
| 454 | api := &fakeToolsAPI{hosts: mixedFleet()} | ||
| 455 | tl := newTestTools(api, &fakeRunner{}) | ||
| 456 | no := false | ||
| 457 | |||
| 458 | out, err := tl.VMCreate(t.Context(), VMCreateIn{Wait: &no}) | ||
| 459 | require.NoError(t, err) | ||
| 460 | require.Len(t, api.created, 1) | ||
| 461 | assert.Empty(t, api.created[0].Network) | ||
| 462 | assert.Empty(t, out.Network) | ||
| 463 | assert.Empty(t, out.NetworkIP) | ||
| 464 | } | ||
| 465 | |||
| 466 | // Both addresses come back: the host-fabric one the ssh command uses, and the | ||
| 467 | // one the named network's own DHCP granted. | ||
| 468 | func TestCreateReportsBothAddresses(t *testing.T) { | ||
| 469 | api := &fakeToolsAPI{hosts: mixedFleet(), phases: []string{"creating", "ready"}, network: "lan", networkIP: "192.168.0.232"} | ||
| 470 | tl := newTestTools(api, &fakeRunner{out: ExecResult{ExitCode: 0}}) | ||
| 471 | |||
| 472 | out, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan"}) | ||
| 473 | require.NoError(t, err) | ||
| 474 | assert.Equal(t, "10.77.1.9", out.IP, "the host-fabric address is still the one ssh reaches it at") | ||
| 475 | assert.Equal(t, "lan", out.Network) | ||
| 476 | assert.Equal(t, "192.168.0.232", out.NetworkIP) | ||
| 477 | } | ||
| 478 | |||
| 479 | // Two sources can name the network: what was asked for, and what the server | ||
| 480 | // recorded. Once the record exists it wins — the echo is only what was asked, | ||
| 481 | // and it would quietly lie the day the server normalizes a stored name. | ||
| 482 | func TestCreateReportsTheServersRecordOfTheNetworkOverTheAsk(t *testing.T) { | ||
| 483 | // The host is named, so placement does not filter on the asked-for spelling | ||
| 484 | // and the request reaches the server exactly as written. | ||
| 485 | api := &fakeToolsAPI{hosts: mixedFleet(), phases: []string{"ready"}, network: "lan", networkIP: "192.168.0.232"} | ||
| 486 | tl := newTestTools(api, &fakeRunner{out: ExecResult{ExitCode: 0}}) | ||
| 487 | |||
| 488 | out, err := tl.VMCreate(t.Context(), VMCreateIn{Host: "onyx", Network: "LAN"}) | ||
| 489 | require.NoError(t, err) | ||
| 490 | require.Len(t, api.created, 1) | ||
| 491 | assert.Equal(t, "LAN", api.created[0].Network, "the ask travels unaltered") | ||
| 492 | assert.Equal(t, "lan", out.Network, "but the answer is the server's record") | ||
| 493 | } | ||
| 494 | |||
| 495 | // Readiness is the guest booting. A site's DHCP server owes eitri no schedule, | ||
| 496 | // so a lease that has not landed by then must not hold the create open or fail | ||
| 497 | // it — the VM is up and usable, and network_ip is simply not known yet. | ||
| 498 | func TestCreateSucceedsWithTheNetworkLeaseStillOutstanding(t *testing.T) { | ||
| 499 | api := &fakeToolsAPI{hosts: mixedFleet(), phases: []string{"creating", "ready"}, network: "lan"} // networkIP never arrives | ||
| 500 | tl := newTestTools(api, &fakeRunner{out: ExecResult{ExitCode: 0}}) | ||
| 501 | |||
| 502 | out, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan"}) | ||
| 503 | require.NoError(t, err) | ||
| 504 | assert.Equal(t, "10.77.1.9", out.IP) | ||
| 505 | assert.Equal(t, "lan", out.Network) | ||
| 506 | assert.Empty(t, out.NetworkIP, "an unanswered lease is an empty field, not a failed create") | ||
| 507 | } | ||
| 508 | |||
| 509 | // Default placement is a free choice, so it is made among hosts that can | ||
| 510 | // actually serve what was asked for: picking the first eligible host and | ||
| 511 | // letting the control plane refuse it would manufacture a failure. | ||
| 512 | func TestCreateWithoutAHostPlacesOnOneServingTheNetwork(t *testing.T) { | ||
| 513 | hosts := mixedFleet() | ||
| 514 | hosts[0].HostNetworks = nil // onyx, first and otherwise eligible, serves none | ||
| 515 | hosts[1].HostNetworks = []string{"lan"} // the second one does | ||
| 516 | api := &fakeToolsAPI{hosts: hosts} | ||
| 517 | tl := newTestTools(api, &fakeRunner{}) | ||
| 518 | no := false | ||
| 519 | |||
| 520 | _, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "lan", Wait: &no}) | ||
| 521 | require.NoError(t, err) | ||
| 522 | require.Len(t, api.created, 1) | ||
| 523 | assert.Equal(t, "h2", api.created[0].HostID) | ||
| 524 | } | ||
| 525 | |||
| 526 | // With nothing in the fleet serving it, the refusal names what IS served — | ||
| 527 | // the one place an MCP caller learns the fleet's network names without | ||
| 528 | // provoking a refusal from the control plane. | ||
| 529 | func TestCreateWithNoHostForTheNetworkNamesWhatIsAdvertised(t *testing.T) { | ||
| 530 | api := &fakeToolsAPI{hosts: mixedFleet()} | ||
| 531 | tl := newTestTools(api, &fakeRunner{}) | ||
| 532 | no := false | ||
| 533 | |||
| 534 | _, err := tl.VMCreate(t.Context(), VMCreateIn{Network: "storage", Wait: &no}) | ||
| 535 | require.Error(t, err) | ||
| 536 | assert.ErrorContains(t, err, `no eligible host advertises network "storage"`) | ||
| 537 | assert.ErrorContains(t, err, "onyx: lan, dmz", "the error names what the fleet DOES advertise") | ||
| 538 | assert.ErrorContains(t, err, "Squirtle.local: none") | ||
| 539 | assert.NotContains(t, err.Error(), "charmander", "an offline host has not been asked what it serves") | ||
| 540 | assert.ErrorContains(t, err, "--host-network storage=<bridge>", "with the remedy in the operator's own vocabulary") | ||
| 541 | assert.ErrorContains(t, err, "NAT underlay") | ||
| 542 | assert.Empty(t, api.created, "nothing is created on a host that cannot serve the request") | ||
| 543 | } | ||
| 544 | |||
| 545 | // A caller who names both a host and a network gets that host: the control | ||
| 546 | // plane owns the pairing and refuses it in one message. Re-deciding it here | ||
| 547 | // would put a second copy of the rule in front of the first. | ||
| 548 | func TestCreateOnANamedHostLeavesTheNetworkToTheControlPlane(t *testing.T) { | ||
| 549 | api := &fakeToolsAPI{hosts: mixedFleet()} | ||
| 550 | tl := newTestTools(api, &fakeRunner{}) | ||
| 551 | no := false | ||
| 552 | |||
| 553 | _, err := tl.VMCreate(t.Context(), VMCreateIn{Host: "Squirtle.local", Network: "lan", Wait: &no}) | ||
| 554 | require.NoError(t, err) | ||
| 555 | require.Len(t, api.created, 1) | ||
| 556 | assert.Equal(t, "h2", api.created[0].HostID) | ||
| 557 | assert.Equal(t, "lan", api.created[0].Network, "the request travels as asked, for the server to judge") | ||
| 558 | } | ||
| 559 | |||
| 405 | func TestCreateWaitTimeoutDoesNotDestroy(t *testing.T) { | 560 | func TestCreateWaitTimeoutDoesNotDestroy(t *testing.T) { |
| 406 | api := &fakeToolsAPI{phases: []string{"creating"}} // never ready | 561 | api := &fakeToolsAPI{phases: []string{"creating"}} // never ready |
| 407 | run := &fakeRunner{} | 562 | run := &fakeRunner{} |
| @@ -468,6 +623,30 @@ func TestReadFileNotReadyVMRejectedWithoutCallingRunner(t *testing.T) { | |||
| 468 | assert.Empty(t, run.reads, "runner must not be called on a non-ready VM") | 623 | assert.Empty(t, run.reads, "runner must not be called on a non-ready VM") |
| 469 | } | 624 | } |
| 470 | 625 | ||
| 626 | // vm_list and vm_info hand back the API's own VM, so a guest's second address | ||
| 627 | // is already in both. This pins that: flattening either into a hand-written | ||
| 628 | // view is exactly how a field goes missing from the surface that is supposed to | ||
| 629 | // be the programmatic one. | ||
| 630 | func TestListAndInfoCarryBothAddresses(t *testing.T) { | ||
| 631 | api := &fakeToolsAPI{vms: []client.VM{{ | ||
| 632 | ID: "abc123", Name: "web-1", Lifecycle: "ready", | ||
| 633 | AssignedIP: "10.77.1.5", Network: "lan", NetworkIP: "192.168.0.232", | ||
| 634 | }}} | ||
| 635 | tl := newTestTools(api, &fakeRunner{}) | ||
| 636 | |||
| 637 | listed, err := tl.VMList(t.Context(), VMListIn{}) | ||
| 638 | require.NoError(t, err) | ||
| 639 | require.Len(t, listed.VMs, 1) | ||
| 640 | assert.Equal(t, "lan", listed.VMs[0].Network) | ||
| 641 | assert.Equal(t, "192.168.0.232", listed.VMs[0].NetworkIP) | ||
| 642 | assert.Equal(t, "10.77.1.5", listed.VMs[0].AssignedIP) | ||
| 643 | |||
| 644 | info, err := tl.VMInfo(t.Context(), VMInfoIn{VM: "web-1"}) | ||
| 645 | require.NoError(t, err) | ||
| 646 | assert.Equal(t, "lan", info.VM.Network) | ||
| 647 | assert.Equal(t, "192.168.0.232", info.VM.NetworkIP, "vm_info is where a caller reads a lease that landed late") | ||
| 648 | } | ||
| 649 | |||
| 471 | // TestVMInfoHintNamespacesConnectName pins that vm_info's ssh_command hint dials | 650 | // TestVMInfoHintNamespacesConnectName pins that vm_info's ssh_command hint dials |
| 472 | // the gate by the <tenant>.<name> connect name (the gate rejects a bare name), | 651 | // the gate by the <tenant>.<name> connect name (the gate rejects a bare name), |
| 473 | // and that the gateless deployment still emits the plain `ssh <user>@<name>`. | 652 | // and that the gateless deployment still emits the plain `ssh <user>@<name>`. |
internal/names/names.go
| Old | New | ||
|---|---|---|---|
| @@ -1,6 +1,10 @@ | |||
| 1 | // Package names validates the DNS-label shape shared across planes: a VM's | 1 | // Package names validates the DNS-label shape shared across planes: a VM's |
| 2 | // name doubles as its guest hostname, so it must be a valid RFC-1123 label. | 2 | // name doubles as its guest hostname, so it must be a valid RFC-1123 label. |
| 3 | // This package is a dependency-free leaf. | 3 | // It also validates the distinct, narrower grammar of a named network's name |
| 4 | // (see IsNetworkName), and holds the one cross-plane fact about which host OS | ||
| 5 | // can serve a named network at all (see OSServesNamedNetworks). This package | ||
| 6 | // is a dependency-free leaf, which is exactly why both facts live here rather | ||
| 7 | // than beside either plane. | ||
| 4 | package names | 8 | package names |
| 5 | 9 | ||
| 6 | import "regexp" | 10 | import "regexp" |
| @@ -22,6 +26,44 @@ var sha256Hex = regexp.MustCompile(`^[a-f0-9]{64}$`) | |||
| 22 | // path or trusting the value, so it lives in this dependency-free leaf. | 26 | // path or trusting the value, so it lives in this dependency-free leaf. |
| 23 | func IsSHA256Hex(s string) bool { return sha256Hex.MatchString(s) } | 27 | func IsSHA256Hex(s string) bool { return sha256Hex.MatchString(s) } |
| 24 | 28 | ||
| 29 | // IsNetworkName reports whether s is a legal named-network name: 1–32 chars | ||
| 30 | // of [a-z0-9-], no leading/trailing hyphen, and not the reserved word "nat" | ||
| 31 | // (the anonymous default underlay's name in prose; a network named "nat" | ||
| 32 | // would make every conversation about it ambiguous). One home for the | ||
| 33 | // grammar — the API validates with it and the agent flag parser refuses | ||
| 34 | // with it, so the two cannot drift. The byte-wise loop below is sound for | ||
| 35 | // multi-byte UTF-8 input too: every legal character here is single-byte | ||
| 36 | // ASCII, so any byte of a multi-byte rune falls outside the allowed ranges | ||
| 37 | // and rejects. | ||
| 38 | func IsNetworkName(s string) bool { | ||
| 39 | if s == "" || s == "nat" || len(s) > 32 { | ||
| 40 | return false | ||
| 41 | } | ||
| 42 | if s[0] == '-' || s[len(s)-1] == '-' { | ||
| 43 | return false | ||
| 44 | } | ||
| 45 | for i := 0; i < len(s); i++ { | ||
| 46 | switch c := s[i]; { | ||
| 47 | case c >= 'a' && c <= 'z', c >= '0' && c <= '9', c == '-': | ||
| 48 | // allowed | ||
| 49 | default: | ||
| 50 | return false | ||
| 51 | } | ||
| 52 | } | ||
| 53 | return true | ||
| 54 | } | ||
| 55 | |||
| 56 | // OSServesNamedNetworks reports whether an agent running on goos (the value | ||
| 57 | // runtime.GOOS reports) can serve a named network at all: only a Linux host | ||
| 58 | // owns a bridge to attach a named network's taps to. This is the one home | ||
| 59 | // for that fact — the agent's flag parser refuses --host-network at startup | ||
| 60 | // with it, and the server words a create's network refusal from it using | ||
| 61 | // store.Host.OS (set once, from this same runtime.GOOS, at the host's | ||
| 62 | // enrollment — see internal/server/api/networks.go for why that source is | ||
| 63 | // trustworthy without a live report), so a third OS gaining support is a | ||
| 64 | // one-line change made once, here, rather than found and fixed twice. | ||
| 65 | func OSServesNamedNetworks(goos string) bool { return goos == "linux" } | ||
| 66 | |||
| 25 | // ConnectName returns a VM's connect name, "<tenant>.<name>" — the form the SSH | 67 | // ConnectName returns a VM's connect name, "<tenant>.<name>" — the form the SSH |
| 26 | // gate resolves and the VM's host certificate is issued for. It splits back | 68 | // gate resolves and the VM's host certificate is issued for. It splits back |
| 27 | // unambiguously on the first dot only because tenant ids and VM names are both | 69 | // unambiguously on the first dot only because tenant ids and VM names are both |
internal/names/names_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,87 @@ | |||
| 1 | package names | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "os" | ||
| 5 | "path/filepath" | ||
| 6 | "strings" | ||
| 7 | "testing" | ||
| 8 | ) | ||
| 9 | |||
| 10 | // TestIsNetworkName exercises the network-name grammar's boundaries directly | ||
| 11 | // against the implementation: the length limits, the leading/trailing-hyphen | ||
| 12 | // rule, the reserved "nat" word, and the lowercase-alnum-and-hyphen charset. | ||
| 13 | func TestIsNetworkName(t *testing.T) { | ||
| 14 | valid := []string{"lan", "lab-2", "a", strings.Repeat("x", 32)} | ||
| 15 | invalid := []string{"", "nat", "LAN", "l_n", "-lan", "lan-", strings.Repeat("x", 33), "br0.7"} | ||
| 16 | for _, s := range valid { | ||
| 17 | if !IsNetworkName(s) { | ||
| 18 | t.Errorf("IsNetworkName(%q) = false, want true", s) | ||
| 19 | } | ||
| 20 | } | ||
| 21 | for _, s := range invalid { | ||
| 22 | if IsNetworkName(s) { | ||
| 23 | t.Errorf("IsNetworkName(%q) = true, want false", s) | ||
| 24 | } | ||
| 25 | } | ||
| 26 | } | ||
| 27 | |||
| 28 | // TestDocsMatchNetworkNameGrammar is the docs-match tripwire for the network- | ||
| 29 | // name grammar. IsNetworkName is the one home for the rule (see its doc | ||
| 30 | // comment), but the agent's flag parser (internal/agent/run/cli.go) and | ||
| 31 | // docs/quickstart.md both restate it in prose for a human reader, in a form | ||
| 32 | // that cannot import the Go function. This test reads the quickstart prose | ||
| 33 | // and fails if its load-bearing fragments go missing, and separately proves | ||
| 34 | // the stated bounds still match IsNetworkName's behavior — so a change to | ||
| 35 | // the grammar cannot silently leave the docs describing the old rule. | ||
| 36 | func TestDocsMatchNetworkNameGrammar(t *testing.T) { | ||
| 37 | quickstart := readRepoFile(t, "docs/quickstart.md") | ||
| 38 | for _, want := range []string{ | ||
| 39 | "1–32 of `[a-z0-9-]`", | ||
| 40 | "no leading", | ||
| 41 | "or trailing hyphen", | ||
| 42 | "`nat` is reserved", | ||
| 43 | } { | ||
| 44 | if !strings.Contains(quickstart, want) { | ||
| 45 | t.Errorf("docs/quickstart.md is missing %q — the network-name grammar prose drifted from IsNetworkName", want) | ||
| 46 | } | ||
| 47 | } | ||
| 48 | |||
| 49 | // Behavioral half: the bounds the prose claims (1-32 chars, no | ||
| 50 | // leading/trailing hyphen, "nat" reserved) must match what the code | ||
| 51 | // actually accepts. TestIsNetworkName already covers this in depth; | ||
| 52 | // these three checks pin the specific numbers the doc sentence quotes. | ||
| 53 | if !IsNetworkName(strings.Repeat("x", 32)) { | ||
| 54 | t.Error("IsNetworkName rejects a 32-char name — docs/quickstart.md's \"1–32\" would be wrong") | ||
| 55 | } | ||
| 56 | if IsNetworkName(strings.Repeat("x", 33)) { | ||
| 57 | t.Error("IsNetworkName accepts a 33-char name — docs/quickstart.md's \"1–32\" would be wrong") | ||
| 58 | } | ||
| 59 | if IsNetworkName("nat") { | ||
| 60 | t.Error("IsNetworkName accepts \"nat\" — docs/quickstart.md's \"nat is reserved\" would be wrong") | ||
| 61 | } | ||
| 62 | } | ||
| 63 | |||
| 64 | // TestOSServesNamedNetworks pins the one platform this fact is true for today | ||
| 65 | // and confirms it is a real comparison, not a stub that always answers true. | ||
| 66 | func TestOSServesNamedNetworks(t *testing.T) { | ||
| 67 | if !OSServesNamedNetworks("linux") { | ||
| 68 | t.Error("OSServesNamedNetworks(\"linux\") = false, want true") | ||
| 69 | } | ||
| 70 | for _, goos := range []string{"darwin", "windows", ""} { | ||
| 71 | if OSServesNamedNetworks(goos) { | ||
| 72 | t.Errorf("OSServesNamedNetworks(%q) = true, want false", goos) | ||
| 73 | } | ||
| 74 | } | ||
| 75 | } | ||
| 76 | |||
| 77 | // readRepoFile reads a repo-root-relative path, locating the root by walking | ||
| 78 | // up from this test file's own location so the result does not depend on the | ||
| 79 | // working directory the test is run from. | ||
| 80 | func readRepoFile(t *testing.T, rel string) string { | ||
| 81 | t.Helper() | ||
| 82 | b, err := os.ReadFile(filepath.Join("..", "..", rel)) | ||
| 83 | if err != nil { | ||
| 84 | t.Fatalf("reading %s: %v", rel, err) | ||
| 85 | } | ||
| 86 | return string(b) | ||
| 87 | } | ||
internal/pb/sync.pb.go
| Old | New | ||
|---|---|---|---|
| @@ -245,6 +245,11 @@ type Hello struct { | |||
| 245 | Capacity *Capacity `protobuf:"bytes,8,opt,name=capacity,proto3" json:"capacity,omitempty"` | 245 | Capacity *Capacity `protobuf:"bytes,8,opt,name=capacity,proto3" json:"capacity,omitempty"` |
| 246 | Credential string `protobuf:"bytes,9,opt,name=credential,proto3" json:"credential,omitempty"` // Bearer host credential, verified in first frame | 246 | Credential string `protobuf:"bytes,9,opt,name=credential,proto3" json:"credential,omitempty"` // Bearer host credential, verified in first frame |
| 247 | Facts *HostFacts `protobuf:"bytes,10,opt,name=facts,proto3" json:"facts,omitempty"` // best-effort static host identity; refreshed each Hello | 247 | Facts *HostFacts `protobuf:"bytes,10,opt,name=facts,proto3" json:"facts,omitempty"` // best-effort static host identity; refreshed each Hello |
| 248 | // The named guest networks this host is configured to serve | ||
| 249 | // (--host-network name=bridge), names only. In Hello rather than the | ||
| 250 | // report because the set changes only with agent configuration, i.e. a | ||
| 251 | // restart. Empty for a Mac, and for a Linux host with none configured. | ||
| 252 | HostNetworks []string `protobuf:"bytes,11,rep,name=host_networks,json=hostNetworks,proto3" json:"host_networks,omitempty"` | ||
| 248 | unknownFields protoimpl.UnknownFields | 253 | unknownFields protoimpl.UnknownFields |
| 249 | sizeCache protoimpl.SizeCache | 254 | sizeCache protoimpl.SizeCache |
| 250 | } | 255 | } |
| @@ -342,6 +347,13 @@ func (x *Hello) GetFacts() *HostFacts { | |||
| 342 | return nil | 347 | return nil |
| 343 | } | 348 | } |
| 344 | 349 | ||
| 350 | func (x *Hello) GetHostNetworks() []string { | ||
| 351 | if x != nil { | ||
| 352 | return x.HostNetworks | ||
| 353 | } | ||
| 354 | return nil | ||
| 355 | } | ||
| 356 | |||
| 345 | type Capacity struct { | 357 | type Capacity struct { |
| 346 | state protoimpl.MessageState `protogen:"open.v1"` | 358 | state protoimpl.MessageState `protogen:"open.v1"` |
| 347 | Vcpus int64 `protobuf:"varint,1,opt,name=vcpus,proto3" json:"vcpus,omitempty"` | 359 | Vcpus int64 `protobuf:"varint,1,opt,name=vcpus,proto3" json:"vcpus,omitempty"` |
| @@ -603,7 +615,7 @@ type ActualVM struct { | |||
| 603 | VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` | 615 | VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` |
| 604 | Power string `protobuf:"bytes,2,opt,name=power,proto3" json:"power,omitempty"` // "running"|"stopped" | 616 | Power string `protobuf:"bytes,2,opt,name=power,proto3" json:"power,omitempty"` // "running"|"stopped" |
| 605 | Phase string `protobuf:"bytes,3,opt,name=phase,proto3" json:"phase,omitempty"` // "creating"|"ready"|"failed"|"quarantined" | 617 | Phase string `protobuf:"bytes,3,opt,name=phase,proto3" json:"phase,omitempty"` // "creating"|"ready"|"failed"|"quarantined" |
| 606 | Ip string `protobuf:"bytes,4,opt,name=ip,proto3" json:"ip,omitempty"` // the address this guest has, however its host came by it | 618 | Ip string `protobuf:"bytes,4,opt,name=ip,proto3" json:"ip,omitempty"` // the guest's address on its host's NAT underlay — every guest has one, from boot |
| 607 | LastError string `protobuf:"bytes,5,opt,name=last_error,json=lastError,proto3" json:"last_error,omitempty"` | 619 | LastError string `protobuf:"bytes,5,opt,name=last_error,json=lastError,proto3" json:"last_error,omitempty"` |
| 608 | // The guest's ed25519 HOST public key. It is generated on the host, and the | 620 | // The guest's ed25519 HOST public key. It is generated on the host, and the |
| 609 | // private half never leaves it — this is the only half that travels. Sent on | 621 | // private half never leaves it — this is the only half that travels. Sent on |
| @@ -618,7 +630,16 @@ type ActualVM struct { | |||
| 618 | // Free text for display, never parsed. Empty is the normal state of a settled | 630 | // Free text for display, never parsed. Empty is the normal state of a settled |
| 619 | // VM and means only "nothing to add": an agent that predates this field sends | 631 | // VM and means only "nothing to add": an agent that predates this field sends |
| 620 | // it never, and a console reading it reads exactly what it read before. | 632 | // it never, and a console reading it reads exactly what it read before. |
| 621 | StatusDetail string `protobuf:"bytes,7,opt,name=status_detail,json=statusDetail,proto3" json:"status_detail,omitempty"` | 633 | StatusDetail string `protobuf:"bytes,7,opt,name=status_detail,json=statusDetail,proto3" json:"status_detail,omitempty"` |
| 634 | // The address the site's DHCP server granted this guest on its SECOND NIC, | ||
| 635 | // the one attached to the named host network its spec asked for (see | ||
| 636 | // VMDesired.network). Empty for the guests that have no such NIC — the | ||
| 637 | // majority — and for one DHCP round-trip after a networked guest boots, | ||
| 638 | // because the host learns it by watching the exchange rather than granting | ||
| 639 | // it. Never a substitute for ip: that one is known before the guest is even | ||
| 640 | // running, which is why the gate path uses it and this field never appears | ||
| 641 | // there. | ||
| 642 | NetworkIp string `protobuf:"bytes,8,opt,name=network_ip,json=networkIp,proto3" json:"network_ip,omitempty"` | ||
| 622 | unknownFields protoimpl.UnknownFields | 643 | unknownFields protoimpl.UnknownFields |
| 623 | sizeCache protoimpl.SizeCache | 644 | sizeCache protoimpl.SizeCache |
| 624 | } | 645 | } |
| @@ -702,6 +723,13 @@ func (x *ActualVM) GetStatusDetail() string { | |||
| 702 | return "" | 723 | return "" |
| 703 | } | 724 | } |
| 704 | 725 | ||
| 726 | func (x *ActualVM) GetNetworkIp() string { | ||
| 727 | if x != nil { | ||
| 728 | return x.NetworkIp | ||
| 729 | } | ||
| 730 | return "" | ||
| 731 | } | ||
| 732 | |||
| 705 | type QuarantinedVM struct { | 733 | type QuarantinedVM struct { |
| 706 | state protoimpl.MessageState `protogen:"open.v1"` | 734 | state protoimpl.MessageState `protogen:"open.v1"` |
| 707 | VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` | 735 | VmId string `protobuf:"bytes,1,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` |
| @@ -936,8 +964,14 @@ type VMDesired struct { | |||
| 936 | // the agent could not tell "the gate is off, boot uncertified" from "your | 964 | // the agent could not tell "the gate is off, boot uncertified" from "your |
| 937 | // certificate has not come back yet" — both are an empty ssh_host_cert. | 965 | // certificate has not come back yet" — both are an empty ssh_host_cert. |
| 938 | HostCertRequired bool `protobuf:"varint,19,opt,name=host_cert_required,json=hostCertRequired,proto3" json:"host_cert_required,omitempty"` | 966 | HostCertRequired bool `protobuf:"varint,19,opt,name=host_cert_required,json=hostCertRequired,proto3" json:"host_cert_required,omitempty"` |
| 939 | unknownFields protoimpl.UnknownFields | 967 | // The named host network this guest attaches to, or "" for the host's |
| 940 | sizeCache protoimpl.SizeCache | 968 | // NAT'd private bridge (the default, and the only behavior agents before |
| 969 | // this field know). A name is only ever placed on a host that advertised | ||
| 970 | // it (see Hello.host_networks); an agent that has the name but no longer | ||
| 971 | // the configuration fails the VM legibly rather than silently NAT-ing it. | ||
| 972 | Network string `protobuf:"bytes,20,opt,name=network,proto3" json:"network,omitempty"` | ||
| 973 | unknownFields protoimpl.UnknownFields | ||
| 974 | sizeCache protoimpl.SizeCache | ||
| 941 | } | 975 | } |
| 942 | 976 | ||
| 943 | func (x *VMDesired) Reset() { | 977 | func (x *VMDesired) Reset() { |
| @@ -1075,6 +1109,13 @@ func (x *VMDesired) GetHostCertRequired() bool { | |||
| 1075 | return false | 1109 | return false |
| 1076 | } | 1110 | } |
| 1077 | 1111 | ||
| 1112 | func (x *VMDesired) GetNetwork() string { | ||
| 1113 | if x != nil { | ||
| 1114 | return x.Network | ||
| 1115 | } | ||
| 1116 | return "" | ||
| 1117 | } | ||
| 1118 | |||
| 1078 | type DesiredStateSnapshot struct { | 1119 | type DesiredStateSnapshot struct { |
| 1079 | state protoimpl.MessageState `protogen:"open.v1"` | 1120 | state protoimpl.MessageState `protogen:"open.v1"` |
| 1080 | Epoch uint64 `protobuf:"varint,1,opt,name=epoch,proto3" json:"epoch,omitempty"` // agents refuse epoch < highest seen | 1121 | Epoch uint64 `protobuf:"varint,1,opt,name=epoch,proto3" json:"epoch,omitempty"` // agents refuse epoch < highest seen |
| @@ -1671,7 +1712,7 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" + | |||
| 1671 | "\bsnapshot\x18\x01 \x01(\v2\x1e.eitri.v1.DesiredStateSnapshotH\x00R\bsnapshot\x12:\n" + | 1712 | "\bsnapshot\x18\x01 \x01(\v2\x1e.eitri.v1.DesiredStateSnapshotH\x00R\bsnapshot\x12:\n" + |
| 1672 | "\fconsole_open\x18\x02 \x01(\v2\x15.eitri.v1.ConsoleOpenH\x00R\vconsoleOpen\x12.\n" + | 1713 | "\fconsole_open\x18\x02 \x01(\v2\x15.eitri.v1.ConsoleOpenH\x00R\vconsoleOpen\x12.\n" + |
| 1673 | "\btcp_open\x18\x03 \x01(\v2\x11.eitri.v1.TCPOpenH\x00R\atcpOpenB\x05\n" + | 1714 | "\btcp_open\x18\x03 \x01(\v2\x11.eitri.v1.TCPOpenH\x00R\atcpOpenB\x05\n" + |
| 1674 | "\x03msg\"\xb8\x02\n" + | 1715 | "\x03msg\"\xdd\x02\n" + |
| 1675 | "\x05Hello\x12\x17\n" + | 1716 | "\x05Hello\x12\x17\n" + |
| 1676 | "\ahost_id\x18\x01 \x01(\tR\x06hostId\x12\x1a\n" + | 1717 | "\ahost_id\x18\x01 \x01(\tR\x06hostId\x12\x1a\n" + |
| 1677 | "\bhostname\x18\x02 \x01(\tR\bhostname\x12\x0e\n" + | 1718 | "\bhostname\x18\x02 \x01(\tR\bhostname\x12\x0e\n" + |
| @@ -1684,7 +1725,8 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" + | |||
| 1684 | "credential\x18\t \x01(\tR\n" + | 1725 | "credential\x18\t \x01(\tR\n" + |
| 1685 | "credential\x12)\n" + | 1726 | "credential\x12)\n" + |
| 1686 | "\x05facts\x18\n" + | 1727 | "\x05facts\x18\n" + |
| 1687 | " \x01(\v2\x13.eitri.v1.HostFactsR\x05factsJ\x04\b\x06\x10\aR\vbridge_cidr\"P\n" + | 1728 | " \x01(\v2\x13.eitri.v1.HostFactsR\x05facts\x12#\n" + |
| 1729 | "\rhost_networks\x18\v \x03(\tR\fhostNetworksJ\x04\b\x06\x10\aR\vbridge_cidr\"P\n" + | ||
| 1688 | "\bCapacity\x12\x14\n" + | 1730 | "\bCapacity\x12\x14\n" + |
| 1689 | "\x05vcpus\x18\x01 \x01(\x03R\x05vcpus\x12\x15\n" + | 1731 | "\x05vcpus\x18\x01 \x01(\x03R\x05vcpus\x12\x15\n" + |
| 1690 | "\x06mem_mb\x18\x02 \x01(\x03R\x05memMb\x12\x17\n" + | 1732 | "\x06mem_mb\x18\x02 \x01(\x03R\x05memMb\x12\x17\n" + |
| @@ -1708,7 +1750,7 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" + | |||
| 1708 | "\fdisk_used_gb\x18\a \x01(\x03R\n" + | 1750 | "\fdisk_used_gb\x18\a \x01(\x03R\n" + |
| 1709 | "diskUsedGb\x12 \n" + | 1751 | "diskUsedGb\x12 \n" + |
| 1710 | "\fdisk_free_gb\x18\b \x01(\x03R\n" + | 1752 | "\fdisk_free_gb\x18\b \x01(\x03R\n" + |
| 1711 | "diskFreeGb\"\xc7\x01\n" + | 1753 | "diskFreeGb\"\xe6\x01\n" + |
| 1712 | "\bActualVM\x12\x13\n" + | 1754 | "\bActualVM\x12\x13\n" + |
| 1713 | "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x14\n" + | 1755 | "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x14\n" + |
| 1714 | "\x05power\x18\x02 \x01(\tR\x05power\x12\x14\n" + | 1756 | "\x05power\x18\x02 \x01(\tR\x05power\x12\x14\n" + |
| @@ -1717,7 +1759,9 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" + | |||
| 1717 | "\n" + | 1759 | "\n" + |
| 1718 | "last_error\x18\x05 \x01(\tR\tlastError\x12&\n" + | 1760 | "last_error\x18\x05 \x01(\tR\tlastError\x12&\n" + |
| 1719 | "\x0fssh_host_pubkey\x18\x06 \x01(\tR\rsshHostPubkey\x12#\n" + | 1761 | "\x0fssh_host_pubkey\x18\x06 \x01(\tR\rsshHostPubkey\x12#\n" + |
| 1720 | "\rstatus_detail\x18\a \x01(\tR\fstatusDetail\"\x81\x01\n" + | 1762 | "\rstatus_detail\x18\a \x01(\tR\fstatusDetail\x12\x1d\n" + |
| 1763 | "\n" + | ||
| 1764 | "network_ip\x18\b \x01(\tR\tnetworkIp\"\x81\x01\n" + | ||
| 1721 | "\rQuarantinedVM\x12\x13\n" + | 1765 | "\rQuarantinedVM\x12\x13\n" + |
| 1722 | "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" + | 1766 | "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" + |
| 1723 | "\x04name\x18\x02 \x01(\tR\x04name\x12\x1f\n" + | 1767 | "\x04name\x18\x02 \x01(\tR\x04name\x12\x1f\n" + |
| @@ -1736,7 +1780,7 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" + | |||
| 1736 | "guest_cidr\x18\b \x01(\tR\tguestCidr\x126\n" + | 1780 | "guest_cidr\x18\b \x01(\tR\tguestCidr\x126\n" + |
| 1737 | "\texposures\x18\t \x03(\v2\x18.eitri.v1.ExposureActualR\texposures\x12(\n" + | 1781 | "\texposures\x18\t \x03(\v2\x18.eitri.v1.ExposureActualR\texposures\x12(\n" + |
| 1738 | "\x10host_uplink_addr\x18\n" + | 1782 | "\x10host_uplink_addr\x18\n" + |
| 1739 | " \x01(\tR\x0ehostUplinkAddr\"\xa2\x04\n" + | 1783 | " \x01(\tR\x0ehostUplinkAddr\"\xbc\x04\n" + |
| 1740 | "\tVMDesired\x12\x13\n" + | 1784 | "\tVMDesired\x12\x13\n" + |
| 1741 | "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" + | 1785 | "\x05vm_id\x18\x01 \x01(\tR\x04vmId\x12\x12\n" + |
| 1742 | "\x04name\x18\x02 \x01(\tR\x04name\x12\x1b\n" + | 1786 | "\x04name\x18\x02 \x01(\tR\x04name\x12\x1b\n" + |
| @@ -1759,7 +1803,8 @@ const file_proto_eitri_v1_sync_proto_rawDesc = "" + | |||
| 1759 | "\x12ssh_authorized_key\x18\f \x01(\tR\x10sshAuthorizedKey\x12\"\n" + | 1803 | "\x12ssh_authorized_key\x18\f \x01(\tR\x10sshAuthorizedKey\x12\"\n" + |
| 1760 | "\rssh_host_cert\x18\x11 \x01(\tR\vsshHostCert\x12<\n" + | 1804 | "\rssh_host_cert\x18\x11 \x01(\tR\vsshHostCert\x12<\n" + |
| 1761 | "\x1bssh_user_ca_authorized_keys\x18\x12 \x03(\tR\x17sshUserCaAuthorizedKeys\x12,\n" + | 1805 | "\x1bssh_user_ca_authorized_keys\x18\x12 \x03(\tR\x17sshUserCaAuthorizedKeys\x12,\n" + |
| 1762 | "\x12host_cert_required\x18\x13 \x01(\bR\x10hostCertRequiredJ\x04\b\r\x10\x0eJ\x04\b\x0e\x10\x0fJ\x04\b\x0f\x10\x10J\x04\b\x10\x10\x11R\x10ssh_host_key_pem\"\xc9\x01\n" + | 1806 | "\x12host_cert_required\x18\x13 \x01(\bR\x10hostCertRequired\x12\x18\n" + |
| 1807 | "\anetwork\x18\x14 \x01(\tR\anetworkJ\x04\b\r\x10\x0eJ\x04\b\x0e\x10\x0fJ\x04\b\x0f\x10\x10J\x04\b\x10\x10\x11R\x10ssh_host_key_pem\"\xc9\x01\n" + | ||
| 1763 | "\x14DesiredStateSnapshot\x12\x14\n" + | 1808 | "\x14DesiredStateSnapshot\x12\x14\n" + |
| 1764 | "\x05epoch\x18\x01 \x01(\x04R\x05epoch\x12%\n" + | 1809 | "\x05epoch\x18\x01 \x01(\x04R\x05epoch\x12%\n" + |
| 1765 | "\x03vms\x18\x02 \x03(\v2\x13.eitri.v1.VMDesiredR\x03vms\x12;\n" + | 1810 | "\x03vms\x18\x02 \x03(\v2\x13.eitri.v1.VMDesiredR\x03vms\x12;\n" + |
internal/server/api/api.go
| Old | New | ||
|---|---|---|---|
| @@ -11,6 +11,7 @@ import ( | |||
| 11 | "fmt" | 11 | "fmt" |
| 12 | "log/slog" | 12 | "log/slog" |
| 13 | "net/http" | 13 | "net/http" |
| 14 | "slices" | ||
| 14 | "strings" | 15 | "strings" |
| 15 | "time" | 16 | "time" |
| 16 | 17 | ||
| @@ -429,12 +430,24 @@ func toHostResponse(h store.Host, st registry.HostState, ok bool, alloc store.Al | |||
| 429 | CPUModel: h.CPUModel, | 430 | CPUModel: h.CPUModel, |
| 430 | Virt: h.Virt, | 431 | Virt: h.Virt, |
| 431 | UplinkAddr: h.UplinkAddr, | 432 | UplinkAddr: h.UplinkAddr, |
| 433 | // A host that has advertised nothing serves an empty list, never JSON | ||
| 434 | // null: the field is declared non-nullable, and "no networks" is the | ||
| 435 | // same answer whether the host said so or has not spoken at all. | ||
| 436 | HostNetworks: []string{}, | ||
| 432 | } | 437 | } |
| 433 | if ok { | 438 | if ok { |
| 434 | hr.Online = st.Online | 439 | hr.Online = st.Online |
| 435 | hr.Stale = st.Stale | 440 | hr.Stale = st.Stale |
| 436 | hr.Sessions = st.Sessions | 441 | hr.Sessions = st.Sessions |
| 437 | hr.AgentVersion = st.AgentVersion | 442 | hr.AgentVersion = st.AgentVersion |
| 443 | // Registry-held like AgentVersion, and for the same reason it is gated | ||
| 444 | // on `ok`: only a host that has connected has said which networks it | ||
| 445 | // serves, and a host this server has not heard from advertises nothing | ||
| 446 | // rather than whatever it once did. The nil check keeps the empty list | ||
| 447 | // above: a connected host that named no networks still serves []. | ||
| 448 | if st.HostNetworks != nil { | ||
| 449 | hr.HostNetworks = st.HostNetworks | ||
| 450 | } | ||
| 438 | // LastSeen unset ⇒ connected but never reported: leave the age fields | 451 | // LastSeen unset ⇒ connected but never reported: leave the age fields |
| 439 | // null rather than emit a bogus "last seen at the zero time". | 452 | // null rather than emit a bogus "last seen at the zero time". |
| 440 | if !st.LastSeen.IsZero() { | 453 | if !st.LastSeen.IsZero() { |
| @@ -577,6 +590,8 @@ func toVMResponse(vm store.VM, actualPower, phase, statusDetail string, destroyA | |||
| 577 | Status: vm.Status, | 590 | Status: vm.Status, |
| 578 | LastError: vm.LastError, | 591 | LastError: vm.LastError, |
| 579 | AssignedIP: vm.AssignedIP, | 592 | AssignedIP: vm.AssignedIP, |
| 593 | Network: vm.Network, | ||
| 594 | NetworkIP: vm.NetworkIP, | ||
| 580 | CreatedAt: vm.CreatedAt, | 595 | CreatedAt: vm.CreatedAt, |
| 581 | Deleted: vm.DeletedAt != nil, | 596 | Deleted: vm.DeletedAt != nil, |
| 582 | ActualPower: actualPower, | 597 | ActualPower: actualPower, |
| @@ -740,6 +755,13 @@ func validateCreateVM(req *types.CreateVMRequest) (string, int) { | |||
| 740 | if req.VCPUs < 1 || req.MemMB < 1 || req.DiskGB < 1 { | 755 | if req.VCPUs < 1 || req.MemMB < 1 || req.DiskGB < 1 { |
| 741 | return "vcpus, mem_mb and disk_gb must each be >= 1", http.StatusBadRequest | 756 | return "vcpus, mem_mb and disk_gb must each be >= 1", http.StatusBadRequest |
| 742 | } | 757 | } |
| 758 | // A network is optional (empty is the NAT underlay), but a name that cannot | ||
| 759 | // be a network name is malformed here rather than a refusal later — the | ||
| 760 | // grammar is names.IsNetworkName, the same one the agent's --host-network | ||
| 761 | // parser refuses with, so a name the API accepts is one a host could serve. | ||
| 762 | if req.Network != "" && !names.IsNetworkName(req.Network) { | ||
| 763 | return "invalid network", http.StatusBadRequest | ||
| 764 | } | ||
| 743 | return "", 0 | 765 | return "", 0 |
| 744 | } | 766 | } |
| 745 | 767 | ||
| @@ -882,6 +904,27 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) { | |||
| 882 | } | 904 | } |
| 883 | } | 905 | } |
| 884 | 906 | ||
| 907 | // Named-network precondition, the fourth refusal of this shape: the VM | ||
| 908 | // asked for a network, and only the agent's own configuration can say | ||
| 909 | // whether this host serves it. Advertisement arrives in the Hello, so — | ||
| 910 | // unlike the offline-tolerant refusals above — silence refuses: an agent | ||
| 911 | // that predates the field advertises nothing and would silently NAT the | ||
| 912 | // guest, which is the one outcome the spec forbids. A VM that asked for | ||
| 913 | // the LAN either gets it or does not exist. | ||
| 914 | // | ||
| 915 | // hostHasSpoken already folds in Online (see above): the server trusts no | ||
| 916 | // Hello-derived fact, including a network name the host may have just | ||
| 917 | // advertised, until the host is Online. A freshly-connected host can | ||
| 918 | // therefore see this refusal fire for one report cadence even though it | ||
| 919 | // did advertise the network — self-healing once the next report lands, | ||
| 920 | // and uniform with the sibling refusals this one sits beside. | ||
| 921 | if req.Network != "" { | ||
| 922 | if !hostHasSpoken || !slices.Contains(hostState.HostNetworks, req.Network) { | ||
| 923 | http.Error(w, noNetworkRefusal(host.Name, req.HostID, req.Network, host.OS, hostState.HostNetworks, hostHasSpoken), http.StatusConflict) | ||
| 924 | return | ||
| 925 | } | ||
| 926 | } | ||
| 927 | |||
| 885 | // Install the SSH key into user-supplied cloud-init. When only one of the | 928 | // Install the SSH key into user-supplied cloud-init. When only one of the |
| 886 | // two is set the seed builder handles it (verbatim user-data, or the | 929 | // two is set the seed builder handles it (verbatim user-data, or the |
| 887 | // generated default template); it's the BOTH case that used to silently | 930 | // generated default template); it's the BOTH case that used to silently |
| @@ -937,6 +980,7 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) { | |||
| 937 | MemMB: req.MemMB, | 980 | MemMB: req.MemMB, |
| 938 | DiskGB: req.DiskGB, | 981 | DiskGB: req.DiskGB, |
| 939 | PowerState: req.PowerState, | 982 | PowerState: req.PowerState, |
| 983 | Network: req.Network, | ||
| 940 | } | 984 | } |
| 941 | 985 | ||
| 942 | // The row carries no host key. A guest's host key is generated by the host | 986 | // The row carries no host key. A guest's host key is generated by the host |
internal/server/api/api_test.go
| Old | New | ||
|---|---|---|---|
| @@ -80,7 +80,7 @@ func TestResponseJSONKeysAreSnakeCase(t *testing.T) { | |||
| 80 | for _, k := range []string{ | 80 | for _, k := range []string{ |
| 81 | "id", "host_id", "name", "image_url", "vcpus", "mem_mb", "disk_gb", | 81 | "id", "host_id", "name", "image_url", "vcpus", "mem_mb", "disk_gb", |
| 82 | "power_state", "status", "status_detail", "last_error", "assigned_ip", | 82 | "power_state", "status", "status_detail", "last_error", "assigned_ip", |
| 83 | "created_at", "deleted", "actual_power", "phase", | 83 | "network", "network_ip", "created_at", "deleted", "actual_power", "phase", |
| 84 | "destroy_at", "lifecycle", "trusted_cas", "injected_key", | 84 | "destroy_at", "lifecycle", "trusted_cas", "injected_key", |
| 85 | } { | 85 | } { |
| 86 | assert.Contains(t, v, k, "vm response must contain key %q", k) | 86 | assert.Contains(t, v, k, "vm response must contain key %q", k) |
| @@ -397,12 +397,20 @@ func enroll(t *testing.T, ts *httptest.Server) map[string]string { | |||
| 397 | // host in the fleet for a moment after the server restarts. | 397 | // host in the fleet for a moment after the server restarts. |
| 398 | func enrollSilent(t *testing.T, ts *httptest.Server) map[string]string { | 398 | func enrollSilent(t *testing.T, ts *httptest.Server) map[string]string { |
| 399 | t.Helper() | 399 | t.Helper() |
| 400 | return enrollSilentOS(t, ts, "linux") | ||
| 401 | } | ||
| 402 | |||
| 403 | // enrollSilentOS is enrollSilent with the host's reported OS as a parameter, | ||
| 404 | // for tests that need a host whose OS itself is the refusal — a Mac cannot | ||
| 405 | // serve a named network no matter what it later advertises. | ||
| 406 | func enrollSilentOS(t *testing.T, ts *httptest.Server, os string) map[string]string { | ||
| 407 | t.Helper() | ||
| 400 | resp := do(t, "POST", ts.URL+"/api/v1/enroll-tokens", testPAT, nil) | 408 | resp := do(t, "POST", ts.URL+"/api/v1/enroll-tokens", testPAT, nil) |
| 401 | require.Equal(t, 201, resp.StatusCode) | 409 | require.Equal(t, 201, resp.StatusCode) |
| 402 | var tok map[string]string | 410 | var tok map[string]string |
| 403 | json.NewDecoder(resp.Body).Decode(&tok) | 411 | json.NewDecoder(resp.Body).Decode(&tok) |
| 404 | resp = do(t, "POST", ts.URL+"/api/v1/enroll", "", map[string]string{ | 412 | resp = do(t, "POST", ts.URL+"/api/v1/enroll", "", map[string]string{ |
| 405 | "token": tok["token"], "name": "host-a", "os": "linux", "arch": "amd64", "provisioner": "cloudhv"}) | 413 | "token": tok["token"], "name": "host-a", "os": os, "arch": "amd64", "provisioner": "cloudhv"}) |
| 406 | require.Equal(t, 201, resp.StatusCode) | 414 | require.Equal(t, 201, resp.StatusCode) |
| 407 | var out map[string]string | 415 | var out map[string]string |
| 408 | json.NewDecoder(resp.Body).Decode(&out) | 416 | json.NewDecoder(resp.Body).Decode(&out) |
internal/server/api/network_api_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,235 @@ | |||
| 1 | package api | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "encoding/json" | ||
| 5 | "net/http" | ||
| 6 | "net/http/httptest" | ||
| 7 | "testing" | ||
| 8 | |||
| 9 | "github.com/stretchr/testify/assert" | ||
| 10 | "github.com/stretchr/testify/require" | ||
| 11 | ) | ||
| 12 | |||
| 13 | // agentAdvertisesNetworks is the Hello of an agent started with --host-network: | ||
| 14 | // the names it will serve, held in the registry beside its version. Followed by | ||
| 15 | // the first report, because only a host that is online has told this server | ||
| 16 | // anything — the same predicate the capacity refusal is judged on. | ||
| 17 | func agentAdvertisesNetworks(hostID string, networks ...string) { | ||
| 18 | testReg.SetHostNetworks(hostID, networks) | ||
| 19 | agentReports(hostID) | ||
| 20 | } | ||
| 21 | |||
| 22 | // createVMOnNetwork posts a create asking for a named network, so a test reads | ||
| 23 | // as the placement it is really making. | ||
| 24 | func createVMOnNetwork(t *testing.T, ts *httptest.Server, hostID, name, network string) *http.Response { | ||
| 25 | t.Helper() | ||
| 26 | return do(t, "POST", ts.URL+"/api/v1/vms", testPAT, map[string]any{ | ||
| 27 | "host_id": hostID, "name": name, "network": network, | ||
| 28 | }) | ||
| 29 | } | ||
| 30 | |||
| 31 | // TestCreateVMBridgedRefusedWhenHostSilent is where this refusal parts company | ||
| 32 | // with its siblings. Capacity and the certified-host-key check tolerate a host | ||
| 33 | // that has not spoken, because desired state is allowed to run ahead of a host | ||
| 34 | // that is merely offline. A network cannot: advertisement arrives only in the | ||
| 35 | // Hello, so silence is indistinguishable from an agent that predates the field | ||
| 36 | // — and that agent would put the guest on NAT without a word, which is the one | ||
| 37 | // outcome this design forbids. | ||
| 38 | func TestCreateVMBridgedRefusedWhenHostSilent(t *testing.T) { | ||
| 39 | ts, st, _ := testServer(t) | ||
| 40 | out := enroll(t, ts) // a Hello and nothing since: online is false | ||
| 41 | |||
| 42 | resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan") | ||
| 43 | require.Equal(t, 409, resp.StatusCode) | ||
| 44 | msg := bodyOf(t, resp) | ||
| 45 | assert.Contains(t, msg, `network "lan"`, "the refusal must name the network asked for") | ||
| 46 | assert.Contains(t, msg, `host "host-a" (`+out["host_id"]+")", "and the host it was asked of") | ||
| 47 | assert.Contains(t, msg, "it is not currently reporting, so its networks cannot be confirmed", | ||
| 48 | "a silent host's networks are unknown, not absent") | ||
| 49 | assert.Contains(t, msg, "create without a network", "the refusal must name a way out") | ||
| 50 | |||
| 51 | vms, err := st.ListVMs() | ||
| 52 | require.NoError(t, err) | ||
| 53 | assert.Empty(t, vms, "a refused create must leave no row behind") | ||
| 54 | } | ||
| 55 | |||
| 56 | // TestCreateVMBridgedRefusedWhenNotAdvertised: the host is right there and | ||
| 57 | // serving, and what it serves is not this. Both shapes of "no" have to be | ||
| 58 | // actionable — an agent configured with no networks at all needs the flag, one | ||
| 59 | // serving other names needs to be told which. | ||
| 60 | func TestCreateVMBridgedRefusedWhenNotAdvertised(t *testing.T) { | ||
| 61 | t.Run("advertises nothing", func(t *testing.T) { | ||
| 62 | ts, st, _ := testServer(t) | ||
| 63 | out := enroll(t, ts) | ||
| 64 | agentAdvertisesNetworks(out["host_id"]) | ||
| 65 | |||
| 66 | resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan") | ||
| 67 | require.Equal(t, 409, resp.StatusCode) | ||
| 68 | msg := bodyOf(t, resp) | ||
| 69 | assert.Contains(t, msg, `network "lan"`) | ||
| 70 | assert.Contains(t, msg, "configured with no named networks") | ||
| 71 | assert.Contains(t, msg, "--host-network lan=<bridge>", "the remedy is the flag that would serve it") | ||
| 72 | |||
| 73 | vms, err := st.ListVMs() | ||
| 74 | require.NoError(t, err) | ||
| 75 | assert.Empty(t, vms) | ||
| 76 | }) | ||
| 77 | |||
| 78 | t.Run("advertises another name", func(t *testing.T) { | ||
| 79 | ts, _, _ := testServer(t) | ||
| 80 | out := enroll(t, ts) | ||
| 81 | agentAdvertisesNetworks(out["host_id"], "lab", "dmz") | ||
| 82 | |||
| 83 | resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan") | ||
| 84 | require.Equal(t, 409, resp.StatusCode) | ||
| 85 | msg := bodyOf(t, resp) | ||
| 86 | assert.Contains(t, msg, `network "lan"`) | ||
| 87 | assert.Contains(t, msg, "it advertises lab, dmz", "what the host does serve is half the answer") | ||
| 88 | assert.Contains(t, msg, "--host-network lan=<bridge>") | ||
| 89 | }) | ||
| 90 | } | ||
| 91 | |||
| 92 | // TestCreateVMBridgedRefusedOnIncapableOS is the refusal for a host whose OS | ||
| 93 | // cannot serve a named network at all, distinct from a host that merely isn't | ||
| 94 | // advertising one: no --host-network remedy exists for a Mac, so the message | ||
| 95 | // must not offer it, and must instead point at what does exist — a Linux | ||
| 96 | // host. This fires even before the host has spoken (see noNetworkRefusal's | ||
| 97 | // os parameter), because an OS, once enrolled, does not change out from under | ||
| 98 | // a host the way its advertised networks do. | ||
| 99 | func TestCreateVMBridgedRefusedOnIncapableOS(t *testing.T) { | ||
| 100 | ts, st, _ := testServer(t) | ||
| 101 | out := enrollSilentOS(t, ts, "darwin") | ||
| 102 | |||
| 103 | resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan") | ||
| 104 | require.Equal(t, 409, resp.StatusCode) | ||
| 105 | msg := bodyOf(t, resp) | ||
| 106 | assert.Contains(t, msg, `network "lan"`) | ||
| 107 | assert.Contains(t, msg, "darwin", "the refusal must name the incapable OS") | ||
| 108 | assert.Contains(t, msg, "Linux host", "the remedy is a different host, not a flag") | ||
| 109 | assert.NotContains(t, msg, "--host-network", "no flag can make a Mac serve a named network") | ||
| 110 | |||
| 111 | vms, err := st.ListVMs() | ||
| 112 | require.NoError(t, err) | ||
| 113 | assert.Empty(t, vms, "a refused create must leave no row behind") | ||
| 114 | } | ||
| 115 | |||
| 116 | // TestCreateVMBridgedNotRefusedOnIncapableOSWhenOSUnknown pins the empty-os | ||
| 117 | // fallthrough: an enroll request that omitted the OS field is not the same | ||
| 118 | // claim as a host known to be incapable, so it must get the ordinary | ||
| 119 | // not-currently-reporting text, not the OS-incapable one. | ||
| 120 | func TestCreateVMBridgedNotRefusedOnIncapableOSWhenOSUnknown(t *testing.T) { | ||
| 121 | ts, _, _ := testServer(t) | ||
| 122 | out := enrollSilentOS(t, ts, "") | ||
| 123 | |||
| 124 | resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan") | ||
| 125 | require.Equal(t, 409, resp.StatusCode) | ||
| 126 | msg := bodyOf(t, resp) | ||
| 127 | assert.Contains(t, msg, "it is not currently reporting", "unknown OS must fall through to the silent-host text") | ||
| 128 | assert.NotContains(t, msg, "cannot serve named networks", "an unset OS is not a known-incapable one") | ||
| 129 | } | ||
| 130 | |||
| 131 | // TestCreateVMBridgedAcceptedWhenAdvertised is the accepting half: a name the | ||
| 132 | // host advertises is placed, and the name lands on the row — the record the | ||
| 133 | // snapshot reads to tell the agent which underlay to build. | ||
| 134 | func TestCreateVMBridgedAcceptedWhenAdvertised(t *testing.T) { | ||
| 135 | ts, st, _ := testServer(t) | ||
| 136 | out := enroll(t, ts) | ||
| 137 | agentAdvertisesNetworks(out["host_id"], "lan") | ||
| 138 | |||
| 139 | resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", "lan") | ||
| 140 | require.Equal(t, 201, resp.StatusCode) | ||
| 141 | var created map[string]string | ||
| 142 | require.NoError(t, json.NewDecoder(resp.Body).Decode(&created)) | ||
| 143 | |||
| 144 | vm, err := st.GetVM(created["id"]) | ||
| 145 | require.NoError(t, err) | ||
| 146 | assert.Equal(t, "lan", vm.Network) | ||
| 147 | } | ||
| 148 | |||
| 149 | // TestCreateVMWithoutANetworkIsNeverJudged: the NAT default is what nearly | ||
| 150 | // every create asks for, and admission must not have made it conditional on a | ||
| 151 | // host saying anything. A silent host still takes a NAT create. | ||
| 152 | func TestCreateVMWithoutANetworkIsNeverJudged(t *testing.T) { | ||
| 153 | ts, st, _ := testServer(t) | ||
| 154 | out := enrollSilent(t, ts) | ||
| 155 | |||
| 156 | resp := do(t, "POST", ts.URL+"/api/v1/vms", testPAT, map[string]any{ | ||
| 157 | "host_id": out["host_id"], "name": "natted", | ||
| 158 | }) | ||
| 159 | require.Equal(t, 201, resp.StatusCode) | ||
| 160 | var created map[string]string | ||
| 161 | require.NoError(t, json.NewDecoder(resp.Body).Decode(&created)) | ||
| 162 | |||
| 163 | vm, err := st.GetVM(created["id"]) | ||
| 164 | require.NoError(t, err) | ||
| 165 | assert.Equal(t, "", vm.Network, "an unnamed network is the NAT underlay") | ||
| 166 | } | ||
| 167 | |||
| 168 | // TestCreateVMInvalidNetworkName: a name that cannot be a network name is a | ||
| 169 | // malformed request, judged before any host is consulted — the grammar lives in | ||
| 170 | // names.IsNetworkName, which the agent's flag parser refuses with too. | ||
| 171 | func TestCreateVMInvalidNetworkName(t *testing.T) { | ||
| 172 | ts, _, _ := testServer(t) | ||
| 173 | out := enroll(t, ts) | ||
| 174 | agentAdvertisesNetworks(out["host_id"], "lan") | ||
| 175 | |||
| 176 | for _, bad := range []string{"NAT!", "nat", "-lan", "lan_1"} { | ||
| 177 | resp := createVMOnNetwork(t, ts, out["host_id"], "bridged", bad) | ||
| 178 | require.Equal(t, 400, resp.StatusCode, "network %q must be refused as malformed", bad) | ||
| 179 | assert.Contains(t, bodyOf(t, resp), "invalid network") | ||
| 180 | } | ||
| 181 | } | ||
| 182 | |||
| 183 | // TestHostViewServesAdvertisedNetworks: the console cannot offer a network the | ||
| 184 | // operator did not configure, so what a host advertises has to reach the host | ||
| 185 | // view — and a host that has said nothing must serve an empty set rather than | ||
| 186 | // the last thing it said. | ||
| 187 | func TestHostViewServesAdvertisedNetworks(t *testing.T) { | ||
| 188 | ts, _, _ := testServer(t) | ||
| 189 | out := enroll(t, ts) | ||
| 190 | agentAdvertisesNetworks(out["host_id"], "lan", "lab") | ||
| 191 | |||
| 192 | resp := do(t, "GET", ts.URL+"/api/v1/hosts", testPAT, nil) | ||
| 193 | require.Equal(t, 200, resp.StatusCode) | ||
| 194 | var hosts []struct { | ||
| 195 | ID string `json:"id"` | ||
| 196 | HostNetworks []string `json:"host_networks"` | ||
| 197 | } | ||
| 198 | require.NoError(t, json.NewDecoder(resp.Body).Decode(&hosts)) | ||
| 199 | require.Len(t, hosts, 1) | ||
| 200 | assert.Equal(t, out["host_id"], hosts[0].ID) | ||
| 201 | assert.Equal(t, []string{"lan", "lab"}, hosts[0].HostNetworks) | ||
| 202 | } | ||
| 203 | |||
| 204 | // TestHostViewServesEmptyNetworksNotNull asserts on the response bytes rather | ||
| 205 | // than a decoded struct because the consumer is TypeScript: the contract is the | ||
| 206 | // literal JSON the browser receives, so that is where it has to be checked. | ||
| 207 | // host_networks is declared non-nullable and the console indexes it without a | ||
| 208 | // guard, so a host with nothing to advertise has to serve [] — the empty answer | ||
| 209 | // — and never null. Both silences produce it: a host that has never connected, | ||
| 210 | // and a connected host started without a single --host-network. | ||
| 211 | func TestHostViewServesEmptyNetworksNotNull(t *testing.T) { | ||
| 212 | for _, tc := range []struct { | ||
| 213 | name string | ||
| 214 | enrolls func(t *testing.T, ts *httptest.Server) | ||
| 215 | }{ | ||
| 216 | {"never connected", func(t *testing.T, ts *httptest.Server) { | ||
| 217 | enrollSilent(t, ts) | ||
| 218 | }}, | ||
| 219 | {"connected, advertising nothing", func(t *testing.T, ts *httptest.Server) { | ||
| 220 | out := enroll(t, ts) | ||
| 221 | agentAdvertisesNetworks(out["host_id"]) | ||
| 222 | }}, | ||
| 223 | } { | ||
| 224 | t.Run(tc.name, func(t *testing.T) { | ||
| 225 | ts, _, _ := testServer(t) | ||
| 226 | tc.enrolls(t, ts) | ||
| 227 | |||
| 228 | resp := do(t, "GET", ts.URL+"/api/v1/hosts", testPAT, nil) | ||
| 229 | require.Equal(t, 200, resp.StatusCode) | ||
| 230 | body := bodyOf(t, resp) | ||
| 231 | assert.Contains(t, body, `"host_networks":[]`, "the empty set is a list, not a null") | ||
| 232 | assert.NotContains(t, body, `"host_networks":null`) | ||
| 233 | }) | ||
| 234 | } | ||
| 235 | } | ||
internal/server/api/networks.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,51 @@ | |||
| 1 | package api | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "fmt" | ||
| 5 | "strings" | ||
| 6 | |||
| 7 | "github.com/a73x/eitri/internal/names" | ||
| 8 | ) | ||
| 9 | |||
| 10 | // noNetworkRefusal explains a create naming a network its host is not | ||
| 11 | // advertising, with the remedy in the operator's vocabulary. Which remedy | ||
| 12 | // depends on what the host said: its OS cannot serve one at all (name a | ||
| 13 | // Linux host instead), nothing at all (wait for it), no networks (configure | ||
| 14 | // one), or other names (pick one, or configure this one). Every branch ends | ||
| 15 | // at the same escape hatch — the NAT underlay is always there. | ||
| 16 | // | ||
| 17 | // os is store.Host.OS, an enrollment-time fact: it is written once, at | ||
| 18 | // RedeemEnrollmentToken, from the joining agent's runtime.GOOS, and | ||
| 19 | // UpdateHostFacts (the per-Hello refresh) deliberately never touches it — so | ||
| 20 | // it needs no spoken/hostHasSpoken gate the way advertised networks do, a | ||
| 21 | // kernel does not change OS between Hellos. An empty os is not a host that | ||
| 22 | // has gone quiet; it is an enroll request that omitted the field (handleEnroll | ||
| 23 | // passes req.OS through unvalidated). That falls through to the existing | ||
| 24 | // branches rather than being told it cannot serve anything: unknown is not | ||
| 25 | // the same claim as known-incapable, and only the latter is worth | ||
| 26 | // interrupting the operator over. | ||
| 27 | // | ||
| 28 | // The comparison stays un-normalised on purpose: os is trusted to be a | ||
| 29 | // literal runtime.GOOS string, because the only thing that ever writes it is | ||
| 30 | // the agent's own enroll request, and every real agent sends exactly what | ||
| 31 | // runtime.GOOS gives it. A malformed value here (say "Linux", capitalized) | ||
| 32 | // would be a different failure than the ones this function is for — an | ||
| 33 | // enroller lying about its platform — and OSServesNamedNetworks would refuse | ||
| 34 | // it exactly like an unsupported OS: permanently, with no flag able to fix | ||
| 35 | // it. That is the right failure mode for a value nothing legitimate can | ||
| 36 | // produce, so it gets no special-cased remedy of its own. | ||
| 37 | func noNetworkRefusal(hostName, hostID, network, os string, advertised []string, spoken bool) string { | ||
| 38 | var b strings.Builder | ||
| 39 | fmt.Fprintf(&b, "host %q (%s) is not advertising network %q", hostName, hostID, network) | ||
| 40 | switch { | ||
| 41 | case os != "" && !names.OSServesNamedNetworks(os): | ||
| 42 | fmt.Fprintf(&b, ": its OS (%s) cannot serve named networks. Place bridged guests on a Linux host, or create without a network for the NAT underlay.", os) | ||
| 43 | case !spoken: | ||
| 44 | b.WriteString(": it is not currently reporting, so its networks cannot be confirmed. Wait for the host to come online, or create without a network for the NAT underlay.") | ||
| 45 | case len(advertised) == 0: | ||
| 46 | b.WriteString(": its agent is configured with no named networks. Restart the agent with --host-network " + network + "=<bridge>, or create without a network for the NAT underlay.") | ||
| 47 | default: | ||
| 48 | fmt.Fprintf(&b, ": it advertises %s. Pick one of those, restart the agent with --host-network %s=<bridge>, or create without a network for the NAT underlay.", strings.Join(advertised, ", "), network) | ||
| 49 | } | ||
| 50 | return b.String() | ||
| 51 | } | ||
internal/server/api/testdata/create-vm-request.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -8,5 +8,6 @@ | |||
| 8 | "power_state": "running", | 8 | "power_state": "running", |
| 9 | "vcpus": 4, | 9 | "vcpus": 4, |
| 10 | "mem_mb": 4096, | 10 | "mem_mb": 4096, |
| 11 | "disk_gb": 20 | 11 | "disk_gb": 20, |
| 12 | "network": "lan" | ||
| 12 | } | 13 | } |
internal/server/api/testdata/host.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -35,6 +35,10 @@ | |||
| 35 | "cpu_model": "AMD Ryzen 9 7950X", | 35 | "cpu_model": "AMD Ryzen 9 7950X", |
| 36 | "virt": "kvm", | 36 | "virt": "kvm", |
| 37 | "uplink_addr": "192.168.0.190", | 37 | "uplink_addr": "192.168.0.190", |
| 38 | "host_networks": [ | ||
| 39 | "lan", | ||
| 40 | "lab" | ||
| 41 | ], | ||
| 38 | "metrics": { | 42 | "metrics": { |
| 39 | "uptime_s": 86400, | 43 | "uptime_s": 86400, |
| 40 | "mem_used_mb": 12000, | 44 | "mem_used_mb": 12000, |
internal/server/api/testdata/snapshot.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -37,6 +37,10 @@ | |||
| 37 | "cpu_model": "AMD Ryzen 9 7950X", | 37 | "cpu_model": "AMD Ryzen 9 7950X", |
| 38 | "virt": "kvm", | 38 | "virt": "kvm", |
| 39 | "uplink_addr": "192.168.0.190", | 39 | "uplink_addr": "192.168.0.190", |
| 40 | "host_networks": [ | ||
| 41 | "lan", | ||
| 42 | "lab" | ||
| 43 | ], | ||
| 40 | "metrics": { | 44 | "metrics": { |
| 41 | "uptime_s": 86400, | 45 | "uptime_s": 86400, |
| 42 | "mem_used_mb": 12000, | 46 | "mem_used_mb": 12000, |
| @@ -62,6 +66,8 @@ | |||
| 62 | "status": "ready", | 66 | "status": "ready", |
| 63 | "last_error": "boot timeout", | 67 | "last_error": "boot timeout", |
| 64 | "assigned_ip": "10.77.1.2", | 68 | "assigned_ip": "10.77.1.2", |
| 69 | "network": "lan", | ||
| 70 | "network_ip": "192.168.0.42", | ||
| 65 | "created_at": "2026-07-27T12:01:00Z", | 71 | "created_at": "2026-07-27T12:01:00Z", |
| 66 | "deleted": true, | 72 | "deleted": true, |
| 67 | "actual_power": "stopped", | 73 | "actual_power": "stopped", |
internal/server/api/testdata/vm-trusted-cas-empty.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -10,6 +10,8 @@ | |||
| 10 | "status": "ready", | 10 | "status": "ready", |
| 11 | "last_error": "boot timeout", | 11 | "last_error": "boot timeout", |
| 12 | "assigned_ip": "10.77.1.2", | 12 | "assigned_ip": "10.77.1.2", |
| 13 | "network": "lan", | ||
| 14 | "network_ip": "192.168.0.42", | ||
| 13 | "created_at": "2026-07-27T12:01:00Z", | 15 | "created_at": "2026-07-27T12:01:00Z", |
| 14 | "deleted": true, | 16 | "deleted": true, |
| 15 | "actual_power": "stopped", | 17 | "actual_power": "stopped", |
internal/server/api/testdata/vm-trusted-cas-null.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -10,6 +10,8 @@ | |||
| 10 | "status": "ready", | 10 | "status": "ready", |
| 11 | "last_error": "boot timeout", | 11 | "last_error": "boot timeout", |
| 12 | "assigned_ip": "10.77.1.2", | 12 | "assigned_ip": "10.77.1.2", |
| 13 | "network": "lan", | ||
| 14 | "network_ip": "192.168.0.42", | ||
| 13 | "created_at": "2026-07-27T12:01:00Z", | 15 | "created_at": "2026-07-27T12:01:00Z", |
| 14 | "deleted": true, | 16 | "deleted": true, |
| 15 | "actual_power": "stopped", | 17 | "actual_power": "stopped", |
internal/server/api/testdata/vm.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -10,6 +10,8 @@ | |||
| 10 | "status": "ready", | 10 | "status": "ready", |
| 11 | "last_error": "boot timeout", | 11 | "last_error": "boot timeout", |
| 12 | "assigned_ip": "10.77.1.2", | 12 | "assigned_ip": "10.77.1.2", |
| 13 | "network": "lan", | ||
| 14 | "network_ip": "192.168.0.42", | ||
| 13 | "created_at": "2026-07-27T12:01:00Z", | 15 | "created_at": "2026-07-27T12:01:00Z", |
| 14 | "deleted": true, | 16 | "deleted": true, |
| 15 | "actual_power": "stopped", | 17 | "actual_power": "stopped", |
internal/server/api/types/types.go
| Old | New | ||
|---|---|---|---|
| @@ -94,6 +94,11 @@ type Host struct { | |||
| 94 | // the address to dial for a published guest port. Empty until the host has | 94 | // the address to dial for a published guest port. Empty until the host has |
| 95 | // said. | 95 | // said. |
| 96 | UplinkAddr string `json:"uplink_addr"` | 96 | UplinkAddr string `json:"uplink_addr"` |
| 97 | // HostNetworks are the named guest networks this host's agent advertised | ||
| 98 | // at connect (--host-network). Registry state: empty when the host has | ||
| 99 | // not spoken since server start. A create naming one of these gains a | ||
| 100 | // second NIC on that network; a create naming anything else is refused. | ||
| 101 | HostNetworks []string `json:"host_networks"` | ||
| 97 | // Metrics is live MEASURED utilization, present only when online. Distinct | 102 | // Metrics is live MEASURED utilization, present only when online. Distinct |
| 98 | // from Allocated (control-plane bookkeeping / VM quotas). | 103 | // from Allocated (control-plane bookkeeping / VM quotas). |
| 99 | Metrics *Metrics `json:"metrics"` | 104 | Metrics *Metrics `json:"metrics"` |
| @@ -104,17 +109,28 @@ type Host struct { | |||
| 104 | // cloud_init, ssh_authorized_key — are deliberately excluded. Every field is | 109 | // cloud_init, ssh_authorized_key — are deliberately excluded. Every field is |
| 105 | // spelled out — no struct embedding. | 110 | // spelled out — no struct embedding. |
| 106 | type VM struct { | 111 | type VM struct { |
| 107 | ID string `json:"id"` | 112 | ID string `json:"id"` |
| 108 | HostID string `json:"host_id"` | 113 | HostID string `json:"host_id"` |
| 109 | Name string `json:"name"` | 114 | Name string `json:"name"` |
| 110 | ImageURL string `json:"image_url"` | 115 | ImageURL string `json:"image_url"` |
| 111 | VCPUs int64 `json:"vcpus"` | 116 | VCPUs int64 `json:"vcpus"` |
| 112 | MemMB int64 `json:"mem_mb"` | 117 | MemMB int64 `json:"mem_mb"` |
| 113 | DiskGB int64 `json:"disk_gb"` | 118 | DiskGB int64 `json:"disk_gb"` |
| 114 | PowerState string `json:"power_state"` | 119 | PowerState string `json:"power_state"` |
| 115 | Status string `json:"status"` | 120 | Status string `json:"status"` |
| 116 | LastError string `json:"last_error"` | 121 | LastError string `json:"last_error"` |
| 117 | AssignedIP string `json:"assigned_ip"` | 122 | // AssignedIP is the guest's address on its host's private fabric — every |
| 123 | // guest has one, from boot: the gate splices to it, exposures publish it, | ||
| 124 | // and its siblings on that host reach it there. Never routable off-host. | ||
| 125 | AssignedIP string `json:"assigned_ip"` | ||
| 126 | // Network is the named host network this guest attaches to with a SECOND | ||
| 127 | // NIC, frozen at create; "" means it has only the private one. | ||
| 128 | Network string `json:"network"` | ||
| 129 | // NetworkIP is the address that named network's own DHCP server granted | ||
| 130 | // that second NIC, as the host snooped it — the address the rest of that | ||
| 131 | // network knows this guest by. Empty until the network answers, and | ||
| 132 | // always empty for a guest that asked for none. | ||
| 133 | NetworkIP string `json:"network_ip"` | ||
| 118 | CreatedAt time.Time `json:"created_at"` | 134 | CreatedAt time.Time `json:"created_at"` |
| 119 | Deleted bool `json:"deleted"` | 135 | Deleted bool `json:"deleted"` |
| 120 | ActualPower string `json:"actual_power"` | 136 | ActualPower string `json:"actual_power"` |
| @@ -254,6 +270,12 @@ type CreateVMRequest struct { | |||
| 254 | VCPUs int64 `json:"vcpus"` | 270 | VCPUs int64 `json:"vcpus"` |
| 255 | MemMB int64 `json:"mem_mb"` | 271 | MemMB int64 `json:"mem_mb"` |
| 256 | DiskGB int64 `json:"disk_gb"` | 272 | DiskGB int64 `json:"disk_gb"` |
| 273 | // Network names a host network to attach this guest to — one of the names | ||
| 274 | // its host advertises in host_networks. Empty (the default) is the NAT | ||
| 275 | // underlay. A name the chosen host is not advertising is a 409: there is | ||
| 276 | // no silent fallback, a VM that asked for the LAN either gets it or is | ||
| 277 | // never created. | ||
| 278 | Network string `json:"network"` | ||
| 257 | } | 279 | } |
| 258 | 280 | ||
| 259 | // Default VM sizes the control plane applies when a create request leaves a | 281 | // Default VM sizes the control plane applies when a create request leaves a |
internal/server/api/wire_golden_test.go
| Old | New | ||
|---|---|---|---|
| @@ -77,6 +77,7 @@ func TestWireGolden(t *testing.T) { | |||
| 77 | CPUModel: "AMD Ryzen 9 7950X", | 77 | CPUModel: "AMD Ryzen 9 7950X", |
| 78 | Virt: "kvm", | 78 | Virt: "kvm", |
| 79 | UplinkAddr: "192.168.0.190", | 79 | UplinkAddr: "192.168.0.190", |
| 80 | HostNetworks: []string{"lan", "lab"}, | ||
| 80 | Metrics: &types.Metrics{ | 81 | Metrics: &types.Metrics{ |
| 81 | UptimeS: 86400, | 82 | UptimeS: 86400, |
| 82 | MemUsedMB: 12000, | 83 | MemUsedMB: 12000, |
| @@ -102,6 +103,8 @@ func TestWireGolden(t *testing.T) { | |||
| 102 | Status: "ready", | 103 | Status: "ready", |
| 103 | LastError: "boot timeout", | 104 | LastError: "boot timeout", |
| 104 | AssignedIP: "10.77.1.2", | 105 | AssignedIP: "10.77.1.2", |
| 106 | Network: "lan", | ||
| 107 | NetworkIP: "192.168.0.42", | ||
| 105 | CreatedAt: base.Add(time.Minute), | 108 | CreatedAt: base.Add(time.Minute), |
| 106 | Deleted: true, | 109 | Deleted: true, |
| 107 | ActualPower: "stopped", | 110 | ActualPower: "stopped", |
| @@ -179,6 +182,7 @@ func TestWireGolden(t *testing.T) { | |||
| 179 | VCPUs: 4, | 182 | VCPUs: 4, |
| 180 | MemMB: 4096, | 183 | MemMB: 4096, |
| 181 | DiskGB: 20, | 184 | DiskGB: 20, |
| 185 | Network: "lan", | ||
| 182 | }) | 186 | }) |
| 183 | 187 | ||
| 184 | goldenCheck(t, "patch-vm-request", types.PatchVMRequest{ | 188 | goldenCheck(t, "patch-vm-request", types.PatchVMRequest{ |
internal/server/registry/registry.go
| Old | New | ||
|---|---|---|---|
| @@ -90,6 +90,13 @@ type HostState struct { | |||
| 90 | // (like Sessions, owned by the connect path, preserved across reports). | 90 | // (like Sessions, owned by the connect path, preserved across reports). |
| 91 | // Empty until a version-reporting agent connects. | 91 | // Empty until a version-reporting agent connects. |
| 92 | AgentVersion string | 92 | AgentVersion string |
| 93 | // HostNetworks are the named guest networks this host's agent was | ||
| 94 | // configured with (--host-network), as advertised in each Hello. Owned by | ||
| 95 | // the connect path exactly like AgentVersion — no report carries them, so | ||
| 96 | // UpdateReport preserves them — and empty for a host that has not spoken | ||
| 97 | // since server start, for a Mac agent, and for any agent that predates the | ||
| 98 | // field. Create-time admission reads this set. | ||
| 99 | HostNetworks []string | ||
| 93 | // The following are derived on each Get from LastSeen and the clock; they | 100 | // The following are derived on each Get from LastSeen and the clock; they |
| 94 | // are not stored. | 101 | // are not stored. |
| 95 | Online bool | 102 | Online bool |
| @@ -118,7 +125,9 @@ func (r *Registry) UpdateReport(hostID string, rep Report) { | |||
| 118 | // whole HostState, and Sessions is owned by RecordConnect, not the report. | 125 | // whole HostState, and Sessions is owned by RecordConnect, not the report. |
| 119 | sessions := r.m[hostID].Sessions | 126 | sessions := r.m[hostID].Sessions |
| 120 | agentVersion := r.m[hostID].AgentVersion | 127 | agentVersion := r.m[hostID].AgentVersion |
| 121 | r.m[hostID] = HostState{Report: rep, LastSeen: r.now(), Sessions: sessions, AgentVersion: agentVersion} | 128 | hostNetworks := r.m[hostID].HostNetworks |
| 129 | r.m[hostID] = HostState{Report: rep, LastSeen: r.now(), Sessions: sessions, | ||
| 130 | AgentVersion: agentVersion, HostNetworks: hostNetworks} | ||
| 122 | } | 131 | } |
| 123 | 132 | ||
| 124 | // RecordConnect increments the host's session counter, marking one agent | 133 | // RecordConnect increments the host's session counter, marking one agent |
| @@ -142,6 +151,23 @@ func (r *Registry) SetAgentVersion(hostID, v string) { | |||
| 142 | r.m[hostID] = st | 151 | r.m[hostID] = st |
| 143 | } | 152 | } |
| 144 | 153 | ||
| 154 | // SetHostNetworks records the named guest networks a host advertised in its | ||
| 155 | // Hello. Connect-path-owned like AgentVersion: reports never carry it, so | ||
| 156 | // UpdateReport preserves it. Each Hello replaces the set — an agent restarted | ||
| 157 | // without its --host-network flags advertises none, and the fleet must see | ||
| 158 | // that rather than the networks it used to serve. | ||
| 159 | // | ||
| 160 | // Stores networks by reference, same convention as the Report slices: the | ||
| 161 | // caller must not retain or mutate the slice after passing it in. Get clones | ||
| 162 | // on the way out, so a reader can never observe or corrupt this copy. | ||
| 163 | func (r *Registry) SetHostNetworks(hostID string, networks []string) { | ||
| 164 | r.mu.Lock() | ||
| 165 | defer r.mu.Unlock() | ||
| 166 | st := r.m[hostID] | ||
| 167 | st.HostNetworks = networks | ||
| 168 | r.m[hostID] = st | ||
| 169 | } | ||
| 170 | |||
| 145 | func (r *Registry) Get(hostID string) (HostState, bool) { | 171 | func (r *Registry) Get(hostID string) (HostState, bool) { |
| 146 | r.mu.RLock() | 172 | r.mu.RLock() |
| 147 | defer r.mu.RUnlock() | 173 | defer r.mu.RUnlock() |
| @@ -166,5 +192,6 @@ func (r *Registry) Get(hostID string) (HostState, bool) { | |||
| 166 | quarantined[i].VMSpecJSON = bytes.Clone(quarantined[i].VMSpecJSON) | 192 | quarantined[i].VMSpecJSON = bytes.Clone(quarantined[i].VMSpecJSON) |
| 167 | } | 193 | } |
| 168 | st.Report.Quarantined = quarantined | 194 | st.Report.Quarantined = quarantined |
| 195 | st.HostNetworks = slices.Clone(st.HostNetworks) | ||
| 169 | return st, true | 196 | return st, true |
| 170 | } | 197 | } |
internal/server/registry/registry_test.go
| Old | New | ||
|---|---|---|---|
| @@ -132,6 +132,27 @@ func TestAgentVersionSurvivesReports(t *testing.T) { | |||
| 132 | } | 132 | } |
| 133 | } | 133 | } |
| 134 | 134 | ||
| 135 | // TestHostNetworksSurviveReports: the advertised network names arrive in the | ||
| 136 | // Hello and never again, so the report path — which replaces the whole state — | ||
| 137 | // must carry them forward. Create-time admission reads them on every create; a | ||
| 138 | // report that blanked them would refuse a bridged VM on the host that serves it. | ||
| 139 | func TestHostNetworksSurviveReports(t *testing.T) { | ||
| 140 | now := time.Now() | ||
| 141 | r := New(func() time.Time { return now }) | ||
| 142 | r.SetHostNetworks("h1", []string{"lan"}) | ||
| 143 | r.UpdateReport("h1", Report{}) | ||
| 144 | st, ok := r.Get("h1") | ||
| 145 | require.True(t, ok) | ||
| 146 | assert.Equal(t, []string{"lan"}, st.HostNetworks, "a report must preserve what the Hello advertised") | ||
| 147 | |||
| 148 | // A reconnect re-advertises: an agent restarted without its --host-network | ||
| 149 | // flags advertises none, and the registry must say so rather than hold the | ||
| 150 | // networks it used to serve. | ||
| 151 | r.SetHostNetworks("h1", nil) | ||
| 152 | st, _ = r.Get("h1") | ||
| 153 | assert.Empty(t, st.HostNetworks, "a Hello with no networks retracts the old set") | ||
| 154 | } | ||
| 155 | |||
| 135 | // TestGetClonesExposureStatuses pins that a caller cannot corrupt registry | 156 | // TestGetClonesExposureStatuses pins that a caller cannot corrupt registry |
| 136 | // state through the slice it is handed — the same rule the VM rows follow. | 157 | // state through the slice it is handed — the same rule the VM rows follow. |
| 137 | func TestGetClonesExposureStatuses(t *testing.T) { | 158 | func TestGetClonesExposureStatuses(t *testing.T) { |
internal/server/store/store.go
| Old | New | ||
|---|---|---|---|
| @@ -79,7 +79,18 @@ type VM struct { | |||
| 79 | ID, HostID, Name, ImageURL, ImageSHA256, CloudInit string | 79 | ID, HostID, Name, ImageURL, ImageSHA256, CloudInit string |
| 80 | VCPUs, MemMB, DiskGB int64 | 80 | VCPUs, MemMB, DiskGB int64 |
| 81 | PowerState, Status, LastError, AssignedIP string | 81 | PowerState, Status, LastError, AssignedIP string |
| 82 | SSHAuthorizedKey string | 82 | // Network is the named host network this VM attaches to, as recorded at |
| 83 | // create — the same name a host advertised in its Hello. '' is the NAT | ||
| 84 | // underlay: every VM created before this column existed, and any VM | ||
| 85 | // created without naming one, backfills to it. | ||
| 86 | Network string | ||
| 87 | // NetworkIP is the address the site's DHCP server granted this guest on | ||
| 88 | // its named-network NIC, as its host snooped it. AssignedIP is the other | ||
| 89 | // address the same guest has, on the host's private fabric — every VM has | ||
| 90 | // that one, and only a VM with a Network can have this one. '' means not | ||
| 91 | // discovered (yet, or ever: a guest configured static never asks). | ||
| 92 | NetworkIP string | ||
| 93 | SSHAuthorizedKey string | ||
| 83 | // SSHHostPubKey is the public half of the guest's SSH host key, as its host | 94 | // SSHHostPubKey is the public half of the guest's SSH host key, as its host |
| 84 | // reported it (authorized_keys form). The private half lives on that host | 95 | // reported it (authorized_keys form). The private half lives on that host |
| 85 | // and the control plane never sees it. SSHHostCert is the certificate the | 96 | // and the control plane never sees it. SSHHostCert is the certificate the |
| @@ -105,7 +116,7 @@ type VM struct { | |||
| 105 | // makes nil unambiguously "unrecorded" rather than "trusts nothing", which | 116 | // makes nil unambiguously "unrecorded" rather than "trusts nothing", which |
| 106 | // is the distinction every reader of this field depends on. | 117 | // is the distinction every reader of this field depends on. |
| 107 | TrustedCAs []TrustedCA | 118 | TrustedCAs []TrustedCA |
| 108 | // The VM's reachable address is AssignedIP (the agent-reported bridge IP). | 119 | // The address the gate and every exposure target is AssignedIP. |
| 109 | CreatedAt time.Time | 120 | CreatedAt time.Time |
| 110 | DeletedAt *time.Time | 121 | DeletedAt *time.Time |
| 111 | // Tenant is the owning tenant, always derived from the host's — never client-set. | 122 | // Tenant is the owning tenant, always derived from the host's — never client-set. |
| @@ -371,6 +382,17 @@ func Open(path, cidrPool string) (*Store, error) { | |||
| 371 | // proto field 9 — the same discipline for the same reason: retire the | 382 | // proto field 9 — the same discipline for the same reason: retire the |
| 372 | // compatibility only once no v0.0.5 can still be out there. | 383 | // compatibility only once no v0.0.5 can still be out there. |
| 373 | {"vms", "persistent", "INTEGER NOT NULL DEFAULT 1"}, | 384 | {"vms", "persistent", "INTEGER NOT NULL DEFAULT 1"}, |
| 385 | // The named host network this VM attaches to, as recorded at create. | ||
| 386 | // '' is the NAT underlay — every row that predates this column | ||
| 387 | // backfills to it, which is exactly what an unset network means. | ||
| 388 | {"vms", "network", "TEXT NOT NULL DEFAULT ''"}, | ||
| 389 | // The address the site's DHCP server granted a guest on its named | ||
| 390 | // network, as its host snooped it. Agent-reported like assigned_ip and | ||
| 391 | // separate from it: assigned_ip is the private-fabric address every | ||
| 392 | // guest has from boot, this one exists only for a guest with a second | ||
| 393 | // NIC and only once that network has answered. '' is "not discovered", | ||
| 394 | // which is also what every row predating the column honestly says. | ||
| 395 | {"vms", "network_ip", "TEXT NOT NULL DEFAULT ''"}, | ||
| 374 | // The address a host presents on the network it reaches the fleet | 396 | // The address a host presents on the network it reaches the fleet |
| 375 | // over. Reported every tick like the guest subnet, and stored for the | 397 | // over. Reported every tick like the guest subnet, and stored for the |
| 376 | // same reason: the console renders `host:port` for every exposure, | 398 | // same reason: the console renders `host:port` for every exposure, |
| @@ -842,12 +864,12 @@ func (s *Store) CreateVM(vm VM) error { | |||
| 842 | _, err = tx.Exec( | 864 | _, err = tx.Exec( |
| 843 | `INSERT INTO vms(id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key, | 865 | `INSERT INTO vms(id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key, |
| 844 | injected_key_type, injected_key_fp, injected_key_comment, trusted_cas, | 866 | injected_key_type, injected_key_fp, injected_key_comment, trusted_cas, |
| 845 | vcpus, mem_mb, disk_gb, power_state, created_at) | 867 | vcpus, mem_mb, disk_gb, power_state, network, created_at) |
| 846 | VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, | 868 | VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, |
| 847 | vm.ID, vm.HostID, vm.Name, vm.Tenant, vm.ImageURL, vm.ImageSHA256, | 869 | vm.ID, vm.HostID, vm.Name, vm.Tenant, vm.ImageURL, vm.ImageSHA256, |
| 848 | vm.CloudInit, vm.SSHAuthorizedKey, | 870 | vm.CloudInit, vm.SSHAuthorizedKey, |
| 849 | vm.InjectedKeyType, vm.InjectedKeyFP, vm.InjectedKeyComment, trustedCAs, | 871 | vm.InjectedKeyType, vm.InjectedKeyFP, vm.InjectedKeyComment, trustedCAs, |
| 850 | vm.VCPUs, vm.MemMB, vm.DiskGB, vm.PowerState, | 872 | vm.VCPUs, vm.MemMB, vm.DiskGB, vm.PowerState, vm.Network, |
| 851 | now.Format(time.RFC3339), | 873 | now.Format(time.RFC3339), |
| 852 | ) | 874 | ) |
| 853 | if err != nil { | 875 | if err != nil { |
| @@ -1387,6 +1409,55 @@ func (s *Store) RecordVMStatus(id, status, lastErr, ip string) (string, error) { | |||
| 1387 | return ip, nil | 1409 | return ip, nil |
| 1388 | } | 1410 | } |
| 1389 | 1411 | ||
| 1412 | // RecordVMNetworkIP persists the address a guest's named-network NIC was | ||
| 1413 | // granted, as its host snooped it from the site's DHCP server. | ||
| 1414 | // | ||
| 1415 | // Separate from RecordVMStatus because it is a separate fact with a separate | ||
| 1416 | // clock: assigned_ip exists the moment the reservation is made, while this one | ||
| 1417 | // arrives whenever the site's server answers — which can be long before the VM | ||
| 1418 | // is ready, the only phase RecordVMStatus writes in. | ||
| 1419 | // | ||
| 1420 | // The CASE is the assigned_ip rule, for the assigned_ip reason: an empty report | ||
| 1421 | // is "not discovered", never "no longer has one", so it keeps what was stored | ||
| 1422 | // rather than erasing it. An unusable address (link-local — what a guest shows | ||
| 1423 | // when nothing answered — loopback, multicast, unparseable) is dropped the same | ||
| 1424 | // way, so a NIC that never got a real lease reports nothing rather than noise. | ||
| 1425 | // | ||
| 1426 | // hostID is part of the WHERE clause rather than a check made before it, the | ||
| 1427 | // same idiom RecordVMHostKey uses and for the same reason: a host may only | ||
| 1428 | // ever speak for the VMs it holds, and a predicate leaves no read-then-write | ||
| 1429 | // window in which the VM could move. A host that reports a lease for a VM | ||
| 1430 | // that is not its own changes no rows and gets sql.ErrNoRows. | ||
| 1431 | // | ||
| 1432 | // network is also in the WHERE: a VM that never asked for a named network can | ||
| 1433 | // never grow a LAN address, however an agent misbehaves — a forged report, or | ||
| 1434 | // one from an agent whose view of the VM has skewed from the server's own. | ||
| 1435 | // Refusing the write makes that pairing unrepresentable in the row rather | ||
| 1436 | // than merely unrendered by the clients that know to hide it. This means | ||
| 1437 | // sql.ErrNoRows now covers three predicates — the row is gone, the wrong host | ||
| 1438 | // reported it, or the VM has no network — and the caller cannot tell which: | ||
| 1439 | // a pre-check that classified the failure would reopen the read-then-write | ||
| 1440 | // window this whole idiom exists to close. The resulting slog.Warn recurs at | ||
| 1441 | // sync cadence by design (the write-through cache only commits on success, so | ||
| 1442 | // it is never rate-limited away); it says only that something is wrong, and | ||
| 1443 | // the VM id in the log line is what lets an operator resolve it in one query. | ||
| 1444 | func (s *Store) RecordVMNetworkIP(id, hostID, ip string) error { | ||
| 1445 | if ip != "" && !usableAddress(ip) { | ||
| 1446 | ip = "" | ||
| 1447 | } | ||
| 1448 | res, err := s.db.Exec( | ||
| 1449 | `UPDATE vms SET network_ip=CASE WHEN ?='' THEN network_ip ELSE ? END WHERE id=? AND host_id=? AND network != ''`, | ||
| 1450 | ip, ip, id, hostID, | ||
| 1451 | ) | ||
| 1452 | if err != nil { | ||
| 1453 | return err | ||
| 1454 | } | ||
| 1455 | if n, _ := res.RowsAffected(); n == 0 { | ||
| 1456 | return sql.ErrNoRows | ||
| 1457 | } | ||
| 1458 | return nil | ||
| 1459 | } | ||
| 1460 | |||
| 1390 | // RecordVMHostKey stores the public host key a host generated for one of its | 1461 | // RecordVMHostKey stores the public host key a host generated for one of its |
| 1391 | // guests, together with the certificate the control plane signed for it. | 1462 | // guests, together with the certificate the control plane signed for it. |
| 1392 | // | 1463 | // |
| @@ -1417,7 +1488,7 @@ func (s *Store) RecordVMHostKey(vmID, hostID, pubkey, cert string) error { | |||
| 1417 | const vmColumns = `id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key, | 1488 | const vmColumns = `id, host_id, name, tenant, image_url, image_sha256, cloud_init, ssh_authorized_key, |
| 1418 | ssh_host_pubkey, ssh_host_cert, | 1489 | ssh_host_pubkey, ssh_host_cert, |
| 1419 | injected_key_type, injected_key_fp, injected_key_comment, trusted_cas, | 1490 | injected_key_type, injected_key_fp, injected_key_comment, trusted_cas, |
| 1420 | vcpus, mem_mb, disk_gb, power_state, status, last_error, assigned_ip, | 1491 | vcpus, mem_mb, disk_gb, power_state, network, status, last_error, assigned_ip, network_ip, |
| 1421 | created_at, deleted_at` | 1492 | created_at, deleted_at` |
| 1422 | 1493 | ||
| 1423 | // scanVM's column order must match vmColumns exactly — it is positional, not | 1494 | // scanVM's column order must match vmColumns exactly — it is positional, not |
| @@ -1432,7 +1503,7 @@ func scanVM(rows *sql.Rows) (VM, error) { | |||
| 1432 | &vm.CloudInit, &vm.SSHAuthorizedKey, &vm.SSHHostPubKey, &vm.SSHHostCert, | 1503 | &vm.CloudInit, &vm.SSHAuthorizedKey, &vm.SSHHostPubKey, &vm.SSHHostCert, |
| 1433 | &vm.InjectedKeyType, &vm.InjectedKeyFP, &vm.InjectedKeyComment, &trustedCAs, | 1504 | &vm.InjectedKeyType, &vm.InjectedKeyFP, &vm.InjectedKeyComment, &trustedCAs, |
| 1434 | &vm.VCPUs, &vm.MemMB, &vm.DiskGB, | 1505 | &vm.VCPUs, &vm.MemMB, &vm.DiskGB, |
| 1435 | &vm.PowerState, &vm.Status, &vm.LastError, &vm.AssignedIP, | 1506 | &vm.PowerState, &vm.Network, &vm.Status, &vm.LastError, &vm.AssignedIP, &vm.NetworkIP, |
| 1436 | &createdAt, &deletedAt, | 1507 | &createdAt, &deletedAt, |
| 1437 | ) | 1508 | ) |
| 1438 | if err != nil { | 1509 | if err != nil { |
internal/server/store/store_test.go
| Old | New | ||
|---|---|---|---|
| @@ -88,6 +88,150 @@ func TestGetVM(t *testing.T) { | |||
| 88 | assert.NotNil(t, got.DeletedAt) | 88 | assert.NotNil(t, got.DeletedAt) |
| 89 | } | 89 | } |
| 90 | 90 | ||
| 91 | // TestVMNetworkRoundTrip pins that the empty string is the NAT default an | ||
| 92 | // unset create stores, and a named network survives the row whole. | ||
| 93 | func TestVMNetworkRoundTrip(t *testing.T) { | ||
| 94 | s := newStore(t) | ||
| 95 | h := enrollHost(t, s) | ||
| 96 | |||
| 97 | require.NoError(t, s.CreateVM(VM{ | ||
| 98 | ID: "vm-lan", HostID: h.ID, Name: "on-lan", | ||
| 99 | ImageURL: "http://img", ImageSHA256: "abc", | ||
| 100 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", | ||
| 101 | Network: "lan", | ||
| 102 | })) | ||
| 103 | got, err := s.GetVM("vm-lan") | ||
| 104 | require.NoError(t, err) | ||
| 105 | assert.Equal(t, "lan", got.Network) | ||
| 106 | |||
| 107 | // A VM created without naming a network gets the NAT underlay, spelled ''. | ||
| 108 | require.NoError(t, s.CreateVM(VM{ | ||
| 109 | ID: "vm-nat", HostID: h.ID, Name: "on-nat", | ||
| 110 | ImageURL: "http://img", ImageSHA256: "abc", | ||
| 111 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", | ||
| 112 | })) | ||
| 113 | got, err = s.GetVM("vm-nat") | ||
| 114 | require.NoError(t, err) | ||
| 115 | assert.Equal(t, "", got.Network) | ||
| 116 | } | ||
| 117 | |||
| 118 | // TestRecordVMNetworkIPKeepsBothAddresses pins that the two addresses a | ||
| 119 | // networked guest has are two independent columns: the named network's lease | ||
| 120 | // lands in network_ip and never touches the private-fabric address the gate | ||
| 121 | // and every exposure aim at. | ||
| 122 | func TestRecordVMNetworkIPKeepsBothAddresses(t *testing.T) { | ||
| 123 | s := newStore(t) | ||
| 124 | h := enrollHost(t, s) | ||
| 125 | require.NoError(t, s.CreateVM(VM{ | ||
| 126 | ID: "vm-lan", HostID: h.ID, Name: "on-lan", ImageURL: "u", ImageSHA256: "s", | ||
| 127 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan", | ||
| 128 | })) | ||
| 129 | |||
| 130 | // A fresh row has the reservation and nothing from the site's DHCP yet. | ||
| 131 | _, err := s.RecordVMStatus("vm-lan", "ready", "", "10.77.1.2") | ||
| 132 | require.NoError(t, err) | ||
| 133 | got, err := s.GetVM("vm-lan") | ||
| 134 | require.NoError(t, err) | ||
| 135 | assert.Equal(t, "10.77.1.2", got.AssignedIP) | ||
| 136 | assert.Empty(t, got.NetworkIP, "nothing has answered on the named NIC yet") | ||
| 137 | |||
| 138 | require.NoError(t, s.RecordVMNetworkIP("vm-lan", h.ID, "192.168.0.42")) | ||
| 139 | got, err = s.GetVM("vm-lan") | ||
| 140 | require.NoError(t, err) | ||
| 141 | assert.Equal(t, "192.168.0.42", got.NetworkIP) | ||
| 142 | assert.Equal(t, "10.77.1.2", got.AssignedIP, "the private address is a separate fact") | ||
| 143 | |||
| 144 | // And the status write does not disturb the discovered one either. | ||
| 145 | _, err = s.RecordVMStatus("vm-lan", "failed", "boom", "10.77.1.2") | ||
| 146 | require.NoError(t, err) | ||
| 147 | got, err = s.GetVM("vm-lan") | ||
| 148 | require.NoError(t, err) | ||
| 149 | assert.Equal(t, "192.168.0.42", got.NetworkIP) | ||
| 150 | } | ||
| 151 | |||
| 152 | // TestRecordVMNetworkIPKeepsPriorOnNothingUsable pins the empty-never-clears | ||
| 153 | // rule assigned_ip already follows: a report with no address, or one naming | ||
| 154 | // nobody, leaves the last known address standing. A guest whose lease renews | ||
| 155 | // to a different address is still followed. | ||
| 156 | func TestRecordVMNetworkIPKeepsPriorOnNothingUsable(t *testing.T) { | ||
| 157 | const prior = "192.168.0.42" | ||
| 158 | for _, tc := range []struct { | ||
| 159 | name string | ||
| 160 | ip string | ||
| 161 | want string | ||
| 162 | }{ | ||
| 163 | {"a new lease on the site's network", "192.168.0.77", "192.168.0.77"}, | ||
| 164 | {"none discovered", "", prior}, | ||
| 165 | {"link-local: the site's DHCP never answered", "169.254.11.2", prior}, | ||
| 166 | {"loopback", "127.0.0.1", prior}, | ||
| 167 | {"unspecified", "0.0.0.0", prior}, | ||
| 168 | {"multicast", "224.0.0.1", prior}, | ||
| 169 | {"unparseable", "not-an-ip", prior}, | ||
| 170 | } { | ||
| 171 | t.Run(tc.name, func(t *testing.T) { | ||
| 172 | s := newStore(t) | ||
| 173 | h := enrollHost(t, s) | ||
| 174 | require.NoError(t, s.CreateVM(VM{ | ||
| 175 | ID: "vm1", HostID: h.ID, Name: "a", ImageURL: "u", ImageSHA256: "s", | ||
| 176 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan", | ||
| 177 | })) | ||
| 178 | require.NoError(t, s.RecordVMNetworkIP("vm1", h.ID, prior)) | ||
| 179 | |||
| 180 | require.NoError(t, s.RecordVMNetworkIP("vm1", h.ID, tc.ip)) | ||
| 181 | vm, err := s.GetVM("vm1") | ||
| 182 | require.NoError(t, err) | ||
| 183 | assert.Equal(t, tc.want, vm.NetworkIP) | ||
| 184 | }) | ||
| 185 | } | ||
| 186 | } | ||
| 187 | |||
| 188 | // TestRecordVMNetworkIPRefusesNetworklessVM pins that a VM which never named | ||
| 189 | // a network cannot grow a network_ip: the pairing is impossible, so the write | ||
| 190 | // itself refuses it rather than leaving it for a client to hide. The same | ||
| 191 | // sql.ErrNoRows a foreign host or a missing row gets is what a report against | ||
| 192 | // the NAT underlay gets too. | ||
| 193 | func TestRecordVMNetworkIPRefusesNetworklessVM(t *testing.T) { | ||
| 194 | s := newStore(t) | ||
| 195 | h := enrollHost(t, s) | ||
| 196 | require.NoError(t, s.CreateVM(VM{ | ||
| 197 | ID: "vm-nat", HostID: h.ID, Name: "on-nat", ImageURL: "u", ImageSHA256: "s", | ||
| 198 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", | ||
| 199 | })) | ||
| 200 | |||
| 201 | err := s.RecordVMNetworkIP("vm-nat", h.ID, "192.168.0.42") | ||
| 202 | require.ErrorIs(t, err, sql.ErrNoRows) | ||
| 203 | got, err := s.GetVM("vm-nat") | ||
| 204 | require.NoError(t, err) | ||
| 205 | assert.Empty(t, got.NetworkIP, "a VM with no named network stores no discovered address") | ||
| 206 | } | ||
| 207 | |||
| 208 | // TestRecordVMNetworkIPUnknownVM pins that a report for a row that is gone is | ||
| 209 | // an error, not a silent no-op — the same answer RecordVMStatus gives. | ||
| 210 | func TestRecordVMNetworkIPUnknownVM(t *testing.T) { | ||
| 211 | s := newStore(t) | ||
| 212 | h := enrollHost(t, s) | ||
| 213 | assert.ErrorIs(t, s.RecordVMNetworkIP("nope", h.ID, "192.168.0.42"), sql.ErrNoRows) | ||
| 214 | } | ||
| 215 | |||
| 216 | // TestRecordVMNetworkIPRefusesAnotherHostsVM pins the same idiom | ||
| 217 | // RecordVMHostKey uses: host_id is a predicate on the UPDATE, not a check made | ||
| 218 | // before it, so a lease reported for a VM that belongs to a different host | ||
| 219 | // changes no rows and gets sql.ErrNoRows rather than silently landing. | ||
| 220 | func TestRecordVMNetworkIPRefusesAnotherHostsVM(t *testing.T) { | ||
| 221 | s := newStore(t) | ||
| 222 | h := enrollHost(t, s) | ||
| 223 | require.NoError(t, s.CreateVM(VM{ | ||
| 224 | ID: "vm1", HostID: h.ID, Name: "on-lan", ImageURL: "u", ImageSHA256: "s", | ||
| 225 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan", | ||
| 226 | })) | ||
| 227 | |||
| 228 | err := s.RecordVMNetworkIP("vm1", "some-other-host", "192.168.0.42") | ||
| 229 | require.ErrorIs(t, err, sql.ErrNoRows) | ||
| 230 | got, err := s.GetVM("vm1") | ||
| 231 | require.NoError(t, err) | ||
| 232 | assert.Empty(t, got.NetworkIP, "a claim from a host that does not hold this VM writes nothing") | ||
| 233 | } | ||
| 234 | |||
| 91 | func TestRecordVMHostKeyCertifiesOnlyItsOwnHostsVM(t *testing.T) { | 235 | func TestRecordVMHostKeyCertifiesOnlyItsOwnHostsVM(t *testing.T) { |
| 92 | s := newStore(t) | 236 | s := newStore(t) |
| 93 | h := enrollHost(t, s) | 237 | h := enrollHost(t, s) |
internal/server/syncsvc/network_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,179 @@ | |||
| 1 | package syncsvc | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "testing" | ||
| 5 | "time" | ||
| 6 | |||
| 7 | "github.com/a73x/eitri/internal/pb" | ||
| 8 | "github.com/a73x/eitri/internal/server/store" | ||
| 9 | "github.com/stretchr/testify/assert" | ||
| 10 | "github.com/stretchr/testify/require" | ||
| 11 | ) | ||
| 12 | |||
| 13 | // TestHelloAdvertisesHostNetworks is the whole basis of named-network | ||
| 14 | // admission: a create is only allowed onto a host that says it serves the | ||
| 15 | // network, and the only place a host ever says so is its Hello. If this stops | ||
| 16 | // reaching the registry, every bridged create on the fleet is refused. | ||
| 17 | func TestHelloAdvertisesHostNetworks(t *testing.T) { | ||
| 18 | f := setup(t) | ||
| 19 | _, err := dialHello(t, f.addr, f.fp, &pb.Hello{ | ||
| 20 | HostId: f.host.ID, Provisioner: "cloudhv", Credential: f.cred, | ||
| 21 | HostNetworks: []string{"lan", "lab"}, | ||
| 22 | }) | ||
| 23 | require.NoError(t, err) | ||
| 24 | |||
| 25 | require.Eventually(t, func() bool { | ||
| 26 | st, ok := f.reg.Get(f.host.ID) | ||
| 27 | return ok && len(st.HostNetworks) == 2 | ||
| 28 | }, 2*time.Second, 20*time.Millisecond, "the Hello's networks must reach the registry") | ||
| 29 | st, _ := f.reg.Get(f.host.ID) | ||
| 30 | assert.Equal(t, []string{"lan", "lab"}, st.HostNetworks) | ||
| 31 | } | ||
| 32 | |||
| 33 | // TestSnapshotCarriesTheVMsNetwork: the row records which underlay a guest was | ||
| 34 | // admitted onto, and the snapshot is the only way that reaches the agent that | ||
| 35 | // builds it. A NAT guest carries the empty name — the agent's default path — | ||
| 36 | // and neither one is inferred anywhere downstream. | ||
| 37 | func TestSnapshotCarriesTheVMsNetwork(t *testing.T) { | ||
| 38 | f := setup(t) | ||
| 39 | require.NoError(t, f.st.CreateVM(store.VM{ | ||
| 40 | ID: "vm1", HostID: f.host.ID, Name: "bridged", ImageURL: "u", ImageSHA256: "s", | ||
| 41 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan", | ||
| 42 | })) | ||
| 43 | require.NoError(t, f.st.CreateVM(store.VM{ | ||
| 44 | ID: "vm2", HostID: f.host.ID, Name: "natted", ImageURL: "u", ImageSHA256: "s", | ||
| 45 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", | ||
| 46 | })) | ||
| 47 | |||
| 48 | snap, err := f.svc.buildSnapshot(f.host.ID) | ||
| 49 | require.NoError(t, err) | ||
| 50 | got := map[string]string{} | ||
| 51 | for _, v := range snap.GetVms() { | ||
| 52 | got[v.GetVmId()] = v.GetNetwork() | ||
| 53 | } | ||
| 54 | assert.Equal(t, map[string]string{"vm1": "lan", "vm2": ""}, got) | ||
| 55 | } | ||
| 56 | |||
| 57 | // networkedVM creates one VM on the named network `lan` for the report tests. | ||
| 58 | func networkedVM(t *testing.T, f *fixture, id string) { | ||
| 59 | t.Helper() | ||
| 60 | require.NoError(t, f.st.CreateVM(store.VM{ | ||
| 61 | ID: id, HostID: f.host.ID, Name: id, ImageURL: "u", ImageSHA256: "s", | ||
| 62 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", Network: "lan", | ||
| 63 | })) | ||
| 64 | } | ||
| 65 | |||
| 66 | // TestApplyReportRecordsBothAddresses pins the inbound leg of the second NIC: | ||
| 67 | // the address the site's DHCP granted is durable beside — never instead of — | ||
| 68 | // the private-fabric address the gate and every exposure aim at. | ||
| 69 | func TestApplyReportRecordsBothAddresses(t *testing.T) { | ||
| 70 | f := setup(t) | ||
| 71 | networkedVM(t, f, "vm1") | ||
| 72 | |||
| 73 | f.svc.applyReport(f.host.ID, &pb.ActualStateReport{ | ||
| 74 | Vms: []*pb.ActualVM{{ | ||
| 75 | VmId: "vm1", Power: "running", Phase: "ready", | ||
| 76 | Ip: "10.77.1.2", NetworkIp: "192.168.0.42", | ||
| 77 | }}, | ||
| 78 | }) | ||
| 79 | |||
| 80 | vm, err := f.st.GetVM("vm1") | ||
| 81 | require.NoError(t, err) | ||
| 82 | assert.Equal(t, "10.77.1.2", vm.AssignedIP) | ||
| 83 | assert.Equal(t, "192.168.0.42", vm.NetworkIP) | ||
| 84 | } | ||
| 85 | |||
| 86 | // TestApplyReportRecordsTheLeaseBeforeTheGuestIsReady: the site's DHCP answers | ||
| 87 | // on its own clock, which is not the lifecycle's. A lease that lands while the | ||
| 88 | // guest is still booting is recorded then — the status write's ready/failed | ||
| 89 | // filter must not hold the address hostage. | ||
| 90 | func TestApplyReportRecordsTheLeaseBeforeTheGuestIsReady(t *testing.T) { | ||
| 91 | f := setup(t) | ||
| 92 | networkedVM(t, f, "vm1") | ||
| 93 | |||
| 94 | f.svc.applyReport(f.host.ID, &pb.ActualStateReport{ | ||
| 95 | Vms: []*pb.ActualVM{{VmId: "vm1", Power: "running", Phase: "creating", NetworkIp: "192.168.0.42"}}, | ||
| 96 | }) | ||
| 97 | |||
| 98 | vm, err := f.st.GetVM("vm1") | ||
| 99 | require.NoError(t, err) | ||
| 100 | assert.Equal(t, "192.168.0.42", vm.NetworkIP) | ||
| 101 | assert.Equal(t, "pending", vm.Status, "the lifecycle write still waits for ready or failed") | ||
| 102 | } | ||
| 103 | |||
| 104 | // TestApplyReportKeepsTheLastKnownLease: an agent restarted before its guest | ||
| 105 | // renews reports no address on the named NIC, and silence is not "the guest | ||
| 106 | // left the network". The stored address stands until a new one replaces it. | ||
| 107 | func TestApplyReportKeepsTheLastKnownLease(t *testing.T) { | ||
| 108 | f := setup(t) | ||
| 109 | networkedVM(t, f, "vm1") | ||
| 110 | ready := func(netIP string) *pb.ActualStateReport { | ||
| 111 | return &pb.ActualStateReport{Vms: []*pb.ActualVM{{ | ||
| 112 | VmId: "vm1", Power: "running", Phase: "ready", Ip: "10.77.1.2", NetworkIp: netIP, | ||
| 113 | }}} | ||
| 114 | } | ||
| 115 | |||
| 116 | f.svc.applyReport(f.host.ID, ready("192.168.0.42")) | ||
| 117 | f.svc.applyReport(f.host.ID, ready("")) | ||
| 118 | vm, err := f.st.GetVM("vm1") | ||
| 119 | require.NoError(t, err) | ||
| 120 | assert.Equal(t, "192.168.0.42", vm.NetworkIP, "an empty report is 'not known', not 'gone'") | ||
| 121 | |||
| 122 | // A renewal onto a different address is followed, not stuck. | ||
| 123 | f.svc.applyReport(f.host.ID, ready("192.168.0.77")) | ||
| 124 | vm, err = f.st.GetVM("vm1") | ||
| 125 | require.NoError(t, err) | ||
| 126 | assert.Equal(t, "192.168.0.77", vm.NetworkIP) | ||
| 127 | |||
| 128 | // An unusable address (what a guest shows when nothing answered) is one | ||
| 129 | // the store drops and keeps the prior value for — the cache still commits | ||
| 130 | // the raw report, so this proves the divergence between what netIPTrack | ||
| 131 | // remembers and what the row holds is harmless: the next GOOD address | ||
| 132 | // still reads as a change against the cache and lands. | ||
| 133 | f.svc.applyReport(f.host.ID, ready("169.254.11.2")) | ||
| 134 | vm, err = f.st.GetVM("vm1") | ||
| 135 | require.NoError(t, err) | ||
| 136 | assert.Equal(t, "192.168.0.77", vm.NetworkIP, "an unusable address is dropped, not stored") | ||
| 137 | |||
| 138 | f.svc.applyReport(f.host.ID, ready("192.168.0.99")) | ||
| 139 | vm, err = f.st.GetVM("vm1") | ||
| 140 | require.NoError(t, err) | ||
| 141 | assert.Equal(t, "192.168.0.99", vm.NetworkIP, "a good address after junk still lands") | ||
| 142 | } | ||
| 143 | |||
| 144 | // TestApplyReportSkipsUnchangedLeases proves the per-VM dedup: a bridged guest | ||
| 145 | // re-reports the same address every tick forever, and the store runs on one | ||
| 146 | // connection. Only a change may cost a write. | ||
| 147 | func TestApplyReportSkipsUnchangedLeases(t *testing.T) { | ||
| 148 | f := setup(t) | ||
| 149 | networkedVM(t, f, "vm1") | ||
| 150 | rep := &pb.ActualStateReport{Vms: []*pb.ActualVM{{ | ||
| 151 | VmId: "vm1", Power: "running", Phase: "ready", Ip: "10.77.1.2", NetworkIp: "192.168.0.42", | ||
| 152 | }}} | ||
| 153 | |||
| 154 | f.svc.applyReport(f.host.ID, rep) | ||
| 155 | f.svc.applyReport(f.host.ID, rep) | ||
| 156 | f.svc.applyReport(f.host.ID, rep) | ||
| 157 | |||
| 158 | f.svc.netIPTrack.mu.Lock() | ||
| 159 | defer f.svc.netIPTrack.mu.Unlock() | ||
| 160 | assert.Equal(t, map[string]string{"vm1": "192.168.0.42"}, f.svc.netIPTrack.last, | ||
| 161 | "the address is remembered once, so the repeats write nothing") | ||
| 162 | } | ||
| 163 | |||
| 164 | // TestApplyReportForgetsAReapedVMsLease bounds the cache: a VM whose row is | ||
| 165 | // hard-deleted must not leave its address remembered forever. | ||
| 166 | func TestApplyReportForgetsAReapedVMsLease(t *testing.T) { | ||
| 167 | f := setup(t) | ||
| 168 | networkedVM(t, f, "vm1") | ||
| 169 | |||
| 170 | f.svc.applyReport(f.host.ID, &pb.ActualStateReport{ | ||
| 171 | Vms: []*pb.ActualVM{{VmId: "vm1", Power: "running", Phase: "ready", NetworkIp: "192.168.0.42"}}, | ||
| 172 | }) | ||
| 173 | require.NoError(t, f.st.TombstoneVM("vm1")) | ||
| 174 | f.svc.applyReport(f.host.ID, &pb.ActualStateReport{Destroyed: []string{"vm1"}}) | ||
| 175 | |||
| 176 | f.svc.netIPTrack.mu.Lock() | ||
| 177 | defer f.svc.netIPTrack.mu.Unlock() | ||
| 178 | assert.Empty(t, f.svc.netIPTrack.last, "a reaped VM's address is dead weight") | ||
| 179 | } | ||
internal/server/syncsvc/syncsvc.go
| Old | New | ||
|---|---|---|---|
| @@ -59,6 +59,10 @@ type Service struct { | |||
| 59 | // the same reason netTrack does: every host reports every tick forever, and | 59 | // the same reason netTrack does: every host reports every tick forever, and |
| 60 | // the store runs on a single connection. | 60 | // the store runs on a single connection. |
| 61 | uplinkTrack *netTracker | 61 | uplinkTrack *netTracker |
| 62 | // netIPTrack remembers the named-network address each VM last had WRITTEN. | ||
| 63 | // Keyed by vmID like certTrack, and for the same reason: a bridged guest | ||
| 64 | // re-reports the address its lease renewed to, every tick, forever. | ||
| 65 | netIPTrack *netTracker | ||
| 62 | // maxCredAge, when non-zero, rejects credentials whose issued-at is older. | 66 | // maxCredAge, when non-zero, rejects credentials whose issued-at is older. |
| 63 | // Zero disables the age check (default: expiry without an auto-renewal | 67 | // Zero disables the age check (default: expiry without an auto-renewal |
| 64 | // channel would force periodic re-enrolls; per-host generation revocation | 68 | // channel would force periodic re-enrolls; per-host generation revocation |
| @@ -106,7 +110,8 @@ func newWithWriteTimeout(st *store.Store, reg *registry.Registry, h *hub.Hub, se | |||
| 106 | } | 110 | } |
| 107 | return &Service{st: st, reg: reg, hub: h, secret: secret, maxCredAge: maxCredAge, writeTimeout: writeTimeout, | 111 | return &Service{st: st, reg: reg, hub: h, secret: secret, maxCredAge: maxCredAge, writeTimeout: writeTimeout, |
| 108 | conns: map[string]quic.Connection{}, recorder: st, tracker: newStatusTracker(), netTrack: newNetTracker(), | 112 | conns: map[string]quic.Connection{}, recorder: st, tracker: newStatusTracker(), netTrack: newNetTracker(), |
| 109 | uplinkTrack: newNetTracker(), certTrack: newNetTracker(), offers: map[string]offer{}, now: time.Now} | 113 | uplinkTrack: newNetTracker(), certTrack: newNetTracker(), netIPTrack: newNetTracker(), |
| 114 | offers: map[string]offer{}, now: time.Now} | ||
| 110 | } | 115 | } |
| 111 | 116 | ||
| 112 | // Serve accepts QUIC connections until ctx is cancelled. | 117 | // Serve accepts QUIC connections until ctx is cancelled. |
| @@ -205,6 +210,7 @@ func (s *Service) handleConn(ctx context.Context, conn quic.Connection) { | |||
| 205 | } | 210 | } |
| 206 | s.reg.RecordConnect(hostID) | 211 | s.reg.RecordConnect(hostID) |
| 207 | s.reg.SetAgentVersion(hostID, h.GetFacts().GetAgentVersion()) | 212 | s.reg.SetAgentVersion(hostID, h.GetFacts().GetAgentVersion()) |
| 213 | s.reg.SetHostNetworks(hostID, h.GetHostNetworks()) | ||
| 208 | s.clearOfferIfDone(hostID, h.GetFacts().GetAgentVersion()) | 214 | s.clearOfferIfDone(hostID, h.GetFacts().GetAgentVersion()) |
| 209 | slog.Info("agent connected", "host", hostID, "provisioner", h.GetProvisioner(), "last_seen_epoch", h.GetLastSeenEpoch()) | 215 | slog.Info("agent connected", "host", hostID, "provisioner", h.GetProvisioner(), "last_seen_epoch", h.GetLastSeenEpoch()) |
| 210 | 216 | ||
| @@ -380,6 +386,11 @@ func (s *Service) buildSnapshot(hostID string) (*pb.DesiredStateSnapshot, error) | |||
| 380 | SshAuthorizedKey: v.SSHAuthorizedKey, | 386 | SshAuthorizedKey: v.SSHAuthorizedKey, |
| 381 | SshUserCaAuthorizedKeys: cas, | 387 | SshUserCaAuthorizedKeys: cas, |
| 382 | SshHostCert: v.SSHHostCert, | 388 | SshHostCert: v.SSHHostCert, |
| 389 | // Which underlay this guest attaches to: a named host network, or | ||
| 390 | // "" for the NAT default. Admission has already proven the host | ||
| 391 | // advertises the name, so the agent is only being told what it | ||
| 392 | // said it could serve. | ||
| 393 | Network: v.Network, | ||
| 383 | // With a CA in hand the fleet issues host certificates, so a guest | 394 | // With a CA in hand the fleet issues host certificates, so a guest |
| 384 | // must present one. The host generates the key, reports the public | 395 | // must present one. The host generates the key, reports the public |
| 385 | // half, and holds the guest at the gate until the certificate for | 396 | // half, and holds the guest at the gate until the certificate for |
| @@ -523,11 +534,33 @@ func (s *Service) applyReport(hostID string, rep *pb.ActualStateReport) { | |||
| 523 | // the cache is updated only after a successful write, so a rejected write | 534 | // the cache is updated only after a successful write, so a rejected write |
| 524 | // never suppresses the next retry. | 535 | // never suppresses the next retry. |
| 525 | for _, v := range rep.GetVms() { | 536 | for _, v := range rep.GetVms() { |
| 537 | vmID := v.GetVmId() | ||
| 538 | // The address the site's own DHCP server granted this guest on its | ||
| 539 | // named network. Recorded in every phase, not just the two below: the | ||
| 540 | // lease can land while the guest is still booting, and it is a fact | ||
| 541 | // about the guest's second NIC rather than about its lifecycle. Empty | ||
| 542 | // is "not discovered", never "gone" — the rule assigned_ip and | ||
| 543 | // guest_cidr both follow, enforced in the UPDATE itself. | ||
| 544 | // | ||
| 545 | // The cache below commits the RAW reported netIP, while the store may | ||
| 546 | // still drop it as unusable and keep the prior value — unlike | ||
| 547 | // statusTracker, this tracker never learns what the store actually | ||
| 548 | // wrote. That divergence is benign: a later good address still reads | ||
| 549 | // as a change and lands, and a repeat of the same unusable address is | ||
| 550 | // just a no-op skip, which is no different from what the store would | ||
| 551 | // have done with it anyway. | ||
| 552 | if netIP := v.GetNetworkIp(); netIP != "" { | ||
| 553 | if err := s.netIPTrack.writeThrough(vmID, netIP, func() error { | ||
| 554 | return s.st.RecordVMNetworkIP(vmID, hostID, netIP) | ||
| 555 | }); err != nil { | ||
| 556 | slog.Warn("record vm network ip", "vm", vmID, "host", hostID, "ip", netIP, "err", err) | ||
| 557 | } | ||
| 558 | } | ||
| 559 | |||
| 526 | phase := v.GetPhase() | 560 | phase := v.GetPhase() |
| 527 | if phase != "ready" && phase != "failed" { | 561 | if phase != "ready" && phase != "failed" { |
| 528 | continue | 562 | continue |
| 529 | } | 563 | } |
| 530 | vmID := v.GetVmId() | ||
| 531 | // Pass the RAW reported ip to RecordVMStatus to preserve its | 564 | // Pass the RAW reported ip to RecordVMStatus to preserve its |
| 532 | // empty-ip-keeps-prior UPDATE semantics, and cache the address it | 565 | // empty-ip-keeps-prior UPDATE semantics, and cache the address it |
| 533 | // reports back rather than the one we sent it. | 566 | // reports back rather than the one we sent it. |
| @@ -593,6 +626,7 @@ func (s *Service) applyReport(hostID string, rep *pb.ActualStateReport) { | |||
| 593 | // memory-bounding thing regardless of whether the delete succeeds). | 626 | // memory-bounding thing regardless of whether the delete succeeds). |
| 594 | s.tracker.forget(id) | 627 | s.tracker.forget(id) |
| 595 | s.certTrack.forget(id) | 628 | s.certTrack.forget(id) |
| 629 | s.netIPTrack.forget(id) | ||
| 596 | if err := s.st.HardDeleteVM(id); err != nil { | 630 | if err := s.st.HardDeleteVM(id); err != nil { |
| 597 | slog.Warn("HardDeleteVM failed", "vm", id, "host", hostID, "err", err) | 631 | slog.Warn("HardDeleteVM failed", "vm", id, "host", hostID, "err", err) |
| 598 | } else { | 632 | } else { |
internal/server/syncsvc/syncsvc_test.go
| Old | New | ||
|---|---|---|---|
| @@ -120,6 +120,13 @@ func dial(t *testing.T, addr, fp, hostID, cred string) (*testConn, error) { | |||
| 120 | // host change backend. | 120 | // host change backend. |
| 121 | func dialAs(t *testing.T, addr, fp, hostID, cred, provisioner string) (*testConn, error) { | 121 | func dialAs(t *testing.T, addr, fp, hostID, cred, provisioner string) (*testConn, error) { |
| 122 | t.Helper() | 122 | t.Helper() |
| 123 | return dialHello(t, addr, fp, &pb.Hello{HostId: hostID, Provisioner: provisioner, Credential: cred}) | ||
| 124 | } | ||
| 125 | |||
| 126 | // dialHello is the connect handshake with the Hello spelled out, for the tests | ||
| 127 | // that care what an agent says about itself on the way in. | ||
| 128 | func dialHello(t *testing.T, addr, fp string, h *pb.Hello) (*testConn, error) { | ||
| 129 | t.Helper() | ||
| 123 | ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) | 130 | ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) |
| 124 | defer cancel() | 131 | defer cancel() |
| 125 | conn, err := quic.DialAddr(ctx, addr, transport.ClientTLS(fp), | 132 | conn, err := quic.DialAddr(ctx, addr, transport.ClientTLS(fp), |
| @@ -131,8 +138,7 @@ func dialAs(t *testing.T, addr, fp, hostID, cred, provisioner string) (*testConn | |||
| 131 | if err != nil { | 138 | if err != nil { |
| 132 | return nil, err | 139 | return nil, err |
| 133 | } | 140 | } |
| 134 | hello := &pb.AgentMessage{Msg: &pb.AgentMessage_Hello{Hello: &pb.Hello{ | 141 | hello := &pb.AgentMessage{Msg: &pb.AgentMessage_Hello{Hello: h}} |
| 135 | HostId: hostID, Provisioner: provisioner, Credential: cred}}} | ||
| 136 | if err := transport.WriteMsg(up, hello); err != nil { | 142 | if err := transport.WriteMsg(up, hello); err != nil { |
| 137 | return nil, err | 143 | return nil, err |
| 138 | } | 144 | } |
internal/transport/fieldnumbers_test.go
| Old | New | ||
|---|---|---|---|
| @@ -55,6 +55,7 @@ var wireSchema = map[string]map[string]protoreflect.FieldNumber{ | |||
| 55 | "capacity": 8, | 55 | "capacity": 8, |
| 56 | "credential": 9, | 56 | "credential": 9, |
| 57 | "facts": 10, | 57 | "facts": 10, |
| 58 | "host_networks": 11, | ||
| 58 | }, | 59 | }, |
| 59 | "Capacity": { | 60 | "Capacity": { |
| 60 | "vcpus": 1, | 61 | "vcpus": 1, |
| @@ -88,6 +89,7 @@ var wireSchema = map[string]map[string]protoreflect.FieldNumber{ | |||
| 88 | "last_error": 5, | 89 | "last_error": 5, |
| 89 | "ssh_host_pubkey": 6, | 90 | "ssh_host_pubkey": 6, |
| 90 | "status_detail": 7, | 91 | "status_detail": 7, |
| 92 | "network_ip": 8, | ||
| 91 | }, | 93 | }, |
| 92 | "QuarantinedVM": { | 94 | "QuarantinedVM": { |
| 93 | "vm_id": 1, | 95 | "vm_id": 1, |
| @@ -123,6 +125,7 @@ var wireSchema = map[string]map[string]protoreflect.FieldNumber{ | |||
| 123 | "ssh_host_cert": 17, | 125 | "ssh_host_cert": 17, |
| 124 | "ssh_user_ca_authorized_keys": 18, | 126 | "ssh_user_ca_authorized_keys": 18, |
| 125 | "host_cert_required": 19, | 127 | "host_cert_required": 19, |
| 128 | "network": 20, | ||
| 126 | }, | 129 | }, |
| 127 | "DesiredStateSnapshot": { | 130 | "DesiredStateSnapshot": { |
| 128 | "epoch": 1, | 131 | "epoch": 1, |
proto/eitri/v1/sync.proto
| Old | New | ||
|---|---|---|---|
| @@ -32,6 +32,11 @@ message Hello { | |||
| 32 | Capacity capacity = 8; | 32 | Capacity capacity = 8; |
| 33 | string credential = 9; // Bearer host credential, verified in first frame | 33 | string credential = 9; // Bearer host credential, verified in first frame |
| 34 | HostFacts facts = 10; // best-effort static host identity; refreshed each Hello | 34 | HostFacts facts = 10; // best-effort static host identity; refreshed each Hello |
| 35 | // The named guest networks this host is configured to serve | ||
| 36 | // (--host-network name=bridge), names only. In Hello rather than the | ||
| 37 | // report because the set changes only with agent configuration, i.e. a | ||
| 38 | // restart. Empty for a Mac, and for a Linux host with none configured. | ||
| 39 | repeated string host_networks = 11; | ||
| 35 | } | 40 | } |
| 36 | 41 | ||
| 37 | message Capacity { | 42 | message Capacity { |
| @@ -69,7 +74,7 @@ message ActualVM { | |||
| 69 | string vm_id = 1; | 74 | string vm_id = 1; |
| 70 | string power = 2; // "running"|"stopped" | 75 | string power = 2; // "running"|"stopped" |
| 71 | string phase = 3; // "creating"|"ready"|"failed"|"quarantined" | 76 | string phase = 3; // "creating"|"ready"|"failed"|"quarantined" |
| 72 | string ip = 4; // the address this guest has, however its host came by it | 77 | string ip = 4; // the guest's address on its host's NAT underlay — every guest has one, from boot |
| 73 | string last_error = 5; | 78 | string last_error = 5; |
| 74 | // The guest's ed25519 HOST public key. It is generated on the host, and the | 79 | // The guest's ed25519 HOST public key. It is generated on the host, and the |
| 75 | // private half never leaves it — this is the only half that travels. Sent on | 80 | // private half never leaves it — this is the only half that travels. Sent on |
| @@ -85,6 +90,15 @@ message ActualVM { | |||
| 85 | // VM and means only "nothing to add": an agent that predates this field sends | 90 | // VM and means only "nothing to add": an agent that predates this field sends |
| 86 | // it never, and a console reading it reads exactly what it read before. | 91 | // it never, and a console reading it reads exactly what it read before. |
| 87 | string status_detail = 7; | 92 | string status_detail = 7; |
| 93 | // The address the site's DHCP server granted this guest on its SECOND NIC, | ||
| 94 | // the one attached to the named host network its spec asked for (see | ||
| 95 | // VMDesired.network). Empty for the guests that have no such NIC — the | ||
| 96 | // majority — and for one DHCP round-trip after a networked guest boots, | ||
| 97 | // because the host learns it by watching the exchange rather than granting | ||
| 98 | // it. Never a substitute for ip: that one is known before the guest is even | ||
| 99 | // running, which is why the gate path uses it and this field never appears | ||
| 100 | // there. | ||
| 101 | string network_ip = 8; | ||
| 88 | } | 102 | } |
| 89 | 103 | ||
| 90 | message QuarantinedVM { | 104 | message QuarantinedVM { |
| @@ -164,6 +178,12 @@ message VMDesired { | |||
| 164 | // the agent could not tell "the gate is off, boot uncertified" from "your | 178 | // the agent could not tell "the gate is off, boot uncertified" from "your |
| 165 | // certificate has not come back yet" — both are an empty ssh_host_cert. | 179 | // certificate has not come back yet" — both are an empty ssh_host_cert. |
| 166 | bool host_cert_required = 19; | 180 | bool host_cert_required = 19; |
| 181 | // The named host network this guest attaches to, or "" for the host's | ||
| 182 | // NAT'd private bridge (the default, and the only behavior agents before | ||
| 183 | // this field know). A name is only ever placed on a host that advertised | ||
| 184 | // it (see Hello.host_networks); an agent that has the name but no longer | ||
| 185 | // the configuration fails the VM legibly rather than silently NAT-ing it. | ||
| 186 | string network = 20; | ||
| 167 | } | 187 | } |
| 168 | 188 | ||
| 169 | message DesiredStateSnapshot { | 189 | message DesiredStateSnapshot { |
scripts/coverage.sh
| Old | New | ||
|---|---|---|---|
| @@ -27,12 +27,22 @@ declare -A FLOOR=( | |||
| 27 | [internal/agent/ipalloc]=90 | 27 | [internal/agent/ipalloc]=90 |
| 28 | [internal/agent/hostinfo]=99 | 28 | [internal/agent/hostinfo]=99 |
| 29 | [internal/agent/imagecache]=80 | 29 | [internal/agent/imagecache]=80 |
| 30 | [internal/agent/netenv]=85 | 30 | [internal/agent/netenv]=88 |
| 31 | # netsnoop's parser, its packet filter and the direction check that decides | ||
| 32 | # whether a lease is real are all covered here; what is left is opening and | ||
| 33 | # binding the AF_PACKET socket, which needs CAP_NET_RAW and a live tap and is | ||
| 34 | # proven on real hardware, so the package number stays in the 60s. | ||
| 35 | [internal/agent/netsnoop]=63 | ||
| 31 | [internal/agent/cloudhv]=86 | 36 | [internal/agent/cloudhv]=86 |
| 32 | [internal/agent/hyperlog]=92 | 37 | [internal/agent/hyperlog]=92 |
| 33 | [internal/agent/pidfile]=100 | 38 | [internal/agent/pidfile]=100 |
| 34 | [internal/agent/vfkit]=88 | 39 | [internal/agent/vfkit]=88 |
| 35 | [internal/agent/syncclient]=82 | 40 | # syncclient has a load-sensitive timing test: 82.3% measured standalone, but |
| 41 | # 81.8–83.1% under the all-package parallel run, where a loaded machine can | ||
| 42 | # cost it a branch. The floor sits under the bottom of that range rather than | ||
| 43 | # under the standalone number — a floor that only holds on an idle machine | ||
| 44 | # fails CI at random (collab issue 4cb268b3). | ||
| 45 | [internal/agent/syncclient]=81 | ||
| 36 | # exposeproxy runs real proxy goroutines; its coverage wobbles run to run | 46 | # exposeproxy runs real proxy goroutines; its coverage wobbles run to run |
| 37 | # (measured 92.5–93.8%), so this floor carries a margin the others don't need. | 47 | # (measured 92.5–93.8%), so this floor carries a margin the others don't need. |
| 38 | [internal/agent/exposeproxy]=91 | 48 | [internal/agent/exposeproxy]=91 |
| @@ -56,7 +66,7 @@ declare -A FLOOR=( | |||
| 56 | [internal/server/release]=93 | 66 | [internal/server/release]=93 |
| 57 | [internal/agent/selfupdate]=70 | 67 | [internal/agent/selfupdate]=70 |
| 58 | [internal/agent/bootstrap]=73 | 68 | [internal/agent/bootstrap]=73 |
| 59 | [internal/agent/run]=40 | 69 | [internal/agent/run]=48 |
| 60 | [internal/server/hosttoken]=100 | 70 | [internal/server/hosttoken]=100 |
| 61 | [internal/server/hub]=94 | 71 | [internal/server/hub]=94 |
| 62 | [internal/server/syncsvc]=85 | 72 | [internal/server/syncsvc]=85 |
| @@ -66,21 +76,21 @@ declare -A FLOOR=( | |||
| 66 | [internal/site]=84 | 76 | [internal/site]=84 |
| 67 | [internal/smoke]=56 | 77 | [internal/smoke]=56 |
| 68 | [internal/cli]=71 | 78 | [internal/cli]=71 |
| 69 | [internal/mcpserver]=73 | 79 | [internal/mcpserver]=76 |
| 70 | [internal/oidcprovider]=79 | 80 | [internal/oidcprovider]=79 |
| 71 | [internal/server/config]=100 | 81 | [internal/server/config]=100 |
| 72 | [internal/cloudinit]=77 | 82 | [internal/cloudinit]=77 |
| 73 | [internal/covsnap]=77 | 83 | [internal/covsnap]=77 |
| 74 | [internal/joinblob]=96 | 84 | [internal/joinblob]=96 |
| 75 | [internal/gateclient]=60 | 85 | [internal/gateclient]=60 |
| 76 | # internal/names and internal/random have no tests of their own: they are | 86 | [internal/names]=74 |
| 77 | # exercised only through the packages that call them, so they report 0.0% | 87 | # internal/random has no tests of its own: it is exercised only through the |
| 78 | # here. A real floor would be a lie. The 0 floor keeps them COUNTED — so the | 88 | # packages that call it, so it reports 0.0% here. A real floor would be a |
| 79 | # renamed/removed-package guard still accounts for them — while asserting | 89 | # lie. The 0 floor keeps it COUNTED — so the renamed/removed-package guard |
| 80 | # nothing about a number nothing measures. Give a package a real floor only | 90 | # still accounts for it — while asserting nothing about a number nothing |
| 81 | # once it has tests of its own. (internal/pb is generated and stays unlisted; | 91 | # measures. Give it a real floor once it has tests of its own. (internal/pb |
| 82 | # internal/guest has no tests AND no coverable line to gate yet.) | 92 | # is generated and stays unlisted; internal/guest has no tests AND no |
| 83 | [internal/names]=0 | 93 | # coverable line to gate yet.) |
| 84 | [internal/random]=0 | 94 | [internal/random]=0 |
| 85 | ) | 95 | ) |
| 86 | 96 | ||
web/src/lib/api-types.ts
| Old | New | ||
|---|---|---|---|
| @@ -1526,6 +1526,7 @@ export interface components { | |||
| 1526 | image_url?: string; | 1526 | image_url?: string; |
| 1527 | mem_mb?: number; | 1527 | mem_mb?: number; |
| 1528 | name?: string; | 1528 | name?: string; |
| 1529 | network?: string; | ||
| 1529 | power_state?: string; | 1530 | power_state?: string; |
| 1530 | ssh_authorized_key?: string; | 1531 | ssh_authorized_key?: string; |
| 1531 | vcpus?: number; | 1532 | vcpus?: number; |
| @@ -1598,6 +1599,7 @@ export interface components { | |||
| 1598 | cpu_model: string; | 1599 | cpu_model: string; |
| 1599 | /** Format: date-time */ | 1600 | /** Format: date-time */ |
| 1600 | enrolled_at: string; | 1601 | enrolled_at: string; |
| 1602 | host_networks: string[]; | ||
| 1601 | id: string; | 1603 | id: string; |
| 1602 | kernel: string; | 1604 | kernel: string; |
| 1603 | /** Format: date-time */ | 1605 | /** Format: date-time */ |
| @@ -1700,6 +1702,8 @@ export interface components { | |||
| 1700 | lifecycle: string; | 1702 | lifecycle: string; |
| 1701 | mem_mb: number; | 1703 | mem_mb: number; |
| 1702 | name: string; | 1704 | name: string; |
| 1705 | network: string; | ||
| 1706 | network_ip: string; | ||
| 1703 | phase: string; | 1707 | phase: string; |
| 1704 | power_state: string; | 1708 | power_state: string; |
| 1705 | status: string; | 1709 | status: string; |
web/src/lib/fleet.svelte.ts
| Old | New | ||
|---|---|---|---|
| @@ -511,6 +511,45 @@ export function vmIP(vm: VM): string { | |||
| 511 | return vm.assigned_ip || '—'; | 511 | return vm.assigned_ip || '—'; |
| 512 | } | 512 | } |
| 513 | 513 | ||
| 514 | /** VM_IP_HINT is the note beside a VM's address. One sentence for every guest, | ||
| 515 | * because every guest has this NIC and it says the same thing on all of them: | ||
| 516 | * an address on the host's private bridge, NAT'd out, reachable off that host | ||
| 517 | * only through the gate or a published port. A named network does not change | ||
| 518 | * this address — it adds a second one, which vmNetworkAddr reports. */ | ||
| 519 | export const VM_IP_HINT = '(host bridge, NAT—not reachable off-host)'; | ||
| 520 | |||
| 521 | /** vmNetworkAddr is the address a guest's named network granted its second | ||
| 522 | * NIC, or '' when there is nothing to show: a guest that asked for no | ||
| 523 | * network, or one whose network has not answered yet. Used by vmNetworkValue | ||
| 524 | * to extend the Network row's value once the address is known, and by the | ||
| 525 | * fleet-table search box, which matches on it directly. | ||
| 526 | * | ||
| 527 | * Both fields are checked because they can disagree: a VM created before | ||
| 528 | * the freeze, or one a stale row otherwise carries a leftover network_ip | ||
| 529 | * for, must not read as networked once vm.network is empty — an address | ||
| 530 | * with no network to hang it on is not a fact worth showing. */ | ||
| 531 | export function vmNetworkAddr(vm: VM): string { | ||
| 532 | return vm.network && vm.network_ip ? vm.network_ip : ''; | ||
| 533 | } | ||
| 534 | |||
| 535 | /** vmNetworkAddrHint names the network that did the addressing, because the | ||
| 536 | * address alone does not say which of the host's networks it came from. */ | ||
| 537 | export function vmNetworkAddrHint(vm: VM): string { | ||
| 538 | return `(on ${vm.network}, addressed by that network's DHCP)`; | ||
| 539 | } | ||
| 540 | |||
| 541 | /** vmNetworkValue is the Network row's whole value: the operator's own name | ||
| 542 | * for the network alone, until that network's DHCP has answered — then the | ||
| 543 | * name and the address it granted, joined the way the console joins a | ||
| 544 | * primary value onto a secondary one (host status's "· offline", the | ||
| 545 | * platform labels, os·kernel, sessionSummary). One row, one fact, and no | ||
| 546 | * word in it that eitri coined; '' when the guest asked for no network. */ | ||
| 547 | export function vmNetworkValue(vm: VM): string { | ||
| 548 | if (!vm.network) return ''; | ||
| 549 | const addr = vmNetworkAddr(vm); | ||
| 550 | return addr ? `${vm.network} · ${addr}` : vm.network; | ||
| 551 | } | ||
| 552 | |||
| 514 | /** vmStatus is the single lifecycle status folded from the orthogonal state axes. | 553 | /** vmStatus is the single lifecycle status folded from the orthogonal state axes. |
| 515 | * | 554 | * |
| 516 | * The server owns this derivation (deriveLifecycle in internal/server/api) | 555 | * The server owns this derivation (deriveLifecycle in internal/server/api) |
| @@ -531,6 +570,19 @@ export function hostStatusLabel(h: Host): string { | |||
| 531 | return `${h.status}${h.online ? '' : ' · offline'}`; | 570 | return `${h.status}${h.online ? '' : ' · offline'}`; |
| 532 | } | 571 | } |
| 533 | 572 | ||
| 573 | /** hostNetworkOptions is the named networks a create may ask a host for: the | ||
| 574 | * last set that host advertised, whether or not it is online this instant. | ||
| 575 | * | ||
| 576 | * Online is deliberately not consulted. Gating on it empties the list for | ||
| 577 | * every SSE tick a host spends dark, and an emptied list is how a create for | ||
| 578 | * a named network turns into a NAT VM without anyone saying so; a create | ||
| 579 | * against a host that really is gone is refused by the server, which is the | ||
| 580 | * right place for that refusal. A host not in the fleet (or none selected) | ||
| 581 | * offers nothing. */ | ||
| 582 | export function hostNetworkOptions(hosts: Host[], hostID: string | undefined): string[] { | ||
| 583 | return hosts.find((h) => h.id === hostID)?.host_networks ?? []; | ||
| 584 | } | ||
| 585 | |||
| 534 | /** shortFingerprint abbreviates an OpenSSH SHA256 fingerprint for a table cell, | 586 | /** shortFingerprint abbreviates an OpenSSH SHA256 fingerprint for a table cell, |
| 535 | * keeping the SHA256: prefix (so it still reads as a fingerprint and not a | 587 | * keeping the SHA256: prefix (so it still reads as a fingerprint and not a |
| 536 | * hash of some other kind) and enough of the digest to tell two CAs apart by | 588 | * hash of some other kind) and enough of the digest to tell two CAs apart by |
web/src/lib/fleet.test.ts
| Old | New | ||
|---|---|---|---|
| @@ -4,6 +4,7 @@ import { | |||
| 4 | fleet, | 4 | fleet, |
| 5 | formatUptime, | 5 | formatUptime, |
| 6 | hostBundle, | 6 | hostBundle, |
| 7 | hostNetworkOptions, | ||
| 7 | hostStatusLabel, | 8 | hostStatusLabel, |
| 8 | joinCommands, | 9 | joinCommands, |
| 9 | sessionSummary, | 10 | sessionSummary, |
| @@ -11,6 +12,9 @@ import { | |||
| 11 | upgradeStuck, | 12 | upgradeStuck, |
| 12 | upgradeStuckNote, | 13 | upgradeStuckNote, |
| 13 | vmDetail, | 14 | vmDetail, |
| 15 | vmNetworkAddr, | ||
| 16 | vmNetworkAddrHint, | ||
| 17 | vmNetworkValue, | ||
| 14 | vmTrustStale, | 18 | vmTrustStale, |
| 15 | UPGRADE_STUCK_S, | 19 | UPGRADE_STUCK_S, |
| 16 | type Exposure, | 20 | type Exposure, |
| @@ -46,6 +50,8 @@ function vmTrusting(trusted_cas: TrustedCA[] | null): VM { | |||
| 46 | lifecycle: 'ready', | 50 | lifecycle: 'ready', |
| 47 | mem_mb: 1024, | 51 | mem_mb: 1024, |
| 48 | name: 'guest', | 52 | name: 'guest', |
| 53 | network: '', | ||
| 54 | network_ip: '', | ||
| 49 | phase: 'ready', | 55 | phase: 'ready', |
| 50 | power_state: 'running', | 56 | power_state: 'running', |
| 51 | status: 'ready', | 57 | status: 'ready', |
| @@ -152,6 +158,7 @@ function hostUpgrading( | |||
| 152 | capacity: { vcpus: 8, mem_mb: 16384, disk_gb: 256 }, | 158 | capacity: { vcpus: 8, mem_mb: 16384, disk_gb: 256 }, |
| 153 | cpu_model: 'AMD Ryzen 9 7950X', | 159 | cpu_model: 'AMD Ryzen 9 7950X', |
| 154 | enrolled_at: '2026-08-01T09:00:00Z', | 160 | enrolled_at: '2026-08-01T09:00:00Z', |
| 161 | host_networks: [], | ||
| 155 | id: 'host-1', | 162 | id: 'host-1', |
| 156 | kernel: '6.15.4-arch1-1', | 163 | kernel: '6.15.4-arch1-1', |
| 157 | last_seen: '2026-08-11T09:00:00Z', | 164 | last_seen: '2026-08-11T09:00:00Z', |
| @@ -282,6 +289,34 @@ describe('hostStatusLabel', () => { | |||
| 282 | }); | 289 | }); |
| 283 | }); | 290 | }); |
| 284 | 291 | ||
| 292 | describe('hostNetworkOptions', () => { | ||
| 293 | /** advertising is one host in the fleet, named and serving `networks`. */ | ||
| 294 | function advertising(id: string, networks: string[], online = true): Host { | ||
| 295 | return { ...hostUpgrading('v0.0.6', null, online), id, host_networks: networks }; | ||
| 296 | } | ||
| 297 | |||
| 298 | test('the selected host offers the networks it advertised', () => { | ||
| 299 | const hosts = [advertising('host-1', ['lan', 'lab']), advertising('host-2', ['dmz'])]; | ||
| 300 | expect(hostNetworkOptions(hosts, 'host-2')).toEqual(['dmz']); | ||
| 301 | }); | ||
| 302 | |||
| 303 | test('a host that has gone dark still offers them', () => { | ||
| 304 | // The regression this guards: a host flapping offline for one snapshot | ||
| 305 | // used to empty the list, which cleared the operator's choice and sent a | ||
| 306 | // create for a named network as a create for NAT. | ||
| 307 | expect(hostNetworkOptions([advertising('host-1', ['lan'], false)], 'host-1')).toEqual(['lan']); | ||
| 308 | }); | ||
| 309 | |||
| 310 | test('a host advertising nothing offers nothing', () => { | ||
| 311 | expect(hostNetworkOptions([advertising('host-1', [])], 'host-1')).toEqual([]); | ||
| 312 | }); | ||
| 313 | |||
| 314 | test('a host not in the fleet offers nothing', () => { | ||
| 315 | expect(hostNetworkOptions([advertising('host-1', ['lan'])], 'host-2')).toEqual([]); | ||
| 316 | expect(hostNetworkOptions([advertising('host-1', ['lan'])], undefined)).toEqual([]); | ||
| 317 | }); | ||
| 318 | }); | ||
| 319 | |||
| 285 | describe('capacityReading', () => { | 320 | describe('capacityReading', () => { |
| 286 | test('a half-full host has room and a calm level', () => { | 321 | test('a half-full host has room and a calm level', () => { |
| 287 | expect(capacityReading(2, 4)).toEqual({ pct: 50, free: 2, over: 0, level: 'ok' }); | 322 | expect(capacityReading(2, 4)).toEqual({ pct: 50, free: 2, over: 0, level: 'ok' }); |
| @@ -436,6 +471,45 @@ describe('vmDetail', () => { | |||
| 436 | }); | 471 | }); |
| 437 | }); | 472 | }); |
| 438 | 473 | ||
| 474 | describe('the two addresses a guest can have', () => { | ||
| 475 | test('a guest with no named network has no discovered address', () => { | ||
| 476 | expect(vmNetworkAddr(vmTrusting(null))).toBe(''); | ||
| 477 | }); | ||
| 478 | |||
| 479 | test('a guest whose network has not answered yet has no discovered address', () => { | ||
| 480 | // The NIC exists; the address does not. A placeholder would present a | ||
| 481 | // fact nobody has. | ||
| 482 | expect(vmNetworkAddr({ ...vmTrusting(null), network: 'lan' })).toBe(''); | ||
| 483 | }); | ||
| 484 | |||
| 485 | test('a leftover network_ip with no network to hang it on has no discovered address', () => { | ||
| 486 | // A stale or pre-freeze row can carry an address with an empty | ||
| 487 | // network — that pairing is not a fact worth showing. | ||
| 488 | expect(vmNetworkAddr({ ...vmTrusting(null), network_ip: '192.168.0.42' })).toBe(''); | ||
| 489 | }); | ||
| 490 | |||
| 491 | test('a discovered address is shown with the network that granted it', () => { | ||
| 492 | const vm = { ...vmTrusting(null), network: 'lan', network_ip: '192.168.0.42' }; | ||
| 493 | expect(vmNetworkAddr(vm)).toBe('192.168.0.42'); | ||
| 494 | expect(vmNetworkAddrHint(vm)).toBe("(on lan, addressed by that network's DHCP)"); | ||
| 495 | }); | ||
| 496 | }); | ||
| 497 | |||
| 498 | describe('vmNetworkValue', () => { | ||
| 499 | test('a guest with no named network has no value', () => { | ||
| 500 | expect(vmNetworkValue(vmTrusting(null))).toBe(''); | ||
| 501 | }); | ||
| 502 | |||
| 503 | test('a guest whose network has not answered yet shows the name alone', () => { | ||
| 504 | expect(vmNetworkValue({ ...vmTrusting(null), network: 'lan' })).toBe('lan'); | ||
| 505 | }); | ||
| 506 | |||
| 507 | test('a discovered address joins onto the name', () => { | ||
| 508 | const vm = { ...vmTrusting(null), network: 'lan', network_ip: '192.168.0.42' }; | ||
| 509 | expect(vmNetworkValue(vm)).toBe('lan · 192.168.0.42'); | ||
| 510 | }); | ||
| 511 | }); | ||
| 512 | |||
| 439 | describe('sessionSummary', () => { | 513 | describe('sessionSummary', () => { |
| 440 | test('a counted port says what it holds and what it has turned away', () => { | 514 | test('a counted port says what it holds and what it has turned away', () => { |
| 441 | expect(sessionSummary(exposure({ active: 7, refused: 12, dropped: 3 }))).toBe( | 515 | expect(sessionSummary(exposure({ active: 7, refused: 12, dropped: 3 }))).toBe( |
web/src/routes/+page.svelte
| Old | New | ||
|---|---|---|---|
| @@ -12,6 +12,7 @@ | |||
| 12 | restoreVM, | 12 | restoreVM, |
| 13 | vmPower, | 13 | vmPower, |
| 14 | vmIP, | 14 | vmIP, |
| 15 | vmNetworkValue, | ||
| 15 | vmPowerAction, | 16 | vmPowerAction, |
| 16 | vmTrustStale, | 17 | vmTrustStale, |
| 17 | deleteConfirm, | 18 | deleteConfirm, |
| @@ -21,6 +22,7 @@ | |||
| 21 | refreshUserCAs, | 22 | refreshUserCAs, |
| 22 | capacityReading, | 23 | capacityReading, |
| 23 | hostStatusLabel, | 24 | hostStatusLabel, |
| 25 | hostNetworkOptions, | ||
| 24 | type CreateVMRequest, | 26 | type CreateVMRequest, |
| 25 | type VM | 27 | type VM |
| 26 | } from '$lib/fleet.svelte'; | 28 | } from '$lib/fleet.svelte'; |
| @@ -84,7 +86,7 @@ | |||
| 84 | const shownVMs = $derived( | 86 | const shownVMs = $derived( |
| 85 | fleet.vms.filter((v) => { | 87 | fleet.vms.filter((v) => { |
| 86 | if (!needle) return true; | 88 | if (!needle) return true; |
| 87 | return `${v.name} ${hostName(v.host_id)} ${vmStatus(v)} ${vmPower(v)} ${vmIP(v)}` | 89 | return `${v.name} ${hostName(v.host_id)} ${vmStatus(v)} ${vmPower(v)} ${vmIP(v)} ${vmNetworkValue(v)}` |
| 88 | .toLowerCase() | 90 | .toLowerCase() |
| 89 | .includes(needle); | 91 | .includes(needle); |
| 90 | }) | 92 | }) |
| @@ -92,6 +94,28 @@ | |||
| 92 | 94 | ||
| 93 | let form = $state<CreateVMRequest>({ host_id: '' }); | 95 | let form = $state<CreateVMRequest>({ host_id: '' }); |
| 94 | 96 | ||
| 97 | // A network belongs to a host, so the choice on offer is the selected host's | ||
| 98 | // advertised set—the last set it advertised, online or not (hostNetworkOptions | ||
| 99 | // carries why online is no part of it). | ||
| 100 | const hostNetworks = $derived(hostNetworkOptions(fleet.hosts, form.host_id)); | ||
| 101 | // A picked name belongs to the host it was picked on, so only a change of | ||
| 102 | // host clears it—not the name's absence from the list above. Clearing on | ||
| 103 | // absence would let a host flapping dark for one tick silently turn a | ||
| 104 | // named-network create into a NAT one, which is the outcome this design | ||
| 105 | // forbids: a VM that asked for a named network either gets it or does not | ||
| 106 | // exist. A name the newly | ||
| 107 | // chosen host does not serve therefore reaches the server, which refuses it | ||
| 108 | // and says which networks that host has. pickedOn only remembers which host | ||
| 109 | // answered for the current choice, and is deliberately not $state: this | ||
| 110 | // effect both reads and writes it, so a reactive latch would make the effect | ||
| 111 | // depend on its own write. Guarded on a real change, the shape Console | ||
| 112 | // resets a reused session with. | ||
| 113 | let pickedOn: string | undefined; | ||
| 114 | $effect(() => { | ||
| 115 | if (pickedOn !== undefined && pickedOn !== form.host_id) form.network = ''; | ||
| 116 | pickedOn = form.host_id; | ||
| 117 | }); | ||
| 118 | |||
| 95 | // noCA is the tenant with nothing registered, which is worth saying before | 119 | // noCA is the tenant with nothing registered, which is worth saying before |
| 96 | // the form is filled in rather than after it is submitted. Held off until | 120 | // the form is filled in rather than after it is submitted. Held off until |
| 97 | // the CAs have actually been fetched, so an empty list that only means | 121 | // the CAs have actually been fetched, so an empty list that only means |
| @@ -270,7 +294,18 @@ | |||
| 270 | {/if} | 294 | {/if} |
| 271 | </td> | 295 | </td> |
| 272 | <td class="num">{vmCountByHost.get(h.id) ?? 0}</td> | 296 | <td class="num">{vmCountByHost.get(h.id) ?? 0}</td> |
| 273 | <td>{h.bridge_cidr || '—'}</td> | 297 | <td> |
| 298 | {h.bridge_cidr || '—'} | ||
| 299 | <!-- The named networks beside the NAT bridge every host has, so | ||
| 300 | that adding --host-network to a unit can be confirmed here | ||
| 301 | rather than inferred from what the create dialog offers. A | ||
| 302 | host serving none says nothing. --> | ||
| 303 | {#if h.host_networks.length > 0} | ||
| 304 | <span class="networks" title="advertised with --host-network" | ||
| 305 | >· {h.host_networks.join(', ')}</span | ||
| 306 | > | ||
| 307 | {/if} | ||
| 308 | </td> | ||
| 274 | <td class="num" class:over={capacityReading(h.allocated.vcpus, h.capacity.vcpus).over > 0} | 309 | <td class="num" class:over={capacityReading(h.allocated.vcpus, h.capacity.vcpus).over > 0} |
| 275 | >{h.allocated.vcpus}/{h.capacity.vcpus || '?'}</td | 310 | >{h.allocated.vcpus}/{h.capacity.vcpus || '?'}</td |
| 276 | > | 311 | > |
| @@ -378,6 +413,17 @@ | |||
| 378 | {/each} | 413 | {/each} |
| 379 | </select> | 414 | </select> |
| 380 | </label> | 415 | </label> |
| 416 | {#if hostNetworks.length > 0} | ||
| 417 | <label> | ||
| 418 | Network | ||
| 419 | <select bind:value={form.network}> | ||
| 420 | <option value="">NAT (default)</option> | ||
| 421 | {#each hostNetworks as n} | ||
| 422 | <option value={n}>{n}</option> | ||
| 423 | {/each} | ||
| 424 | </select> | ||
| 425 | </label> | ||
| 426 | {/if} | ||
| 381 | <label> | 427 | <label> |
| 382 | Name (optional) | 428 | Name (optional) |
| 383 | <input bind:value={form.name} placeholder="auto: sandbox-xxxx" /> | 429 | <input bind:value={form.name} placeholder="auto: sandbox-xxxx" /> |
| @@ -501,6 +547,12 @@ | |||
| 501 | color: var(--bad); | 547 | color: var(--bad); |
| 502 | cursor: help; | 548 | cursor: help; |
| 503 | } | 549 | } |
| 550 | /* Faint, like every other secondary row fact: the bridge is the reading, | ||
| 551 | the names are what else this host can put a guest on. */ | ||
| 552 | .networks { | ||
| 553 | color: var(--faint); | ||
| 554 | cursor: help; | ||
| 555 | } | ||
| 504 | .actions { | 556 | .actions { |
| 505 | display: flex; | 557 | display: flex; |
| 506 | gap: 0.5em; | 558 | gap: 0.5em; |
web/src/routes/vms/[id]/+page.svelte
| Old | New | ||
|---|---|---|---|
| @@ -12,6 +12,10 @@ | |||
| 12 | teardownApprox, | 12 | teardownApprox, |
| 13 | vmPower, | 13 | vmPower, |
| 14 | vmIP, | 14 | vmIP, |
| 15 | VM_IP_HINT, | ||
| 16 | vmNetworkValue, | ||
| 17 | vmNetworkAddr, | ||
| 18 | vmNetworkAddrHint, | ||
| 15 | vmDetail, | 19 | vmDetail, |
| 16 | sessionSummary, | 20 | sessionSummary, |
| 17 | vmPowerAction, | 21 | vmPowerAction, |
| @@ -248,7 +252,13 @@ | |||
| 248 | </td> | 252 | </td> |
| 249 | </tr> | 253 | </tr> |
| 250 | <tr><th>Power</th><td>{vmPower(vm)} (desired: {vm.power_state})</td></tr> | 254 | <tr><th>Power</th><td>{vmPower(vm)} (desired: {vm.power_state})</td></tr> |
| 251 | <tr><th>IP</th><td>{vmIP(vm)} <span class="hint">(host bridge, NAT—not reachable off-host)</span></td></tr> | 255 | <!-- One row for the whole story: the network the operator named at |
| 256 | create, and — once that network's own DHCP has answered — the | ||
| 257 | second address it granted this guest, joined onto the same | ||
| 258 | value. Nothing here is a word eitri invented: the label matches | ||
| 259 | the API field, and the name in the value is the operator's own. --> | ||
| 260 | {#if vmNetworkValue(vm)}<tr><th>Network</th><td>{vmNetworkValue(vm)}{#if vmNetworkAddr(vm)} <span class="hint">{vmNetworkAddrHint(vm)}</span>{/if}</td></tr>{/if} | ||
| 261 | <tr><th>IP</th><td>{vmIP(vm)} <span class="hint">{VM_IP_HINT}</span></td></tr> | ||
| 252 | <tr><th>Resources</th><td>{vm.vcpus}c / {vm.mem_mb}MB / {vm.disk_gb}GB</td></tr> | 262 | <tr><th>Resources</th><td>{vm.vcpus}c / {vm.mem_mb}MB / {vm.disk_gb}GB</td></tr> |
| 253 | <tr><th>Image</th><td class="wrap">{vm.image_url}</td></tr> | 263 | <tr><th>Image</th><td class="wrap">{vm.image_url}</td></tr> |
| 254 | <tr> | 264 | <tr> |