508d3d97

docs: bridged guest networking

a73x 2026-08-06 09:12

Commit message

docs: bridged guest networking

A Linux host can put its guests on the operator's own network: it is told
which bridge to attach taps to, the site's DHCP addresses the guests, and
the agent discovers the result instead of allocating it. Validation of the
bridge it was handed replaces the bridge it used to build.

Hosts already on NAT keep it, recorded in their identity rather than
re-inferred each start. A host that is itself an eitri guest stays there:
its network's DHCP is an eitri agent, which answers only reservations it
holds.

docs/superpowers/plans/2026-08-05-bridged-guest-networking.md

@@ @@ -0,0 +1,309 @@ @@
+# Bridged by default: a guest is a machine on the network
+**Status:** parked — demand-driven. Service exposure covers reaching a guest's
+services; what this plan alone provides is LAN citizenship — broadcast-domain
+discovery (mDNS/SSDP) and operator-owned addressing. It is revived by the
+first workload that needs those. A UDP want alone is not that workload:
+`protocol: udp` on exposures is the smaller answer.
+**Branch:** `v0.0.3`
+**Ethos:** the guest owns the guest, eitri owns the fleet — and the network
+belongs to whoever runs it.
+## The decision
+A Linux host puts its guests on the operator's network. It does not build a
+network of its own, does not hand out addresses, and does not masquerade. It is
+told which bridge to attach taps to, and it refuses to start without one.
+This is a default, not a mode: there is no `--guest-net` flag and nothing in the
+console flips it. A Linux host that has a bridge is bridged; a Linux host that
+does not is a host that will not serve.
+Two things it is not. It is not fleet-wide — macOS cannot bridge, so the NAT
+shape survives on Macs whatever we do here (below). And it is not a licence to
+rebuild the operator's uplink: eitri never enslaves an interface it did not
+create. The bridge is the operator's, built by whatever manages that host's
+network, and the agent's only claim on it is a tap.
+## What is there now
+Every guest sits behind NAT on a host-private bridge. `netenv.Net` builds
+`eitri0`, takes `.1`, enables forwarding, installs one `nft` masquerade rule
+(`netenv.go:172`), and runs eitri's own DHCP responder with a per-VM
+reservation keyed on a deterministic MAC (`netenv.go:240`, `state.go:94`).
+The seam this all hangs off is already the right one. `cloudhv` does not import
+`netenv`: it declares a five-method consumer-owned `Network` interface
+(`cloudhv.go:46`) — `ReserveIP`, `Address`, `CreateTap`, `DeleteTap`,
+`TapName` — and `netenv` happens to satisfy it. The VMM sees one line of the
+whole arrangement, `--net tap=eit-xxxxxxxx,mac=52:54:00:…`
+(`cloudhv.go:151`), and that line is identical under either regime.
+So bridged is a second implementation of an interface that already exists.
+`reconcile`, `buildArgs`, the pumps and the pidfiles do not move.
+## The three things that actually change
+### 1. The mode is derived from the identity, never set
+`state.Identity` (`state.go:43`) gains two fields — `GuestNet string` (`nat` or
+`bridged`) and `Bridge string` — and the answer comes from the record, with no
+flag to disagree with:
+- `GuestNet` set → that, and for `bridged`, on the bridge named in `Bridge`.
+- unset, `BridgeCIDR != ""` → a host enrolled by an older agent. NAT, recorded
+  as `GuestNet: "nat"` on this first start and read from the record thereafter.
+- unset, nothing else → a new host. `--bridge` is required; refuse to start.
+The recording matters more than it looks. Deriving "this host is on NAT" from a
+missing field works exactly once — the next change to what absence means would
+silently re-home hosts that were never asked. Writing the answer down the first
+time a host resolves it makes an existing host's NAT a decision the fleet holds,
+not an inference it re-makes every start. It is the same one-time persist
+`persistGuestCIDR` already does for the subnet (`guestcidr.go:51`).
+This is the precedence `resolveGuestCIDR` already enforces for the subnet
+(`guestcidr.go:33`), and for the reason stated there: a flag that overrides a
+persisted identity moves the network under live guests and every one of them
+loses its address at the next restart. Changing a host's bridge is an edit to
+`identity.json`, exactly as re-homing its subnet is.
+Where the answer is given: `join`. The mode is identity and join is the
+ceremony that mints identity, so `eitri-agent join --bridge br0` is the
+operator answering at the moment the host is asked. Serve accepts the same
+flag as backfill only — consulted when the record is silent, the role
+`cfg.BridgeCIDR` already plays in `resolveGuestCIDR` — and a restart that
+passes it lands in one of three rows:
+- record silent → the flag answers and is persisted.
+- record agrees → no-op; restating the recorded answer is harmless.
+- record contradicts (`nat` recorded, or a different bridge named) → refuse,
+  loudly. Obeying would re-home every guest as a side effect of a restart
+  flag; ignoring would silently discard an instruction the operator stopped
+  the agent to give. The refusal names the record and the sanctioned path:
+  edit `identity.json` with the agent stopped, or decommission and re-join.
+Two orderings that matter. The resolver runs before anything persists a
+fabricated CIDR — `newPlatform` persists `lastResortGuestCIDR` today, and a
+fresh host that reached that code first would classify itself as legacy NAT.
+And `--bridge` with `--bridge-cidr` is a contradiction in terms — one names a
+network the operator owns, the other a network eitri would build — and is
+refused rather than resolved.
+The refusal is the report. The machine that knows which bridge to name is
+the host itself, so when `--bridge` is missing the agent enumerates before it
+exits. A host with bridges gets the candidates and the evidence that matters —
+`br0: up, ports enp4s0` against `docker0: up, no physical port` — and the
+operator reads their own answer off the list. A host with none gets the name
+of the interface they are about to enslave: the default-route interface,
+offered as a suggestion — wrong on an odd topology, and costing nothing
+there, because a suggestion can be ignored. What the agent never does is
+choose. A filter that picked the one plausible candidate would ship guests
+onto the wrong segment silently on exactly the host where it guessed wrong,
+and the whole design rests on the mode being an answer the operator gave.
+Suggest loudly, choose never.
+### 2. Addressing flips from allocate to observe
+There is no reservation to make. The site's DHCP server hands the lease, and the
+agent's job is to find out what it was.
+`ReserveIP` becomes a no-op returning `""` — `attachNet` passes its result
+straight into `CreateTap` (`cloudhv.go:317`) and bridged `CreateTap` ignores it.
+`Address` becomes discovery, and returns `""` until it has an answer.
+Nothing above this needs teaching, because macOS already taught it.
+`noteAddress` treats an empty answer as "I don't know yet" and refuses to blank
+a known address (`reconcile.go:563`); the control plane's guard stopped asking
+about topology; the host reports its own subnet. The consumers that poll for
+`assigned_ip` — the console column, `eitri-mcp`'s wait, the smoke gate — already
+wait for a Mac and will wait the same way here.
+**Discovery mechanism.** Two sources, cheapest first:
+- `/proc/net/arp` — free, no exec, scoped by its Device column, and populated
+  for any guest the host has already spoken to.
+- A passive ARP watcher on the bridge for the cold case, which is the common
+  one: nothing has dialled the guest yet, and dialling is what needs the address.
+  An `AF_PACKET` socket filtered to `ETH_P_ARP` yields sender-MAC → sender-IP,
+  matched against `state.MAC(vmID)`. Every guest broadcasts within seconds of
+  its lease — an RFC 5227 announcement for its own address, and a request for
+  its gateway — so the answer arrives unprompted and updates itself if the
+  site renumbers the guest. The watcher drops unspecified sender addresses: a
+*probe* carries sender-IP `0.0.0.0` by definition — junk that
+  `noteAddress` has no reason to refuse and the tunnel would then dial.
+ARP rather than DHCP snooping deliberately: a DHCP reply may be unicast to the
+guest's MAC, which the bridge will not deliver to the host without putting the
+interface in promiscuous mode. ARP is broadcast and needs no such thing.
+A discovered address is already durable — `reconcile` persists `rec.IP` — so an
+agent restart keeps the last known address while the watcher re-warms, and
+`syncclient`'s tunnel (which refuses an empty address, `client.go:530`) never
+sees a gap it would not have seen on a Mac.
+### 3. Most of the host-wide setup stops running
+Under bridged, `newPlatform` (`wire_linux.go:84`) does none of this:
+| Step | Why it goes |
+|---|---|
+| `EnsureBridge` | the bridge is the operator's; we validate, never create |
+| `StartDHCP` | an eitri DHCP server on the LAN hijacks the site's |
+| `replayReservations` | there are no reservations to replay |
+| `CheckUplinkCollision` | it refuses precisely the bridged case, by definition |
+| `lastResortGuestCIDR` | a fabricated subnet is only right where something builds one |
+`CheckUplinkCollision` (`uplink.go:67`) deserves a note rather than a deletion.
+It fails when the guest subnet contains an address the host is configured with,
+which is the definition of bridged — but it is the correct check for the shape
+it was written for, and the failure it prevents (the host's own address becomes
+local, it stops answering, and the machine best placed to fix it is the one that
+just lost the route) is real. It stays, scoped to legacy NAT.
+What replaces it is validation of what we were given, at startup, before a
+guest can fail illegibly later — split on the line `CheckUplinkCollision`
+already draws: refuse on facts, warn on diagnostics. The facts are that the
+named link exists, is a bridge (`/sys/class/net/<name>/bridge`), and is up.
+These are never wrong under any topology; an operator whose bridge fails them
+typed a name that does not hold. Whether the bridge leads anywhere is not a
+fact but the operator's intent: the default route may sit on the bridge
+itself (the canonical shape — the host's address lives on `br0` and the
+uplink is a port), on another interface entirely with a guests-only bridge on
+a second NIC, or nowhere by design on an isolated segment. Any rule that
+consults the routing table refuses one of the legitimate shapes, so the
+routing table is not consulted. The one inference worth making is a warning:
+a bridge with no physical port enslaved — the `docker0` signature, nothing
+under `brif/` but taps and veths — is logged plainly at startup as guests
+that may have no way off this host. A guest that then never gets an address
+is a legible symptom whose explanation is already in the log, which is the
+whole point of checking.
+One more warning belongs beside it: `br_netfilter`. With that module loaded —
+Docker loads it — frames bridged through the operator's bridge traverse
+iptables, and a `FORWARD` policy of `DROP` (Docker again) eats them: tap up,
+ARP seen, no traffic, nothing to look at. The ruleset cannot be proven safe
+from here, but the module's presence can be named at startup, and the FAQ
+carries the rest.
+`GuestCIDR` reports the bridge's own IPv4 network, masked to its own prefix
+length. This is the shape `vfkit.GuestCIDR` already uses (`vfkit/guestnet.go`)
+and the same rule applies: no answer leaves the fleet's record alone.
+## macOS is untouched
+Virtualization.framework needs `com.apple.vm.networking` to bridge — a
+restricted entitlement, granted by Apple on request, and it lives in the code
+signature of the binary holding it. That binary is vfkit, not ours, and vfkit
+ships `nat` (`vfkit.go:236`). A Mac keeps vmnet, keeps discovering its address
+from `/var/db/dhcpd_leases`, and keeps reporting `192.168.64.0/24`.
+So the honest summary of the end state: bridged on Linux, NAT on macOS, and no
+user-facing toggle anywhere.
+## The hosts that are on NAT stay on NAT
+Not a grace period and not a deprecation: a supported configuration, chosen for
+the hosts that hold it. crucible and mewtwo keep the network they have, across
+this change and across every agent upgrade after it, with no operator action and
+no re-homing. The agent states which shape it is running at startup, at info,
+once — a fact about the host, not a warning about it.
+So `netenv`'s NAT half, `internal/agent/dhcp` and `internal/agent/ipalloc` are
+not on a path to deletion. They are the implementation of one of the two shapes
+a Linux host can have, and they keep their tests and their maintenance.
+What "default" means, precisely, is that a host with nothing recorded gets
+bridged and must be told which bridge. It says nothing about a host that has
+already answered the question.
+The one edge worth knowing: a host that is decommissioned and re-enrolled has no
+identity, so it is a new host and needs `--bridge`. Moving crucible or mewtwo to
+bridged is that, or an edit to `identity.json` — deliberately, and for the reason
+in §1. The edit route completes per guest, not per host: the tap master check
+in `bridgenet` (step 2) re-wires each tap under the new bridge at the next
+converge, but a migrated guest keeps its NAT address until DHCP next makes it
+ask — a renewal the site's server will not honour, or a power-cycle.
+## Steps
+. `state.Identity.GuestNet` and `.Bridge`, `--bridge` on join and serve, and
+   the resolver beside `resolveGuestCIDR` in `internal/agent/run` — including
+   the one-time persist, which is what an existing host's first start under
+   this agent depends on, and the resolver running before anything persists a
+   fabricated CIDR. Table-driven over the identity shapes: the upgrade case
+   (a v0.0.2 identity, untouched), the flag that agrees, the flag that
+   contradicts (refused), and `--bridge` with `--bridge-cidr` (refused), each
+   pinned by its own test.
+. `internal/agent/bridgenet`: the second `cloudhv.Network` implementation.
+   `CreateTap` creates the tap and enslaves it to the named bridge;
+   `DeleteTap` removes it; `ReserveIP` returns `""`. Idempotency verifies the
+   tap's master, not its existence (still rejecting a non-tap squatter, as
+   `netenv` does): a tap left enslaved to `eitri0` by a host migrated off NAT
+   passes an existence check and leaves its guest wired to a dead bridge —
+   right master is done, wrong master is re-enslaved.
+. Bridge validation and the refusal enumeration, with the sysfs root
+   injectable so every mode is testable without a bridge: refuse (missing,
+   not-a-bridge, down), warn (no physical port, `br_netfilter` loaded),
+   enumerate (interfaces in, message out).
+. The ARP watcher, as its own package with the socket injectable — parsing is
+   pure and testable against captured frames, including the dropped
+   `0.0.0.0`-sender probe.
+. `Address`: `/proc/net/arp` first, watcher second.
+. `wire_linux.go` picks the implementation; the NAT-only startup steps move
+   behind the legacy branch. `--bridge` joins `Config`; `proposeGuestCIDR`
+   returns nil under bridged (the fleet has nothing useful to suggest).
+. `GuestCIDR` from the bridge's own address, and the shape said out loud:
+   `GuestNet` and the bridge name in hostinfo, so the console's host page
+   reads "bridged on br0" beside the network it already names.
+. Docs: `faq.md`'s network answer is now false in its first two sentences,
+   and gains the `br_netfilter` entry; `quickstart.md` gains the host-prep
+   step, written in the reversible form (`netplan try` or the distro
+   equivalent — converting the uplink into a bridge over SSH is how you lose
+   the box, and the docs say so); the join card notes the `--bridge`
+   ingredient it cannot pre-fill — the console mints the command before the
+   host exists, and the refusal enumeration is what backfills the name; an
+   `assumptions.md` entry for the ARP-discovery bet (IPv4-only, trusts the
+   LAN) and one for the tenancy consequence below.
+## Gate
+The boot-gate needs no new leg: `internal/smoke/scenario.go` polls for
+`assigned_ip` and then SSHes to it through the gate, so a green gate is
+discovery working, the tap on the right bridge, and the tunnel reaching a guest
+the site's DHCP addressed. The problem is where it runs.
+The gate fleet is a physical host on the real LAN, whose guests a real DHCP
+server can address — so this work gates on the ordinary `make deploy` whenever
+it is picked up. A bridged guest on a nested host can never gate: its "LAN" is
+its parent's guest subnet, and the DHCP server there is an eitri agent that
+fail-closes on a MAC it holds no reservation for (`dhcp.go:106`) — the same
+fact recorded in the costs below.
+## What this costs, stated plainly
+- **L2 isolation is gone.** Two tenants' guests on one host share a broadcast
+  domain with each other and with everything else on the operator's LAN. This is
+  a self-hosted posture. A fleet that needs tenant isolation wants the bridge on
+  a tagged VLAN sub-interface, which this design permits and does not manage.
+- **`52:54:00` + 24 bits of sha256 now has to be unique on a real network**, not
+  just within one host's private bridge. Widening it renumbers every existing
+  guest, so it stays as it is — but the blast radius of a collision grew from
+  one host to one site.
+- **No sticky addresses** unless the site's DHCP has reservations. A rebooted
+  guest gets what it is given.
+- **Discovery trusts the LAN.** Anything in the broadcast domain can claim a
+  guest's MAC and re-point the recorded address. The SSH host CA makes the
+  hijack fail closed at the crypto layer — misdirection and denial, not
+  compromise — the same shared-broadcast-domain posture as the first bullet.
+- **Discovery is IPv4-only.** ARP and DHCPv4. An IPv6-only site gets no
+  address column.
+- **A host that is itself an eitri guest cannot be bridged.** Its LAN is its
+  parent's guest subnet, and the DHCP server there is an eitri agent that
+  answers only reservations it holds — by design, and correctly. Bridged assumes
+  a site DHCP that will address a stranger. Nested hosts stay on NAT, which is
+  the shape they already have and keep.
+- **mewtwo runs dnsmasq proxy-DHCP and TFTP on `enp4s0`.** Bridged guests join
+  that broadcast domain. They boot UEFI from disk, so PXE should never win the
+  race — but it is now a race that exists, and it is worth watching on the first
+  guest booted there.