docs/superpowers/plans/2026-08-05-bridged-guest-networking.md

Ref: main v0.0.5   Size: 17.5 KiB   History

# Bridged by default: a guest is a machine on the network

**Status:** parked — demand-driven. Service exposure covers reaching a guest's
services; what this plan alone provides is LAN citizenship — broadcast-domain
discovery (mDNS/SSDP) and operator-owned addressing. It is revived by the
first workload that needs those. A UDP want alone is not that workload:
`protocol: udp` on exposures is the smaller answer.
**Branch:** `v0.0.3`
**Ethos:** the guest owns the guest, eitri owns the fleet — and the network
belongs to whoever runs it.

## The decision

A Linux host puts its guests on the operator's network. It does not build a
network of its own, does not hand out addresses, and does not masquerade. It is
told which bridge to attach taps to, and it refuses to start without one.

This is a default, not a mode: there is no `--guest-net` flag and nothing in the
console flips it. A Linux host that has a bridge is bridged; a Linux host that
does not is a host that will not serve.

Two things it is not. It is not fleet-wide — macOS cannot bridge, so the NAT
shape survives on Macs whatever we do here (below). And it is not a licence to
rebuild the operator's uplink: eitri never enslaves an interface it did not
create. The bridge is the operator's, built by whatever manages that host's
network, and the agent's only claim on it is a tap.

## What is there now

Every guest sits behind NAT on a host-private bridge. `netenv.Net` builds
`eitri0`, takes `.1`, enables forwarding, installs one `nft` masquerade rule
(`netenv.go:172`), and runs eitri's own DHCP responder with a per-VM
reservation keyed on a deterministic MAC (`netenv.go:240`, `state.go:94`).

The seam this all hangs off is already the right one. `cloudhv` does not import
`netenv`: it declares a five-method consumer-owned `Network` interface
(`cloudhv.go:46`) — `ReserveIP`, `Address`, `CreateTap`, `DeleteTap`,
`TapName` — and `netenv` happens to satisfy it. The VMM sees one line of the
whole arrangement, `--net tap=eit-xxxxxxxx,mac=52:54:00:…`
(`cloudhv.go:151`), and that line is identical under either regime.

So bridged is a second implementation of an interface that already exists.
`reconcile`, `buildArgs`, the pumps and the pidfiles do not move.

## The three things that actually change

### 1. The mode is derived from the identity, never set

`state.Identity` (`state.go:43`) gains two fields — `GuestNet string` (`nat` or
`bridged`) and `Bridge string` — and the answer comes from the record, with no
flag to disagree with:

- `GuestNet` set → that, and for `bridged`, on the bridge named in `Bridge`.
- unset, `BridgeCIDR != ""` → a host enrolled by an older agent. NAT, recorded
  as `GuestNet: "nat"` on this first start and read from the record thereafter.
- unset, nothing else → a new host. `--bridge` is required; refuse to start.

The recording matters more than it looks. Deriving "this host is on NAT" from a
missing field works exactly once — the next change to what absence means would
silently re-home hosts that were never asked. Writing the answer down the first
time a host resolves it makes an existing host's NAT a decision the fleet holds,
not an inference it re-makes every start. It is the same one-time persist
`persistGuestCIDR` already does for the subnet (`guestcidr.go:51`).

This is the precedence `resolveGuestCIDR` already enforces for the subnet
(`guestcidr.go:33`), and for the reason stated there: a flag that overrides a
persisted identity moves the network under live guests and every one of them
loses its address at the next restart. Changing a host's bridge is an edit to
`identity.json`, exactly as re-homing its subnet is.

Where the answer is given: `join`. The mode is identity and join is the
ceremony that mints identity, so `eitri-agent join --bridge br0` is the
operator answering at the moment the host is asked. Serve accepts the same
flag as backfill only — consulted when the record is silent, the role
`cfg.BridgeCIDR` already plays in `resolveGuestCIDR` — and a restart that
passes it lands in one of three rows:

- record silent → the flag answers and is persisted.
- record agrees → no-op; restating the recorded answer is harmless.
- record contradicts (`nat` recorded, or a different bridge named) → refuse,
  loudly. Obeying would re-home every guest as a side effect of a restart
  flag; ignoring would silently discard an instruction the operator stopped
  the agent to give. The refusal names the record and the sanctioned path:
  edit `identity.json` with the agent stopped, or decommission and re-join.

Two orderings that matter. The resolver runs before anything persists a
fabricated CIDR — `newPlatform` persists `lastResortGuestCIDR` today, and a
fresh host that reached that code first would classify itself as legacy NAT.
And `--bridge` with `--bridge-cidr` is a contradiction in terms — one names a
network the operator owns, the other a network eitri would build — and is
refused rather than resolved.

The refusal is the report. The machine that knows which bridge to name is
the host itself, so when `--bridge` is missing the agent enumerates before it
exits. A host with bridges gets the candidates and the evidence that matters —
`br0: up, ports enp4s0` against `docker0: up, no physical port` — and the
operator reads their own answer off the list. A host with none gets the name
of the interface they are about to enslave: the default-route interface,
offered as a suggestion — wrong on an odd topology, and costing nothing
there, because a suggestion can be ignored. What the agent never does is
choose. A filter that picked the one plausible candidate would ship guests
onto the wrong segment silently on exactly the host where it guessed wrong,
and the whole design rests on the mode being an answer the operator gave.
Suggest loudly, choose never.

### 2. Addressing flips from allocate to observe

There is no reservation to make. The site's DHCP server hands the lease, and the
agent's job is to find out what it was.

`ReserveIP` becomes a no-op returning `""` — `attachNet` passes its result
straight into `CreateTap` (`cloudhv.go:317`) and bridged `CreateTap` ignores it.
`Address` becomes discovery, and returns `""` until it has an answer.

Nothing above this needs teaching, because macOS already taught it.
`noteAddress` treats an empty answer as "I don't know yet" and refuses to blank
a known address (`reconcile.go:563`); the control plane's guard stopped asking
about topology; the host reports its own subnet. The consumers that poll for
`assigned_ip` — the console column, `eitri-mcp`'s wait, the smoke gate — already
wait for a Mac and will wait the same way here.

**Discovery mechanism.** Two sources, cheapest first:

- `/proc/net/arp` — free, no exec, scoped by its Device column, and populated
  for any guest the host has already spoken to.
- A passive ARP watcher on the bridge for the cold case, which is the common
  one: nothing has dialled the guest yet, and dialling is what needs the address.
  An `AF_PACKET` socket filtered to `ETH_P_ARP` yields sender-MAC → sender-IP,
  matched against `state.MAC(vmID)`. Every guest broadcasts within seconds of
  its lease — an RFC 5227 announcement for its own address, and a request for
  its gateway — so the answer arrives unprompted and updates itself if the
  site renumbers the guest. The watcher drops unspecified sender addresses: a
  5227 *probe* carries sender-IP `0.0.0.0` by definition — junk that
  `noteAddress` has no reason to refuse and the tunnel would then dial.

ARP rather than DHCP snooping deliberately: a DHCP reply may be unicast to the
guest's MAC, which the bridge will not deliver to the host without putting the
interface in promiscuous mode. ARP is broadcast and needs no such thing.

A discovered address is already durable — `reconcile` persists `rec.IP` — so an
agent restart keeps the last known address while the watcher re-warms, and
`syncclient`'s tunnel (which refuses an empty address, `client.go:530`) never
sees a gap it would not have seen on a Mac.

### 3. Most of the host-wide setup stops running

Under bridged, `newPlatform` (`wire_linux.go:84`) does none of this:

| Step | Why it goes |
|---|---|
| `EnsureBridge` | the bridge is the operator's; we validate, never create |
| `StartDHCP` | an eitri DHCP server on the LAN hijacks the site's |
| `replayReservations` | there are no reservations to replay |
| `CheckUplinkCollision` | it refuses precisely the bridged case, by definition |
| `lastResortGuestCIDR` | a fabricated subnet is only right where something builds one |

`CheckUplinkCollision` (`uplink.go:67`) deserves a note rather than a deletion.
It fails when the guest subnet contains an address the host is configured with,
which is the definition of bridged — but it is the correct check for the shape
it was written for, and the failure it prevents (the host's own address becomes
local, it stops answering, and the machine best placed to fix it is the one that
just lost the route) is real. It stays, scoped to legacy NAT.

What replaces it is validation of what we were given, at startup, before a
guest can fail illegibly later — split on the line `CheckUplinkCollision`
already draws: refuse on facts, warn on diagnostics. The facts are that the
named link exists, is a bridge (`/sys/class/net/<name>/bridge`), and is up.
These are never wrong under any topology; an operator whose bridge fails them
typed a name that does not hold. Whether the bridge leads anywhere is not a
fact but the operator's intent: the default route may sit on the bridge
itself (the canonical shape — the host's address lives on `br0` and the
uplink is a port), on another interface entirely with a guests-only bridge on
a second NIC, or nowhere by design on an isolated segment. Any rule that
consults the routing table refuses one of the legitimate shapes, so the
routing table is not consulted. The one inference worth making is a warning:
a bridge with no physical port enslaved — the `docker0` signature, nothing
under `brif/` but taps and veths — is logged plainly at startup as guests
that may have no way off this host. A guest that then never gets an address
is a legible symptom whose explanation is already in the log, which is the
whole point of checking.

One more warning belongs beside it: `br_netfilter`. With that module loaded —
Docker loads it — frames bridged through the operator's bridge traverse
iptables, and a `FORWARD` policy of `DROP` (Docker again) eats them: tap up,
ARP seen, no traffic, nothing to look at. The ruleset cannot be proven safe
from here, but the module's presence can be named at startup, and the FAQ
carries the rest.

`GuestCIDR` reports the bridge's own IPv4 network, masked to its own prefix
length. This is the shape `vfkit.GuestCIDR` already uses (`vfkit/guestnet.go`)
and the same rule applies: no answer leaves the fleet's record alone.

## macOS is untouched

Virtualization.framework needs `com.apple.vm.networking` to bridge — a
restricted entitlement, granted by Apple on request, and it lives in the code
signature of the binary holding it. That binary is vfkit, not ours, and vfkit
ships `nat` (`vfkit.go:236`). A Mac keeps vmnet, keeps discovering its address
from `/var/db/dhcpd_leases`, and keeps reporting `192.168.64.0/24`.

So the honest summary of the end state: bridged on Linux, NAT on macOS, and no
user-facing toggle anywhere.

## The hosts that are on NAT stay on NAT

Not a grace period and not a deprecation: a supported configuration, chosen for
the hosts that hold it. crucible and mewtwo keep the network they have, across
this change and across every agent upgrade after it, with no operator action and
no re-homing. The agent states which shape it is running at startup, at info,
once — a fact about the host, not a warning about it.

So `netenv`'s NAT half, `internal/agent/dhcp` and `internal/agent/ipalloc` are
not on a path to deletion. They are the implementation of one of the two shapes
a Linux host can have, and they keep their tests and their maintenance.

What "default" means, precisely, is that a host with nothing recorded gets
bridged and must be told which bridge. It says nothing about a host that has
already answered the question.

The one edge worth knowing: a host that is decommissioned and re-enrolled has no
identity, so it is a new host and needs `--bridge`. Moving crucible or mewtwo to
bridged is that, or an edit to `identity.json` — deliberately, and for the reason
in §1. The edit route completes per guest, not per host: the tap master check
in `bridgenet` (step 2) re-wires each tap under the new bridge at the next
converge, but a migrated guest keeps its NAT address until DHCP next makes it
ask — a renewal the site's server will not honour, or a power-cycle.

## Steps

1. `state.Identity.GuestNet` and `.Bridge`, `--bridge` on join and serve, and
   the resolver beside `resolveGuestCIDR` in `internal/agent/run` — including
   the one-time persist, which is what an existing host's first start under
   this agent depends on, and the resolver running before anything persists a
   fabricated CIDR. Table-driven over the identity shapes: the upgrade case
   (a v0.0.2 identity, untouched), the flag that agrees, the flag that
   contradicts (refused), and `--bridge` with `--bridge-cidr` (refused), each
   pinned by its own test.
2. `internal/agent/bridgenet`: the second `cloudhv.Network` implementation.
   `CreateTap` creates the tap and enslaves it to the named bridge;
   `DeleteTap` removes it; `ReserveIP` returns `""`. Idempotency verifies the
   tap's master, not its existence (still rejecting a non-tap squatter, as
   `netenv` does): a tap left enslaved to `eitri0` by a host migrated off NAT
   passes an existence check and leaves its guest wired to a dead bridge —
   right master is done, wrong master is re-enslaved.
3. Bridge validation and the refusal enumeration, with the sysfs root
   injectable so every mode is testable without a bridge: refuse (missing,
   not-a-bridge, down), warn (no physical port, `br_netfilter` loaded),
   enumerate (interfaces in, message out).
4. The ARP watcher, as its own package with the socket injectable — parsing is
   pure and testable against captured frames, including the dropped
   `0.0.0.0`-sender probe.
5. `Address`: `/proc/net/arp` first, watcher second.
6. `wire_linux.go` picks the implementation; the NAT-only startup steps move
   behind the legacy branch. `--bridge` joins `Config`; `proposeGuestCIDR`
   returns nil under bridged (the fleet has nothing useful to suggest).
7. `GuestCIDR` from the bridge's own address, and the shape said out loud:
   `GuestNet` and the bridge name in hostinfo, so the console's host page
   reads "bridged on br0" beside the network it already names.
8. Docs: `faq.md`'s network answer is now false in its first two sentences,
   and gains the `br_netfilter` entry; `quickstart.md` gains the host-prep
   step, written in the reversible form (`netplan try` or the distro
   equivalent — converting the uplink into a bridge over SSH is how you lose
   the box, and the docs say so); the join card notes the `--bridge`
   ingredient it cannot pre-fill — the console mints the command before the
   host exists, and the refusal enumeration is what backfills the name; an
   `assumptions.md` entry for the ARP-discovery bet (IPv4-only, trusts the
   LAN) and one for the tenancy consequence below.

## Gate

The boot-gate needs no new leg: `internal/smoke/scenario.go` polls for
`assigned_ip` and then SSHes to it through the gate, so a green gate is
discovery working, the tap on the right bridge, and the tunnel reaching a guest
the site's DHCP addressed. The problem is where it runs.

The gate fleet is a physical host on the real LAN, whose guests a real DHCP
server can address — so this work gates on the ordinary `make deploy` whenever
it is picked up. A bridged guest on a nested host can never gate: its "LAN" is
its parent's guest subnet, and the DHCP server there is an eitri agent that
fail-closes on a MAC it holds no reservation for (`dhcp.go:106`) — the same
fact recorded in the costs below.

## What this costs, stated plainly

- **L2 isolation is gone.** Two tenants' guests on one host share a broadcast
  domain with each other and with everything else on the operator's LAN. This is
  a self-hosted posture. A fleet that needs tenant isolation wants the bridge on
  a tagged VLAN sub-interface, which this design permits and does not manage.
- **`52:54:00` + 24 bits of sha256 now has to be unique on a real network**, not
  just within one host's private bridge. Widening it renumbers every existing
  guest, so it stays as it is — but the blast radius of a collision grew from
  one host to one site.
- **No sticky addresses** unless the site's DHCP has reservations. A rebooted
  guest gets what it is given.
- **Discovery trusts the LAN.** Anything in the broadcast domain can claim a
  guest's MAC and re-point the recorded address. The SSH host CA makes the
  hijack fail closed at the crypto layer — misdirection and denial, not
  compromise — the same shared-broadcast-domain posture as the first bullet.
- **Discovery is IPv4-only.** ARP and DHCPv4. An IPv6-only site gets no
  address column.
- **A host that is itself an eitri guest cannot be bridged.** Its LAN is its
  parent's guest subnet, and the DHCP server there is an eitri agent that
  answers only reservations it holds — by design, and correctly. Bridged assumes
  a site DHCP that will address a stranger. Nested hosts stay on NAT, which is
  the shape they already have and keep.
- **mewtwo runs dnsmasq proxy-DHCP and TFTP on `enp4s0`.** Bridged guests join
  that broadcast domain. They boot UEFI from disk, so PXE should never win the
  race — but it is now a race that exists, and it is worth watching on the first
  guest booted there.