3f10d17c

feat(agent): a Mac reads the subnet vmnet gave it

a73x 2026-08-06 09:12

Commit message

feat(agent): a Mac reads the subnet vmnet gave it

A macOS host reports the subnet its guests are actually on, so the fleet records
where they are rather than the allocation it handed out.

Not from vmnet's own configuration, which states the guest network outright in
/Library/Preferences/SystemConfiguration/com.apple.vmnet.plist and is root-only.
The agent does not run as root on macOS and should not start: the guest network
belongs to Virtualization.framework and the entitlement to vfkit, so that one
file would be the only reason it ever needed privileges.

The same two facts are available unprivileged from elsewhere. macOS's lease
database is world-readable and holds addresses vmnet actually handed out; the
interface list needs no privileges and carries each network's own mask. The
guest subnet is the host network containing a leased address — an address inside
it, and a prefix length that is read rather than assumed, which is what reading
a lease alone could not give.

It matches on containment, not on an interface name. bridge100 is not
contractual — a second vmnet network is bridge101 — and a Mac carries bridges
that have nothing to do with guests, Thunderbolt's among them. A network holding
an address vmnet leased to a guest is the guest network, whatever it is called.

A host with no guests reports nothing, and that is the honest answer rather than
a failure: vmnet's bridge exists only while a guest is attached, and nothing has
leased an address. The subnet arrives with the first guest and every report
after it, which is precisely the late answer polling was chosen to allow.

The agent logs its guest subnet when it changes, and warns when it goes away. An
empty answer is correct on the wire — the fleet keeps its existing record — and
that correctness is what makes a broken reader invisible: a host that cannot see
its own network looks exactly like one that could.

docs/assumptions.md

Old		New
@@ -238,12 +238,51 @@ Mac-local control plane: `assigned_ip = 192.168.64.10`, an address no fleet
238	allocation contains, accepted and stored. The Linux gate cannot show this — its	238	allocation contains, accepted and stored. The Linux gate cannot show this — its
239	guests sit inside their own host's `bridge_cidr` and satisfy the old guard too.	239	guests sit inside their own host's `bridge_cidr` and satisfy the old guard too.
240		240
241	What remains untrue is the host row itself. That same run reports the Mac's	241	Inverted 2026-08-04, which is what the entry above was asking for: the
242	`bridge_cidr` as 10.102.1.0/24 while its guests live on 192.168.64.x: the fleet	242	subnet a host's guests are on is now reported BY the host, in every
243	still allocates a subnet to a host whose OS already owns one. The address no	243	`ActualStateReport`, and the fleet records what it is told. Proven on the same
244	longer dies of it, but the row asserts something false, and a host whose OS owns	244	M1 — its row read 10.102.1.0/24, an allocation from a pool it never used, and
245	addressing should be telling the fleet which subnet its guests are on rather	245	now reads 192.168.64.0/24, which is where its guests actually are.
246	than being told.	246
		247	The transition is worth keeping, because it is the empty-means-no-answer rule
		248	made visible rather than asserted. With a guest running: `guest subnet
		249	cidr=192.168.64.0/24`. Twenty seconds after it was reaped, vmnet's bridge gone
		250	with it: `guest subnet unknown — the fleet keeps this host's existing record`.
		251	The row still reads 192.168.64.0/24. A host that cannot see its own network
		252	leaves the fleet's memory alone rather than erasing it.
		253
		254	### A Mac can read the configuration of its own guest network
		255
		256	vmnet states the guest network outright in
		257	`/Library/Preferences/SystemConfiguration/com.apple.vmnet.plist` —
		258	`Shared_Net_Address` and `Shared_Net_Mask`, an address and a netmask, exactly
		259	what a host needs to report its subnet. Underpinned reading it there.
		260
		261	DISPROVEN 2026-08-04: that file is root-only, and the agent does not run as
		262	root on macOS. It has no reason to — the guest network belongs to
		263	Virtualization.framework and the entitlement to vfkit — so reading one config
		264	file would have been the only thing it ever needed privileges for, which is a
		265	bad trade. Two implementations died on this. The first asked `defaults`, which
		266	resolves domains through cfprefsd rather than opening the path it is given, and
		267	answered "The domain/default pair does not exist" for a file `plutil` reads
		268	happily. The second asked `plutil`, which opens the file and is refused.
		269
		270	What replaced it uses nothing privileged: macOS's lease database is
		271	world-readable and holds addresses vmnet has actually handed out, and the
		272	interface list carries each network's own mask. The guest subnet is the host
		273	network containing a leased address — an address inside it, and a prefix length
		274	read rather than assumed, which is the half a lease alone could never supply.
		275
		276	The cost is that a Mac with no guests running reports no subnet, because vmnet's
		277	bridge exists only while a guest is attached to it. That is honest, and the poll
		278	design makes it free: the answer arrives with the first guest.
		279
		280	The thing that made this expensive to find was that all three readers fail
		281	IDENTICALLY and correctly. An unreadable configuration reports no subnet, the
		282	fleet keeps its existing record, and nothing anywhere says so — a wrong command
		283	and a Mac with no network configured are indistinguishable. The agent now logs
		284	its guest subnet when it changes and warns when it goes away, which is what
		285	turned the third attempt into a single run rather than a fourth guess.
247		286
248	### Apple Silicon runs arm64 guests only	287	### Apple Silicon runs arm64 guests only
249		288

internal/agent/run/wire_darwin.go

Old		New
@@ -44,8 +44,9 @@ func newPlatform(_ context.Context, cfg Config, st *state.Store) (platform, erro
44	pumps := serialpump.NewManager(vfkit.NewConsoleSource(prov.SocketPath), st.SerialLogPath)	44	pumps := serialpump.NewManager(vfkit.NewConsoleSource(prov.SocketPath), st.SerialLogPath)
45	prov.Pumps = pumps	45	prov.Pumps = pumps
46		46
47	// Reading vmnet's subnet lands in a later commit; until then this host has	47	// Asked once per report. Two file reads and an interface walk — no
48	// no answer, which by the empty-means-no-answer rule leaves the fleet's	48	// subprocess, no privileges, and nothing worth caching over a ten-second
49	// record alone rather than erasing it.	49	// poll, where a cache would reintroduce exactly the staleness polling exists
50	return platform{Prov: prov, Pumps: pumps, GuestCIDR: func() string { return "" }}, nil	50	// to avoid.
		51	return platform{Prov: prov, Pumps: pumps, GuestCIDR: vfkit.GuestCIDR}, nil
51	}	52	}

internal/agent/syncclient/client.go

Old		New
@@ -345,6 +345,12 @@ func (c *Client) session(ctx context.Context) error {
345	sessCtx, cancel := context.WithCancel(ctx)	345	sessCtx, cancel := context.WithCancel(ctx)
346	defer cancel()	346	defer cancel()
347		347
		348	// Remembered across ticks so the subnet is logged when it CHANGES rather
		349	// than every ten seconds. A host that cannot say which network its guests
		350	// are on is otherwise indistinguishable from one that can: the empty answer
		351	// is correct on the wire — the fleet keeps its existing record — and so is
		352	// silent everywhere an operator might look.
		353	lastCIDR := unreported
348	step := func() error {	354	step := func() error {
349	mu.Lock()	355	mu.Lock()
350	snap := latest	356	snap := latest
@@ -361,6 +367,7 @@ func (c *Client) session(ctx context.Context) error {
361	// leaves the fleet's record alone.	367	// leaves the fleet's record alone.
362	if c.GuestCIDR != nil {	368	if c.GuestCIDR != nil {
363	rep.GuestCidr = c.GuestCIDR()	369	rep.GuestCidr = c.GuestCIDR()
		370	logGuestCIDR(rep.GuestCidr, &lastCIDR)
364	}	371	}
365	return transport.WriteMsg(up, &pb.AgentMessage{Msg: &pb.AgentMessage_Report{Report: rep}})	372	return transport.WriteMsg(up, &pb.AgentMessage{Msg: &pb.AgentMessage_Report{Report: rep}})
366	}	373	}
@@ -570,6 +577,27 @@ func (c *Client) dial(ip string) (net.Conn, error) {
570	return net.Dial("tcp", net.JoinHostPort(ip, "22"))	577	return net.Dial("tcp", net.JoinHostPort(ip, "22"))
571	}	578	}
572		579
		580	// unreported distinguishes "no report has been sent yet" from a report that
		581	// carried no subnet, so the first answer is always logged — including the first
		582	// empty one.
		583	const unreported = "\x00never reported"
		584
		585	// logGuestCIDR reports a change in this host's guest subnet, once, at the tick
		586	// it changes. An answer that goes away is a WARNING: the fleet will keep
		587	// answering from its existing record, which is right, and quietly wrong-looking
		588	// if the host's network has genuinely moved.
		589	func logGuestCIDR(cidr string, last *string) {
		590	if cidr == *last {
		591	return
		592	}
		593	*last = cidr
		594	if cidr == "" {
		595	slog.Warn("guest subnet unknown — the fleet keeps this host's existing record")
		596	return
		597	}
		598	slog.Info("guest subnet", "cidr", cidr)
		599	}
		600
573	// classifyErr distinguishes a permanent auth rejection from a transient error so	601	// classifyErr distinguishes a permanent auth rejection from a transient error so
574	// Run() can log loudly and avoid a tight reconnect loop on a dead credential.	602	// Run() can log loudly and avoid a tight reconnect loop on a dead credential.
575	func classifyErr(err error) error {	603	func classifyErr(err error) error {

internal/agent/syncclient/client_test.go

+	close(block)
+// TestLogGuestCIDROnlyOnChange pins the observability that was missing while
+// two implementations of the macOS reader silently reported nothing. An empty
+// answer is CORRECT on the wire — the fleet keeps its existing record — which
+// is exactly why it has to be visible somewhere: a host that cannot see its own
+// network otherwise looks identical to one that can.
+func TestLogGuestCIDROnlyOnChange(t *testing.T) {
+	last := unreported
+	// The first answer always logs, including the first empty one.
+	logGuestCIDR("", &last)
+	if last != "" {
+		t.Fatalf("last = %q, want the empty answer recorded", last)
+	}
+	// A repeat does not: this runs every tick, forever.
+	logGuestCIDR("", &last)
+	if last != "" {
+		t.Fatalf("last = %q, want unchanged", last)
+	}
+	// An answer arriving is a change.
+	logGuestCIDR("192.168.64.0/24", &last)
+	if last != "192.168.64.0/24" {
+		t.Fatalf("last = %q, want the new subnet", last)
+	}
+	// And an answer going away is a change too — the interesting one.
+	logGuestCIDR("", &last)
+	if last != "" {
+		t.Fatalf("last = %q, want the loss recorded", last)
+	}
+}

internal/agent/vfkit/guestnet.go

@@ @@ -0,0 +1,125 @@ @@
+package vfkit
+import (
+	"net"
+	"net/netip"
+	"os"
+	"strings"
+)
+// GuestCIDR reports the subnet this Mac's guests are on: the network of
+// whichever host interface contains an address vmnet has actually handed out.
+// It returns "" when it cannot say, which by the empty-means-no-answer rule
+// leaves the fleet's record alone and retries on the next report.
+//
+// Both halves are things an unprivileged agent can see, which is the whole
+// reason for this shape. The obvious source — vmnet's own configuration at
+// /Library/Preferences/SystemConfiguration/com.apple.vmnet.plist, which states
+// the address and netmask outright — is ROOT-ONLY, and the agent deliberately
+// does not run as root on macOS: the guest network belongs to
+// Virtualization.framework and the entitlement to vfkit, so there is nothing
+// else it would need privileges for. Reading one config file is a poor reason
+// to take them. (`defaults` cannot read that file either, for an unrelated
+// reason: it resolves domains through cfprefsd rather than opening the path.)
+//
+// The lease database is world-readable and the interface list needs no
+// privileges at all, and between them they give exactly what the plist would
+// have: an address inside the guest network, and a netmask to size it with.
+// Neither is guessed — in particular the prefix length is the interface's own,
+// so nothing here assumes a /24 the way deriving from a lease alone would.
+//
+// A host with no guests reports nothing. vmnet's bridge exists only while a
+// guest is attached to it, and the lease database is empty until one has asked
+// for an address, so a Mac that has never run a guest has no guest network to
+// describe. That is honest rather than a failure, and it costs nothing: the
+// answer arrives with the first guest, and on every report after it.
+func GuestCIDR() string {
+	return guestCIDR(readFile(defaultLeasesPath), hostNetworks())
+}
+// guestCIDR is GuestCIDR with both readings injected, so the pairing is
+// testable on a machine that has neither of them.
+func guestCIDR(leases string, networks []netip.Prefix) string {
+	for _, addr := range leasedAddresses(leases) {
+		for _, pfx := range networks {
+			if pfx.Contains(addr) {
+				return pfx.String()
+			}
+		}
+	}
+	return ""
+}
+// hostNetworks returns every IPv4 network configured on this host's interfaces,
+// each masked to its own prefix length. Addresses are flattened across
+// interfaces deliberately: which interface holds the guest network is not
+// something this needs to know, and the name vmnet uses is not contractual —
+// bridge100 today, bridge101 for a second vmnet network, while bridge0 is
+// Thunderbolt's and holds no IPv4 address at all. A network containing an
+// address vmnet leased to a guest IS the guest network, whatever it is called.
+func hostNetworks() []netip.Prefix {
+	ifaces, err := net.Interfaces()
+	if err != nil {
+		return nil
+	}
+	var out []netip.Prefix
+	for _, iface := range ifaces {
+		addrs, err := iface.Addrs()
+		if err != nil {
+			continue
+		}
+		for _, a := range addrs {
+			if pfx, ok := prefixOf(a); ok {
+				out = append(out, pfx)
+			}
+		}
+	}
+	return out
+}
+// prefixOf reduces one interface address to the IPv4 network it sits on.
+func prefixOf(a net.Addr) (netip.Prefix, bool) {
+	ipnet, ok := a.(*net.IPNet)
+	if !ok || ipnet.IP.To4() == nil {
+		return netip.Prefix{}, false
+	}
+	ones, bits := ipnet.Mask.Size()
+	// A non-contiguous mask reports (0, 0), and a /0 on a host interface is not
+	// a network anything is meaningfully "on".
+	if bits != 32 || ones == 0 {
+		return netip.Prefix{}, false
+	}
+	host, ok := netip.AddrFromSlice(ipnet.IP.To4())
+	if !ok {
+		return netip.Prefix{}, false
+	}
+	return netip.PrefixFrom(host, ones).Masked(), true
+}
+// readFile returns path's contents, or "" when it cannot be read.
+func readFile(path string) string {
+	raw, err := os.ReadFile(path)
+	if err != nil {
+		return ""
+	}
+	return string(raw)
+}
+// leasedAddresses returns every address in the lease database. Any one of them
+// locates the guest network — they are all on it — so the caller takes the
+// first that lands inside a live interface. Stale entries from destroyed VMs
+// are useful here rather than a problem: they still name the subnet, and one
+// that no longer matches any interface is simply skipped.
+func leasedAddresses(leases string) []netip.Addr {
+	var out []netip.Addr
+	for _, line := range strings.Split(leases, "\n") {
+		key, value, ok := strings.Cut(strings.TrimSpace(line), "=")
+		if !ok || key != "ip_address" {
+			continue
+		}
+		if a, err := netip.ParseAddr(parseIP(value)); err == nil && a.Is4() {
+			out = append(out, a)
+		}
+	}
+	return out
+}

internal/agent/vfkit/guestnet_test.go

@@ @@ -0,0 +1,167 @@ @@
+package vfkit
+import (
+	"net"
+	"net/netip"
+	"testing"
+)
+// realLeases is the shape of macOS's lease database, taken from an M1 running
+// eitri guests: brace-delimited stanzas, unpadded MAC octets behind an ARP
+// hardware-type prefix, expiry in hex.
+const realLeases = `{
+	name=proof-1
+	ip_address=192.168.64.10
+	hw_address=1,52:54:0:9d:c1:4
+	identifier=1,52:54:0:9d:c1:4
+	lease=0x68b0a1f2
+}
+{
+	name=proof-2
+	ip_address=192.168.64.11
+	hw_address=1,52:54:0:3a:7b:2
+	lease=0x68b0a2c9
+}
+`
+// prefixes is shorthand for the host networks a reading of the interface list
+// would produce.
+func prefixes(t *testing.T, ss ...string) []netip.Prefix {
+	t.Helper()
+	var out []netip.Prefix
+	for _, s := range ss {
+		p, err := netip.ParsePrefix(s)
+		if err != nil {
+			t.Fatalf("bad test prefix %q: %v", s, err)
+		}
+		out = append(out, p)
+	}
+	return out
+}
+// TestGuestCIDRPairsALeaseWithItsInterface pins the whole idea: an address
+// vmnet actually handed out, matched against the networks this host has
+// configured, gives the guest subnet AND its prefix length without guessing
+// either.
+func TestGuestCIDRPairsALeaseWithItsInterface(t *testing.T) {
+	// What a Mac running a guest looks like: loopback, the uplink, Thunderbolt's
+	// bridge (no IPv4, so absent here), and vmnet's.
+	host := prefixes(t, "127.0.0.0/8", "192.168.1.0/24", "192.168.64.0/24")
+	if got := guestCIDR(realLeases, host); got != "192.168.64.0/24" {
+		t.Errorf("guestCIDR() = %q, want 192.168.64.0/24", got)
+	}
+}
+// TestGuestCIDRTakesThePrefixFromTheInterface pins that the prefix length is
+// the interface's own and never assumed. Deriving from a lease alone would
+// force a guess at /24, and vmnet's network is configurable.
+func TestGuestCIDRTakesThePrefixFromTheInterface(t *testing.T) {
+	for _, tc := range []struct{ name, network, want string }{
+		{"the usual /24", "192.168.64.0/24", "192.168.64.0/24"},
+		{"a wider network", "192.168.0.0/16", "192.168.0.0/16"},
+		{"a narrower one", "192.168.64.0/26", "192.168.64.0/26"},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := guestCIDR(realLeases, prefixes(t, tc.network)); got != tc.want {
+				t.Errorf("guestCIDR() = %q, want %q", got, tc.want)
+			}
+		})
+	}
+}
+// TestGuestCIDRSaysNothingRatherThanGuessing pins the empty-means-no-answer
+// rule at its source. Each of these leaves the fleet's record alone and retries
+// on the next report, rather than reporting something invented.
+func TestGuestCIDRSaysNothingRatherThanGuessing(t *testing.T) {
+	host := prefixes(t, "127.0.0.0/8", "192.168.1.0/24", "192.168.64.0/24")
+	t.Run("no lease database", func(t *testing.T) {
+		// A Mac that has never run a guest: nothing has asked for an address, so
+		// there is no guest network to describe yet.
+		if got := guestCIDR("", host); got != "" {
+			t.Errorf("guestCIDR() = %q, want empty", got)
+		}
+	})
+	t.Run("no interface holds the leased network", func(t *testing.T) {
+		// vmnet's bridge exists only while a guest is attached, so a lease left
+		// over from a destroyed VM matches nothing. Skipped, not guessed at.
+		if got := guestCIDR(realLeases, prefixes(t, "127.0.0.0/8", "192.168.1.0/24")); got != "" {
+			t.Errorf("guestCIDR() = %q, want empty", got)
+		}
+	})
+	t.Run("no host networks at all", func(t *testing.T) {
+		if got := guestCIDR(realLeases, nil); got != "" {
+			t.Errorf("guestCIDR() = %q, want empty", got)
+		}
+	})
+	t.Run("leases with no usable address", func(t *testing.T) {
+		if got := guestCIDR("{\n\tname=x\n\thw_address=1,52:54:0:1:2:3\n}\n", host); got != "" {
+			t.Errorf("guestCIDR() = %q, want empty", got)
+		}
+	})
+}
+// TestGuestCIDRSkipsStaleLeasesForLiveOnes pins that a lease database
+// outliving its VMs is an asset rather than a hazard: an entry matching no live
+// interface is skipped, and a later one that does match is used.
+func TestGuestCIDRSkipsStaleLeasesForLiveOnes(t *testing.T) {
+	leases := "{\n\tip_address=10.9.9.9\n}\n" + realLeases
+	if got := guestCIDR(leases, prefixes(t, "192.168.64.0/24")); got != "192.168.64.0/24" {
+		t.Errorf("guestCIDR() = %q, want the live network", got)
+	}
+}
+// TestPrefixOfRejectsWhatIsNotANetwork pins the reading of one interface
+// address, including shapes a host really carries.
+func TestPrefixOfRejectsWhatIsNotANetwork(t *testing.T) {
+	ipnet := func(ip string, mask net.IPMask) net.Addr {
+		return &net.IPNet{IP: net.ParseIP(ip), Mask: mask}
+	}
+	for _, tc := range []struct {
+		name string
+		addr net.Addr
+		want string // "" = rejected
+	}{
+		{"a normal /24", ipnet("192.168.64.1", net.CIDRMask(24, 32)), "192.168.64.0/24"},
+		{"masked to its network", ipnet("192.168.64.200", net.CIDRMask(24, 32)), "192.168.64.0/24"},
+		{"IPv6", ipnet("fd00::1", net.CIDRMask(64, 128)), ""},
+		{"a /0", ipnet("10.0.0.1", net.CIDRMask(0, 32)), ""},
+		{"non-contiguous mask", ipnet("10.0.0.1", net.IPMask{255, 0, 255, 0}), ""},
+		{"not an IPNet at all", &net.TCPAddr{IP: net.ParseIP("10.0.0.1")}, ""},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			pfx, got := prefixOf(tc.addr)
+			if tc.want == "" {
+				if got {
+					t.Errorf("prefixOf() accepted %v as %s", tc.addr, pfx)
+				}
+				return
+			}
+			if !got || pfx.String() != tc.want {
+				t.Errorf("prefixOf() = %s, %v; want %s", pfx, got, tc.want)
+			}
+		})
+	}
+}
+// TestHostNetworksReadsThisMachine is a smoke test against the real interface
+// list. It cannot assert a subnet — it runs on whatever CI is — but it proves
+// the reading needs no privileges and returns something well-formed, which is
+// the property that sent the previous two implementations back.
+func TestHostNetworksReadsThisMachine(t *testing.T) {
+	nets := hostNetworks()
+	if len(nets) == 0 {
+		t.Skip("no IPv4 interfaces on this machine")
+	}
+	for _, p := range nets {
+		if !p.Addr().Is4() {
+			t.Errorf("hostNetworks returned a non-IPv4 prefix: %s", p)
+		}
+		if p != p.Masked() {
+			t.Errorf("hostNetworks returned an unmasked prefix: %s", p)
+		}
+	}
+}