d2323cb8

fix: a burst of dials past the observer table waits in the backlog

a73x 2026-09-05 19:04

Commit message

fix: a burst of dials past the observer table waits in the backlog

`acceptConn` parked every new connection in one of four observer slots
and, when all four were taken, accepted the next one and closed it. Eight
tile pumps dial within 150 µs of each other, and the accept loop ran ahead
of their attach frames: the slots filled with peers whose frame was still
in flight and the fifth was closed at accept, SIGPIPE on its first write,
tile ended with nothing painted. An eight-tile wall lost one tile at birth
about one run in four (the refused-tile leg of e2e_13_birth, found rebasing
onto main; main's own build loses the same 118 of 240 in an eight-way
burst rig, and a ReleaseSafe release 18).

`Server.freeObserverSlot` is asked before the accept and `pumpOnce` polls
the listener only while it answers, so a fifth dial waits in the kernel's
128-deep backlog until a promotion or an idle drop frees a slot. The
backlog is the queue; the observer table is what the daemon is reading.
Zero lost of 240 after. Pinned by an observer-burst test: eight connects
with no frame behind them, exactly four held, all eight seated and served
once the frames arrive; red on the old accept. The harness's three-at-a-
time fill rule was the workaround and stays, with its comment rewritten.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

CLAUDE.md

Old		New
@@ -447,6 +447,16 @@ native session panes; it is built only by the native steps and is never installe
447	plus 68 bytes plus the pid's digits, and neither of those directories	447	plus 68 bytes plus the pid's digits, and neither of those directories
448	leaves room for it in 103 (measured 2026-09-03, docs/decisions.md).	448	leaves room for it in 103 (measured 2026-09-03, docs/decisions.md).
449	- No socket stealing: `mux d start` refuses a path another daemon owns.	449	- No socket stealing: `mux d start` refuses a path another daemon owns.
		450	- **A dial the observer table cannot hold waits in the kernel backlog; it
		451	is never accepted and closed.** Every new connection lands in one of four
		452	observer slots until its attach frame promotes it, and eight tile pumps
		453	dial within 150 µs of each other, so the accept loop used to fill the
		454	four with peers whose frame was still in flight and close the fifth —
		455	one tile of an eight-tile wall lost at birth with nothing painted, one
		456	run in four (2026-09-05). `Server.freeObserverSlot` is asked before the
		457	accept and `pumpOnce` polls the listener only while it answers. The
		458	harness's three-at-a-time fill rule in `test/e2e_lib.sh` was the
		459	workaround and is now only a habit.
450	- **A deleted socket path is taken back within a second, and the log says	460	- **A deleted socket path is taken back within a second, and the log says
451	so.** A unix listener outlives its path: delete the file and the daemon	461	so.** A unix listener outlives its path: delete the file and the daemon
452	keeps every session on an inode nothing can reach by name (2026-09-04, a	462	keeps every session on an inode nothing can reach by name (2026-09-04, a

docs/decisions.md

@@ @@ -9295,3 +9295,48 @@ local to the box. @@
+- `make check` flaked once on `Pty.adopt`'s 5 s exit-code wait while a
+  sibling worktree's `make ci` was loading the box; green on the re-run,
+  and unrelated to any file this branch touched.
+## 2026-09-05 — a burst of dials past the observer table waits in the backlog
+Found rebasing the socket-lifecycle branch onto main: `make ci` failed the
+refused-tile leg of `test/e2e_13_birth.sh` at one run in four, on a tree
+whose every other run was green, and on main's own build. The leg seeds an
+eight-tile wall on one daemon, fills the table, and births a ninth to be
+refused; the ninth was SEATED instead, on a slot one of the eight had let
+go of at birth. strace on the wall showed the eight pumps dialling within
+µs of each other and one of them taking SIGPIPE on its first write 34 µs
+after its connect returned: the daemon's accept loop had run ahead of the
+attach frames, filled the four observer slots with peers whose frame was
+still in flight, and closed the fifth at accept — `conn.close(); // out of
+slots`. The pump ended that tile quietly, nothing painted, and the wall
+stood at seven tiles with a slot to spare.
+**Decided:** `Server.freeObserverSlot` is asked BEFORE the accept, and
+`pumpOnce` polls the listener only while it answers. A fifth simultaneous
+dial waits in the kernel's listen backlog (128 deep) until a promotion or
+an idle drop frees a slot — the backlog is the queue and the observer table
+is the set of connections the daemon is reading, and accepting a connection
+it cannot read was never a service. The idle rule stands: four silent peers
+now delay the fifth by up to `observer_idle_ms` rather than close it with
+no diagnostic. The harness's "three at a time" fill rule in `test/e2e_lib.sh`
+was the workaround for this, measured 2026-08; it stays as written, since
+the arithmetic beside it is built for the batch, but it is no longer what
+keeps a fill alive.
+**Measured** (`burst8.sh`: eight piped attaches to one fresh daemon at
+once, thirty trials, count of "connection to the daemon lost"):
+| daemon | lost / 240 |
+|---|---|
+| this branch before the fix, Debug | 118 |
+| main `ad746c0f`, Debug | 118 |
+| main's parent's blocking dial swapped back in, Debug | 118 |
+| installed v0.0.1-17, ReleaseSafe | 18 |
+| this branch after the fix, Debug | 0 |
+The blocking-dial row is the A/B that cleared main's new
+`open_wait.connectUnix`: the loss is the daemon's, and a release build only
+narrows the window. Pinned by the observer-burst test in
+`server_test_attach.zig`: eight connects with no frame behind them, exactly
+four held after twenty pumps, and all eight seated and served once the
+frames arrive.

src/server/server.zig

Old		New
@@ -995,7 +995,12 @@ pub const Server = struct {
995	for (&self.sessions.table, 0..) \|*slot, si\| {	995	for (&self.sessions.table, 0..) \|*slot, si\| {
996	fds[si] = pollIn(if (slot.) \|s\| s.pty.master else -1);	996	fds[si] = pollIn(if (slot.) \|s\| s.pty.master else -1);
997	}	997	}
998	fds[listener_idx] = pollIn(self.bound.fd);	998	// The listener is polled only while an observer slot is free to
		999	// land the connection in. With every slot taken, a peer waits in
		1000	// the kernel's listen backlog until a promotion or an idle drop
		1001	// frees one; polling the listener anyway would wake this loop
		1002	// every pass for a connection `acceptConn` could not take.
		1003	fds[listener_idx] = pollIn(if (self.freeObserverSlot() != null) self.bound.fd else -1);
999	for (&self.clients, 0..) \|*slot, i\| {	1004	for (&self.clients, 0..) \|*slot, i\| {
1000	if (slot.) \|c\| {	1005	if (slot.) \|c\| {
1001	// POLLOUT only while something is owed: asking for it on an	1006	// POLLOUT only while something is owed: asking for it on an
@@ -1157,7 +1162,25 @@ pub const Server = struct {
1157	}	1162	}
1158	}	1163	}
1159		1164
		1165	/// The observer slot the next accepted connection lands in, or null
		1166	/// when all four are taken. `pumpOnce` asks it before polling the
		1167	/// listener, so a burst of dials is never accepted faster than the
		1168	/// slots drain: the fifth peer of a burst waits in the kernel backlog
		1169	/// (128 deep) until a promotion or an idle drop frees a slot. Accepting
		1170	/// it and closing it, which is what this loop did until 2026-09-05,
		1171	/// lost the fifth tile of an eight-tile wall at birth with no
		1172	/// diagnostic, because eight pumps dial within a hundred microseconds
		1173	/// and the accept loop ran ahead of their attach frames — measured at
		1174	/// 118 of 240 attaches lost in eight-way bursts on a Debug daemon.
		1175	fn freeObserverSlot(self: *const Server) ?usize {
		1176	for (self.observers, 0..) \|slot, i\| {
		1177	if (slot == null) return i;
		1178	}
		1179	return null;
		1180	}
		1181
1160	fn acceptConn(self: *Server) void {	1182	fn acceptConn(self: *Server) void {
		1183	const slot = self.freeObserverSlot() orelse return;
1161	// CLOEXEC on the accepted fd, which is what std.net.Server.accept	1184	// CLOEXEC on the accepted fd, which is what std.net.Server.accept
1162	// did here before: the daemon forks a shell per session and a live	1185	// did here before: the daemon forks a shell per session and a live
1163	// client connection leaked into one outlives every close of ours.	1186	// client connection leaked into one outlives every close of ours.
@@ -1172,13 +1195,7 @@ pub const Server = struct {
1172	conn.close();	1195	conn.close();
1173	return;	1196	return;
1174	};	1197	};
1175	for (&self.observers) \|*slot\| {	1198	self.observers[slot] = .{ .fd = conn.handle, .since_ms = monoMs() };
1176	if (slot.* == null) {
1177	slot.* = .{ .fd = conn.handle, .since_ms = monoMs() };
1178	return;
1179	}
1180	}
1181	conn.close(); // out of slots
1182	}	1199	}
1183		1200
1184	/// Once a second: does `sock_path` still name the socket we bound? A	1201	/// Once a second: does `sock_path` still name the socket we bound? A

src/server/server_test_attach.zig

+    });
+    try std.testing.expect(5 + req.len < srv_mod.observer_inbound_max);
+fn seatedClients(s: *const Server) usize {
+    var n: usize = 0;
+    for (s.clients) |c| {
+        if (c != null) n += 1;
+    }
+    return n;
+}
+fn heldObservers(s: *const Server) usize {
+    var n: usize = 0;
+    for (s.observers) |o| {
+        if (o != null) n += 1;
+    }
+    return n;
+}
+fn eightSeated(s: *Server) bool {
+    return seatedClients(s) == 8;
+}
+test "Server: a burst of dials past the observer table waits in the backlog, and every one is seated" {
+    const alloc = std.testing.allocator;
+    var td = try h.TestDaemon.init(alloc, "burst", .{ .shell = "/bin/sh" });
+    defer td.deinit();
+    // Eight connects with no attach frame behind any of them yet: twice the
+    // observer table, and the shape of an eight-tile wall whose pumps dial
+    // within a hundred microseconds of each other. The gap between the
+    // connect and the frame is the race: the daemon's accept loop used to
+    // run ahead of the frames, fill the four slots with silent peers and
+    // close the fifth outright, which lost one tile of eight at birth with
+    // nothing painted (test/e2e_13_birth.sh's refused-tile leg, flaky at
+    // one run in four on 2026-09-05).
+    var conns: [8]std.net.Stream = undefined;
+    for (&conns) |*c| c.* = try dial.dial(td.sock_path);
+    defer for (conns) |c| c.close();
+    // Let the daemon accept what it will before any frame arrives. Exactly
+    // the table's worth lands; the other four are the kernel's to hold.
+    for (0..20) |_| try td.srv.pumpOnce(5);
+    try std.testing.expectEqual(srv_mod.max_observers, heldObservers(&td.srv));
+    try std.testing.expectEqual(@as(usize, 0), seatedClients(&td.srv));
+    for (conns) |c| try proto.writeFrame(c.handle, .attach, &proto.encodeAttach(80, 24, 0, 0));
+    try std.testing.expect(try h.pumpUntil(&td.srv, 5000, &td.srv, eightSeated));
+    try std.testing.expectEqual(@as(usize, 0), heldObservers(&td.srv));
+    // Seated is not served: every one of the eight reads its snapshot,
+    // the four that waited in the backlog included.
+    for (0..40) |_| try td.srv.pumpOnce(5);
+    for (conns) |c| {
+        try std.testing.expect((try h.firstStateFrame(alloc, c.handle, 2000)) != null);
+    }
+}

test/e2e_lib.sh

Old		New
@@ -481,12 +481,17 @@ await_out() {
481	# that made it, which is what lets these detach immediately.	481	# that made it, which is what lets these detach immediately.
482	#	482	#
483	# THREE at a time, and the number is `max_observers` minus one, not	483	# THREE at a time, and the number is `max_observers` minus one, not
484	# `max_clients` minus one: `acceptConn` parks EVERY new connection in an	484	# `max_clients` minus one. `acceptConn` parks every new connection in an
485	# observer slot and promotes it to a client only when its attach frame	485	# observer slot and promotes it to a client only when its attach frame
486	# arrives, so simultaneous dials contend for the four observer slots, and	486	# arrives, so simultaneous dials contend for the four observer slots.
487	# the ones with nowhere to land are closed outright. Measured at batch 7	487	# Until 2026-09-05 the ones with nowhere to land were closed outright —
488	# on a 32-slot daemon: 11 of 31 fills died with "connection to the daemon lost"	488	# measured at batch 7 on a 32-slot daemon: 11 of 31 fills died with
489	# and the table never filled; at batch 3, 31 of 31 land.	489	# "connection to the daemon lost" and the table never filled; at batch 3,
		490	# 31 of 31 land. The daemon now leaves a fifth dial in the kernel backlog
		491	# instead (`Server.freeObserverSlot`), so the batch is no longer what
		492	# keeps the fills alive; it stays because the count arithmetic below is
		493	# written for it and a fixture that races the daemon on purpose belongs
		494	# in a leg that says so, not in a helper every group calls.
490	#	495	#
491	# STATE is a scratch home so the fills never write a wall line the leg	496	# STATE is a scratch home so the fills never write a wall line the leg
492	# later counts. The last name is asked back through mux a so a fill that	497	# later counts. The last name is asked back through mux a so a fill that