29cbd304

fix: a QUIC client sends CONNECTION_CLOSE on teardown, so a poll frees its slot at once

a73x 2026-09-03 05:20

Commit message

fix: a QUIC client sends CONNECTION_CLOSE on teardown, so a poll frees its slot at once

Every QUIC connection takes a daemon client slot at the handshake, and
the wall polls each QUIC host once a second on a connection of its own.
quic.Client.deinit deleted the connection and closed the socket without
a goodbye, so the daemon freed each poll's slot only at its 15 s idle
timer: eight polls filled the eight slots, and a laptop whose only
clients were another wall's polls refused its own wall (clients=8 with
one client per session, found 2026-09-02).

sayGoodbye writes CONNECTION_CLOSE before the socket closes, mirroring
the listener's closeAll; the daemon's read path already kills a peer
that closed. Pinned by a listener test (slot freed within 1 s at a 5 s
idle) and a handoff-group leg that samples the daemon's clients= gauge
while a hub polls it (109 scenarios now).

CLAUDE.md

Old		New
@@ -187,6 +187,14 @@ own. Test fixtures in `test/`:
187	tile locally, because a pump parked in `dial` reads no ask, a pending pane	187	tile locally, because a pump parked in `dial` reads no ask, a pending pane
188	has no pump at all, and birthing onto an `unreachable` row is a designed	188	has no pump at all, and birthing onto an `unreachable` row is a designed
189	path.	189	path.
		190	- A QUIC client says goodbye. Every QUIC connection takes one of the
		191	daemon's eight client slots at the handshake, attached or not, and the
		192	wall polls each QUIC host once a second on a connection of its own.
		193	`quic.Client.deinit` therefore writes CONNECTION_CLOSE before it closes
		194	the socket; a teardown that only dropped the socket left the daemon to
		195	learn from its 15 s idle timer, and eight polls filled the eight slots
		196	in eight seconds — every real attach after that was refused, on a
		197	laptop whose only clients were another wall's polls (2026-09-02).
190	- **A pid-named leftover is reaped by its successor, never by a signal	198	- **A pid-named leftover is reaped by its successor, never by a signal
191	handler.** The daemon's `mux-agent-PID-` and `mux-shellint-PID-`	199	handler.** The daemon's `mux-agent-PID-` and `mux-shellint-PID-`
192	directories and a wall's `mux-ask-PID.sock` are unlinked by their owner	200	directories and a wall's `mux-ask-PID.sock` are unlinked by their owner

docs/decisions.md

+`id` to undefined, which cost one red round) and by the boot group's
+restart leg, which now asserts the SIGKILLed daemon's agent directory is
+there before the restart and gone after it.
+**A laptop that refused its own wall.** Reported the same afternoon: `mux`
+on the laptop read `refused` while a wall over ssh was fine. `mux d stats`
+there said `clients=8 attaches=6 sessions=2`, with one client on each
+session — six slots held by connections attached to nothing. From this
+box, `ss -uanp` showed the wall's two tile connections to the laptop plus a
+third that changed port every second: the session poll, which opens a QUIC
+connection per tick. `quic.Client.deinit` deleted the connection and closed
+the socket without CONNECTION_CLOSE, so the daemon learned of each poll's
+end from its 15 s idle timer; at one poll a second the eight slots were
+gone in eight seconds, and a hub polling a scratch daemon reproduced the
+climb 2, 4, 6, 8 in as many seconds. The fix is `sayGoodbye` on the
+client's teardown, the mirror of the listener's `closeAll`. The daemon
+needed nothing: its read path already kills a connection whose peer
+closed. Pinned by a listener test (slot freed within 1 s at a 5 s idle)
+and a handoff-group leg sampling the daemon's gauge while a hub polls —
+whose first draft passed against the bug, because `mux d stats` puts
+`session 0 clients=1` on the SAME line and a greedy match read that
+instead of `clients=N attaches=`. The mutant's series, 1 2 3 4 5 6 7 8,
+is what the leg now prints when it fails.

src/quic.zig

Old		New
@@ -1061,6 +1061,7 @@ pub const Client = struct {
1061	}	1061	}
1062		1062
1063	pub fn deinit(self: *Client) void {	1063	pub fn deinit(self: *Client) void {
		1064	self.sayGoodbye();
1064	self.in.deinit(self.alloc);	1065	self.in.deinit(self.alloc);
1065	// ngtcp2 before the egress ring: it holds vectors into that ring for	1066	// ngtcp2 before the egress ring: it holds vectors into that ring for
1066	// anything unacknowledged, so freeing the ring first leaves it	1067	// anything unacknowledged, so freeing the ring first leaves it
@@ -1074,6 +1075,36 @@ pub const Client = struct {
1074	self.alloc.destroy(self);	1075	self.alloc.destroy(self);
1075	}	1076	}
1076		1077
		1078	/// CONNECTION_CLOSE on the way out, so the daemon frees this
		1079	/// connection's client slot NOW rather than when its idle timer expires
		1080	/// (15 s by default). The wall polls every QUIC host once a second on a
		1081	/// connection of its own; a teardown that just dropped the socket left
		1082	/// eight of those holding a daemon's eight slots inside eight seconds,
		1083	/// and every real attach after that was refused — found on a laptop
		1084	/// whose only clients were another wall's polls (2026-09-02). Best
		1085	/// effort: a peer that never handshook or already closed gets nothing.
		1086	fn sayGoodbye(self: *Client) void {
		1087	const conn = self.conn orelse return;
		1088	if (self.dead) return;
		1089	var ccerr: c.ngtcp2_ccerr = undefined;
		1090	c.ngtcp2_ccerr_default(&ccerr);
		1091	var buf: [max_udp]u8 = undefined;
		1092	var ps: c.ngtcp2_path_storage = undefined;
		1093	c.ngtcp2_path_storage_zero(&ps);
		1094	var pi: c.ngtcp2_pkt_info = .{ .ecn = 0 };
		1095	const n = c.ngtcp2_conn_write_connection_close_versioned(
		1096	conn,
		1097	&ps.path,
		1098	c.NGTCP2_PKT_INFO_VERSION,
		1099	&pi,
		1100	&buf,
		1101	buf.len,
		1102	&ccerr,
		1103	timestampNs(),
		1104	);
		1105	if (n > 0) _ = std.posix.send(self.fd, buf[0..@intCast(n)], 0) catch {};
		1106	}
		1107
1077	/// Readable means a datagram arrived, not that a frame is waiting.	1108	/// Readable means a datagram arrived, not that a frame is waiting.
1078	pub fn pollFd(self: *const Client) std.posix.fd_t {	1109	pub fn pollFd(self: *const Client) std.posix.fd_t {
1079	return self.fd;	1110	return self.fd;

src/server/quic_server.zig

@@ @@ -1692,3 +1692,41 @@ test "closeAll: a connected client hears the goodbye" { @@
+test {
+    std.testing.refAllDecls(@This());
+test "Listener: a peer's deinit frees its slot at once, not at the idle timeout" {
+    const alloc = std.testing.allocator;
+    const key: quic.Key = .{ .bytes = [_]u8{0x3D} ** quic.key_len };
+    // Long enough that an idle-timer reap inside this test's budget is
+    // impossible: the close observed below has to be the peer's goodbye.
+    const idle_ms = 5000;
+    var owner: EchoOwner = .{};
+    defer owner.deinit();
+    const setup = try loopbackListener(alloc, key, &owner, idle_ms);
+    defer setup.l.deinit();
+    var cl = try TestPeer.init(setup.addr, key);
+    try std.testing.expect(pump(setup.l, &cl, 5000, struct {
+        fn f(o: *EchoOwner, t: *TestPeer) bool {
+            return t.cl.handshake_done and o.opened > 0;
+        }
+    }.f, &owner));
+    try std.testing.expectEqual(@as(usize, 0), owner.closed);
+    // The peer leaves the way every wall poll does: deinit, nothing else.
+    // Without CONNECTION_CLOSE the listener would hold this slot for
+    // `idle_ms`, and a daemon's eight slots are eight seconds of polling.
+    cl.deinit();
+    var waited: u64 = 0;
+    while (waited < 1000 and owner.closed == 0) : (waited += 10) {
+        var fds = [_]std.posix.pollfd{
+            .{ .fd = setup.l.fd, .events = std.posix.POLL.IN, .revents = 0 },
+        };
+        _ = std.posix.poll(&fds, 10) catch break;
+        if (fds[0].revents != 0) setup.l.readable();
+        setup.l.tick();
+        for (setup.l.conns) |slot| {
+            if (slot) |cn| setup.l.drain(cn);
+        }
+    }
+    try std.testing.expectEqual(@as(usize, 1), owner.closed);
+}

test/e2e.sh

Old		New
@@ -170,8 +170,8 @@ done
170	# one of those and adds a convergence point would be pinning a fact every	170	# one of those and adds a convergence point would be pinning a fact every
171	# leg above already establishes.	171	# leg above already establishes.
172		172
173	[ "$OK_COUNT" = "108" ] \|\| {	173	[ "$OK_COUNT" = "109" ] \|\| {
174	echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 108 —"	174	echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 109 —"
175	echo " a scenario was added (update the pin) or silently lost"	175	echo " a scenario was added (update the pin) or silently lost"
176	exit 1	176	exit 1
177	}	177	}

test/e2e_04_handoff.sh

@@ @@ -355,6 +355,43 @@ chmod 600 "$HKEY" @@
+start_daemon "$SOCK17" "$OUT.d17.d" "key-mismatch daemon never bound" --shell /bin/sh \
+    --quic "127.0.0.1:$HQPORT" --key "$HKEY"
+HDPID=$DPID
+# ---- a QUIC poller holds one client slot, not one per second -----------
+#
+# Every QUIC connection takes a client slot at the handshake, and the wall
+# polls each QUIC host once a second on a connection of its own. A client
+# that dropped its socket without CONNECTION_CLOSE left the daemon to
+# learn of it from the 15 s idle timer: eight polls filled the eight
+# slots, and every attach after that was refused — found on a laptop whose
+# only clients were another wall's polls (2026-09-02). Sampled off
+# `mux d stats`, which counts slots HELD, while a hub polls this daemon.
+HPOLLSTATE="${TMPDIR:-/tmp}/mux-e2e-hpoll-$$"
+defer_rm "$HPOLLSTATE"
+mkdir -p "$HPOLLSTATE"
+XDG_STATE_HOME="$HPOLLSTATE" "$MUX" web "quic://127.0.0.1:$HQPORT" --key "$HKEY" \
+    --port "$((HQPORT + 1))" > "$OUT.hpoll" 2>&1 &
+HPOLLPID=$!
+defer_kill "$HPOLLPID"
+# The daemon's gauge is the `clients=N attaches=` pair; the same line goes
+# on to say `session 0 clients=1`, which a greedy match reads instead (the
+# first draft of this leg did, and passed against the bug).
+HPOLLMAX=0
+HPOLLSEEN=""
+for _ in 1 2 3 4 5 6 7 8; do
+    sleep 1
+    HPOLLNOW=$("$MUX" d stats --sock "$SOCK17" 2>/dev/null | sed -n 's/.* clients=\([0-9]*\) attaches=.*/\1/p' | head -1)
+    HPOLLSEEN="$HPOLLSEEN ${HPOLLNOW:-?}"
+    [ "${HPOLLNOW:-0}" -gt "$HPOLLMAX" ] && HPOLLMAX=$HPOLLNOW
+done
+softkill "$HPOLLPID"
+# At most three: the hub's tile on session 0, the poll in flight, and the
+# one before it still draining. Without the goodbye this reads 8 by the
+# eighth second.
+[ "$HPOLLMAX" -le 3 ] || {
+    echo "e2e FAIL: eight seconds of polling held $HPOLLMAX client slots at once (want at most 3); per second:$HPOLLSEEN"
+    cat "$OUT.hpoll"; exit 1; }
+ok "a QUIC poller holds one client slot, not one per second"
+HSHIMS_D=$(wc -l < "$SSHIM_PIDLOG")
+HT0=$(date +%s%N)
+pipe_mux "$OUT.h4" "$OUT.h4.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN2" PATH="$HPATH" timeout 40 \