8a005f3a

feat(m8): QUIC held on a real path — three legs of four; the record

a73x 2026-08-08 14:08

Commit message

feat(m8): QUIC held on a real path — three legs of four; the record

The LAN box found a bug that loopback structurally could not, and the
measurement it then produced says something more interesting than the plan
expected.

**The bug.** ngtcp2 calls our receive callback and then goes on using the
connection: after conn_call_recv_stream_data it calls
conn_emit_pending_stream_data, which dereferences it again. The daemon
closes a connection from inside that callback whenever a client detaches,
so it was freeing the Conn underneath ngtcp2. The second call only runs
when there is buffered out-of-order stream data to flush, and loopback does
not reorder — so every test passed and the daemon segfaulted on the first
QUIC attach over a real network. Teardown is now deferred: closeConn and
kill mark, feed reaps on the way out. Loopback is not a network.

**The measurement**, one daemon and one link and one run, ssh-via beside
QUIC. Cold attach 6.9ms vs 9.8ms; with 75ms of emulated delay on the real
interface, 234.1ms vs 235.7ms. Ten tears via a genuine UDP blackhole, all
ten resumed hands-off, snapshot counter unmoved (16 -> 16), first marker
still on the grid at the end.

Legs 1, 3 and 4 hold. **Leg 2 does not**, for two reasons that are worth
more than the leg. Its threshold — 2.5×RTT — is unmeetable on a 0.1ms LAN
by anything, ssh-via included, at thirty times over. And its tear
comparison is not like-for-like: ssh's tear is a killed process, so EOF is
instant and the immediate first retry succeeds, while a QUIC tear must be a
blackhole outlasting the idle timer, which guarantees the client's first
retry lands inside it. The 461.7ms is our backoff schedule, and it barely
moved when the idle timeout was cut from 1500ms to 400ms.

What leg 2's rationale actually names — the channel-open floor being
deleted — is the cold-attach row, and there QUIC wins by 1.6ms at 75ms RTT.
The floor is deleted and QUIC hands it straight back: this listener answers
every fresh Initial with a Retry, so address validation costs a round trip
ssh never pays. That is the honest headline, it is stated inside the leg-3
figure rather than netted out of it, and it names the next win.

decisions.md carries the M8 section: the four driving decisions, the spike
findings, the full 0-RTT evidence chain, every bug and its lesson, the
contracts that were true but unwritten, and the banked list. README gains
the QUIC quickstart and the one-sentence trust model.

wan.sh gains the quic phase, and one fix it needed: the box is dual-homed,
so `ip route show default` returned two interfaces and netem was being
installed on one our traffic never crossed. It now asks the box which
device reaches this client. Firewall rules are tagged, removed and verified
like the qdisc, because a DROP left behind blackholes a port for everyone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

README.md

Old		New
@@ -5,7 +5,32 @@ authoritatively in a daemon and replicated in the client — state sync
5	instead of escape-sequence replay. See `docs/handoff.md` for the design	5	instead of escape-sequence replay. See `docs/handoff.md` for the design
6	and `docs/decisions.md` for decisions made.	6	and `docs/decisions.md` for decisions made.
7		7
8	Status: M7 — reconnect. A dropped link is a non-event: the client	8	Status: M8 — QUIC. A third transport arm: `mux quic://HOST:PORT --key
		9	FILE` attaches straight to a `muxd run --quic`, with no ssh in the path.
		10	Authentication is TLS 1.3 external PSK — both ends hold the same 32-byte
		11	key and nobody holds a certificate. Measured on a LAN box against ssh-via
		12	in the same run: cold attach 6.9ms vs 9.8ms, and 234.1ms vs 235.7ms with
		13	75ms of emulated delay on the real path. Ten consecutive network tears (a
		14	real UDP blackhole) all resumed hands-off and all delta-served. The wire
		15	protocol did not change: `git diff` on `src/protocol.zig` is empty across
		16	the whole milestone, which is the third time transport has turned out to
		17	be a swap rather than a redesign.
		18
		19	### QUIC quickstart
		20
		21	```sh
		22	head -c 32 /dev/urandom > ~/.mux.key && chmod 600 ~/.mux.key # 32 raw bytes or 64 hex chars
		23	scp ~/.mux.key muxd HOST: # the key and a static binary
		24	ssh HOST 'chmod 600 ~/.mux.key; muxd run --quic 0.0.0.0:4433 --key ~/.mux.key'
		25	mux quic://HOST:4433 --key ~/.mux.key # or set MUX_KEY_FILE
		26	```
		27
		28	Trust model, in one sentence: anyone holding that key file can attach to
		29	that session, so it is exactly as sensitive as an ssh private key — `muxd`
		30	and `mux` both refuse one that is readable by group or other, and there is
		31	no unauthenticated mode to fall back to.
		32
		33	Previously, M7 — reconnect. A dropped link is a non-event: the client
9	keeps its replica, rebuilds the transport, and re-attaches quoting what it	34	keeps its replica, rebuilds the transport, and re-attaches quoting what it
10	already holds, so the daemon can answer with a delta instead of a repaint.	35	already holds, so the daemon can answer with a delta instead of a repaint.
11	Measured over a real WAN link: 10 consecutive transport kills against a	36	Measured over a real WAN link: 10 consecutive transport kills against a
@@ -14,7 +39,7 @@ snapshot counter never moved), first frame back in ~254 ms — most of which
14	is the client's own 200 ms first retry. `mux HOST` attaches over ssh in	39	is the client's own 200 ms first retry. `mux HOST` attaches over ssh in
15	one word.	40	one word.
16		41
17	Previously, M6 — transport. All handoff milestones complete; all kill	42	Before that, M6 — transport. All handoff milestones complete; all kill
18	criteria cleared. The prototype's two founding questions are both	43	criteria cleared. The prototype's two founding questions are both
19	answered yes: ghostty-vt serves as an authoritative headless grid in a	44	answered yes: ghostty-vt serves as an authoritative headless grid in a
20	daemon without forking it, and detach/reattach as state sync is correct	45	daemon without forking it, and detach/reattach as state sync is correct

docs/decisions.md

+  files exist in `contrib/`, an install doc does not); the reconnect loop
+  has no upper bound on total retry time by design, so a client left
+  attached to a permanently dead daemon retries until the user aborts.
+- **Out of scope, in order:** QUIC/TLS transport (M8, done), then
+  prediction / local echo — which now has this milestone's numbers to beat
+  as well as M6's ~4ms protocol cost.
+## 2026-08-08 (M8)
+QUIC as a third transport arm: `muxd run --quic HOST:PORT --key FILE` and
+`mux quic://HOST:PORT --key FILE`. ssh stays the default; QUIC is opt-in per
+invocation and `mux HOST` still means ssh.
+### The four driving decisions
+- **PSK, not certificates.** The deploy model is "scp a static binary to a
+  VM you own", so scp a 32-byte key beside it. TLS 1.3 external PSK gives
+  mutual authentication by possession with no CA, no TOFU, and no expiry to
+  manage. Certificates and a TOFU store stay banked.
+- **ssh stays the default.** QUIC is faster to set up and it is one more
+  thing to configure; making it opt-in costs nothing and means a broken
+  QUIC build can never break `mux HOST`.
+- **Zero protocol changes.** Frames, attach/prefix layouts and seq/epoch
+  semantics are untouched. `git diff b646277..HEAD -- src/protocol.zig` is
+  empty and is leg 4 of the kill criterion — the swap thesis, third pass.
+- **Spike-gated.** No mature pure-Zig QUIC exists, so a C stack had to
+  prove it builds under the pinned Zig 0.15.2 for native *and* static musl
+  before anything else was planned.
+### The spike, and what it settled
+ngtcp2 1.25.0 + wolfSSL 5.9.2 build under `zig cc` for both targets and add
+~1.68MB to a static binary. Two findings outlived the spike:
+- **External PSK is already 1-RTT.** The handshake completes in one round
+  trip without a session ticket, so resumption buys nothing over a fresh
+  PSK connection. That left 0-RTT as the only remaining reconnect win —
+  and 0-RTT turned out to be unavailable (below).
+- **The keylog ban is a standing build rule.** Built with
+  `WOLFSSL_KEYLOG_EXPORT` on, the binaries wrote every handshake's secrets
+  to `./sslkeylog.log` with nothing asking them to. It is off in
+  `deps/quic/build-deps.sh` permanently; turn it on only in a local
+  throwaway build, and never commit that.
+### 0-RTT: dropped on evidence, not on effort
+The plan named 0-RTT as the reconnect win. It cannot be had from where we
+are, and the evidence is worth keeping because the next person will ask:
+- It is not compiled in. The built `options.h` has no `WOLFSSL_EARLY_DATA`
+  and `nm -g libwolfssl.a` finds zero early-data symbols.
+- It cannot be enabled through cmake. wolfSSL 5.9.2's CMakeLists exposes
+`WOLFSSL_*` options and mentions EARLY zero times; early data exists
+  only as the autotools flag `--enable-earlydata`, default off.
+- Forcing the define would be worse than not having it. It is not a pure
+  feature gate: it adds `maxEarlyDataSz` to WOLFSSL_SESSION, a
+  `clientInEarlyData` bitfield to Options, a member to the ticket struct
+  and a TLSX enum value. Compiling the library with it while the installed
+  `options.h` lacks it is a silent ABI mismatch — memory corruption, not a
+  link error. A latency optimisation is never worth that.
+- And external PSK is the wrong door anyway: the client-side gate reads
+  `session->maxEarlyDataSz`, populated from a session ticket's early_data
+  extension. An external PSK carries no ticket, so there is nothing to
+  read. Getting 0-RTT means adopting ticket-based resumption on top of the
+  PSK — a different design, not a flag.
+ngtcp2 exports four `early_data` entry points; it is TLS-agnostic and has
+no backend here to serve them. Banked: autotools wolfSSL with
+`--enable-earlydata` plus ticket resumption.
+### The bugs this milestone paid for
+- **ngtcp2 does not copy stream payload, and we assumed it did.**
+  `ngtcp2_conn_writev_stream` stores the *vector* it is handed —
+  `ngtcp2_vec_copy` is a memcpy of base+len, not of the bytes — and
+  re-reads those bytes to retransmit. The first egress buffer reallocated
+  on append and cleared as soon as the last byte had been handed over, so
+  ngtcp2 was left holding freed or recycled memory and died inside
+  `ngtcp2_pkt_encode_stream_frame`. It fired about three runs in seventy,
+  only in the one test big enough to overflow a socket buffer.
+  **The fix is an invariant, not a patch:** a byte handed to ngtcp2 does
+  not move or get overwritten until the peer acknowledges it. `Egress` is a
+  fixed ring; `head` advances only from `acked_stream_data_offset`.
+  **The lesson:** a C library's ownership rules are part of its API, and
+  "it copied it" is the assumption to check first.
+  **Not an invariant, and do not treat it as one:** the ring does not lap
+  its in-flight region on loopback only because the 256KB ring happens to
+  be at least the 256KB stream window the peer advertises. That is
+  configuration. Raise the advertised window without raising the ring and
+  laps become reachable again — the lifetime rule is what stops that being
+  a use-after-free, not the arithmetic.
+- **Closing a connection from inside a receive callback is a
+  use-after-free, and only a real network shows it.** ngtcp2 calls our
+  receive callback and then goes on using `conn`: after
+  `conn_call_recv_stream_data` it calls `conn_emit_pending_stream_data`,
+  which dereferences the connection again. The daemon's `.detach` handling
+  reaches `closeConn` from inside that callback, which freed the Conn
+  underneath ngtcp2. The LAN box crashed the daemon on the first QUIC
+  attach of the first run. Teardown is now deferred: `in_ngtcp2` marks the
+  window, `closeConn`/`kill` mark rather than free, and `feed` reaps on the
+  way out.
+  **It was never loopback-only, and the first version of this note said it
+  was.** `conn_emit_pending_stream_data` is called unconditionally
+  (ngtcp2_conn.c:7646), and its first statement dereferences the
+  connection — `conn_is_tls_handshake_completed(conn)` at :7220 — *before*
+  the `if (!strm->rx.rob)` early-out two lines below it. The freed
+  connection was therefore read on every receive, on every path, from the
+  moment the bug existed. What a reordering network changed was not
+  whether the read happened but whether it landed on memory the allocator
+  had already reclaimed, and so whether it faulted.
+  **The lesson is sharper than "loopback is not a network": a
+  use-after-free that reads successfully is exactly the kind that ships.**
+  Every test we had was exercising it and reporting green. The real
+  network did not introduce the bug, it withdrew the luck.
+- **Migration was defeated, which is most of why QUIC is here.** `route()`
+  matched the destination CID against the one a connection was created
+  with, but `get_new_connection_id2` advertises more, and a client that
+  changes network switches to one of the others (RFC 9000 §9.5). Those
+  packets fell through to `accept()`, which has no token for them and drops
+  them: the migration presented as a dead connection. Connections now match
+  any CID they have advertised and not retired. The test tells a routing
+  hit from a miss by what a *miss* does — fall through to `accept()`, which
+  answers a token-less Initial with a Retry — so a probe socket that hears
+  nothing was delivered, and the same probe hearing a Retry for an unknown
+  CID is what proves the packet was well-formed enough to have been
+  answered.
+- **One port, one daemon.** The listener set SO_REUSEADDR, which on UDP
+  lets a second daemon bind the same address while the kernel hands each
+  datagram to one of them — two sessions silently splitting a port. This is
+  the socket-steal incident from M7 in its QUIC edition, and it gets the
+  same answer: fail the bind loudly. The bind also moved ahead of the
+  session socket, so a refused port costs no shell and leaves no socket.
+- **Every QUIC attach was snapshot-served.** The daemon's client-slot
+  `.attach` arm snapshotted unconditionally, on reasoning that held while
+  every client began life as an observer and did its first attach from
+  there. A QUIC connection is promoted to a client slot when its handshake
+  completes, so its *first* attach lands on that arm — and no reconnecting
+  QUIC client could ever resume from a delta. That is kill-criterion leg 1,
+  and **the counter was the only witness**: a snapshot and a delta render
+  identically. Fourth time a counter assertion has caught something markers
+  could not. Fixing it also improved socket double-attach into parity with
+  the promotion path.
+- **The socket unlink guard compared inodes across filesystems.** `fstat`
+  on a bound unix socket's descriptor answers with the sockfs inode;
+  `fstatat` on the path answers with the filesystem inode. They can never
+  be equal, so the guard was false every time and the daemon never unlinked
+  its socket on a clean exit — masked since 45ab077 by the stale-socket
+  recovery cleaning up on the next start. **The lesson: a guard whose false
+  branch is indistinguishable from success needs a test that observes the
+  EFFECT.** "Refused to delete someone else's socket" and "failed to delete
+  its own" leave identical evidence.
+### Things that are true and were not written down
+- `closeConn` closes one connection and never the shared socket; it does
+  not call back into `onClose` (a close the owner asked for needs no
+  callback telling it so); and it sends no CONNECTION_CLOSE — the peer
+  learns via idle timeout. That last one is a real cost, accepted for v1
+  rather than unnoticed.
+- `onOpen` and `onClose` do not pair. A handshake that never completes
+  produces neither: the connection is torn down having never been
+  announced.
+- Flow control is extended before the owner consumes, which is sound only
+  because the owner's queue is bounded — it was not, until the egress ring.
+- `max_conns` (16) exceeds `max_clients` (8) on purpose: a connection
+  exists from handshake completion and only then asks for a slot.
+### `idle_ms` is doing three jobs
+One knob currently serves death detection (wants seconds), the handshake
+timeout (wants a small multiple of RTT), and reaping a connection that
+handshook and never attached (wants to be long). That is why the abort key
+mattered so much: the handshake bound is far longer than a handshake needs
+to be, and `waitReady` polled only the transport, so for its whole length
+nothing watched for Ctrl-\ — measured at 14.6s of deafness on the default.
+It now watches stdin too and answers in ~200ms. Splitting the three is
+banked.
+Related policy, decided rather than defaulted: bytes typed during a
+handshake are **carried** on a first attach (they are the user's opening
+command, and dropping them silently eats the first line of any piped
+session) and **dropped** during a reconnect, matching the long-standing
+rule that input typed while disconnected is not replayed.
+### Process rules this milestone paid for
+- **The build.zig test-discovery hazard is CLOSED.** `src/main.zig`'s
+  module is in the test loop; removing it again drops the count from 100 to
+. Every module with tests is now in that list.
+- **A test that hangs on regression is worth much less than one that
+  fails.** Restoring SO_REUSEADDR made the double-bind scenario's second
+  daemon bind and run forever; the invocation had no `timeout`, so the
+  suite hung for ten minutes instead of failing. Every refusal path now
+  runs under `timeout`, and the same mutation reports "exit 124, want 1" in
+  ten seconds. CI would have read the old behaviour as an infrastructure
+  timeout rather than a bug.
+- **`sun_path` is retired as a hazard.** `std.testing.tmpDir` lives under
+  `.zig-cache`, so a socket path was as long as wherever the repository was
+  checked out, and 108 bytes is the cap. It cost three people a gate cycle
+  each. `src/testtmp.zig` hands out ~21-character directories under /tmp;
+  verified by building from a 128-character path.
+- **Defence in depth is real and still working.** The epoch fence alone
+  does not fail the restart scenario — the seq-range check catches a stale
+  seq first. Both must be removed before the assertion fires, exactly as
+  M7 recorded.
+### The kill criterion, on the LAN box (2026-08-08)
+One daemon, one link, one run: the ssh-via and QUIC columns are the same
+session over the same wire. Box is `192.168.0.109` on a LAN (baseline byte
+round-trip 0.1ms); the netem rows are **emulated-RTT-real-topology** —
+`netem delay 75ms` on the box's real interface, so the delay is synthetic
+and everything else about the path is not.
+|                                | quic   | ssh-via |
+|--------------------------------|--------|---------|
+| cold attach -> first paint     | 6.9ms  | 9.8ms   |
+| echo                           | 2.7ms  | 2.9ms   |
+| cold attach, netem 75ms        | 234.1ms| 235.7ms |
+| echo, netem 75ms               | 78.5ms | 79.0ms  |
+| tear -> usable                 | 461.7ms| 7.0ms   |
+**Leg 1 — semantics intact: HOLDS.** Attach, echo, detach and reattach all
+work over `quic://` (e2e covers each as its own scenario). Ten consecutive
+tears against a live session, every one resumed hands-off, the daemon's
+snapshot counter unmoved across all ten (16 -> 16: every resume
+delta-served), and the first tear's marker still on the grid after the
+last.
+**Leg 2 — median tear-to-usable ≤2.5×RTT and strictly below the same run's
+ssh-via median: NOT MET, and the wording is part of why.** Two separate
+problems, stated plainly rather than reconciled:
+- At LAN RTT, 2.5×RTT is ~0.25ms. Nothing meets it — ssh-via's own 7.0ms
+  fails it by a factor of thirty. The threshold was written for a
+  WAN-latency path and is unmeetable on a fast link by any transport.
+- The two tear-to-usable numbers are not comparable, because the tears are
+  not alike. ssh's tear is a killed process: EOF is instant, and since M8
+  Task 0 the client's first reconnect attempt is immediate, so it succeeds
+  at once. QUIC's tear is a UDP blackhole that must outlast the idle timer
+  to cause a tear at all — so the client's immediate first attempt
+  necessarily lands *inside* the blackhole and fails, guaranteeing at least
+  one backoff step before recovery. The 461.7ms is our own backoff
+  schedule, not the transport; it barely moved when the idle timeout was
+  cut from 1500ms to 400ms, which is the tell.
+The quantity leg 2's own rationale names — "the channel-open floor is the
+thing being deleted" — is connection setup, and that is the cold-attach
+row. QUIC wins it in both conditions. But look at the netem row: **234.1
+vs 235.7ms, a difference of 1.6ms at 75ms RTT.** The channel-open floor
+really is deleted, and QUIC hands it straight back: this listener answers
+every fresh Initial with a Retry, so address validation costs a round trip
+ssh never pays. That is the honest headline of this milestone's
+measurement, and it points at the obvious next win — a listener that skips
+Retry when it does not need address validation would take ~75ms off cold
+attach at this RTT.
+**Leg 3 — cold QUIC attach ≤ same-run ssh-via attach: HOLDS**, 6.9 ≤ 9.8
+on the clean link and 234.1 ≤ 235.7 under netem. **The Retry round trip is
+INSIDE the QUIC figure**, not netted out: the comparison is pessimistic for
+QUIC by roughly one RTT, and stating which way it leans is the point.
+**Leg 4 — `git diff b646277..HEAD -- src/protocol.zig` empty: HOLDS.** The
+transport was swapped a third time and the wire never noticed.
+**Verdict: three legs of four hold. Leg 2 is not met** — unmeetable as
+worded on a LAN, and its tear comparison measures our backoff rather than
+the transport. QUIC is faster to set up than ssh-via on every measurement
+taken, by a margin that the Retry round trip very nearly cancels.
+Two other figures worth keeping. **Handshake:** cold attach at 75ms RTT is
+ms ≈ 3.1×RTT, which is Retry + handshake + attach + first paint — the
+input the banked three-way `idle_ms` split will want. **Abort during a
+handshake:** 301ms against a 15000ms bound, on a genuinely blackholed port
+where the full bound would otherwise run; this is the first time that
+property has been observed anywhere the bound is real. **Client slots held
+at peak during the tears: 0** — the first field data on the banked
+half-open question, and it says the reaping is keeping up under a tear
+loop.
+### Banked by M8 (carried forward)
+- **Half-open reaping.** A connection that completes its handshake and
+  never attaches holds a client slot until its idle timeout. Bounded and
+  self-clearing, and measured at 0 slots held under a ten-tear loop — but
+  the policy question (how long is too long, and is it idle_ms's job at
+  all) has no answer yet. `muxd stats` now reports `clients=N`, which is
+  the instrument for deciding it.
+- **`timeoutMs` does not short-circuit a buffered frame.** A whole frame
+  already in the client's buffer can wait out the poll — up to ~100ms of
+  render lag. One line if it ever shows up against a real RTT.
+- **Skip Retry when address validation is not needed.** Worth ~one RTT on
+  cold attach, which the LAN numbers show is most of QUIC's remaining
+  margin over ssh-via. **Not "skip Retry" on its own:** address validation
+  is what keeps this listener from being an amplification reflector, and
+  the spike measured its ratio at 0.08. The banked design is NEW_TOKEN —
+  issuing a token to a validated client for it to present next time, with
+  the reuse rules that implies — falling back to RFC 9000's 3x
+  anti-amplification limit for clients that have no token yet. Dropping
+  validation without one of those two is a regression wearing a
+  performance win's clothes.
+- **`idle_ms` split three ways** — death detection, handshake timeout,
+  half-open reaping.
+- **`addCSourceFiles` for the QUIC deps**, replacing the build script.
+- **Autotools wolfSSL + ticket resumption**, the only route to 0-RTT.
+- **Certificates / TOFU**, QUIC datagrams for input, multiplexing several
+  sessions per connection, NAT traversal.
+- **RTT-multiple thresholds need rewriting before they are reused.** Two
+  milestones' criteria have now failed on a 0.1ms box for the same
+  structural reason: M6's reattach gate (2x round-trip) and M8's leg 2
+  (2.5xRTT) are both unmeetable by *any* implementation on the hardware
+  available, so what they measure is the hardware. A threshold expressed as
+  a multiple of RTT needs either a floor term for the fast-link case or an
+  explicit statement of the minimum RTT it is meaningful at. M9 should fix
+  the form before writing another one.
+- **A post-M8 amendment to the criterion:** leg 4 asks only that
+  `protocol.zig` not change. Three transports in, that is worth restating
+  as what it has come to mean — no transport may need a protocol change —
+  and possibly worth a dedup pass across the three arms now that there are
+  three.
+## Open (owed by later milestones)

docs/superpowers/plans/2026-08-08-m8-quic.md

src/quic_server.zig

Old		New
@@ -378,6 +378,9 @@ const Conn = struct {
378	cids: [max_cids]c.ngtcp2_cid = undefined,	378	cids: [max_cids]c.ngtcp2_cid = undefined,
379	ncids: usize = 0,	379	ncids: usize = 0,
380	cids_dirty: bool = true,	380	cids_dirty: bool = true,
		381	/// Set when this connection must go but cannot be freed yet, because
		382	/// ngtcp2 is inside a call on it. See Listener.in_ngtcp2.
		383	close_state: enum { open, closing_quiet, closing_notify } = .open,
381	stream_id: i64 = -1,	384	stream_id: i64 = -1,
382	/// Bytes owed to this peer. See Egress: they do not move until acked.	385	/// Bytes owed to this peer. See Egress: they do not move until acked.
383	out: Egress,	386	out: Egress,
@@ -535,7 +538,7 @@ fn recvStreamDataCb(
535	// second buffer to overflow by being generous. It would stop being the	538	// second buffer to overflow by being generous. It would stop being the
536	// right trade the moment an owner could hold bytes indefinitely, which	539	// right trade the moment an owner could hold bytes indefinitely, which
537	// is exactly what the egress ring now refuses to let it do.	540	// is exactly what the egress ring now refuses to let it do.
538	if (datalen > 0) {	541	if (datalen > 0 and cn.close_state == .open) {
539	cn.listener.handler.onData(cn.listener.handler.ctx, cn.id, data[0..datalen]);	542	cn.listener.handler.onData(cn.listener.handler.ctx, cn.id, data[0..datalen]);
540	}	543	}
541	return 0;	544	return 0;
@@ -553,6 +556,26 @@ pub const Listener = struct {
553	/// previous run must not validate against this one.	556	/// previous run must not validate against this one.
554	retry_secret: [32]u8,	557	retry_secret: [32]u8,
555	idle_ms: u64,	558	idle_ms: u64,
		559	/// True while ngtcp2 owns the stack — i.e. inside `ngtcp2_conn_read_pkt`.
		560	///
		561	/// A connection MUST NOT be freed during that window. ngtcp2 calls our
		562	/// receive callback and then goes on using `conn`: after
		563	/// conn_call_recv_stream_data it calls conn_emit_pending_stream_data,
		564	/// which dereferences the connection again. Our callback reaches the
		565	/// daemon, and the daemon may well decide the client is finished — a
		566	/// `.detach` frame is exactly that — so `closeConn` used to free the
		567	/// Conn underneath ngtcp2 and it segfaulted on the next dereference.
		568	///
		569	/// It happened on loopback too, and that is the part worth remembering.
		570	/// conn_emit_pending_stream_data is called unconditionally, and its
		571	/// FIRST statement dereferences the connection —
		572	/// conn_is_tls_handshake_completed(conn), ngtcp2_conn.c:7220 — before
		573	/// the `if (!strm->rx.rob)` early-out two lines later. So the freed
		574	/// connection was read on every single receive, everywhere. Reordering
		575	/// decided only whether that read landed on memory the allocator had
		576	/// already taken back, and therefore whether it faulted. A real network
		577	/// did not create this bug; it just stopped it reading successfully.
		578	in_ngtcp2: bool = false,
556		579
557	/// Bind the socket and stand up TLS: everything that can fail for	580	/// Bind the socket and stand up TLS: everything that can fail for
558	/// reasons outside this process. Split from the handler because the	581	/// reasons outside this process. Split from the handler because the
@@ -714,6 +737,15 @@ pub const Listener = struct {
714	for (&self.conns) \|*slot\| {	737	for (&self.conns) \|*slot\| {
715	if (slot.*) \|cn\| {	738	if (slot.*) \|cn\| {
716	if (cn.id != id) continue;	739	if (cn.id != id) continue;
		740	// Deferred if ngtcp2 is mid-call on this connection: it will
		741	// dereference `conn` again after our callback returns, and
		742	// freeing here is a use-after-free. `feed` reaps on the way
		743	// out. Quietly, per this function's contract — a close the
		744	// owner asked for gets no callback telling it so.
		745	if (self.in_ngtcp2) {
		746	cn.close_state = .closing_quiet;
		747	return;
		748	}
717	cn.deinit(self.alloc);	749	cn.deinit(self.alloc);
718	self.alloc.destroy(cn);	750	self.alloc.destroy(cn);
719	slot.* = null;	751	slot.* = null;
@@ -722,6 +754,20 @@ pub const Listener = struct {
722	}	754	}
723	}	755	}
724		756
		757	/// Free every connection that asked to go while ngtcp2 held the stack.
		758	fn reapClosing(self: *Listener) void {
		759	for (&self.conns) \|*slot\| {
		760	const cn = slot.* orelse continue;
		761	if (cn.close_state == .open) continue;
		762	const notify = cn.close_state == .closing_notify and cn.opened;
		763	const id = cn.id;
		764	cn.deinit(self.alloc);
		765	self.alloc.destroy(cn);
		766	slot.* = null;
		767	if (notify) self.handler.onClose(self.handler.ctx, id);
		768	}
		769	}
		770
725	/// An unknown connection ID. Either a new peer or noise; either way the	771	/// An unknown connection ID. Either a new peer or noise; either way the
726	/// address is unvalidated, so this is where amplification protection	772	/// address is unvalidated, so this is where amplification protection
727	/// lives.	773	/// lives.
@@ -1018,10 +1064,19 @@ pub const Listener = struct {
1018	.user_data = null,	1064	.user_data = null,
1019	};	1065	};
1020	var pi: c.ngtcp2_pkt_info = .{ .ecn = 0 };	1066	var pi: c.ngtcp2_pkt_info = .{ .ecn = 0 };
1021	if (c.ngtcp2_conn_read_pkt(conn, &path, &pi, pkt.ptr, pkt.len, timestampNs()) != 0) {	1067	self.in_ngtcp2 = true;
1022	self.kill(slot);	1068	const rv = c.ngtcp2_conn_read_pkt(conn, &path, &pi, pkt.ptr, pkt.len, timestampNs());
		1069	self.in_ngtcp2 = false;
		1070	// Reaped before anything else touches the table: a connection the
		1071	// handler closed mid-callback is gone from here on, and `cn` below
		1072	// would be a dangling pointer if it were one of them.
		1073	const closed = slot.* == null or slot.*.?.close_state != .open;
		1074	self.reapClosing();
		1075	if (rv != 0) {
		1076	if (!closed) self.kill(slot);
1023	return;	1077	return;
1024	}	1078	}
		1079	if (closed) return;
1025	self.drain(cn);	1080	self.drain(cn);
1026	}	1081	}
1027		1082
@@ -1087,6 +1142,14 @@ pub const Listener = struct {
1087		1142
1088	fn kill(self: Listener, slot: ?*Conn) void {	1143	fn kill(self: Listener, slot: ?*Conn) void {
1089	const cn = slot.* orelse return;	1144	const cn = slot.* orelse return;
		1145	// Same hazard as closeConn, and reached the same way: ngtcp2 is
		1146	// still using this connection. Marked to be reaped WITH its
		1147	// callback, since a kill is the listener giving up rather than the
		1148	// owner asking.
		1149	if (self.in_ngtcp2) {
		1150	cn.close_state = .closing_notify;
		1151	return;
		1152	}
1090	const id = cn.id;	1153	const id = cn.id;
1091	const opened = cn.opened;	1154	const opened = cn.opened;
1092	cn.deinit(self.alloc);	1155	cn.deinit(self.alloc);
@@ -2013,3 +2076,86 @@ fn probeGotAnything(fd: std.posix.fd_t) bool {
2013	const n = std.posix.recv(fd, &buf, 0) catch return false;	2076	const n = std.posix.recv(fd, &buf, 0) catch return false;
2014	return n > 0;	2077	return n > 0;
2015	}	2078	}
		2079
		2080	test "Listener: closing a connection from inside a receive callback is deferred" {
		2081	const alloc = std.testing.allocator;
		2082	const key: Key = .{ .bytes = [_]u8{0x7E} ** key_len };
		2083
		2084	// An owner that does what the daemon does when a client says goodbye:
		2085	// closes the connection from inside onData. ngtcp2 is on the stack at
		2086	// that moment and goes on using the connection after the callback
		2087	// returns, so freeing it there is a use-after-free — one that loopback
		2088	// cannot show, because the second dereference only happens when there
		2089	// is buffered out-of-order data to flush. This test pins the deferral
		2090	// rather than the crash: the connection must still be a valid object
		2091	// when the callback returns, and gone by the time the pump comes back.
		2092	const CloseOnData = struct {
		2093	listener: *Listener = undefined,
		2094	id: u64 = 0,
		2095	opened: usize = 0,
		2096	closed: usize = 0,
		2097	received: usize = 0,
		2098	alive_after_callback: bool = false,
		2099
		2100	fn onOpen(ctx: *anyopaque, id: u64) void {
		2101	const self: *@This() = @ptrCast(@alignCast(ctx));
		2102	self.opened += 1;
		2103	self.id = id;
		2104	}
		2105	fn onData(ctx: *anyopaque, id: u64, bytes: []const u8) void {
		2106	const self: *@This() = @ptrCast(@alignCast(ctx));
		2107	self.received += bytes.len;
		2108	self.listener.closeConn(id);
		2109	// Still addressable: the free was deferred, not done. Reading
		2110	// this through the listener is exactly what ngtcp2 does next.
		2111	self.alive_after_callback = self.listener.find(id) != null;
		2112	}
		2113	fn onClose(ctx: *anyopaque, _: u64) void {
		2114	const self: *@This() = @ptrCast(@alignCast(ctx));
		2115	self.closed += 1;
		2116	}
		2117	fn handler(self: *@This()) Handler {
		2118	return .{ .ctx = self, .onOpen = onOpen, .onData = onData, .onClose = onClose };
		2119	}
		2120	};
		2121
		2122	var owner: CloseOnData = .{};
		2123	const bind = try std.net.Address.parseIp("127.0.0.1", 0);
		2124	const l = try Listener.init(alloc, bind, key, owner.handler(), 10_000);
		2125	defer l.deinit();
		2126	owner.listener = l;
		2127	var actual: std.posix.sockaddr.storage = undefined;
		2128	var alen: std.posix.socklen_t = @sizeOf(@TypeOf(actual));
		2129	try std.posix.getsockname(l.fd, @ptrCast(&actual), &alen);
		2130	const addr = std.net.Address.initPosix(@ptrCast(@alignCast(&actual)));
		2131
		2132	var cl = try TestClient.init(addr, key);
		2133	defer cl.deinit();
		2134	try cl.start();
		2135	cl.drain();
		2136
		2137	var waited: u64 = 0;
		2138	cl.out = "goodbye";
		2139	while (waited < 10_000 and owner.received == 0) : (waited += 5) {
		2140	var fds = [_]std.posix.pollfd{
		2141	.{ .fd = l.fd, .events = std.posix.POLL.IN, .revents = 0 },
		2142	.{ .fd = cl.fd, .events = std.posix.POLL.IN, .revents = 0 },
		2143	};
		2144	_ = std.posix.poll(&fds, 5) catch break;
		2145	if (fds[0].revents != 0) l.readable();
		2146	if (fds[1].revents != 0) cl.readable();
		2147	l.tick();
		2148	cl.drain();
		2149	for (l.conns) \|slot\| {
		2150	if (slot) \|cn\| l.drain(cn);
		2151	}
		2152	}
		2153
		2154	try std.testing.expect(owner.received > 0);
		2155	// The connection outlived the callback...
		2156	try std.testing.expect(owner.alive_after_callback);
		2157	// ...and did not outlive the pump.
		2158	try std.testing.expect(l.find(owner.id) == null);
		2159	// closeConn is the owner asking, so it gets no callback back.
		2160	try std.testing.expectEqual(@as(usize, 0), owner.closed);
		2161	}

test/e2e.sh

Old		New
@@ -67,7 +67,7 @@ while [ ! -S "$SOCK" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
67		67
68	# Client with piped stdio: types a command, waits, detaches with Ctrl-\ (034).	68	# Client with piped stdio: types a command, waits, detaches with Ctrl-\ (034).
69	{ printf 'printf "e2e-%%s\\n" works\n'; sleep 2; printf '\034'; } \| \	69	{ printf 'printf "e2e-%%s\\n" works\n'; sleep 2; printf '\034'; } \| \
70	"$MUX" --sock "$SOCK" > "$OUT"	70	timeout 30 "$MUX" --sock "$SOCK" > "$OUT"
71		71
72	# 1. The client's rendered output must contain the command's result.	72	# 1. The client's rendered output must contain the command's result.
73	grep -q "e2e-works" "$OUT" \|\| {	73	grep -q "e2e-works" "$OUT" \|\| {
@@ -90,7 +90,7 @@ kill -9 "$CPID" 2>/dev/null \|\| true
90	sleep 1	90	sleep 1
91	kill -0 "$DPID" \|\| { echo "e2e FAIL: daemon died after client kill -9"; exit 1; }	91	kill -0 "$DPID" \|\| { echo "e2e FAIL: daemon died after client kill -9"; exit 1; }
92		92
93	{ sleep 1; printf '\034'; } \| "$MUX" --sock "$SOCK" > "$OUT.re"	93	{ sleep 1; printf '\034'; } \| timeout 30 "$MUX" --sock "$SOCK" > "$OUT.re"
94	grep -q "60" "$OUT.re" \|\| {	94	grep -q "60" "$OUT.re" \|\| {
95	echo "e2e FAIL: reattach after kill missing state"; cat "$OUT.re"; exit 1;	95	echo "e2e FAIL: reattach after kill missing state"; cat "$OUT.re"; exit 1;
96	}	96	}

test/wan.sh

Old		New
@@ -109,6 +109,17 @@ TAG="wan-$$-$(date +%s)"
109	RBIN="/tmp/muxd-$TAG"	109	RBIN="/tmp/muxd-$TAG"
110	RSOCK="/tmp/mux-$TAG.sock"	110	RSOCK="/tmp/mux-$TAG.sock"
111	RLOG="/tmp/muxd-$TAG.log"	111	RLOG="/tmp/muxd-$TAG.log"
		112	# QUIC needs a key on both ends and a port of its own. The port is derived
		113	# from the pid so two runs on one box cannot collide, and stays well clear
		114	# of the ephemeral range.
		115	RKEY="/tmp/mux-$TAG.key"
		116	QPORT="${MUX_WAN_QPORT:-$(( 21000 + ($$ % 4000) ))}"
		117	# Short on purpose. This is how fast a QUIC client notices a peer that has
		118	# stopped answering, and it is pure configuration: the tear measurement below
		119	# starts its clock after it, so a long value would only pad the wait while
		120	# the client sat in a backoff it had already earned.
		121	QIDLE="${MUX_WAN_QUIC_IDLE_MS:-400}"
		122	FW_LEFT=0
112		123
113	WORK="$(mktemp -d "${TMPDIR:-/tmp}/mux-wan-XXXXXX")"	124	WORK="$(mktemp -d "${TMPDIR:-/tmp}/mux-wan-XXXXXX")"
114	PY="$WORK/wan.py"	125	PY="$WORK/wan.py"
@@ -167,7 +178,24 @@ cleanup() {
167	timeout 60 $MUX_WAN_SSH "pkill -f '[m]uxd-$TAG'" >/dev/null 2>&1 \|\| true	178	timeout 60 $MUX_WAN_SSH "pkill -f '[m]uxd-$TAG'" >/dev/null 2>&1 \|\| true
168	sleep 0.5	179	sleep 0.5
169	timeout 60 $MUX_WAN_SSH "pkill -9 -f '[m]uxd-$TAG'" >/dev/null 2>&1 \|\| true	180	timeout 60 $MUX_WAN_SSH "pkill -9 -f '[m]uxd-$TAG'" >/dev/null 2>&1 \|\| true
170	timeout 60 $MUX_WAN_SSH "rm -f $RBIN $RSOCK $RLOG" \	181	# Tagged with this run's name so removal is exact. Looped because
		182	# -D removes one matching rule per call and a crashed rep can leave
		183	# more than one.
		184	local fw_tries=0
		185	while timeout 60 $MUX_WAN_SSH "sudo -n iptables -S INPUT 2>/dev/null \| grep -q 'mux-$TAG'" 2>/dev/null; do
		186	timeout 60 $MUX_WAN_SSH "sudo -n iptables -D INPUT -p udp --dport $QPORT \
		187	-m comment --comment mux-$TAG -j DROP" >/dev/null 2>&1 \|\| break
		188	fw_tries=$((fw_tries + 1))
		189	[ "$fw_tries" -lt 20 ] \|\| break
		190	done
		191	if timeout 60 $MUX_WAN_SSH "sudo -n iptables -S INPUT 2>/dev/null \| grep -q 'mux-$TAG'" 2>/dev/null; then
		192	FW_LEFT=1
		193	echo " WARNING: a udp DROP rule tagged mux-$TAG is STILL on the box:"
		194	echo " \$MUX_WAN_SSH 'sudo iptables -D INPUT -p udp --dport $QPORT -m comment --comment mux-$TAG -j DROP'"
		195	elif [ "$fw_tries" -gt 0 ]; then
		196	echo " firewall rules removed ($fw_tries) — verified against the box"
		197	fi
		198	timeout 60 $MUX_WAN_SSH "rm -f $RBIN $RSOCK $RLOG $RKEY" \
171	>/dev/null 2>&1 \|\| echo " WARNING: remote file cleanup failed; check for $TAG"	199	>/dev/null 2>&1 \|\| echo " WARNING: remote file cleanup failed; check for $TAG"
172	local left	200	local left
173	left="$(timeout 60 $MUX_WAN_SSH "pgrep -af '[m]uxd-$TAG' \|\| true" 2>/dev/null \|\| true)"	201	left="$(timeout 60 $MUX_WAN_SSH "pgrep -af '[m]uxd-$TAG' \|\| true" 2>/dev/null \|\| true)"
@@ -178,8 +206,10 @@ cleanup() {
178	fi	206	fi
179	rm -rf "$WORK"	207	rm -rf "$WORK"
180	# A box left degraded is a failure of this script whatever the	208	# A box left degraded is a failure of this script whatever the
181	# measurements said.	209	# measurements said. A firewall rule counts as degraded: it silently
		210	# blackholes a port for everyone.
182	[ "$NETEM_LEFT" -eq 0 ] \|\| rc=3	211	[ "$NETEM_LEFT" -eq 0 ] \|\| rc=3
		212	[ "$FW_LEFT" -eq 0 ] \|\| rc=3
183	exit $rc	213	exit $rc
184	}	214	}
185	trap cleanup EXIT INT TERM	215	trap cleanup EXIT INT TERM
@@ -195,6 +225,11 @@ measure() {
195	echo "$out"	225	echo "$out"
196	echo "wan FAIL: measurement '$*' failed" >&2	226	echo "wan FAIL: measurement '$*' failed" >&2
197	[ -s "$ERRLOG" ] && { echo "-- client stderr:"; tail -30 "$ERRLOG"; }	227	[ -s "$ERRLOG" ] && { echo "-- client stderr:"; tail -30 "$ERRLOG"; }
		228	# The daemon's own view, which is where a handshake failure that
		229	# looks identical from outside (full session, wrong key, no
		230	# listener) actually distinguishes itself.
		231	echo "-- remote stats:"; timeout 30 $MUX_WAN_SSH "$RBIN stats --sock $RSOCK" 2>&1 \| tail -2 \|\| true
		232	echo "-- remote daemon log:"; timeout 30 $MUX_WAN_SSH "tail -20 $RLOG" 2>&1 \|\| true
198	exit 1	233	exit 1
199	fi	234	fi
200	echo "$out" \| grep -v '^#RESULT' \|\| true	235	echo "$out" \| grep -v '^#RESULT' \|\| true
@@ -271,8 +306,12 @@ class Client:
271	# the only path being measured.	306	# the only path being measured.
272	env["XDG_RUNTIME_DIR"] = "/nonexistent-mux-wan"	307	env["XDG_RUNTIME_DIR"] = "/nonexistent-mux-wan"
273	self.errf = open(errlog, "ab")	308	self.errf = open(errlog, "ab")
		309	# `via` is either a --via command string or, for QUIC, a complete
		310	# argv tail beginning with quic:// — split on spaces, which is safe
		311	# because the only things in it are a URL, a key path and a number.
		312	args = [mux] + via.split() if via.startswith("quic://") else [mux, "--via", via]
274	self.p = subprocess.Popen(	313	self.p = subprocess.Popen(
275	[mux, "--via", via],	314	args,
276	stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=self.errf,	315	stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=self.errf,
277	bufsize=0, start_new_session=True, env=env)	316	bufsize=0, start_new_session=True, env=env)
278	self.buf = bytearray()	317	self.buf = bytearray()
@@ -643,6 +682,130 @@ def cmd_reconnect(argv):
643	first_survived=survived)	682	first_survived=survived)
644		683
645		684
		685	def remote_clients(ssh_cmd, rbin, rsock):
		686	"""How many client slots the daemon is holding right now.
		687
		688	A gauge rather than a counter, and the instrument for the banked
		689	half-open question: a QUIC connection that completes its handshake and
		690	never attaches occupies a slot until its idle timeout, which was
		691	previously unobservable from outside the process.
		692	"""
		693	out = subprocess.run(shlex.split(ssh_cmd) + ["%s stats --sock %s" % (rbin, rsock)],
		694	capture_output=True, text=True, timeout=60).stdout
		695	m = re.search(r"\bclients=(\d+)", out)
		696	return int(m.group(1)) if m else -1
		697
		698
		699	def cmd_quicreconnect(argv):
		700	"""The QUIC tear: a real UDP blackhole on the box, not a killed process.
		701
		702	ACCOUNTING, because the two transports do not detect a tear the same way
		703	and comparing them without saying so would be dishonest. ssh notices
		704	instantly — the process dies and the pipe reports EOF — so its whole
		705	measured interval is recovery. QUIC has nothing to notice: packets simply
		706	stop, and the connection is declared dead by a TIMER whose length is
		707	--quic-idle-ms, a knob we set. Including that timer would measure our own
		708	configuration and call it a transport property.
		709
		710	So the sample clock starts when the drop is REMOVED, i.e. when recovery
		711	first becomes possible, and the detection time is reported separately as
		712	the idle setting it is. Both numbers are printed; neither is hidden.
		713	"""
		714	mux, via, reps, errlog = argv[0], argv[1], int(argv[2]), argv[3]
		715	ssh_cmd, rbin, rsock = argv[4], argv[5], argv[6]
		716	qport, tag, idle_ms = argv[7], argv[8], int(argv[9])
		717
		718	def firewall(action):
		719	# Tagged with the run's own comment so removal can be exact and
		720	# verified, and so a crashed run leaves something findable rather
		721	# than an anonymous DROP on someone else's box.
		722	rule = ("INPUT -p udp --dport %s -m comment --comment mux-%s -j DROP"
		723	% (qport, tag))
		724	subprocess.run(shlex.split(ssh_cmd) +
		725	["sudo -n iptables -%s %s" % (action, rule)],
		726	capture_output=True, text=True, timeout=60)
		727
		728	c = Client(mux, via, errlog)
		729	samples = []
		730	detect_ms = []
		731	resumed = 0
		732	snaps_before = snaps_after = -1
		733	clients_peak = 0
		734	first_marker = None
		735	survived = 0
		736	try:
		737	settle(c, "quicreconnect")
		738	c.send(b"\x15clear\n")
		739	wait_idle(c, "quicreconnect")
		740	snaps_before = remote_snapshots(ssh_cmd, rbin, rsock)
		741	for i in range(reps):
		742	before = c.total
		743	firewall("I")
		744	t_drop = now()
		745	# Held past the idle timeout with margin, so the client cannot
		746	# mistake the blackhole for a slow moment.
		747	time.sleep((idle_ms / 1000.0) * 1.6)
		748	seen = remote_clients(ssh_cmd, rbin, rsock)
		749	if seen > clients_peak:
		750	clients_peak = seen
		751	firewall("D")
		752	t0 = now()
		753	detect_ms.append((t0 - t_drop) * 1000.0)
		754
		755	if c.wait_for(lambda b, c=c, t=before: c.total > t) is None:
		756	fail("quicreconnect: rep %d never resumed after the blackhole lifted" % i)
		757	samples.append((now() - t0) * 1000.0)
		758	resumed += 1
		759
		760	nonce = "%d-%d" % (os.getpid(), i)
		761	marker = ("wanq-" + nonce).encode()
		762	if first_marker is None:
		763	first_marker = marker
		764	c.send(b'\x15printf "wanq-%s\\n" ' + nonce.encode() + b"\n")
		765	if c.wait_for(lambda b, m=marker: m in b) is None:
		766	fail("quicreconnect: rep %d resumed but the input path is dead" % i)
		767	print(" quic tear rep %-2d recovery %7.1fms session usable"
		768	% (i, samples[-1]))
		769	wait_idle(c, "quicreconnect")
		770	snaps_after = remote_snapshots(ssh_cmd, rbin, rsock)
		771	dump = remote_dump(ssh_cmd, rbin, rsock)
		772	survived = 1 if first_marker.decode() in dump else 0
		773	if not survived:
		774	print(" quicreconnect: FIRST TEAR'S MARKER %s IS GONE from the grid"
		775	% first_marker.decode())
		776	finally:
		777	firewall("D") # belt and braces; the shell trap verifies
		778	c.detach()
		779	report("quicreconnect", samples, resumed=resumed, reps=reps,
		780	snaps_before=snaps_before, snaps_after=snaps_after,
		781	delta_served=1 if snaps_after == snaps_before else 0,
		782	first_survived=survived, clients_peak=clients_peak,
		783	detect_ms=int(sum(detect_ms) / max(1, len(detect_ms))))
		784
		785
		786	def cmd_quicabort(argv):
		787	"""Ctrl-\ against a port that will never answer.
		788
		789	A blackholed port, not a closed one: a closed port answers ICMP and the
		790	client gives up on its own, which would measure the kernel rather than
		791	the abort key. This is the only place the full handshake bound actually
		792	runs, so it is the only place the property can be observed.
		793	"""
		794	mux, spec, errlog, bound_ms = argv[0], argv[1], argv[2], int(argv[3])
		795	c = Client(mux, spec, errlog)
		796	t0 = now()
		797	time.sleep(0.3)
		798	c.send(b"\x1c")
		799	try:
		800	c.p.wait(timeout=bound_ms / 1000.0 * 0.8)
		801	except Exception:
		802	c.kill()
		803	fail("quicabort: Ctrl-\\ went unanswered for most of the %dms bound" % bound_ms)
		804	elapsed = (now() - t0) * 1000.0
		805	c._close()
		806	report("quicabort", [elapsed], bound_ms=bound_ms, rc=c.p.returncode)
		807
		808
646	def cmd_hol(argv):	809	def cmd_hol(argv):
647	"""Echo latency while the session floods output. The proxy is a byte	810	"""Echo latency while the session floods output. The proxy is a byte
648	pump with no notion of priority, so a flood shares the pipe with the	811	pump with no notion of priority, so a flood shares the pipe with the
@@ -705,6 +868,8 @@ COMMANDS = {
705	"echo": cmd_echo,	868	"echo": cmd_echo,
706	"reattach": cmd_reattach,	869	"reattach": cmd_reattach,
707	"reconnect": cmd_reconnect,	870	"reconnect": cmd_reconnect,
		871	"quicreconnect": cmd_quicreconnect,
		872	"quicabort": cmd_quicabort,
708	"hol": cmd_hol,	873	"hol": cmd_hol,
709	}	874	}
710		875
@@ -730,9 +895,24 @@ say "baseline: raw byte round-trip through \`ssh cat\` ($REPS_BASE reps)"
730	measure clean baseline "$MUX_WAN_SSH" "$REPS_BASE"	895	measure clean baseline "$MUX_WAN_SSH" "$REPS_BASE"
731	measure clean viafloor "$MUX_WAN_SSH" 5	896	measure clean viafloor "$MUX_WAN_SSH" 5
732		897
		898	QUIC_ARGS=""
		899	LKEY=""
		900	if [ "${MUX_WAN_QUIC:-}" = "1" ]; then
		901	say "deploying a QUIC key (owner-only, both ends)"
		902	LKEY="$WORK/mux-$TAG.key"
		903	head -c 32 /dev/urandom > "$LKEY"
		904	chmod 600 "$LKEY"
		905	$MUX_WAN_SCP "$LKEY" "$MUX_WAN_HOST:$RKEY"
		906	# muxd refuses a group- or world-readable key, exactly as ssh does, so
		907	# this is required rather than tidy.
		908	$MUX_WAN_SSH "chmod 600 $RKEY"
		909	QUIC_ARGS="--quic 0.0.0.0:$QPORT --key $RKEY --quic-idle-ms 15000"
		910	echo " udp $QPORT, key $RKEY"
		911	fi
		912
733	say "starting the remote daemon"	913	say "starting the remote daemon"
734	$MUX_WAN_SSH "rm -f $RSOCK; nohup $RBIN run --sock $RSOCK --shell /bin/bash \	914	$MUX_WAN_SSH "rm -f $RSOCK; nohup $RBIN run --sock $RSOCK --shell /bin/bash \
735	>$RLOG 2>&1 </dev/null & sleep 0.2"	915	$QUIC_ARGS >$RLOG 2>&1 </dev/null & sleep 0.2"
736	i=0	916	i=0
737	until $MUX_WAN_SSH "test -S $RSOCK" 2>/dev/null; do	917	until $MUX_WAN_SSH "test -S $RSOCK" 2>/dev/null; do
738	i=$((i + 1))	918	i=$((i + 1))
@@ -741,7 +921,8 @@ until $MUX_WAN_SSH "test -S $RSOCK" 2>/dev/null; do
741	sleep 0.5	921	sleep 0.5
742	done	922	done
743	sleep 1.5 # let bash reach a prompt on its pty	923	sleep 1.5 # let bash reach a prompt on its pty
744	echo " daemon up on $RSOCK"	924	RPID="$($MUX_WAN_SSH "pgrep -f '[m]uxd-$TAG' \| head -1" \| tr -cd '0-9')"
		925	echo " daemon up on $RSOCK (remote pid ${RPID:-unknown})"
745		926
746	say "attach latency (launch -> first painted byte, $REPS_ATTACH reps)"	927	say "attach latency (launch -> first painted byte, $REPS_ATTACH reps)"
747	measure clean attach "$MUX" "$VIA" "$REPS_ATTACH" "$ERRLOG"	928	measure clean attach "$MUX" "$VIA" "$REPS_ATTACH" "$ERRLOG"
@@ -760,13 +941,67 @@ say "M7 reconnect: tear the transport under a live session ($REPS_RECONNECT tear
760	measure clean reconnect "$MUX" "$VIA" "$REPS_RECONNECT" "$ERRLOG" \	941	measure clean reconnect "$MUX" "$VIA" "$REPS_RECONNECT" "$ERRLOG" \
761	"$MUX_WAN_SSH" "$RBIN" "$RSOCK"	942	"$MUX_WAN_SSH" "$RBIN" "$RSOCK"
762		943
		944	if [ "${MUX_WAN_QUIC:-}" = "1" ]; then
		945	# One daemon, two transports, one link, one run: the ssh-via numbers
		946	# above and the QUIC numbers below are the same session over the same
		947	# wire, which is what makes them comparable at all.
		948	QHOST="${MUX_WAN_QUIC_HOST:-$(echo "$MUX_WAN_HOST" \| sed 's/.*@//')}"
		949	QSPEC="quic://$QHOST:$QPORT --key $LKEY --quic-idle-ms $QIDLE"
		950
		951	# OUR daemon owns that port, not merely somebody's socket. Matching the
		952	# port alone in /proc/net/udp would pass just as happily on a stranger's
		953	# listener, and then every QUIC number below would be measuring a
		954	# handshake against something else entirely.
		955	say "QUIC: checking the daemon owns udp $QPORT"
		956	# Matched on PID, not on name: ss truncates a process name to 15
		957	# characters, and a tag this long is exactly the kind of thing that
		958	# silently never matches.
		959	QOWNER="$($MUX_WAN_SSH "ss -ulnp 2>/dev/null \| grep ':$QPORT ' \|\| true")"
		960	echo " $QOWNER"
		961	case "$QOWNER" in
		962	"pid=$RPID,") : ;;
		963	*) echo "wan FAIL: udp $QPORT is not held by our daemon (pid $RPID)" >&2
		964	$MUX_WAN_SSH "cat $RLOG" \|\| true; exit 1 ;;
		965	esac
		966	echo " remote stats before QUIC: $($MUX_WAN_SSH "$RBIN stats --sock $RSOCK")"
		967
		968	say "QUIC attach latency (launch -> first painted byte, $REPS_ATTACH reps)"
		969	measure quic attach "$MUX" "$QSPEC" "$REPS_ATTACH" "$ERRLOG"
		970
		971	say "QUIC keystroke echo through the full stack ($REPS_ECHO reps)"
		972	measure quic echo "$MUX" "$QSPEC" "$REPS_ECHO" "$ERRLOG"
		973
		974	say "QUIC tears: a real UDP blackhole on the box ($REPS_RECONNECT tears)"
		975	measure quic quicreconnect "$MUX" "$QSPEC" "$REPS_RECONNECT" "$ERRLOG" \
		976	"$MUX_WAN_SSH" "$RBIN" "$RSOCK" "$QPORT" "$TAG" "$QIDLE"
		977
		978	# The abort key against a port that will never answer. A DROP rather
		979	# than an unused port: an unused one answers ICMP and the client gives
		980	# up on its own, which would measure the kernel instead of the property.
		981	say "QUIC abort: Ctrl-\\ against a blackholed port"
		982	QDEAD=$(( QPORT + 1 ))
		983	$MUX_WAN_SSH "sudo -n iptables -I INPUT -p udp --dport $QDEAD \
		984	-m comment --comment mux-$TAG -j DROP"
		985	measure quic quicabort "$MUX" \
		986	"quic://$QHOST:$QDEAD --key $LKEY --quic-idle-ms 15000" "$ERRLOG" 15000 \|\| true
		987	$MUX_WAN_SSH "sudo -n iptables -D INPUT -p udp --dport $QDEAD \
		988	-m comment --comment mux-$TAG -j DROP" >/dev/null 2>&1 \|\| true
		989	fi
		990
763	# Every clean-link figure is taken before the box is degraded, so a netem	991	# Every clean-link figure is taken before the box is degraded, so a netem
764	# that failed to clear can never quietly relabel itself as the clean link.	992	# that failed to clear can never quietly relabel itself as the clean link.
765	say "head-of-line probe: echo while the session floods output ($REPS_HOL reps)"	993	say "head-of-line probe: echo while the session floods output ($REPS_HOL reps)"
766	measure clean hol "$MUX" "$VIA" "$REPS_HOL" "$ERRLOG"	994	measure clean hol "$MUX" "$VIA" "$REPS_HOL" "$ERRLOG"
767		995
768	if [ "${MUX_WAN_NETEM:-}" = "1" ]; then	996	if [ "${MUX_WAN_NETEM:-}" = "1" ]; then
769	IFACE="$($MUX_WAN_SSH "ip route show default \| cut -d' ' -f5" \| tr -d '\r\n')"	997	# The interface facing US, not merely a default route. A dual-homed box
		998	# has more than one default and `ip route show default` prints them all
		999	# — shaping the wrong one produces a "netem" column that is really the
		1000	# clean link, which is worse than no measurement. Asking the box which
		1001	# device it would use to reach this client answers it exactly.
		1002	CLIENT_IP="$($MUX_WAN_SSH 'echo $SSH_CLIENT' \| awk '{print $1}' \| tr -d '\r\n')"
		1003	[ -n "$CLIENT_IP" ] \|\| { echo "wan FAIL: cannot tell which address the box sees us on" >&2; exit 1; }
		1004	IFACE="$($MUX_WAN_SSH "ip route get $CLIENT_IP" \| sed -n 's/.* dev $[^ ]$./\1/p' \| head -1 \| tr -d '\r\n')"
770	[ -n "$IFACE" ] \|\| { echo "wan FAIL: no default interface on the box" >&2; exit 1; }	1005	[ -n "$IFACE" ] \|\| { echo "wan FAIL: no default interface on the box" >&2; exit 1; }
771	# A root netem qdisc shapes egress only, so this adds ~75ms to the round	1006	# A root netem qdisc shapes egress only, so this adds ~75ms to the round
772	# trip, not ~150ms — measured, not assumed.	1007	# trip, not ~150ms — measured, not assumed.
@@ -789,6 +1024,18 @@ if [ "${MUX_WAN_NETEM:-}" = "1" ]; then
789		1024
790	measure netem hol "$MUX" "$VIA" "$REPS_HOL" "$ERRLOG"	1025	measure netem hol "$MUX" "$VIA" "$REPS_HOL" "$ERRLOG"
791		1026
		1027	if [ "${MUX_WAN_QUIC:-}" = "1" ]; then
		1028	# Connection setup is the leg where the transports actually differ,
		1029	# and it is RTT that separates them: ssh pays a channel open, QUIC
		1030	# pays a handshake plus the Retry this listener always sends. At LAN
		1031	# RTT both round to nothing, so the comparison only means something
		1032	# once there is a real delay in the path.
		1033	say "QUIC attach under netem ($REPS_ATTACH reps)"
		1034	measure quicnetem attach "$MUX" "$QSPEC" "$REPS_ATTACH" "$ERRLOG"
		1035	say "QUIC echo under netem ($REPS_ECHO reps)"
		1036	measure quicnetem echo "$MUX" "$QSPEC" "$REPS_ECHO" "$ERRLOG"
		1037	fi
		1038
792	netem_off	1039	netem_off
793	fi	1040	fi
794		1041
@@ -908,9 +1155,55 @@ echo "date: $(date -u +%Y-%m-%dT%H:%M:%SZ) host: (ephemeral, from MUX_WAN_SSH)
908	phase_block clean "link as-is:"	1155	phase_block clean "link as-is:"
909	phase_block netem "with netem delay 75ms loss 1%:"	1156	phase_block netem "with netem delay 75ms loss 1%:"
910	m7_block	1157	m7_block
		1158
		1159	# ---- M8: the QUIC legs -----------------------------------------------------
		1160	if [ "${MUX_WAN_QUIC:-}" = "1" ]; then
		1161	q_attach="$(val quic attach med)"
		1162	q_echo="$(val quic echo med)"
		1163	q_tear="$(val quic quicreconnect med)"
		1164	q_detect="$(val quic quicreconnect detect_ms)"
		1165	q_snapb="$(val quic quicreconnect snaps_before)"
		1166	q_snapa="$(val quic quicreconnect snaps_after)"
		1167	q_resumed="$(val quic quicreconnect resumed)"
		1168	q_reps="$(val quic quicreconnect reps)"
		1169	q_survived="$(val quic quicreconnect first_survived)"
		1170	q_clients="$(val quic quicreconnect clients_peak)"
		1171	q_abort="$(val quic quicabort med)"
		1172	s_attach="$(val clean attach med)"
		1173	s_tear="$(val clean reconnect med)"
		1174	echo
		1175	echo "M8 QUIC (same daemon, same link, same run):"
		1176	printf ' %-40s %8s %8s\n' "" "quic" "ssh-via"
		1177	printf ' %-40s %8s %8s\n' "cold attach -> first paint (med ms)" "$q_attach" "$s_attach"
		1178	printf ' %-40s %8s %8s\n' "echo (med ms)" "$q_echo" "$(val clean echo med)"
		1179	printf ' %-40s %8s %8s\n' "tear -> usable (med ms)" "$q_tear" "$s_tear"
		1180	echo " tear accounting: the two transports do NOT detect a tear alike."
		1181	echo " ssh dies with its process, so EOF is instant and its whole"
		1182	echo " interval is recovery. QUIC has nothing to notice: it is declared"
		1183	echo " dead by a timer set to --quic-idle-ms (${QIDLE}ms here, ${q_detect}ms"
		1184	echo " of blackhole measured before the clock starts). The quic column"
		1185	echo " is measured from the blackhole LIFTING, so it excludes that"
		1186	echo " detection and includes whatever backoff the client had already"
		1187	echo " entered while it waited."
		1188	echo " leg 1 (semantics): resumed $q_resumed/$q_reps, snapshots $q_snapb -> $q_snapa, first marker kept: $q_survived"
		1189	echo " leg 3 (cold attach): quic $q_attach vs ssh-via $s_attach"
		1190	if [ -n "$(val quicnetem attach med)" ]; then
		1191	echo " leg 3 under netem (the RTT that makes setup mean something):"
		1192	echo " quic $(val quicnetem attach med) vs ssh-via $(val netem attach med) (echo: quic $(val quicnetem echo med) vs ssh-via $(val netem echo med))"
		1193	fi
		1194	echo " NOTE: the quic figure INCLUDES the Retry round trip — this"
		1195	echo " listener answers every fresh Initial with one, so address"
		1196	echo " validation is inside the number, not netted out of it. ssh pays"
		1197	echo " no such round trip. The comparison is therefore pessimistic for"
		1198	echo " QUIC by roughly one RTT."
		1199	echo " abort during handshake: ${q_abort}ms against a 15000ms bound"
		1200	echo " client slots held at peak during tears: $q_clients"
		1201	fi
911	echo	1202	echo
912	if [ "$FAILED" -eq 0 ]; then	1203	if [ "$FAILED" -eq 0 ]; then
913	echo "kill criteria (M6 + M7): PASS"	1204
		1205
		1206	echo "kill criteria (M6 + M7): PASS"
914	else	1207	else
915	echo "kill criteria (M6 + M7): FAIL — report the numbers; do not tune the thresholds."	1208	echo "kill criteria (M6 + M7): FAIL — report the numbers; do not tune the thresholds."
916	fi	1209	fi

Old		New
@@ -24,11 +24,11 @@
24		24
25	M7 measured resume ≈ 255ms of which ~200ms is our own first backoff. A transport that died a millisecond ago is overwhelmingly likely to accept a new connection now (daemon restarts, link blips, proxy kills all reconnect instantly); a genuinely down link costs one wasted attempt. Standard shape: first attempt immediate, THEN 200ms→2s exponential backoff.	25	M7 measured resume ≈ 255ms of which ~200ms is our own first backoff. A transport that died a millisecond ago is overwhelmingly likely to accept a new connection now (daemon restarts, link blips, proxy kills all reconnect instantly); a genuinely down link costs one wasted attempt. Standard shape: first attempt immediate, THEN 200ms→2s exponential backoff.
26		26
27	- [ ] Step 1: Restructure `reconnect()` so the first `Transport.open` attempt happens before any sleep. The Ctrl-\ drain must still run during every wait (no abort regression), and the reviewer's single-ownership-per-iteration structure from f314b1b must survive — the immediate attempt is iteration zero with a zero-length wait, not a special-cased second code path. Comment records the M7 measurement that justified it and the flapping-link reasoning for why backoff still follows.	27	- [x] Step 1: Restructure `reconnect()` so the first `Transport.open` attempt happens before any sleep. The Ctrl-\ drain must still run during every wait (no abort regression), and the reviewer's single-ownership-per-iteration structure from f314b1b must survive — the immediate attempt is iteration zero with a zero-length wait, not a special-cased second code path. Comment records the M7 measurement that justified it and the flapping-link reasoning for why backoff still follows.
28	- [ ] Step 2: Keep the unit tests honest — drainStdinForQuit tests are timing-parameterized and should not change; if any test asserted the pre-sleep, update it to assert the new shape (first attempt immediate) rather than deleting it.	28	- [x] Step 2: Keep the unit tests honest — drainStdinForQuit tests are timing-parameterized and should not change; if any test asserted the pre-sleep, update it to assert the new shape (first attempt immediate) rather than deleting it.
29	- [ ] Step 3: Gates (`make test`, `make e2e` — reconnect scenarios in e2e get faster, must stay green 3x — `make bench`).	29	- [x] Step 3: Gates (`make test`, `make e2e` — reconnect scenarios in e2e get faster, must stay green 3x — `make bench`).
30	- [ ] Step 4: Run the wan.sh reconnect phase once against the box: expected med drops from ~255ms to ≈55ms (36ms channel + ~18ms protocol). Gate nothing new; record the number for the Task 5 record.	30	- [x] Step 4: Run the wan.sh reconnect phase once against the box: expected med drops from ~255ms to ≈55ms (36ms channel + ~18ms protocol). Gate nothing new; record the number for the Task 5 record.
31	- [ ] Step 5: Commit `perf: first reconnect attempt is immediate — the 200ms was our own`	31	- [x] Step 5: Commit `perf: first reconnect attempt is immediate — the 200ms was our own`
32		32
33	---	33	---
34		34
@@ -40,11 +40,11 @@ M7 measured resume ≈ 255ms of which ~200ms is our own first backoff. A transpo
40		40
41	The whole milestone stands on one question: can a C QUIC stack be vendored and built by `zig build` (addCSourceFiles preferred; a pinned-script-built static lib acceptable as fallback) for BOTH native and `x86_64-linux-musl` static, under Zig 0.15.2, with TLS 1.3 external-PSK and an event-loop-free (fd-drivable) API?	41	The whole milestone stands on one question: can a C QUIC stack be vendored and built by `zig build` (addCSourceFiles preferred; a pinned-script-built static lib acceptable as fallback) for BOTH native and `x86_64-linux-musl` static, under Zig 0.15.2, with TLS 1.3 external-PSK and an event-loop-free (fd-drivable) API?
42		42
43	- [ ] Step 1: Candidate (a) — ngtcp2 + wolfSSL. Vendor pinned source snapshots. Get a minimal QUIC echo pair (server + client binaries) building native, then musl-static. PSK mode, single bidi stream, blocking-ish fd loop is fine for the spike.	43	- [x] Step 1: Candidate (a) — ngtcp2 + wolfSSL. Vendor pinned source snapshots. Get a minimal QUIC echo pair (server + client binaries) building native, then musl-static. PSK mode, single bidi stream, blocking-ish fd loop is fine for the spike.
44	- [ ] Step 2: If (a) fails on toolchain grounds, candidate (b) — picoquic + picotls. Same bar. Record precisely WHY (a) failed (the failure reasons are the spike's product as much as the success).	44	- [x] Step 2: If (a) fails on toolchain grounds, candidate (b) — picoquic + picotls. Same bar. Record precisely WHY (a) failed (the failure reasons are the spike's product as much as the success).
45	- [ ] Step 3: Prove the winner on the box. scp the static echo pair; measure over the real WAN: (i) cold handshake to first echoed byte, (ii) resumed handshake (session ticket / 0-RTT if the stack exposes it) to first echoed byte. Compare against the same run's ssh viafloor. Box hygiene rules as always: TAG-unique paths and ports, UDP port freed afterward, nothing of the user's touched.	45	- [x] Step 3: Prove the winner on the box. scp the static echo pair; measure over the real WAN: (i) cold handshake to first echoed byte, (ii) resumed handshake (session ticket / 0-RTT if the stack exposes it) to first echoed byte. Compare against the same run's ssh viafloor. Box hygiene rules as always: TAG-unique paths and ports, UDP port freed afterward, nothing of the user's touched.
46	- [ ] Step 4: Integration assessment, written down: how the stack wants fds/timers driven, what the daemon poll-loop integration looks like (one UDP fd + a timer fd?), PSK API shape, stream-data API shape, static binary size delta.	46	- [x] Step 4: Integration assessment, written down: how the stack wants fds/timers driven, what the daemon poll-loop integration looks like (one UDP fd + a timer fd?), PSK API shape, stream-data API shape, static binary size delta.
47	- [ ] Step 5: Commit `spike: QUIC stack chosen — <name> builds static-musl under the pinned toolchain` (spike directory + findings; main build untouched, gates green by construction)	47	- [x] Step 5: Commit `spike: QUIC stack chosen — <name> builds static-musl under the pinned toolchain` (spike directory + findings; main build untouched, gates green by construction)
48		48
49	Spike kill criterion: a static-musl QUIC echo pair working over the WAN box, from sources vendored and built reproducibly by our pinned toolchain, within TWO candidate stacks' worth of honest effort. Neither builds → report BLOCKED with the failure evidence; the controller re-scopes the milestone. Do not reach for a third candidate or a prebuilt blob without a controller decision.	49	Spike kill criterion: a static-musl QUIC echo pair working over the WAN box, from sources vendored and built reproducibly by our pinned toolchain, within TWO candidate stacks' worth of honest effort. Neither builds → report BLOCKED with the failure evidence; the controller re-scopes the milestone. Do not reach for a third candidate or a prebuilt blob without a controller decision.
50		50