a73x

531494b3

feat(m7): reconnect held over the WAN — 10 tears, 10 delta resumes; the record

a73x   2026-08-08 14:08

Commit message
feat(m7): reconnect held over the WAN — 10 tears, 10 delta resumes; the record
README.md
Old New
@@ -5,7 +5,16 @@ authoritatively in a daemon and replicated in the client — state sync
5 instead of escape-sequence replay. See `docs/handoff.md` for the design 5 instead of escape-sequence replay. See `docs/handoff.md` for the design
6 and `docs/decisions.md` for decisions made. 6 and `docs/decisions.md` for decisions made.
7 7
8 Status: **M6 — transport.** All handoff milestones complete; all kill 8 Status: **M7 — reconnect.** A dropped link is a non-event: the client
9 keeps its replica, rebuilds the transport, and re-attaches quoting what it
10 already holds, so the daemon can answer with a delta instead of a repaint.
11 Measured over a real WAN link: 10 consecutive transport kills against a
12 live session, all 10 resumed hands-off, all 10 delta-served (the daemon's
13 snapshot counter never moved), first frame back in ~254 ms — most of which
14 is the client's own 200 ms first retry. `mux HOST` attaches over ssh in
15 one word.
16
17 Previously, **M6 — transport.** All handoff milestones complete; all kill
9 criteria cleared. The prototype's two founding questions are both 18 criteria cleared. The prototype's two founding questions are both
10 answered yes: ghostty-vt serves as an authoritative headless grid in a 19 answered yes: ghostty-vt serves as an authoritative headless grid in a
11 daemon without forking it, and detach/reattach as state sync is correct 20 daemon without forking it, and detach/reattach as state sync is correct
@@ -30,8 +39,18 @@ toolchain, override with `make ZIG=...`.
30 39
31 Attach over a network: 40 Attach over a network:
32 41
42 ./zig-out/bin/mux HOST # sugar for the --via line below
33 ./zig-out/bin/mux --via "ssh host /path/muxd proxy --sock /path/muxd.sock" 43 ./zig-out/bin/mux --via "ssh host /path/muxd proxy --sock /path/muxd.sock"
34 44
45 `mux HOST` runs `ssh HOST muxd proxy`, so muxd must be on HOST's PATH;
46 anything more specific (a custom socket, a non-default path) is what
47 `--via` is for. `mux` itself does not parse HOST, so ssh's own config —
48 host aliases, ports, ProxyJump — keeps working. It is interpolated into a
49 shell command, though, exactly as the `--via` string is: **HOST must be
50 trusted**. Feeding it unreviewed from an inventory file or a cloud API is
51 handing that source a shell. If the link drops, the client reconnects on
52 its own and resumes the session; Ctrl-\ gives up waiting.
53
35 `--via` runs the protocol over any command that exposes the daemon's 54 `--via` runs the protocol over any command that exposes the daemon's
36 socket on its stdio — `muxd proxy` is one such command, and it is a 55 socket on its stdio — `muxd proxy` is one such command, and it is a
37 frame-agnostic byte pump that contains no protocol knowledge at all. 56 frame-agnostic byte pump that contains no protocol knowledge at all.
docs/decisions.md
Old New
@@ -479,6 +479,131 @@ path, and it is transport work, which is the verdict restated.
479 cost measured here is what a predictor must beat to be worth its 479 cost measured here is what a predictor must beat to be worth its
480 complexity on a link this good. 480 complexity on a link this good.
481 481
482 ## 2026-08-07 (M7)
483
484 - **Kill criterion: PASSED on the real WAN box.** 10 consecutive transport
485 kills against a live session over ssh (~16.5ms round trip, via a jump
486 host): 10 of 10 resumed with zero manual action, and the remote daemon's
487 `snapshots` counter did not move across any of them (11 → 11) — every
488 resume was delta-served. First frame after a tear: min 254.0ms, med
489 254.5ms, max 255.3ms (n=10). The M6 criteria still pass in the same run
490 (echo med 20.9ms against a 136.5ms budget; reattach protocol share
491 22.0ms against 33.0ms). Re-confirmed on a second run after the
492 convergence clause below was narrowed and partly closed: 10/10 again,
493 snapshots 5 → 5, first frame med 256.4ms, state-survival check passed.
494 - **The criterion was NARROWED, and this is the record of it.** The plan
495 named four elements; the fourth — "replica converged after the last tear
496 (client render matches `muxd dump`)" — is **not** what the harness
497 checks, and the first version of this record restated the criterion
498 without saying so. What was substituted: each tear round-trips a marker
499 out to the remote shell and back through the rebuilt transport, and
500 (added when this was caught) after the last tear the **first** tear's
501 marker must still be in the daemon's grid — accumulated state survived
502 all ten resumes. Why: the client emits a stream of paints, not a grid, so
503 comparing its render to a dump means running a second VT parser inside
504 the measurement harness, whose own bugs would be indistinguishable from
505 the ones it exists to catch. That comparison IS made against a real
506 replica Engine in src/server.zig's tests — over a socket, not over the
507 WAN. The substitute is weaker than the plan's wording. **Full
508 convergence-over-the-WAN stays banked**, and the honest one-line summary
509 of M7 is "done, with the convergence clause narrowed on the record".
510 - **That 254ms is nearly all our own first backoff.** It decomposes as
511 ~200ms of client backoff + 36.4ms of ssh channel open (`viafloor`,
512 measured in the same run) + ~18ms of protocol. The wall clock is printed
513 and not gated for the same reason the M6 reattach ruling gives — gating
514 it would rule on ssh, not on this design — but the split is worth
515 keeping in view: **a shorter first retry would cut resume time roughly
516 fourfold on this link**, and nothing measured here argues against trying
517 it. That multiple is link-specific and must not be quoted as universal —
518 it is the ratio of a fixed 200ms backoff to a fast link's costs, so on
519 the +75ms netem profile the same change is worth roughly 2x, and on a
520 slow enough link it rounds to nothing. Banked rather than tuned, because
521 200ms was chosen for a link that might be flapping and this run cannot
522 tell us how it behaves on one that is.
523 - **Transport death is a non-event; transport *absence* is not.** The
524 client keeps its replica, rebuilds the transport from the recipe it was
525 launched with, and re-attaches quoting `(have_seq, have_epoch)`; the
526 daemon's existing `sendResync` decides delta-or-snapshot. Retry is
527 uniform and generous with no cap, split on one thing only: **whether a
528 session was ever established** (`session_epoch != 0`). A transport that
529 dies before the first snapshot carried no session — a typo'd hostname, a
530 remote muxd that is not there — and exits immediately with the old
531 message. Retry policy is deliberately NOT split on transport type: a
532 dropped ssh link and a killed local daemon are the same event to a
533 client, and the abort key plus the indicator answer both.
534 - **Policy decisions, recorded because they are choices and not
535 consequences:** input typed while disconnected is **dropped**, not
536 queued (replaying a burst of stale keystrokes into a shell is worse than
537 losing them; prediction is the principled fix, and it is a later
538 milestone). A reconnect landing on a **different epoch repaints silently**
539 — new epoch means new session, and a fresh shell is self-evident without
540 a modal "session lost" state. An attach **refusal during a reconnect** is
541 retried for ~5s (the daemon may not have reaped our own dead
542 predecessor's slot yet); a refusal on first attach still exits loudly.
543 Ctrl-\ during a reconnect exits 0 with "detached while reconnecting" —
544 the user chose to leave and the session really is still running.
545 - **`session_epoch` graduated from parsed-but-unused**, closing the M6
546 banked item: it is now the value that lets a reconnect resume a stream
547 instead of re-attaching from zero, and the value that makes "never
548 established" expressible at all.
549 - **`mux HOST` is sugar for `--via "ssh HOST muxd proxy"`** — the VM-hop
550 use case in one word. Nothing parses inside HOST, so ssh's own config
551 (aliases, ports, ProxyJump) keeps working. **It presumes a compatible
552 muxd on the remote PATH and there is no version negotiation anywhere in
553 this protocol: mismatched binaries are undefined behaviour, not a
554 handled error.**
555
556 ### Process rules this milestone paid for
557
558 - **build.zig test discovery is a LIVE hazard.** Tests only run for modules
559 listed in the `test` step's loop. `mux_main.zig` was not in it, so its
560 five new parser tests compiled and silently never ran — the suite
561 reported 79/79 green with them absent, and 84/84 once the module was
562 added. **`src/main.zig` (muxd's entry point) is still not in that list**,
563 so the first test anyone adds there will silently not run. Adding a test
564 file is not enough; check the module is in the loop, and check the test
565 count moved.
566 - **Counters are asserted because markers are blind to *how* a resume was
567 served.** A snapshot-served resume renders identically to a delta-served
568 one, so `snapshots` is the only witness — this is now proven three times
569 by mutation (a client reconnecting with `have_seq=0` passes every marker
570 assertion and fails only the counter). The restart scenario's mirror
571 assertion (`snapshots >= 1` on the *new* daemon) needs **both** the epoch
572 fence and the seq-range check removed before it fires, because
573 `buildDeltaSince` has no out-of-range guard of its own: a stale seq
574 yields a silent zero-row delta that renders fine. It is deliberate
575 defence in depth against those two guards ever being collapsed into one,
576 and it is recorded here so the next reviewer does not repeat the
577 "can't mutate it, is it dead?" cycle.
578 - **`pkill -f` is banned in this suite.** The client's own argv contains
579 the entire `--via` command string, so any pattern matching the proxy also
580 matches the client under test — which killed its own subject once here
581 before the rule existed. Select by pid: `comm == muxd` plus argv, or the
582 parent-pid of the client. The remote-side equivalent is the existing
583 bracketed-pattern trick in `wan.sh`.
584 - **The never-established gate has e2e-only coverage.** The property is
585 process-level (a client that must exit rather than loop, with no tty to
586 rescue it), so it cannot be unit-tested honestly. `make e2e` is therefore
587 a **permanently required gate**, not an optional one: with unit tests
588 alone, a regression to the infinite-retry behaviour ships green.
589 - **Mutation testing here has a stale-binary trap.** `make test` does not
590 rebuild the `mux`/`muxd` binaries, and a failed build leaves the previous
591 ones in place, so a mutation run can report a confident false green
592 against code that was never compiled. Always assert the build's exit
593 status and the binary's freshness before believing a mutation result.
594
595 - **Banked (M7 additions, carrying the M6 list forward intact):** a shorter
596 first reconnect backoff (see above); repeated `--via` is untested while
597 repeated `--sock` is covered — an asymmetry in the parser tests, not in
598 the parser; no version negotiation between `mux` and a remote `muxd`
599 (see above); systemd socket-activation packaging for the VMs (the unit
600 files exist in `contrib/`, an install doc does not); the reconnect loop
601 has no upper bound on total retry time by design, so a client left
602 attached to a permanently dead daemon retries until the user aborts.
603 - **Out of scope, in order:** QUIC/TLS transport (M8, next per user
604 direction), then prediction / local echo — which now has this
605 milestone's numbers to beat as well as M6's ~4ms protocol cost.
606
482 ## Open (owed by later milestones) 607 ## Open (owed by later milestones)
483 608
484 - Scrollback retention *tuning*. The policy itself was decided in M1 and 609 - Scrollback retention *tuning*. The policy itself was decided in M1 and
docs/superpowers/plans/2026-08-07-m7-reconnect.md
Old New
@@ -25,7 +25,7 @@
25 25
26 The via-child plumbing lives inline in `attach()` (src/client.zig:27-51) and the socket path right below it; neither can be re-established mid-session. Extract both into one struct. **No behavior change in this task** — same messages, same exit codes, same child cleanup order (stdin close → kill; the comment at client.zig:40-49 explains why — keep it). 26 The via-child plumbing lives inline in `attach()` (src/client.zig:27-51) and the socket path right below it; neither can be re-established mid-session. Extract both into one struct. **No behavior change in this task** — same messages, same exit codes, same child cleanup order (stdin close → kill; the comment at client.zig:40-49 explains why — keep it).
27 27
28 - [ ] **Step 1: Write the struct and constructor** 28 - [x] **Step 1: Write the struct and constructor**
29 29
30 ```zig 30 ```zig
31 /// One live connection to a muxd, however reached. Reopenable: the 31 /// One live connection to a muxd, however reached. Reopenable: the
@@ -62,13 +62,13 @@ const Transport = struct {
62 }; 62 };
63 ``` 63 ```
64 64
65 - [ ] **Step 2: Rewire `attach()` to use it, keeping today's error messages** 65 - [x] **Step 2: Rewire `attach()` to use it, keeping today's error messages**
66 66
67 `attach()` opens the first Transport, prints the same "cannot start --via command"/"cannot connect to {s} (is muxd running?)" messages on failure, and passes `sock_path`, `via`, and the Transport down to `session()` (signature grows; the recipe is needed there next task). 67 `attach()` opens the first Transport, prints the same "cannot start --via command"/"cannot connect to {s} (is muxd running?)" messages on failure, and passes `sock_path`, `via`, and the Transport down to `session()` (signature grows; the recipe is needed there next task).
68 68
69 - [ ] **Step 3: `make test && make e2e` — both green, zero behavior change** 69 - [x] **Step 3: `make test && make e2e` — both green, zero behavior change**
70 70
71 - [ ] **Step 4: Commit** `refactor: transport is a reopenable value, recipe travels with the session` 71 - [x] **Step 4: Commit** `refactor: transport is a reopenable value, recipe travels with the session`
72 72
73 --- 73 ---
74 74
@@ -77,15 +77,15 @@ const Transport = struct {
77 **Files:** 77 **Files:**
78 - Modify: `src/client.zig` (session loop) 78 - Modify: `src/client.zig` (session loop)
79 79
80 - [ ] **Step 1: Track `last_seq`** 80 - [x] **Step 1: Track `last_seq`**
81 81
82 In the session loop state (near `session_epoch`, client.zig:145): `var last_seq: u64 = 0;`. Update from both authoritative sources: `prefix.seq` in the snapshot arm, `composed.header.seq` in the delta arm. Fix the now-stale comment at client.zig:142-144 ("Nothing reads it yet") — something reads it now. 82 In the session loop state (near `session_epoch`, client.zig:145): `var last_seq: u64 = 0;`. Update from both authoritative sources: `prefix.seq` in the snapshot arm, `composed.header.seq` in the delta arm. Fix the now-stale comment at client.zig:142-144 ("Nothing reads it yet") — something reads it now.
83 83
84 - [ ] **Step 2: Route every transport-loss exit into `reconnect()`** 84 - [x] **Step 2: Route every transport-loss exit into `reconnect()`**
85 85
86 The three exit sites that today set `exit_msg = "mux: connection to muxd lost"` on a *dead transport* (readFrame null at client.zig:180; the input-write catch at :292; the winch-resize catch at :157, plus the scroll-request catches) instead do: `if (!try reconnect(...)) { exit_msg = ...; return 1; }` and `continue` the main loop. The malformed-delta re-attach (client.zig:217-224) is **not** one of these — the transport is alive there, and its `have_seq=0` is deliberate (untrusted replica must force a snapshot); leave it, but extend its comment to say why it differs from reconnect(). 86 The three exit sites that today set `exit_msg = "mux: connection to muxd lost"` on a *dead transport* (readFrame null at client.zig:180; the input-write catch at :292; the winch-resize catch at :157, plus the scroll-request catches) instead do: `if (!try reconnect(...)) { exit_msg = ...; return 1; }` and `continue` the main loop. The malformed-delta re-attach (client.zig:217-224) is **not** one of these — the transport is alive there, and its `have_seq=0` is deliberate (untrusted replica must force a snapshot); leave it, but extend its comment to say why it differs from reconnect().
87 87
88 - [ ] **Step 3: Write `reconnect()`** 88 - [x] **Step 3: Write `reconnect()`**
89 89
90 ```zig 90 ```zig
91 /// The transport died but the session (daemon-side) very likely did not. 91 /// The transport died but the session (daemon-side) very likely did not.
@@ -124,13 +124,13 @@ fn reconnect(
124 124
125 Details the implementer owns: `paintBanner` (factor from `renderScrollback`'s marker paint, client.zig:394-398); `drainStdinForQuit` (poll stdin with the backoff as timeout; read; return true iff 0x1c seen — bytes are otherwise discarded); no unbounded retry cap on *transport* failures (the user has the abort key and the indicator; a laptop asleep for an hour should still resume on wake — that IS the use case), but see Step 4 for the refusal cap. 125 Details the implementer owns: `paintBanner` (factor from `renderScrollback`'s marker paint, client.zig:394-398); `drainStdinForQuit` (poll stdin with the backoff as timeout; read; return true iff 0x1c seen — bytes are otherwise discarded); no unbounded retry cap on *transport* failures (the user has the abort key and the indicator; a laptop asleep for an hour should still resume on wake — that IS the use case), but see Step 4 for the refusal cap.
126 126
127 - [ ] **Step 4: The refusal-during-reconnect grace** 127 - [x] **Step 4: The refusal-during-reconnect grace**
128 128
129 After a reconnect-attach, if the next frame is `exit_status` before any state arrives (the session-full refusal shape, client.zig:241-243), and we were reconnecting (a flag alongside `got_state`), treat it as retryable for up to ~5s total (ghost-slot race: cap=2 and the daemon may not have reaped our dead predecessor). Past the grace: exit with the existing refusal message. A *real* exit_status (after state) still exits normally — the shell exiting while we reconnect must not loop. 129 After a reconnect-attach, if the next frame is `exit_status` before any state arrives (the session-full refusal shape, client.zig:241-243), and we were reconnecting (a flag alongside `got_state`), treat it as retryable for up to ~5s total (ghost-slot race: cap=2 and the daemon may not have reaped our dead predecessor). Past the grace: exit with the existing refusal message. A *real* exit_status (after state) still exits normally — the shell exiting while we reconnect must not loop.
130 130
131 - [ ] **Step 5: `make test && make e2e && make bench` green** 131 - [x] **Step 5: `make test && make e2e && make bench` green**
132 132
133 - [ ] **Step 6: Commit** `feat: transport death is a non-event — reconnect with have_seq, resume by delta` 133 - [x] **Step 6: Commit** `feat: transport death is a non-event — reconnect with have_seq, resume by delta`
134 134
135 --- 135 ---
136 136
@@ -139,7 +139,7 @@ After a reconnect-attach, if the next frame is `exit_status` before any state ar
139 **Files:** 139 **Files:**
140 - Modify: `test/e2e.sh` 140 - Modify: `test/e2e.sh`
141 141
142 - [ ] **Step 1: Scenario A — proxy killed, delta resume** 142 - [x] **Step 1: Scenario A — proxy killed, delta resume**
143 143
144 Append after the M6 --via scenario. Shape (implementer adapts to the file's conventions — set -eu, cleanup traps, `|| true` on the piped client): 144 Append after the M6 --via scenario. Shape (implementer adapts to the file's conventions — set -eu, cleanup traps, `|| true` on the piped client):
145 145
@@ -165,13 +165,13 @@ SNAPS_AFTER=$("$MUXD" stats --sock "$SOCK" | sed -n 's/.*snapshots=\([0-9]*\).*/
165 165
166 Timing note for the implementer: the pkill must land while the client is mid-`sleep 6` (attached, idle). If flaky, drive the tear from a marker in `$OUT.m7` instead of sleep. The pkill pattern must not match the daemon itself — verify with a comment and a `kill -0 $DPID` assertion right after. 166 Timing note for the implementer: the pkill must land while the client is mid-`sleep 6` (attached, idle). If flaky, drive the tear from a marker in `$OUT.m7` instead of sleep. The pkill pattern must not match the daemon itself — verify with a comment and a `kill -0 $DPID` assertion right after.
167 167
168 - [ ] **Step 2: Scenario B — daemon killed and restarted, snapshot resume with the new epoch** 168 - [x] **Step 2: Scenario B — daemon killed and restarted, snapshot resume with the new epoch**
169 169
170 Kill -9 the daemon under an attached client, restart it on the same socket path, and assert the client resumes into the NEW session: a marker typed after restart echoes back, and the pre-restart marker is absent from the new daemon's dump. (This is the epoch fence doing its job end-to-end: same path, different epoch, have_seq rightly refused, snapshot served.) Note: daemon restart on the same path relies on the stale-socket recovery (`ECONNREFUSED` → unlink) — coordinate with the socket-steal fix if it hasn't landed first. 170 Kill -9 the daemon under an attached client, restart it on the same socket path, and assert the client resumes into the NEW session: a marker typed after restart echoes back, and the pre-restart marker is absent from the new daemon's dump. (This is the epoch fence doing its job end-to-end: same path, different epoch, have_seq rightly refused, snapshot served.) Note: daemon restart on the same path relies on the stale-socket recovery (`ECONNREFUSED` → unlink) — coordinate with the socket-steal fix if it hasn't landed first.
171 171
172 - [ ] **Step 3: `make e2e` green, three runs (timing-sensitive scenarios must not flake)** 172 - [x] **Step 3: `make e2e` green, three runs (timing-sensitive scenarios must not flake)**
173 173
174 - [ ] **Step 4: Commit** `test: e2e proves resume-by-delta across a transport tear and resume-by-snapshot across a daemon restart` 174 - [x] **Step 4: Commit** `test: e2e proves resume-by-delta across a transport tear and resume-by-snapshot across a daemon restart`
175 175
176 --- 176 ---
177 177
@@ -182,17 +182,17 @@ Kill -9 the daemon under an attached client, restart it on the same socket path,
182 182
183 The use case is "attach to any of my VMs": `mux vm1`, `mux ubuntu@sandbox-9b70e9`. Sugar for `--via "ssh <host> muxd proxy"` (muxd must be on the remote PATH; document in the usage string). 183 The use case is "attach to any of my VMs": `mux vm1`, `mux ubuntu@sandbox-9b70e9`. Sugar for `--via "ssh <host> muxd proxy"` (muxd must be on the remote PATH; document in the usage string).
184 184
185 - [ ] **Step 1: Extract arg parsing into a testable function** 185 - [x] **Step 1: Extract arg parsing into a testable function**
186 186
187 `fn parseArgs(args: []const [:0]const u8) ParseResult` where ParseResult is a tagged union: `{ sock: ?[]const u8, via: ?[]const u8 } | usage_error | conflict`. A single positional arg (not starting with `-`) becomes `via = "ssh <arg> muxd proxy"` (allocPrint in main after parse, or return the host and let main format). Positional + --sock or --via = conflict, same as the existing pair. 187 `fn parseArgs(args: []const [:0]const u8) ParseResult` where ParseResult is a tagged union: `{ sock: ?[]const u8, via: ?[]const u8 } | usage_error | conflict`. A single positional arg (not starting with `-`) becomes `via = "ssh <arg> muxd proxy"` (allocPrint in main after parse, or return the host and let main format). Positional + --sock or --via = conflict, same as the existing pair.
188 188
189 - [ ] **Step 2: Tests for parseArgs** — plain, `--sock P`, `--via C`, `host`, `user@host`, `host --sock P` (conflict), `--sock P --via C` (conflict), unknown flag (usage). Mutation check: break the positional branch, the host tests must fail. 189 - [x] **Step 2: Tests for parseArgs** — plain, `--sock P`, `--via C`, `host`, `user@host`, `host --sock P` (conflict), `--sock P --via C` (conflict), unknown flag (usage). Mutation check: break the positional branch, the host tests must fail.
190 190
191 - [ ] **Step 3: Update usage string** to `usage: mux [HOST | --sock PATH | --via CMD]\n HOST attaches over "ssh HOST muxd proxy" (muxd must be on HOST's PATH)` 191 - [x] **Step 3: Update usage string** to `usage: mux [HOST | --sock PATH | --via CMD]\n HOST attaches over "ssh HOST muxd proxy" (muxd must be on HOST's PATH)`
192 192
193 - [ ] **Step 4: `make test && make e2e` green** 193 - [x] **Step 4: `make test && make e2e` green**
194 194
195 - [ ] **Step 5: Commit** `feat: mux HOST attaches over ssh — the VM hop is one word` 195 - [x] **Step 5: Commit** `feat: mux HOST attaches over ssh — the VM hop is one word`
196 196
197 --- 197 ---
198 198
@@ -202,17 +202,17 @@ The use case is "attach to any of my VMs": `mux vm1`, `mux ubuntu@sandbox-9b70e9
202 - Modify: `test/wan.sh` (env-gated phase, same MUX_WAN_* convention) 202 - Modify: `test/wan.sh` (env-gated phase, same MUX_WAN_* convention)
203 - Modify: `docs/decisions.md` (M7 section) 203 - Modify: `docs/decisions.md` (M7 section)
204 204
205 - [ ] **Step 1: wan.sh reconnect phase** 205 - [x] **Step 1: wan.sh reconnect phase**
206 206
207 Attached over real ssh via, kill the local ssh child N=10 times in a loop; each iteration: assert resumption (marker echoes), record time-to-first-frame after respawn, assert daemon snapshots counter unchanged across all 10 (all delta-served). Print per-iteration and summary numbers labeled like the M6 phases. Gate: all 10 resumed hands-off AND delta-served. (Resume wall-clock will be dominated by ssh channel-open exactly as M6 measured for reattach — print it labeled, don't gate on it; the M6 ruling comment in wan.sh applies and should be referenced.) 207 Attached over real ssh via, kill the local ssh child N=10 times in a loop; each iteration: assert resumption (marker echoes), record time-to-first-frame after respawn, assert daemon snapshots counter unchanged across all 10 (all delta-served). Print per-iteration and summary numbers labeled like the M6 phases. Gate: all 10 resumed hands-off AND delta-served. (Resume wall-clock will be dominated by ssh channel-open exactly as M6 measured for reattach — print it labeled, don't gate on it; the M6 ruling comment in wan.sh applies and should be referenced.)
208 208
209 - [ ] **Step 2: Run it against the sandbox** (`MUX_WAN_SSH`/`MUX_WAN_SCP`/`MUX_WAN_HOST` as in mux-wan-box memory / wan.sh header comment), record the numbers. 209 - [x] **Step 2: Run it against the sandbox** (`MUX_WAN_SSH`/`MUX_WAN_SCP`/`MUX_WAN_HOST` as in mux-wan-box memory / wan.sh header comment), record the numbers.
210 210
211 - [ ] **Step 3: decisions.md M7 section** 211 - [x] **Step 3: decisions.md M7 section**
212 212
213 Record: kill criterion + measured result; the three policy decisions from the plan header (dropped input, silent epoch-crossing, refusal grace); `session_epoch` graduating from parsed-but-unused; `mux HOST` sugar; what stays banked (QUIC/TLS → M8 next per user, prediction after). 213 Record: kill criterion + measured result; the three policy decisions from the plan header (dropped input, silent epoch-crossing, refusal grace); `session_epoch` graduating from parsed-but-unused; `mux HOST` sugar; what stays banked (QUIC/TLS → M8 next per user, prediction after).
214 214
215 - [ ] **Step 4: Commit** `feat(m7): reconnect held over the WAN — 10 tears, 10 delta resumes; the record` 215 - [x] **Step 4: Commit** `feat(m7): reconnect held over the WAN — 10 tears, 10 delta resumes; the record`
216 216
217 --- 217 ---
218 218
test/wan.sh
Old New
@@ -15,6 +15,30 @@
15 # check pre-kill session state is in that first paint 15 # check pre-kill session state is in that first paint
16 # hol echo latency while the session floods output (probes the 16 # hol echo latency while the session floods output (probes the
17 # proxy's known head-of-line blocking; recorded, not gated) 17 # proxy's known head-of-line blocking; recorded, not gated)
18 # reconnect M7: kill the ssh under a LIVE client N times; every tear must
19 # resume hands-off and be served by a delta (the remote
20 # daemon's snapshots counter must not move)
21 #
22 # M7 kill criterion (docs/superpowers/plans/2026-08-07-m7-reconnect.md):
23 # 10 consecutive transport kills against a live session, every one resumed
24 # with zero manual action, every one delta-served. The resume wall clock is
25 # printed, not gated: it necessarily includes the client's first backoff
26 # and an ssh channel open, so gating it would rule on ssh — the same
27 # reasoning as the M6 reattach ruling below.
28 #
29 # NARROWED, deliberately and on the record: the plan's criterion had a
30 # fourth element — "replica converged after the last tear (client render
31 # matches `muxd dump`)" — and this harness does NOT check that. The client
32 # emits a stream of paints, not a grid, so comparing it to a dump means
33 # parsing VT in the harness: a second terminal emulator, in Python, whose
34 # own bugs would then be indistinguishable from the ones it is meant to
35 # catch. (src/server.zig's tests do make that comparison, against a real
36 # replica Engine — over a socket, not over the WAN.) What is checked here
37 # instead: each tear round-trips a marker out to the remote shell and back
38 # through the rebuilt transport, and after the last tear the FIRST tear's
39 # marker must still be in the daemon's grid — accumulated state survived
40 # all ten resumes. That is the substitution, it is weaker than the plan's
41 # wording, and the full convergence check stays banked.
18 # 42 #
19 # Kill criterion (docs/superpowers/plans/2026-08-07-m6-transport.md): 43 # Kill criterion (docs/superpowers/plans/2026-08-07-m6-transport.md):
20 # median echo <= baseline median + 120ms, and reattach <= ~2x the link 44 # median echo <= baseline median + 120ms, and reattach <= ~2x the link
@@ -43,6 +67,7 @@
43 # MUX_WAN_SCP matching scp command string (same ControlPath) 67 # MUX_WAN_SCP matching scp command string (same ControlPath)
44 # MUX_WAN_HOST scp target prefix, e.g. user@host 68 # MUX_WAN_HOST scp target prefix, e.g. user@host
45 # MUX_WAN_NETEM 1 to also measure under `netem delay 75ms loss 1%` 69 # MUX_WAN_NETEM 1 to also measure under `netem delay 75ms loss 1%`
70 # MUX_WAN_REPS_RECONNECT M7 tears (default 10 — that IS the criterion)
46 # ZIG cross-compiling zig 0.15.2 (the Makefile's default) 71 # ZIG cross-compiling zig 0.15.2 (the Makefile's default)
47 set -euo pipefail 72 set -euo pipefail
48 73
@@ -74,6 +99,9 @@ REPS_BASE="${MUX_WAN_REPS_BASE:-20}"
74 REPS_ATTACH="${MUX_WAN_REPS_ATTACH:-3}" 99 REPS_ATTACH="${MUX_WAN_REPS_ATTACH:-3}"
75 REPS_REATTACH="${MUX_WAN_REPS_REATTACH:-3}" 100 REPS_REATTACH="${MUX_WAN_REPS_REATTACH:-3}"
76 REPS_HOL="${MUX_WAN_REPS_HOL:-5}" 101 REPS_HOL="${MUX_WAN_REPS_HOL:-5}"
102 # M7: the kill criterion names 10 consecutive tears; the default is the
103 # criterion, not a convenience.
104 REPS_RECONNECT="${MUX_WAN_REPS_RECONNECT:-10}"
77 105
78 # Unique remote names: the box may be shared, and a crashed run must never 106 # Unique remote names: the box may be shared, and a crashed run must never
79 # leave a socket another run mistakes for its own. 107 # leave a socket another run mistakes for its own.
@@ -187,6 +215,7 @@ numbers are tens of milliseconds and must be taken around a single write().
187 Each subcommand prints human-readable lines plus one '#RESULT <name> k=v...' 215 Each subcommand prints human-readable lines plus one '#RESULT <name> k=v...'
188 line that wan.sh banks for the summary.""" 216 line that wan.sh banks for the summary."""
189 import os 217 import os
218 import re
190 import select 219 import select
191 import shlex 220 import shlex
192 import signal 221 import signal
@@ -496,6 +525,124 @@ def cmd_reattach(argv):
496 report("reattach", samples, marker_ok=marker_ok) 525 report("reattach", samples, marker_ok=marker_ok)
497 526
498 527
528 def transport_pid(client_pid):
529 """The transport process under a client.
530
531 `mux --via CMD` spawns /bin/sh -c CMD, which execs ssh, so the client's
532 only child IS the transport. Selected by PARENT PID, never by a
533 command-line pattern: the client's own argv contains the whole --via
534 string, so a pattern kill takes out the very client under test. (That
535 mistake has already been made once in this project's e2e work; it is a
536 rule now, not a preference.)
537 """
538 out = subprocess.run(["ps", "--ppid", str(client_pid), "-o", "pid="],
539 capture_output=True, text=True).stdout.split()
540 return int(out[0]) if out else None
541
542
543 def remote_dump(ssh_cmd, rbin, rsock):
544 """The daemon's authoritative grid, as it sees it."""
545 return subprocess.run(shlex.split(ssh_cmd) + ["%s dump --sock %s" % (rbin, rsock)],
546 capture_output=True, text=True, timeout=60).stdout
547
548
549 def remote_snapshots(ssh_cmd, rbin, rsock):
550 """The remote daemon's snapshots counter.
551
552 This is the whole M7 assertion: a resume served by snapshot renders
553 exactly like one served by delta, so the counter is the only witness to
554 which of the two actually happened.
555 """
556 out = subprocess.run(shlex.split(ssh_cmd) + ["%s stats --sock %s" % (rbin, rsock)],
557 capture_output=True, text=True, timeout=60).stdout
558 m = re.search(r"\bsnapshots=(\d+)", out)
559 if m is None:
560 fail("reconnect: no snapshots counter in remote stats %r" % out.strip())
561 return int(m.group(1))
562
563
564 def cmd_reconnect(argv):
565 """M7: tear the transport under a live session N times over the real
566 link. Every tear must resume hands-off — nothing relaunches the client,
567 nothing retypes for it — and every resume must be served by a delta.
568
569 Timing note: the marker is typed only AFTER the resume is observed, not
570 before. Input typed while disconnected is dropped by policy, so typing
571 into the gap would measure the policy rather than the reconnect.
572
573 On the convergence check: the plan's criterion asked for the client's
574 render to match `muxd dump` after the last tear. That is NOT what this
575 measures — see the header note. What it does measure is that the FIRST
576 tear's marker is still in the daemon's grid after the LAST one, which
577 is the accumulated-state half of that claim and costs one ssh call.
578 """
579 mux, via, reps, errlog = argv[0], argv[1], int(argv[2]), argv[3]
580 ssh_cmd, rbin, rsock = argv[4], argv[5], argv[6]
581
582 c = Client(mux, via, errlog)
583 samples = []
584 resumed = 0
585 snaps_before = snaps_after = -1
586 first_marker = None
587 survived = 0
588 try:
589 settle(c, "reconnect")
590 # Start the run on a cleared screen so all `reps` markers fit in one
591 # 24-row viewport (two rows each: the echoed command and its
592 # output). Without this the survival check below would be a coin
593 # flip on whether the first marker had scrolled out of the grid —
594 # `muxd dump` shows the viewport, not the scrollback.
595 c.send(b"\x15clear\n")
596 wait_idle(c, "reconnect")
597 # Taken after the session is up and quiet, so the first attach's own
598 # snapshot is already counted and anything after it is a resume.
599 snaps_before = remote_snapshots(ssh_cmd, rbin, rsock)
600 for i in range(reps):
601 child = transport_pid(c.p.pid)
602 if child is None:
603 fail("reconnect: rep %d found no transport to tear" % i)
604 before = c.total
605 t0 = now()
606 os.kill(child, signal.SIGKILL)
607
608 # First frame after the tear: the client rebuilt the transport,
609 # re-attached, and was answered. Hands-off from the kill to here.
610 if c.wait_for(lambda b, c=c, t=before: c.total > t) is None:
611 fail("reconnect: rep %d never resumed after the transport died" % i)
612 samples.append((now() - t0) * 1000.0)
613 resumed += 1
614
615 # ...and the session is really usable again, not merely painted:
616 # a round trip through the new transport, out to bash and back.
617 nonce = "%d-%d" % (os.getpid(), i)
618 marker = ("wanre-" + nonce).encode()
619 if first_marker is None:
620 first_marker = marker
621 c.send(b'\x15printf "wanre-%s\\n" ' + nonce.encode() + b"\n")
622 if c.wait_for(lambda b, m=marker: m in b) is None:
623 fail("reconnect: rep %d resumed but the input path is dead" % i)
624 print(" reconnect rep %-2d first frame %7.1fms session usable"
625 % (i, samples[-1]))
626 wait_idle(c, "reconnect")
627 snaps_after = remote_snapshots(ssh_cmd, rbin, rsock)
628
629 # Accumulated state survived every resume: work from before the
630 # first tear is still on the daemon's grid after the last one. The
631 # per-rep markers cannot show this — each only proves the session
632 # was usable at that moment, not that nothing was lost along the way.
633 dump = remote_dump(ssh_cmd, rbin, rsock)
634 survived = 1 if first_marker.decode() in dump else 0
635 if not survived:
636 print(" reconnect: FIRST TEAR'S MARKER %s IS GONE from the grid"
637 % first_marker.decode())
638 finally:
639 c.detach()
640 report("reconnect", samples, resumed=resumed, reps=reps,
641 snaps_before=snaps_before, snaps_after=snaps_after,
642 delta_served=1 if snaps_after == snaps_before else 0,
643 first_survived=survived)
644
645
499 def cmd_hol(argv): 646 def cmd_hol(argv):
500 """Echo latency while the session floods output. The proxy is a byte 647 """Echo latency while the session floods output. The proxy is a byte
501 pump with no notion of priority, so a flood shares the pipe with the 648 pump with no notion of priority, so a flood shares the pipe with the
@@ -557,6 +704,7 @@ COMMANDS = {
557 "attach": cmd_attach, 704 "attach": cmd_attach,
558 "echo": cmd_echo, 705 "echo": cmd_echo,
559 "reattach": cmd_reattach, 706 "reattach": cmd_reattach,
707 "reconnect": cmd_reconnect,
560 "hol": cmd_hol, 708 "hol": cmd_hol,
561 } 709 }
562 710
@@ -604,6 +752,14 @@ measure clean echo "$MUX" "$VIA" "$REPS_ECHO" "$ERRLOG"
604 say "reattach after kill -9 ($REPS_REATTACH reps)" 752 say "reattach after kill -9 ($REPS_REATTACH reps)"
605 measure clean reattach "$MUX" "$VIA" "$REPS_REATTACH" "$ERRLOG" 753 measure clean reattach "$MUX" "$VIA" "$REPS_REATTACH" "$ERRLOG"
606 754
755 # M7. Measured on the clean link only: the criterion is about resuming at
756 # all, hands-off, and being served a delta while doing it — none of which
757 # netem's added delay makes more or less true, and each tear costs a real
758 # ssh channel open.
759 say "M7 reconnect: tear the transport under a live session ($REPS_RECONNECT tears)"
760 measure clean reconnect "$MUX" "$VIA" "$REPS_RECONNECT" "$ERRLOG" \
761 "$MUX_WAN_SSH" "$RBIN" "$RSOCK"
762
607 # Every clean-link figure is taken before the box is degraded, so a netem 763 # Every clean-link figure is taken before the box is degraded, so a netem
608 # that failed to clear can never quietly relabel itself as the clean link. 764 # that failed to clear can never quietly relabel itself as the clean link.
609 say "head-of-line probe: echo while the session floods output ($REPS_HOL reps)" 765 say "head-of-line probe: echo while the session floods output ($REPS_HOL reps)"
@@ -704,16 +860,57 @@ phase_block() {
704 printf ' attach (fresh client, same path): med %s\n' "$attach_med" 860 printf ' attach (fresh client, same path): med %s\n' "$attach_med"
705 } 861 }
706 862
863 # M7's own criterion: resumption is pass/fail, and the resume kind is read
864 # off the daemon's counters. The wall-clock column is printed and not gated,
865 # for the same reason the M6 reattach ruling above gives — every resume pays
866 # an ssh channel open (`viafloor`) before a protocol byte can move, so gating
867 # it would rule on ssh, not on this design.
868 m7_block() {
869 local resumed reps snaps_b snaps_a delta_served verdict floor survived
870 resumed="$(val clean reconnect resumed)"
871 [ -n "$resumed" ] || return 0
872 reps="$(val clean reconnect reps)"
873 snaps_b="$(val clean reconnect snaps_before)"
874 snaps_a="$(val clean reconnect snaps_after)"
875 delta_served="$(val clean reconnect delta_served)"
876 floor="$(val clean viafloor med)"
877
878 echo
879 echo "M7 reconnect (transport torn under a live session):"
880 printf ' %-34s %8s %8s %8s %5s\n' "" min med max n
881 printf ' %-34s %8s %8s %8s %5s\n' "first frame after tear" \
882 "$(val clean reconnect min)" "$(val clean reconnect med)" \
883 "$(val clean reconnect max)" "$(val clean reconnect n)"
884 echo " (milliseconds; includes the ~200ms first backoff and one ssh"
885 echo " channel open, measured just above as viafloor med ${floor}ms —"
886 echo " printed, not gated, per the reattach ruling above)"
887 printf ' resumed hands-off: %s of %s\n' "$resumed" "$reps"
888 printf ' daemon snapshots across all tears: %s -> %s (%s)\n' \
889 "$snaps_b" "$snaps_a" \
890 "$([ "$delta_served" = 1 ] && echo "unchanged: every resume delta-served" \
891 || echo "CHANGED: a resume fell back to a full snapshot")"
892 survived="$(val clean reconnect first_survived)"
893 printf ' first tear'"'"'s marker still on the grid after the last: %s\n' \
894 "$([ "$survived" = 1 ] && echo yes || echo NO)"
895 verdict=FAIL
896 [ "$resumed" = "$reps" ] && [ "$delta_served" = 1 ] && [ "$survived" = 1 ] &&
897 verdict=PASS
898 printf ' M7 criterion: %s tears, all resumed, all delta-served, state kept -> %s\n' \
899 "$reps" "$verdict"
900 [ "$verdict" = PASS ] || FAILED=1
901 }
902
707 echo 903 echo
708 echo "=================== M6 WAN measurement summary ===================" 904 echo "=================== WAN measurement summary ======================"
709 echo "date: $(date -u +%Y-%m-%dT%H:%M:%SZ) host: (ephemeral, from MUX_WAN_SSH)" 905 echo "date: $(date -u +%Y-%m-%dT%H:%M:%SZ) host: (ephemeral, from MUX_WAN_SSH)"
710 phase_block clean "link as-is:" 906 phase_block clean "link as-is:"
711 phase_block netem "with netem delay 75ms loss 1%:" 907 phase_block netem "with netem delay 75ms loss 1%:"
908 m7_block
712 echo 909 echo
713 if [ "$FAILED" -eq 0 ]; then 910 if [ "$FAILED" -eq 0 ]; then
714 echo "M6 kill criterion: PASS" 911 echo "kill criteria (M6 + M7): PASS"
715 else 912 else
716 echo "M6 kill criterion: FAIL — report the numbers; do not tune the thresholds." 913 echo "kill criteria (M6 + M7): FAIL — report the numbers; do not tune the thresholds."
717 fi 914 fi
718 echo "==================================================================" 915 echo "=================================================================="
719 exit "$FAILED" 916 exit "$FAILED"