a73x

625c6b51

test(e2e): four helpers, and the marker polls grow the assert they lost

a73x   2026-08-13 14:46

Commit message
test(e2e): four helpers, and the marker polls grow the assert they lost

M17 simplify round, test batch: e2e.sh.

The Important is I2. Scenarios (b) and (c) copied scenario (a)'s
marker-poll loop and dropped its assert, so three loops fell out
silently after 10s and carried on: a session that died at birth was
discovered by whatever ran next, 15s later, and blamed on the
WebSocket leg. wait_grid is the same poll with an UNSKIPPABLE assert,
and it prints the grid the daemon actually holds.

The other three helpers are the shapes this file had already repeated
into an idiom:

  - wait_sock PATH LOG LABEL — bind-poll, assert, print the spawn's own
    log. 19 sites, from M7 through M-web; the ones with no capture of
    their own pass "".
  - assert_stopped SOCK PID LABEL ERRFILE — the four-assertion `muxd
    stop` teardown (rc 0, the stopped line, the socket gone, the pid
    observed dead). Six sites, two of which were partial: scenario (c)
    checked neither the stopped line nor, on its first stop, the
    socket. It is now as strict as (a) and (b).
  - assert_ws_converged WSOUT SOCK LABEL — the sed/diff against the
    daemon's grid plus the doctored-dump control. This GIVES (c) the
    cannot-fail control it never had.

The M13 `muxd stop` scenarios keep their spelled-out assertions on
purpose: `muxd stop` is their subject, and a test should not assert the
thing it exists to pin through machinery that other tests could
pressure into loosening.

assert_ws_converged deliberately does not touch CONV_COUNT — that pin
counts assert_converged call sites, and folding two different checks
into one number would stop either from meaning anything. Both pins
still read 23 scenarios / 35 convergence points.

Also: (c) wrote both its stop captures to $OUT.webstop3, the second
overwriting the first before anything read it, which is how the
evidence for a failure at the first stop disappeared. The second is now
$OUT.webstop4, and both — plus assert_ws_converged's .doc — are in the
trap's rm list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test/e2e.sh
Old New
@@ -230,6 +230,40 @@ wait_gone() {
230 done 230 done
231 } 231 }
232 232
233 # wait_sock PATH LOG LABEL — poll until a daemon has bound PATH (5s), then
234 # ASSERT it. Every `muxd run &` in this file needs this: the bind happens
235 # after the fork, so the very next command would otherwise race it. LOG is
236 # the spawn's own capture, printed on failure because a daemon that failed
237 # to bind almost always said why; pass "" for the spawns that have none.
238 wait_sock() {
239 _i=0
240 while [ ! -S "$1" ] && [ "$_i" -lt 50 ]; do sleep 0.1; _i=$((_i+1)); done
241 [ -S "$1" ] || {
242 echo "e2e FAIL: $3"
243 # An `if` rather than `[ -n "$2" ] && cat "$2"`, for converged_quiet's
244 # reason: a false guard must not become this block's exit status.
245 if [ -n "$2" ]; then cat "$2"; fi
246 exit 1
247 }
248 }
249
250 # wait_grid SOCK NEEDLE LABEL — poll the daemon's own grid for NEEDLE (10s)
251 # and ASSERT it landed. The assert is the whole point. The bare form of this
252 # loop — poll, `&& break`, carry on — falls out silently when the marker
253 # never arrives, so a session that died at birth is discovered by whatever
254 # runs next, 15s later and in the wrong layer. Three M-web scenarios had
255 # grown exactly that shape.
256 wait_grid() {
257 _i=0
258 while [ "$_i" -lt 100 ]; do
259 "$MUXD" dump --sock "$1" 2>/dev/null | grep -q "$2" && return 0
260 sleep 0.1; _i=$((_i+1))
261 done
262 echo "e2e FAIL: $3: '$2' never reached the daemon's grid; it holds:"
263 "$MUXD" dump --sock "$1" || echo "(nothing answers on $1)"
264 exit 1
265 }
266
233 # wait_pid_gone PID LABEL — poll until a tracked pid is gone (2s, the same 267 # wait_pid_gone PID LABEL — poll until a tracked pid is gone (2s, the same
234 # budget `muxd stop` gives itself). Its own helper rather than wait_gone's 268 # budget `muxd stop` gives itself). Its own helper rather than wait_gone's
235 # socket probe, because the two answer different questions: `muxd: stopped` 269 # socket probe, because the two answer different questions: `muxd: stopped`
@@ -252,6 +286,33 @@ wait_pid_gone() {
252 done 286 done
253 } 287 }
254 288
289 # assert_stopped SOCK PID LABEL ERRFILE — teardown by the sanctioned verb,
290 # asserted four ways. An exit code says the command RETURNED, not that it
291 # did the job: `muxd: stopped` is the daemon's own account of it, the absent
292 # socket is the filesystem's, and only the pid can say the process itself
293 # ended (see wait_pid_gone). A stop that exited 0 while leaving any of the
294 # three behind is precisely the regression this shape exists to catch.
295 # ERRFILE holds the stop's stderr — captured rather than let through,
296 # because a stop that fails exits 1 and would abort under `set -e` with no
297 # line of its own, after which the trap deletes the evidence.
298 #
299 # The caller nulls its own pid variable afterwards, the way every scenario
300 # here does: the trap reads those variables, and a pid it still holds is
301 # how a failing run says which daemon it leaked.
302 assert_stopped() {
303 set +e
304 "$MUXD" stop --sock "$1" 2> "$4"
305 _rc=$?
306 set -e
307 [ "$_rc" = "0" ] || {
308 echo "e2e FAIL: $3: stop exited $_rc, want 0"; cat "$4"; exit 1; }
309 grep -q '^muxd: stopped' "$4" || {
310 echo "e2e FAIL: $3: stop did not report stopped"; cat "$4"; exit 1; }
311 [ ! -S "$1" ] || {
312 echo "e2e FAIL: $3: stop left the socket"; ls -l "$1"; exit 1; }
313 wait_pid_gone "$2" "$3: stop reported stopped"
314 }
315
255 # The transport child for a given socket: a process named muxd running the 316 # The transport child for a given socket: a process named muxd running the
256 # `proxy` subcommand on that path. NEVER `pkill -f proxy` — the client's own 317 # `proxy` subcommand on that path. NEVER `pkill -f proxy` — the client's own
257 # argv contains the --via command string, so a pattern kill takes out the 318 # argv contains the --via command string, so a pattern kill takes out the
@@ -330,6 +391,33 @@ assert_converged() {
330 } 391 }
331 } 392 }
332 393
394 # assert_ws_converged WSOUT SOCK LABEL — the WebSocket leg's convergence
395 # check. WSOUT is the stand-in's `dumpexit` file: its replica's grid, in
396 # `muxd dump`'s own format, built from frames that crossed the hub. Diff it
397 # against the daemon's grid, then doctor a copy of that grid and assert the
398 # SAME diff catches the doctored one — the wan.sh rule (M9): a convergence
399 # check that cannot fail proves nothing.
400 #
401 # Deliberately NOT counted in CONV_COUNT. That pin counts assert_converged
402 # call sites — render-replays-the-client-stream — and this is different
403 # machinery answering a different question. Folding the two counts together
404 # would make either pin's number stop meaning anything.
405 assert_ws_converged() {
406 "$MUXD" dump --sock "$2" > "$1.dump"
407 # Trailing whitespace is a formatting difference between two correct
408 # grids, exactly as in converged_quiet.
409 sed 's/[[:space:]]*$//' "$1" > "$1.n"
410 sed 's/[[:space:]]*$//' "$1.dump" > "$1.dump.n"
411 diff -u "$1.dump.n" "$1.n" > "$1.diff" || {
412 echo "e2e FAIL: $3: wsclient replica diverges from daemon grid (-daemon +ws):"
413 head -40 "$1.diff"; exit 1; }
414 cp "$1.dump.n" "$1.doc"
415 printf 'doctored-row\n' >> "$1.doc"
416 if diff -u "$1.doc" "$1.n" > /dev/null 2>&1; then
417 echo "e2e FAIL: $3: the convergence diff cannot fail (doctored dump passed)"; exit 1
418 fi
419 }
420
333 # Scenario checkpoints. The suite's final line asserts the COUNT of these 421 # Scenario checkpoints. The suite's final line asserts the COUNT of these
334 # against a literal: adding or removing a scenario means updating that 422 # against a literal: adding or removing a scenario means updating that
335 # literal, and the friction is the feature — a scenario that silently 423 # literal, and the friction is the feature — a scenario that silently
@@ -464,7 +552,8 @@ cleanup() {
464 "$OUT.webc" "$OUT.webc.err" "$OUT.webc.d" "$OUT.webc2" "$OUT.webc2.err" \ 552 "$OUT.webc" "$OUT.webc.err" "$OUT.webc.d" "$OUT.webc2" "$OUT.webc2.err" \
465 "$OUT.webc2.d" "$OUT.webh2" "$OUT.webws2" "$OUT.webws2.err" \ 553 "$OUT.webc2.d" "$OUT.webh2" "$OUT.webws2" "$OUT.webws2.err" \
466 "$OUT.webws2.n" "$OUT.webws2.dump" "$OUT.webws2.dump.n" \ 554 "$OUT.webws2.n" "$OUT.webws2.dump" "$OUT.webws2.dump.n" \
467 "$OUT.webws2.diff" "$OUT.webstop" "$OUT.webstop2" "$OUT.webstop3" 555 "$OUT.webws2.diff" "$OUT.webws2.doc" \
556 "$OUT.webstop" "$OUT.webstop2" "$OUT.webstop3" "$OUT.webstop4"
468 # The convergence files a FAILING assert_converged leaves behind 557 # The convergence files a FAILING assert_converged leaves behind
469 # (.render/.dump/.rvt/.dvt/.diff for that capture) are deliberately not 558 # (.render/.dump/.rvt/.dvt/.diff for that capture) are deliberately not
470 # chased here: on a failing run they are the evidence. 559 # chased here: on a failing run they are the evidence.
@@ -503,9 +592,7 @@ ok "keygen creates once, 0600 in a 0700 dir, refuses twice"
503 "$MUXD" run --sock "$SOCK" --shell /bin/sh & 592 "$MUXD" run --sock "$SOCK" --shell /bin/sh &
504 DPID=$! 593 DPID=$!
505 594
506 i=0 595 wait_sock "$SOCK" "" "socket never appeared"
507 while [ ! -S "$SOCK" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
508 [ -S "$SOCK" ] || { echo "e2e FAIL: socket never appeared"; exit 1; }
509 596
510 # Client with piped stdio: types a command, waits, detaches with Ctrl-\ (034). 597 # Client with piped stdio: types a command, waits, detaches with Ctrl-\ (034).
511 { printf 'printf "e2e-%%s\\n" works\n'; sleep 2; printf '\034'; } | \ 598 { printf 'printf "e2e-%%s\\n" works\n'; sleep 2; printf '\034'; } | \
@@ -691,9 +778,7 @@ rm -f "$OUT.dead" "$OUT.long" "$OUT.longc"
691 # so there is nothing left to dump against. 778 # so there is nothing left to dump against.
692 "$MUXD" run --sock "$SOCK2" --shell /bin/sh & 779 "$MUXD" run --sock "$SOCK2" --shell /bin/sh &
693 D2PID=$! 780 D2PID=$!
694 i=0 781 wait_sock "$SOCK2" "" "second socket never appeared"
695 while [ ! -S "$SOCK2" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
696 [ -S "$SOCK2" ] || { echo "e2e FAIL: second socket never appeared"; exit 1; }
697 782
698 set +e 783 set +e
699 { printf 'echo m7-abort-live\n'; sleep 3; printf '\034'; sleep 2; } | \ 784 { printf 'echo m7-abort-live\n'; sleep 3; printf '\034'; sleep 2; } | \
@@ -775,9 +860,7 @@ rm -f "$OUT.m7" "$OUT.m7.err"
775 # fresh session instead. Needs its own daemon, since this one gets killed. 860 # fresh session instead. Needs its own daemon, since this one gets killed.
776 "$MUXD" run --sock "$SOCK3" --shell /bin/sh & 861 "$MUXD" run --sock "$SOCK3" --shell /bin/sh &
777 D3PID=$! 862 D3PID=$!
778 i=0 863 wait_sock "$SOCK3" "" "restart-scenario socket never appeared"
779 while [ ! -S "$SOCK3" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
780 [ -S "$SOCK3" ] || { echo "e2e FAIL: restart-scenario socket never appeared"; exit 1; }
781 864
782 set +e 865 set +e
783 { sleep 0.5; printf 'printf "m7b-%%s\\n" pre-restart\n'; sleep 8; \ 866 { sleep 0.5; printf 'printf "m7b-%%s\\n" pre-restart\n'; sleep 8; \
@@ -796,9 +879,7 @@ kill -9 "$D3PID"
796 sleep 0.5 879 sleep 0.5
797 "$MUXD" run --sock "$SOCK3" --shell /bin/sh & 880 "$MUXD" run --sock "$SOCK3" --shell /bin/sh &
798 D3PID=$! 881 D3PID=$!
799 i=0 882 wait_sock "$SOCK3" "" "daemon did not rebind the stale socket"
800 while [ ! -S "$SOCK3" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
801 [ -S "$SOCK3" ] || { echo "e2e FAIL: daemon did not rebind the stale socket"; exit 1; }
802 883
803 set +e 884 set +e
804 wait "$M7BPID" 885 wait "$M7BPID"
@@ -914,9 +995,7 @@ ok "no key anywhere is refused, and says how to make one"
914 "$MUXD" run --sock "$SOCK4" --shell /bin/sh \ 995 "$MUXD" run --sock "$SOCK4" --shell /bin/sh \
915 --quic "127.0.0.1:$QPORT" --key "$QKEY" --quic-idle-ms 3000 & 996 --quic "127.0.0.1:$QPORT" --key "$QKEY" --quic-idle-ms 3000 &
916 D4PID=$! 997 D4PID=$!
917 i=0 998 wait_sock "$SOCK4" "" "--quic daemon never bound its session socket"
918 while [ ! -S "$SOCK4" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
919 [ -S "$SOCK4" ] || { echo "e2e FAIL: --quic daemon never bound its session socket"; exit 1; }
920 999
921 # The UDP port is actually held. /proc/net/udp rather than ss or lsof: it is 1000 # The UDP port is actually held. /proc/net/udp rather than ss or lsof: it is
922 # always there on the platform this daemon runs on, and needs no privileges. 1001 # always there on the platform this daemon runs on, and needs no privileges.
@@ -1174,18 +1253,13 @@ sleep 0.5
1174 "$MUXD" run --sock "$SOCK4" --shell /bin/sh \ 1253 "$MUXD" run --sock "$SOCK4" --shell /bin/sh \
1175 --quic "127.0.0.1:$QPORT" --key "$QKEY" --quic-idle-ms 15000 > "$OUT.q" 2>&1 & 1254 --quic "127.0.0.1:$QPORT" --key "$QKEY" --quic-idle-ms 15000 > "$OUT.q" 2>&1 &
1176 D4PID=$! 1255 D4PID=$!
1177 i=0
1178 while [ ! -S "$SOCK4" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1179 # Nearly vacuous, and kept only because a MISSING file would still be worth 1256 # Nearly vacuous, and kept only because a MISSING file would still be worth
1180 # saying out loud: kill -9 leaves the old socket file behind, so this 1257 # saying out loud: kill -9 leaves the old socket file behind, so this
1181 # passes on the dead daemon's leavings. What proves the unix socket serves 1258 # passes on the dead daemon's leavings. What proves the unix socket serves
1182 # again is further down, where `muxd dump` and `muxd stats` answer on it; 1259 # again is further down, where `muxd dump` and `muxd stats` answer on it;
1183 # what proves the daemon serves SESSIONS again is the post-restart marker 1260 # what proves the daemon serves SESSIONS again is the post-restart marker
1184 # the resumed client gets back. 1261 # the resumed client gets back.
1185 [ -S "$SOCK4" ] || { 1262 wait_sock "$SOCK4" "$OUT.q" "restarted --quic daemon never rebound its session socket"
1186 echo "e2e FAIL: restarted --quic daemon never rebound its session socket"
1187 cat "$OUT.q"; exit 1;
1188 }
1189 # The UDP port really came back, and to THIS daemon. Without SO_REUSEADDR a 1263 # The UDP port really came back, and to THIS daemon. Without SO_REUSEADDR a
1190 # bind that collided would have failed loudly instead. 1264 # bind that collided would have failed loudly instead.
1191 # 1265 #
@@ -1259,10 +1333,7 @@ D4PID=""
1259 env MUX_KEY_FILE="$QKEY" "$MUXD" run --sock "$SOCK9" --shell /bin/sh \ 1333 env MUX_KEY_FILE="$QKEY" "$MUXD" run --sock "$SOCK9" --shell /bin/sh \
1260 --quic "127.0.0.1:$QPORT2" --quic-idle-ms 3000 & 1334 --quic "127.0.0.1:$QPORT2" --quic-idle-ms 3000 &
1261 D9PID=$! 1335 D9PID=$!
1262 i=0 1336 wait_sock "$SOCK9" "" "MUX_KEY_FILE daemon never bound its session socket"
1263 while [ ! -S "$SOCK9" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1264 [ -S "$SOCK9" ] || {
1265 echo "e2e FAIL: MUX_KEY_FILE daemon never bound its session socket"; exit 1; }
1266 1337
1267 set +e 1338 set +e
1268 { printf 'printf "envkey-%%s\\n" ok\n'; sleep 2; printf '\034'; } | \ 1339 { printf 'printf "envkey-%%s\\n" ok\n'; sleep 2; printf '\034'; } | \
@@ -1289,10 +1360,7 @@ D9PID=""
1289 env MUX_KEY_FILE="$QKEY.wrong" "$MUXD" run --sock "$SOCK10" --shell /bin/sh \ 1360 env MUX_KEY_FILE="$QKEY.wrong" "$MUXD" run --sock "$SOCK10" --shell /bin/sh \
1290 --quic "127.0.0.1:$QPORT3" --key "$QKEY" --quic-idle-ms 3000 & 1361 --quic "127.0.0.1:$QPORT3" --key "$QKEY" --quic-idle-ms 3000 &
1291 D10PID=$! 1362 D10PID=$!
1292 i=0 1363 wait_sock "$SOCK10" "" "--key-beats-env daemon never bound its session socket"
1293 while [ ! -S "$SOCK10" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1294 [ -S "$SOCK10" ] || {
1295 echo "e2e FAIL: --key-beats-env daemon never bound its session socket"; exit 1; }
1296 1364
1297 set +e 1365 set +e
1298 { printf 'printf "flagwins-%%s\\n" ok\n'; sleep 2; printf '\034'; } | \ 1366 { printf 'printf "flagwins-%%s\\n" ok\n'; sleep 2; printf '\034'; } | \
@@ -1505,9 +1573,7 @@ PDELAY=300
1505 # Own file, not the client capture: the daemon's fd survives the client's truncation and would write into the replayed stream. 1573 # Own file, not the client capture: the daemon's fd survives the client's truncation and would write into the replayed stream.
1506 "$MUXD" run --sock "$SOCK5" --shell /bin/cat > "$OUT.p1.d" 2>&1 & 1574 "$MUXD" run --sock "$SOCK5" --shell /bin/cat > "$OUT.p1.d" 2>&1 &
1507 D5PID=$! 1575 D5PID=$!
1508 i=0 1576 wait_sock "$SOCK5" "$OUT.p1.d" "prediction daemon never bound"
1509 while [ ! -S "$SOCK5" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1510 [ -S "$SOCK5" ] || { echo "e2e FAIL: prediction daemon never bound"; cat "$OUT.p1.d"; exit 1; }
1511 1577
1512 # 1. Line mode: the glyph is on screen before the round trip could have 1578 # 1. Line mode: the glyph is on screen before the round trip could have
1513 # delivered it. 1579 # delivered it.
@@ -1636,9 +1702,7 @@ chmod +x "$PWSH"
1636 1702
1637 "$MUXD" run --sock "$SOCK6" --shell "$PWSH" > "$OUT.pw.d" 2>&1 & 1703 "$MUXD" run --sock "$SOCK6" --shell "$PWSH" > "$OUT.pw.d" 2>&1 &
1638 D6PID=$! 1704 D6PID=$!
1639 i=0 1705 wait_sock "$SOCK6" "$OUT.pw.d" "password daemon never bound"
1640 while [ ! -S "$SOCK6" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1641 [ -S "$SOCK6" ] || { echo "e2e FAIL: password daemon never bound"; cat "$OUT.pw.d"; exit 1; }
1642 1706
1643 # One character per write, and NOT as a single `printf hunter2`. A whole 1707 # One character per write, and NOT as a single `printf hunter2`. A whole
1644 # word in one write reaches the client as a multi-byte chunk, which is 1708 # word in one write reaches the client as a multi-byte chunk, which is
@@ -1685,9 +1749,7 @@ D6PID=""
1685 # 0x00, after which it consumes input and prints nothing. 1749 # 0x00, after which it consumes input and prints nothing.
1686 "$MUXD" run --sock "$SOCK7" --shell "$RAWMODE" > "$OUT.rw.d" 2>&1 & 1750 "$MUXD" run --sock "$SOCK7" --shell "$RAWMODE" > "$OUT.rw.d" 2>&1 &
1687 D7PID=$! 1751 D7PID=$!
1688 i=0 1752 wait_sock "$SOCK7" "$OUT.rw.d" "rawmode daemon never bound"
1689 while [ ! -S "$SOCK7" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1690 [ -S "$SOCK7" ] || { echo "e2e FAIL: rawmode daemon never bound"; cat "$OUT.rw.d"; exit 1; }
1691 1753
1692 set +e 1754 set +e
1693 { sleep 2; for c in a b c; do printf '%s' "$c"; sleep 0.5; done; \ 1755 { sleep 2; for c in a b c; do printf '%s' "$c"; sleep 0.5; done; \
@@ -1736,9 +1798,7 @@ D7PID=""
1736 # that no longer exists. 1798 # that no longer exists.
1737 "$MUXD" run --sock "$SOCK5" --shell /bin/cat > "$OUT.pr.d" 2>&1 & 1799 "$MUXD" run --sock "$SOCK5" --shell /bin/cat > "$OUT.pr.d" 2>&1 &
1738 D5PID=$! 1800 D5PID=$!
1739 i=0 1801 wait_sock "$SOCK5" "$OUT.pr.d" "reconnect daemon never bound"
1740 while [ ! -S "$SOCK5" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1741 [ -S "$SOCK5" ] || { echo "e2e FAIL: reconnect daemon never bound"; cat "$OUT.pr.d"; exit 1; }
1742 1802
1743 set +e 1803 set +e
1744 { sleep 2; printf 'p'; sleep 5; printf 'q'; sleep 4; printf '\034'; } | \ 1804 { sleep 2; printf 'p'; sleep 5; printf 'q'; sleep 4; printf '\034'; } | \
@@ -1887,9 +1947,7 @@ ok "ptyclient controls: pty echo roundtrips, impossible expect fails loudly, std
1887 # answering the resize reproduces the OLD geometry. 1947 # answering the resize reproduces the OLD geometry.
1888 "$MUXD" run --sock "$SOCK12" --shell /bin/sh > "$OUT.tp2.d" 2>&1 & 1948 "$MUXD" run --sock "$SOCK12" --shell /bin/sh > "$OUT.tp2.d" 2>&1 &
1889 D12PID=$! 1949 D12PID=$!
1890 i=0 1950 wait_sock "$SOCK12" "$OUT.tp2.d" "tp2 daemon never bound"
1891 while [ ! -S "$SOCK12" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
1892 [ -S "$SOCK12" ] || { echo "e2e FAIL: tp2 daemon never bound"; cat "$OUT.tp2.d"; exit 1; }
1893 1951
1894 # Two rules govern every script below, and breaking either one produces a 1952 # Two rules govern every script below, and breaking either one produces a
1895 # capture that is honestly SHORT rather than wrong — a divergence whose 1953 # capture that is honestly SHORT rather than wrong — a divergence whose
@@ -2051,9 +2109,7 @@ EOF
2051 chmod +x "$TP1SH" 2109 chmod +x "$TP1SH"
2052 "$MUXD" run --sock "$SOCK13" --shell "$TP1SH" > "$OUT.tp1.d" 2>&1 & 2110 "$MUXD" run --sock "$SOCK13" --shell "$TP1SH" > "$OUT.tp1.d" 2>&1 &
2053 D13PID=$! 2111 D13PID=$!
2054 i=0 2112 wait_sock "$SOCK13" "$OUT.tp1.d" "tp1 daemon never bound"
2055 while [ ! -S "$SOCK13" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
2056 [ -S "$SOCK13" ] || { echo "e2e FAIL: tp1 daemon never bound"; cat "$OUT.tp1.d"; exit 1; }
2057 # Attach only after seq finished: the client must be served a snapshot of 2113 # Attach only after seq finished: the client must be served a snapshot of
2058 # the TAIL, so the scrollback page it fetches later is content it has never 2114 # the TAIL, so the scrollback page it fetches later is content it has never
2059 # been sent. Observed at 80x24: the attach snapshot carries rows 78..100 and 2115 # been sent. Observed at 80x24: the attach snapshot carries rows 78..100 and
@@ -2669,9 +2725,7 @@ chmod 600 "$HKEY"
2669 "$MUXD" run --sock "$SOCK17" --shell /bin/sh \ 2725 "$MUXD" run --sock "$SOCK17" --shell /bin/sh \
2670 --quic "127.0.0.1:$HQPORT" --key "$HKEY" & 2726 --quic "127.0.0.1:$HQPORT" --key "$HKEY" &
2671 HDPID=$! 2727 HDPID=$!
2672 i=0 2728 wait_sock "$SOCK17" "" "key-mismatch daemon never bound"
2673 while [ ! -S "$SOCK17" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
2674 [ -S "$SOCK17" ] || { echo "e2e FAIL: key-mismatch daemon never bound"; exit 1; }
2675 HSHIMS_D=$(wc -l < "$SSHIM_PIDLOG") 2729 HSHIMS_D=$(wc -l < "$SSHIM_PIDLOG")
2676 HT0=$(date +%s%N) 2730 HT0=$(date +%s%N)
2677 { printf 'printf "fallback-%%s\\n" ok\n'; sleep 6; printf '\034'; } | \ 2731 { printf 'printf "fallback-%%s\\n" ok\n'; sleep 6; printf '\034'; } | \
@@ -2825,38 +2879,10 @@ assert_converged "$OUT.h5" "$SOCK16" "announce-none stays on ssh"
2825 ok "announce-none stays on ssh: no deadline paid (${HMS_E}ms), no fallback line" 2879 ok "announce-none stays on ssh: no deadline paid (${HMS_E}ms), no fallback line"
2826 2880
2827 # Both daemons down by the sanctioned verb, and OBSERVED dead by pid — the 2881 # Both daemons down by the sanctioned verb, and OBSERVED dead by pid — the
2828 # M13 teardown shape, including the labelled RC capture: a stop that fails 2882 # M13 teardown shape, all four assertions of it (see assert_stopped).
2829 # exits 1, which under `set -e` would abort with no line of its own and the 2883 assert_stopped "$SOCK16" "$HAPID" "handoff daemon" "$OUT.stop"
2830 # trap would then remove the evidence.
2831 set +e
2832 "$MUXD" stop --sock "$SOCK16" 2> "$OUT.stop"
2833 RC_STOP=$?
2834 set -e
2835 [ "$RC_STOP" = "0" ] || {
2836 echo "e2e FAIL: handoff daemon stop exited $RC_STOP, want 0"; cat "$OUT.stop"; exit 1; }
2837 grep -q '^muxd: stopped' "$OUT.stop" || {
2838 echo "e2e FAIL: handoff daemon stop did not report stopped"; cat "$OUT.stop"; exit 1; }
2839 [ ! -S "$SOCK16" ] || {
2840 echo "e2e FAIL: stop left the handoff daemon's socket"; ls -l "$SOCK16"; exit 1; }
2841 wait_pid_gone "$HAPID" "handoff daemon: stop reported stopped"
2842 HAPID="" 2884 HAPID=""
2843 set +e 2885 assert_stopped "$SOCK17" "$HDPID" "key-mismatch daemon" "$OUT.stop"
2844 "$MUXD" stop --sock "$SOCK17" 2> "$OUT.stop"
2845 RC_STOP=$?
2846 set -e
2847 [ "$RC_STOP" = "0" ] || {
2848 echo "e2e FAIL: key-mismatch daemon stop exited $RC_STOP, want 0"; cat "$OUT.stop"; exit 1; }
2849 # The same three assertions the SOCK16 leg makes, and for the same reason:
2850 # an exit code says the command returned, not that it did the job. The
2851 # stopped line is the daemon's own account of it and the absent socket is
2852 # the filesystem's — a stop that exited 0 while leaving either behind is
2853 # the regression this shape exists to catch, and it would be invisible
2854 # here without them.
2855 grep -q '^muxd: stopped' "$OUT.stop" || {
2856 echo "e2e FAIL: key-mismatch daemon stop did not report stopped"; cat "$OUT.stop"; exit 1; }
2857 [ ! -S "$SOCK17" ] || {
2858 echo "e2e FAIL: stop left the key-mismatch daemon's socket"; ls -l "$SOCK17"; exit 1; }
2859 wait_pid_gone "$HDPID" "key-mismatch daemon: stop reported stopped"
2860 HDPID="" 2886 HDPID=""
2861 rm -f "$OUT.h1" "$OUT.h1.err" "$OUT.h2" "$OUT.h2.err" "$OUT.h3" "$OUT.h3.err" \ 2887 rm -f "$OUT.h1" "$OUT.h1.err" "$OUT.h2" "$OUT.h2.err" "$OUT.h3" "$OUT.h3.err" \
2862 "$OUT.h4" "$OUT.h4.err" "$OUT.h5" "$OUT.h5.err" "$OUT.stop" "$HKEY" "$HCFGBAD" 2888 "$OUT.h4" "$OUT.h4.err" "$OUT.h5" "$OUT.h5.err" "$OUT.stop" "$HKEY" "$HCFGBAD"
@@ -2872,9 +2898,7 @@ rm -rf "$SSHIM_DIR" "$HRUN" "$HRUN2"
2872 # own row, so grep itself is the geometry assertion. 2898 # own row, so grep itself is the geometry assertion.
2873 "$MUXD" run --sock "$SOCK18" --shell /bin/sh > "$OUT.weba.d" 2>&1 & 2899 "$MUXD" run --sock "$SOCK18" --shell /bin/sh > "$OUT.weba.d" 2>&1 &
2874 D14PID=$! 2900 D14PID=$!
2875 i=0 2901 wait_sock "$SOCK18" "$OUT.weba.d" "web passivity daemon never bound"
2876 while [ ! -S "$SOCK18" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
2877 [ -S "$SOCK18" ] || { echo "e2e FAIL: web passivity daemon never bound"; cat "$OUT.weba.d"; exit 1; }
2878 2902
2879 # The 80x24 client: types the first marker, holds the session open long 2903 # The 80x24 client: types the first marker, holds the session open long
2880 # enough for the 1x1 attacher to come and go, types the second marker 2904 # enough for the 1x1 attacher to come and go, types the second marker
@@ -2882,13 +2906,7 @@ while [ ! -S "$SOCK18" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
2882 { printf 'printf "wall-%%s\\n" pin\n'; sleep 4; printf 'printf "zz-%%s\\n" web\n'; sleep 2; printf '\034'; } | \ 2906 { printf 'printf "wall-%%s\\n" pin\n'; sleep 4; printf 'printf "zz-%%s\\n" web\n'; sleep 2; printf '\034'; } | \
2883 timeout 40 "$MUX" --sock "$SOCK18" > "$OUT.weba" 2> "$OUT.weba.err" & 2907 timeout 40 "$MUX" --sock "$SOCK18" > "$OUT.weba" 2> "$OUT.weba.err" &
2884 WCLIPID=$! 2908 WCLIPID=$!
2885 i=0 2909 wait_grid "$SOCK18" "wall-pin" "web passivity: the first marker"
2886 while [ "$i" -lt 100 ]; do
2887 "$MUXD" dump --sock "$SOCK18" 2>/dev/null | grep -q "wall-pin" && break
2888 sleep 0.1; i=$((i+1))
2889 done
2890 "$MUXD" dump --sock "$SOCK18" | grep -q "wall-pin" || {
2891 echo "e2e FAIL: web passivity: first marker never landed"; exit 1; }
2892 2910
2893 # The 1x1 attacher, on a real 1x1 pty. Its attach is answered with a 2911 # The 1x1 attacher, on a real 1x1 pty. Its attach is answered with a
2894 # unicast snapshot (the alt-screen enter proves the first frame came), 2912 # unicast snapshot (the alt-screen enter proves the first frame came),
@@ -2928,23 +2946,14 @@ WCLIPID=""
2928 [ "$RC" -eq 0 ] || { echo "e2e FAIL: web passivity: 80x24 client exited $RC"; cat "$OUT.weba.err"; exit 1; } 2946 [ "$RC" -eq 0 ] || { echo "e2e FAIL: web passivity: 80x24 client exited $RC"; cat "$OUT.weba.err"; exit 1; }
2929 assert_converged "$OUT.weba" "$SOCK18" "web passivity: the 80x24 client never glitched" 2947 assert_converged "$OUT.weba" "$SOCK18" "web passivity: the 80x24 client never glitched"
2930 2948
2931 set +e 2949 assert_stopped "$SOCK18" "$D14PID" "web passivity" "$OUT.webstop"
2932 "$MUXD" stop --sock "$SOCK18" 2> "$OUT.webstop"
2933 RC_STOP=$?
2934 set -e
2935 [ "$RC_STOP" = "0" ] || { echo "e2e FAIL: web passivity stop exited $RC_STOP"; cat "$OUT.webstop"; exit 1; }
2936 grep -q '^muxd: stopped' "$OUT.webstop" || { echo "e2e FAIL: web passivity stop did not report stopped"; cat "$OUT.webstop"; exit 1; }
2937 [ ! -S "$SOCK18" ] || { echo "e2e FAIL: web passivity stop left the socket"; ls -l "$SOCK18"; exit 1; }
2938 wait_pid_gone "$D14PID" "web passivity: stop reported stopped"
2939 D14PID="" 2950 D14PID=""
2940 ok "a 1x1 attach is refused the grid and can never claim it" 2951 ok "a 1x1 attach is refused the grid and can never claim it"
2941 2952
2942 # --- M-web (b): the hub pumps a real session; wrong Origin refused. 2953 # --- M-web (b): the hub pumps a real session; wrong Origin refused.
2943 "$MUXD" run --sock "$SOCK19" --shell /bin/sh > "$OUT.webb.d" 2>&1 & 2954 "$MUXD" run --sock "$SOCK19" --shell /bin/sh > "$OUT.webb.d" 2>&1 &
2944 D15PID=$! 2955 D15PID=$!
2945 i=0 2956 wait_sock "$SOCK19" "$OUT.webb.d" "web hub daemon never bound"
2946 while [ ! -S "$SOCK19" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
2947 [ -S "$SOCK19" ] || { echo "e2e FAIL: web hub daemon never bound"; cat "$OUT.webb.d"; exit 1; }
2948 "$MUXWEB" --sock "$SOCK19" --port "$WPORT" > "$OUT.webh" 2>&1 & 2957 "$MUXWEB" --sock "$SOCK19" --port "$WPORT" > "$OUT.webh" 2>&1 &
2949 W1PID=$! 2958 W1PID=$!
2950 wait_for "$OUT.webh" "serving" 10 || { 2959 wait_for "$OUT.webh" "serving" 10 || {
@@ -2953,11 +2962,7 @@ wait_for "$OUT.webh" "serving" 10 || {
2953 { printf 'printf "web-%%s\\n" b1\n'; sleep 4; printf '\034'; } | \ 2962 { printf 'printf "web-%%s\\n" b1\n'; sleep 4; printf '\034'; } | \
2954 timeout 40 "$MUX" --sock "$SOCK19" > "$OUT.webb" 2> "$OUT.webb.err" & 2963 timeout 40 "$MUX" --sock "$SOCK19" > "$OUT.webb" 2> "$OUT.webb.err" &
2955 WCLIPID=$! 2964 WCLIPID=$!
2956 i=0 2965 wait_grid "$SOCK19" "web-b1" "web hub: the typing client's marker"
2957 while [ "$i" -lt 100 ]; do
2958 "$MUXD" dump --sock "$SOCK19" 2>/dev/null | grep -q "web-b1" && break
2959 sleep 0.1; i=$((i+1))
2960 done
2961 2966
2962 # The browser stand-in: a passive 1x1 wall tile through the hub. Its 2967 # The browser stand-in: a passive 1x1 wall tile through the hub. Its
2963 # final grid is the daemon's replica as REPLAYED THROUGH the WebSocket 2968 # final grid is the daemon's replica as REPLAYED THROUGH the WebSocket
@@ -2976,21 +2981,7 @@ set -e
2976 echo "e2e FAIL: web hub: wsclient exited $RC" 2981 echo "e2e FAIL: web hub: wsclient exited $RC"
2977 cat -v "$OUT.webws.err" 2>/dev/null; cat "$OUT.webh"; exit 1; } 2982 cat -v "$OUT.webws.err" 2>/dev/null; cat "$OUT.webh"; exit 1; }
2978 2983
2979 "$MUXD" dump --sock "$SOCK19" > "$OUT.webws.dump" 2984 assert_ws_converged "$OUT.webws" "$SOCK19" "web hub"
2980 sed 's/[[:space:]]*$//' "$OUT.webws" > "$OUT.webws.n"
2981 sed 's/[[:space:]]*$//' "$OUT.webws.dump" > "$OUT.webws.dump.n"
2982 diff -u "$OUT.webws.dump.n" "$OUT.webws.n" > "$OUT.webws.diff" || {
2983 echo "e2e FAIL: web hub: wsclient replica diverges from daemon grid (-daemon +ws):"
2984 head -40 "$OUT.webws.diff"; exit 1; }
2985
2986 # The cannot-fail control (the wan.sh rule, M9): doctor a copy of the
2987 # daemon dump and assert the SAME diff catches it. A convergence check
2988 # that cannot fail proves nothing.
2989 cp "$OUT.webws.dump.n" "$OUT.webws.doc"
2990 printf 'doctored-row\n' >> "$OUT.webws.doc"
2991 if diff -u "$OUT.webws.doc" "$OUT.webws.n" > /dev/null 2>&1; then
2992 echo "e2e FAIL: web hub: the convergence diff cannot fail (doctored dump passed)"; exit 1
2993 fi
2994 2985
2995 # Wrong Origin: refused at HTTP, before any upgrade — the wsclient sees 2986 # Wrong Origin: refused at HTTP, before any upgrade — the wsclient sees
2996 # a non-101 and exits 4 — and the daemon never sees a client for it. 2987 # a non-101 and exits 4 — and the daemon never sees a client for it.
@@ -3021,14 +3012,7 @@ assert_converged "$OUT.webb" "$SOCK19" "web hub: the typing client"
3021 kill "$W1PID" 2>/dev/null || true 3012 kill "$W1PID" 2>/dev/null || true
3022 wait_pid_gone "$W1PID" "web hub: killed by tracked pid" 3013 wait_pid_gone "$W1PID" "web hub: killed by tracked pid"
3023 W1PID="" 3014 W1PID=""
3024 set +e 3015 assert_stopped "$SOCK19" "$D15PID" "web hub" "$OUT.webstop2"
3025 "$MUXD" stop --sock "$SOCK19" 2> "$OUT.webstop2"
3026 RC_STOP=$?
3027 set -e
3028 [ "$RC_STOP" = "0" ] || { echo "e2e FAIL: web hub stop exited $RC_STOP"; cat "$OUT.webstop2"; exit 1; }
3029 grep -q '^muxd: stopped' "$OUT.webstop2" || { echo "e2e FAIL: web hub stop did not report stopped"; cat "$OUT.webstop2"; exit 1; }
3030 [ ! -S "$SOCK19" ] || { echo "e2e FAIL: web hub stop left the socket"; ls -l "$SOCK19"; exit 1; }
3031 wait_pid_gone "$D15PID" "web hub: stop reported stopped"
3032 D15PID="" 3016 D15PID=""
3033 ok "hub pumps a real session; wrong origin refused" 3017 ok "hub pumps a real session; wrong origin refused"
3034 3018
@@ -3040,9 +3024,7 @@ ok "hub pumps a real session; wrong origin refused"
3040 # that converges on the new session's content. 3024 # that converges on the new session's content.
3041 "$MUXD" run --sock "$SOCK20" --shell /bin/sh > "$OUT.webc.d" 2>&1 & 3025 "$MUXD" run --sock "$SOCK20" --shell /bin/sh > "$OUT.webc.d" 2>&1 &
3042 D16PID=$! 3026 D16PID=$!
3043 i=0 3027 wait_sock "$SOCK20" "$OUT.webc.d" "web tear daemon never bound"
3044 while [ ! -S "$SOCK20" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
3045 [ -S "$SOCK20" ] || { echo "e2e FAIL: web tear daemon never bound"; cat "$OUT.webc.d"; exit 1; }
3046 "$MUXWEB" --sock "$SOCK20" --port "$WPORT2" > "$OUT.webh2" 2>&1 & 3028 "$MUXWEB" --sock "$SOCK20" --port "$WPORT2" > "$OUT.webh2" 2>&1 &
3047 W2PID=$! 3029 W2PID=$!
3048 wait_for "$OUT.webh2" "serving" 10 || { 3030 wait_for "$OUT.webh2" "serving" 10 || {
@@ -3050,11 +3032,7 @@ wait_for "$OUT.webh2" "serving" 10 || {
3050 3032
3051 { printf 'printf "web-%%s\\n" c1\n'; sleep 1.5; printf '\034'; } | \ 3033 { printf 'printf "web-%%s\\n" c1\n'; sleep 1.5; printf '\034'; } | \
3052 timeout 30 "$MUX" --sock "$SOCK20" > "$OUT.webc" 2> "$OUT.webc.err" 3034 timeout 30 "$MUX" --sock "$SOCK20" > "$OUT.webc" 2> "$OUT.webc.err"
3053 i=0 3035 wait_grid "$SOCK20" "web-c1" "web tear: the first session's marker"
3054 while [ "$i" -lt 100 ]; do
3055 "$MUXD" dump --sock "$SOCK20" 2>/dev/null | grep -q "web-c1" && break
3056 sleep 0.1; i=$((i+1))
3057 done
3058 3036
3059 timeout 90 "$WSCLIENT" --port "$WPORT2" --tile 0 --out "$OUT.webws2" --err "$OUT.webws2.err" <<'EOF' & 3037 timeout 90 "$WSCLIENT" --port "$WPORT2" --tile 0 --out "$OUT.webws2" --err "$OUT.webws2.err" <<'EOF' &
3060 attach 1 1 3038 attach 1 1
@@ -3072,19 +3050,12 @@ WCLIPID=$!
3072 # it waits for is already on the grid, so one settle-length is plenty. 3050 # it waits for is already on the grid, so one settle-length is plenty.
3073 sleep 1 3051 sleep 1
3074 3052
3075 set +e 3053 assert_stopped "$SOCK20" "$D16PID" "web tear: the first daemon, under a live tile" "$OUT.webstop3"
3076 "$MUXD" stop --sock "$SOCK20" 2> "$OUT.webstop3"
3077 RC_STOP=$?
3078 set -e
3079 [ "$RC_STOP" = "0" ] || { echo "e2e FAIL: web tear stop exited $RC_STOP"; cat "$OUT.webstop3"; exit 1; }
3080 wait_pid_gone "$D16PID" "web tear: first daemon stopped under the tile"
3081 D16PID="" 3054 D16PID=""
3082 3055
3083 "$MUXD" run --sock "$SOCK20" --shell /bin/sh > "$OUT.webc2.d" 2>&1 & 3056 "$MUXD" run --sock "$SOCK20" --shell /bin/sh > "$OUT.webc2.d" 2>&1 &
3084 D17PID=$! 3057 D17PID=$!
3085 i=0 3058 wait_sock "$SOCK20" "$OUT.webc2.d" "web tear restart never bound"
3086 while [ ! -S "$SOCK20" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
3087 [ -S "$SOCK20" ] || { echo "e2e FAIL: web tear restart never bound"; cat "$OUT.webc2.d"; exit 1; }
3088 3059
3089 { printf 'printf "web-%%s\\n" c2\n'; sleep 1.5; printf '\034'; } | \ 3060 { printf 'printf "web-%%s\\n" c2\n'; sleep 1.5; printf '\034'; } | \
3090 timeout 30 "$MUX" --sock "$SOCK20" > "$OUT.webc2" 2> "$OUT.webc2.err" 3061 timeout 30 "$MUX" --sock "$SOCK20" > "$OUT.webc2" 2> "$OUT.webc2.err"
@@ -3098,23 +3069,15 @@ WCLIPID=""
3098 echo "e2e FAIL: web tear: wsclient exited $RC" 3069 echo "e2e FAIL: web tear: wsclient exited $RC"
3099 cat -v "$OUT.webws2.err" 2>/dev/null; cat "$OUT.webh2"; exit 1; } 3070 cat -v "$OUT.webws2.err" 2>/dev/null; cat "$OUT.webh2"; exit 1; }
3100 3071
3101 "$MUXD" dump --sock "$SOCK20" > "$OUT.webws2.dump" 3072 assert_ws_converged "$OUT.webws2" "$SOCK20" "web tear: after the epoch crossing"
3102 sed 's/[[:space:]]*$//' "$OUT.webws2" > "$OUT.webws2.n"
3103 sed 's/[[:space:]]*$//' "$OUT.webws2.dump" > "$OUT.webws2.dump.n"
3104 diff -u "$OUT.webws2.dump.n" "$OUT.webws2.n" > "$OUT.webws2.diff" || {
3105 echo "e2e FAIL: web tear: replica diverges after the epoch crossing (-daemon +ws):"
3106 head -40 "$OUT.webws2.diff"; exit 1; }
3107 3073
3108 kill "$W2PID" 2>/dev/null || true 3074 kill "$W2PID" 2>/dev/null || true
3109 wait_pid_gone "$W2PID" "web tear: hub killed by tracked pid" 3075 wait_pid_gone "$W2PID" "web tear: hub killed by tracked pid"
3110 W2PID="" 3076 W2PID=""
3111 set +e 3077 # Its own capture, not a second write to $OUT.webstop3: that file still
3112 "$MUXD" stop --sock "$SOCK20" 2> "$OUT.webstop3" 3078 # holds the FIRST stop's stderr, and overwriting it before anything reads
3113 RC_STOP=$? 3079 # it is how the evidence for a failure up there disappears.
3114 set -e 3080 assert_stopped "$SOCK20" "$D17PID" "web tear: the restarted daemon" "$OUT.webstop4"
3115 [ "$RC_STOP" = "0" ] || { echo "e2e FAIL: web tear final stop exited $RC_STOP"; cat "$OUT.webstop3"; exit 1; }
3116 [ ! -S "$SOCK20" ] || { echo "e2e FAIL: web tear final stop left the socket"; ls -l "$SOCK20"; exit 1; }
3117 wait_pid_gone "$D17PID" "web tear: restarted daemon stopped"
3118 D17PID="" 3081 D17PID=""
3119 ok "hub narrates the tear; the replica re-attaches across an epoch" 3082 ok "hub narrates the tear; the replica re-attaches across an epoch"
3120 3083