a73x

d0550135

test: e2e — ssh shim + five handoff scenarios; pins 15->20 / 28->33

a73x   2026-08-11 15:44

Commit message
test: e2e — ssh shim + five handoff scenarios; pins 15->20 / 28->33

The M14 handoff shipped with no e2e coverage: five real mutations survived
the whole suite. These scenarios kill them.

An ssh shim on PATH drops the HOST argument and execs the rest locally, so
`ssh HOST muxd endpoint` runs the real binary against a socket of the
suite's own making — the client's handoff code runs end to end with no
network. Presence or absence of ssh is read off the shim's own pid log,
never `ps | grep ssh`, which would count the operator's sessions.

  cold        ssh fetches, QUIC carries: the shim pid is OBSERVED dead
              while a second marker still round-trips, and the cached port
              is checked against /proc/net/udp rather than believed
  warm        cache hit dials QUIC with the pid log unmoved, stderr silent
  stale       a poisoned cache self-heals in one refetch with no fallback
              line — the control that proves the fallback grep can fail is
              scenario (d), which asserts the same pattern present
  mismatch    a key the daemon does not hold: one deadline (bounded on
              both sides), one line, session over the pipe
  none        a config home that cannot yield a key: `endpoint none`, no
              deadline paid, no fallback line

Two things the scenarios had to be built around, both measured rather than
assumed. `waitReady` answers Ctrl-\ during a handshake, so a detach byte
landing inside the 2s dial aborts the attach and leaves a scenario passing
having tested nothing — hence six-second scripts where a dial is expected.
And the poisoned port is drawn from BELOW the ephemeral range: the daemon's
lazy bind takes a kernel-assigned port, and a collision would make the
"dead" port live.

wait_pid_gone's message drops its stop-specific wording; the two call sites
that meant "stop said it stopped" now say so in their labels, and the
handoff's caller means "QUIC said it had taken over".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test/e2e.sh
Old New
@@ -18,7 +18,11 @@ OUT="${TMPDIR:-/tmp}/mux-e2e-out-$$"
18 # never the developer's real ~/.config/mux/key. 18 # never the developer's real ~/.config/mux/key.
19 XDG_CONFIG_HOME="${TMPDIR:-/tmp}/mux-e2e-cfg-$$" 19 XDG_CONFIG_HOME="${TMPDIR:-/tmp}/mux-e2e-cfg-$$"
20 XDG_STATE_HOME="${TMPDIR:-/tmp}/mux-e2e-state-$$" 20 XDG_STATE_HOME="${TMPDIR:-/tmp}/mux-e2e-state-$$"
21 export XDG_CONFIG_HOME XDG_STATE_HOME 21 # M14: and the same for the handoff's per-host cache. It holds a KEY, and
22 # the M14 scenarios both read and poison it — neither of which may ever
23 # touch the developer's real ~/.cache/mux.
24 XDG_CACHE_HOME="${TMPDIR:-/tmp}/mux-e2e-cache-$$"
25 export XDG_CONFIG_HOME XDG_STATE_HOME XDG_CACHE_HOME
22 # Second daemon, used only by the M7 abort scenario; declared here so the 26 # Second daemon, used only by the M7 abort scenario; declared here so the
23 # trap below can reference them under `set -u` before they are ever started. 27 # trap below can reference them under `set -u` before they are ever started.
24 SOCK2="${TMPDIR:-/tmp}/muxd-e2e-abort-$$.sock" 28 SOCK2="${TMPDIR:-/tmp}/muxd-e2e-abort-$$.sock"
@@ -85,6 +89,35 @@ SOCK14="${TMPDIR:-/tmp}/muxd-e2e-astart-$$.sock"
85 SOCK15="${TMPDIR:-/tmp}/muxd-e2e-aspty-$$.sock" 89 SOCK15="${TMPDIR:-/tmp}/muxd-e2e-aspty-$$.sock"
86 APID="" 90 APID=""
87 PAPID="" 91 PAPID=""
92 # M14 ssh→QUIC handoff. The shim that stands in for ssh (built where the
93 # scenarios run, declared here so the trap can reach it), and a RUNTIME DIR
94 # per daemon: the handoff dials the DEFAULT socket on the far side, and the
95 # far side is this box, so each scenario's default socket has to be one of
96 # ours rather than the operator's. Two daemons, no more — the one a cold
97 # attach auto-starts, and the key-mismatch one that cannot be shared
98 # because it holds a key nothing else on this box has.
99 SSHIM_DIR="${TMPDIR:-/tmp}/muxd-e2e-sshim-$$"
100 SSHIM_PIDLOG="$SSHIM_DIR/pids"
101 export SSHIM_PIDLOG
102 HRUN="${TMPDIR:-/tmp}/mux-e2e-hrun-$$"
103 HRUN2="${TMPDIR:-/tmp}/mux-e2e-hrun2-$$"
104 SOCK16="$HRUN/muxd.sock"
105 SOCK17="$HRUN2/muxd.sock"
106 HKEY="${TMPDIR:-/tmp}/mux-e2e-hkey-$$"
107 # A config home that is a FILE: `muxd endpoint` cannot create a key under
108 # it and cannot find one, which is the announce-none scenario's lever.
109 HCFGBAD="${TMPDIR:-/tmp}/mux-e2e-hnokey-$$"
110 # Bands of their own, like every other port here, so a concurrent suite
111 # cannot collide — and BELOW the ephemeral range (32768–60999 here), which
112 # for HDEADPORT is not housekeeping but the scenario's correctness. It is
113 # the port the poisoned cache points at, and the daemon it is poisoning
114 # against holds a KERNEL-ASSIGNED ephemeral port: land on that one and the
115 # "dead" port is live, the dial succeeds, and the self-heal scenario
116 # quietly asserts a heal that never had to happen.
117 HQPORT=$(( 11000 + ($$ % 4000) ))
118 HDEADPORT=$(( 16000 + ($$ % 4000) ))
119 HAPID=""
120 HDPID=""
88 121
89 # One counter out of a MUX_PREDICT_STATS line. The client prints exactly one 122 # One counter out of a MUX_PREDICT_STATS line. The client prints exactly one
90 # such line on exit; every field is a key=value pair, so a rename or reorder 123 # such line on exit; every field is a key=value pair, so a rename or reorder
@@ -154,11 +187,17 @@ wait_gone() {
154 # being unlinked, and the process can still be a fraction behind that. Only 187 # being unlinked, and the process can still be a fraction behind that. Only
155 # a pid can say the daemon itself ended, which is the assertion the M13 188 # a pid can say the daemon itself ended, which is the assertion the M13
156 # teardowns owe — never the stop command's own claim. 189 # teardowns owe — never the stop command's own claim.
190 #
191 # M14 gave it a second kind of caller: the cold-handoff scenario asks the
192 # same question of the ssh the client is supposed to have killed once QUIC
193 # took over. So the message names the pid and leaves the EXPECTATION to
194 # each call site's label — the two are "stop said it stopped" and "QUIC
195 # said it had taken over", and one wording cannot honestly claim both.
157 wait_pid_gone() { 196 wait_pid_gone() {
158 _i=0 197 _i=0
159 while kill -0 "$1" 2>/dev/null; do 198 while kill -0 "$1" 2>/dev/null; do
160 _i=$((_i + 1)); [ "$_i" -lt 40 ] || { 199 _i=$((_i + 1)); [ "$_i" -lt 40 ] || {
161 echo "e2e FAIL: $2: stop reported stopped but pid $1 still runs"; exit 1; } 200 echo "e2e FAIL: $2: pid $1 is still running 2s later"; exit 1; }
162 sleep 0.05 201 sleep 0.05
163 done 202 done
164 } 203 }
@@ -290,6 +329,16 @@ cleanup() {
290 # `muxd stop` on a path nobody serves is a no-op that exits 0. 329 # `muxd stop` on a path nobody serves is a no-op that exits 0.
291 [ -n "${MUXD:-}" ] && "$MUXD" stop --sock "$SOCK14" >/dev/null 2>&1 || true 330 [ -n "${MUXD:-}" ] && "$MUXD" stop --sock "$SOCK14" >/dev/null 2>&1 || true
292 [ -n "${MUXD:-}" ] && "$MUXD" stop --sock "$SOCK15" >/dev/null 2>&1 || true 331 [ -n "${MUXD:-}" ] && "$MUXD" stop --sock "$SOCK15" >/dev/null 2>&1 || true
332 # M14's two, the same way and for the same reason: the cold-handoff
333 # daemon is spawned by `muxd endpoint` and is nobody's child here, so
334 # its pid is read off the up-line and the socket stop backstops the
335 # window before that read. The stops come BEFORE the rm -rf below —
336 # unlinking a runtime dir first would leave a live daemon this trap
337 # could no longer reach by path.
338 [ -n "$HDPID" ] && kill "$HDPID" 2>/dev/null || true
339 [ -n "$HAPID" ] && kill "$HAPID" 2>/dev/null || true
340 [ -n "${MUXD:-}" ] && "$MUXD" stop --sock "$SOCK16" >/dev/null 2>&1 || true
341 [ -n "${MUXD:-}" ] && "$MUXD" stop --sock "$SOCK17" >/dev/null 2>&1 || true
293 rm -f "$SOCK8" "$SOCK8T" "$SOCK11" "$OUT.start" "$OUT.start2" "$OUT.s8" \ 342 rm -f "$SOCK8" "$SOCK8T" "$SOCK11" "$OUT.start" "$OUT.start2" "$OUT.s8" \
294 "$OUT.ra" "$OUT.rb" "$OUT.goal" "$OUT.g9" "$OUT.dead" \ 343 "$OUT.ra" "$OUT.rb" "$OUT.goal" "$OUT.g9" "$OUT.dead" \
295 "$SOCK9" "$SOCK10" "$OUT.env1" "$OUT.env2" \ 344 "$SOCK9" "$SOCK10" "$OUT.env1" "$OUT.env2" \
@@ -328,10 +377,17 @@ cleanup() {
328 "$OUT.as" "$OUT.as.err" "$OUT.as2" "$OUT.as2.err" \ 377 "$OUT.as" "$OUT.as.err" "$OUT.as2" "$OUT.as2.err" \
329 "$OUT.stop" "$OUT.stop2" \ 378 "$OUT.stop" "$OUT.stop2" \
330 "$OUT.pa" "$OUT.pa.err" "$OUT.pa.log" 379 "$OUT.pa" "$OUT.pa.err" "$OUT.pa.log"
380 # M14 handoff. The shim, both runtime dirs (each holding its daemon's
381 # socket), the second key, the unusable config home, and the five
382 # captures.
383 rm -f "$HKEY" "$HCFGBAD" \
384 "$OUT.h1" "$OUT.h1.err" "$OUT.h2" "$OUT.h2.err" "$OUT.h3" "$OUT.h3.err" \
385 "$OUT.h4" "$OUT.h4.err" "$OUT.h5" "$OUT.h5.err"
386 rm -rf "$SSHIM_DIR" "$HRUN" "$HRUN2"
331 # The convergence files a FAILING assert_converged leaves behind 387 # The convergence files a FAILING assert_converged leaves behind
332 # (.render/.dump/.rvt/.dvt/.diff for that capture) are deliberately not 388 # (.render/.dump/.rvt/.dvt/.diff for that capture) are deliberately not
333 # chased here: on a failing run they are the evidence. 389 # chased here: on a failing run they are the evidence.
334 rm -rf "$XDG_CONFIG_HOME" "$XDG_STATE_HOME" "${NOKEY_CFG:-}" 390 rm -rf "$XDG_CONFIG_HOME" "$XDG_STATE_HOME" "$XDG_CACHE_HOME" "${NOKEY_CFG:-}"
335 } 391 }
336 trap cleanup EXIT INT TERM 392 trap cleanup EXIT INT TERM
337 393
@@ -2098,7 +2154,7 @@ grep -q '^muxd: stopped' "$OUT.stop" || {
2098 echo "e2e FAIL: stop did not report stopped"; cat "$OUT.stop"; exit 1; } 2154 echo "e2e FAIL: stop did not report stopped"; cat "$OUT.stop"; exit 1; }
2099 [ ! -S "$SOCK14" ] || { 2155 [ ! -S "$SOCK14" ] || {
2100 echo "e2e FAIL: stop left the socket behind"; ls -l "$SOCK14"; exit 1; } 2156 echo "e2e FAIL: stop left the socket behind"; ls -l "$SOCK14"; exit 1; }
2101 wait_pid_gone "$APID" "proxy auto-start" 2157 wait_pid_gone "$APID" "proxy auto-start: stop reported stopped"
2102 APID="" 2158 APID=""
2103 2159
2104 # Idempotence control: stop with nothing there is exit 0 and says so. 2160 # Idempotence control: stop with nothing there is exit 0 and says so.
@@ -2172,22 +2228,396 @@ grep -q '^muxd: stopped' "$OUT.stop" || {
2172 echo "e2e FAIL: pty leg stop did not report stopped"; cat "$OUT.stop"; exit 1; } 2228 echo "e2e FAIL: pty leg stop did not report stopped"; cat "$OUT.stop"; exit 1; }
2173 [ ! -S "$SOCK15" ] || { 2229 [ ! -S "$SOCK15" ] || {
2174 echo "e2e FAIL: stop left the pty leg's socket"; ls -l "$SOCK15"; exit 1; } 2230 echo "e2e FAIL: stop left the pty leg's socket"; ls -l "$SOCK15"; exit 1; }
2175 wait_pid_gone "$PAPID" "pty leg" 2231 wait_pid_gone "$PAPID" "pty leg: stop reported stopped"
2176 PAPID="" 2232 PAPID=""
2177 rm -f "$OUT.pa" "$OUT.pa.err" "$OUT.pa.log" "$OUT.stop" 2233 rm -f "$OUT.pa" "$OUT.pa.err" "$OUT.pa.log" "$OUT.stop"
2178 ok "local mux auto-start under the pty fixture" 2234 ok "local mux auto-start under the pty fixture"
2179 2235
2236 # --- M14: the ssh→QUIC handoff -----------------------------------------
2237 #
2238 # `mux HOST` fetches QUIC coordinates over ssh once, caches them, and
2239 # attaches over pure QUIC thereafter — falling back to that same ssh, one
2240 # deadline later, when QUIC cannot get through. Five scenarios: cold, warm,
2241 # a poisoned cache, a key mismatch, and a remote that can offer nothing.
2242 #
2243 # The shim IS ssh as far as the client can tell: `ssh HOST CMD...` drops
2244 # HOST and execs CMD here, so `ssh whatever muxd endpoint` runs the real
2245 # binary against a socket of this suite's own making and every line of the
2246 # client's handoff code runs for real, with no network anywhere.
2247 #
2248 # Every invocation appends its pid to SSHIM_PIDLOG BEFORE the exec, and
2249 # that log is how these scenarios know whether ssh ran. Never `ps | grep
2250 # ssh`: this box belongs to someone who is probably ssh'd into something,
2251 # and a warm attach's entire claim is that no ssh existed.
2252 mkdir -p "$SSHIM_DIR" "$HRUN" "$HRUN2"
2253 : > "$SSHIM_PIDLOG"
2254 cat > "$SSHIM_DIR/ssh" <<'SHIM'
2255 #!/bin/sh
2256 echo $$ >> "${SSHIM_PIDLOG:?}"
2257 shift
2258 exec "$@"
2259 SHIM
2260 chmod +x "$SSHIM_DIR/ssh"
2261 # The PATH every scenario below runs the client under: the shim shadows any
2262 # real ssh, and `muxd` resolves to the binary under test rather than to
2263 # whatever is installed.
2264 HPATH="$SSHIM_DIR:$(dirname "$MUXD"):$PATH"
2265 # `user@` on the host is not decoration. The QUIC dial strips it
2266 # (handoff.dialHost) and gets a loopback literal, which is what makes the
2267 # dial reach the daemon at all; the CACHE keys on the whole word, which is
2268 # what keeps these three scenarios' cache file distinct from scenario (d)'s
2269 # bare `127.0.0.1`. A fake name would resolve to nothing and every QUIC
2270 # attempt below would fail at DNS without ever dialling.
2271 HHOST="mux-e2e@127.0.0.1"
2272 HCACHE="$XDG_CACHE_HOME/mux/hosts/$HHOST"
2273 # Nothing may be set in the environment that redirects the key resolution:
2274 # `muxd endpoint` reads MUX_KEY_FILE first, and an operator who has one
2275 # exported would silently change which key gets announced.
2276 unset MUX_KEY_FILE
2277
2278 # (a) COLD: no daemon, no cache. The attach has to produce the daemon (via
2279 # `muxd endpoint`'s own auto-start), fetch coordinates over the shim, and
2280 # END UP ON QUIC — and that last part is proven by observation rather than
2281 # inference: the ssh the announce arrived on is killed by the client on
2282 # QUIC success, so once its pid is OBSERVED gone, a second marker still
2283 # making the round trip can only be riding QUIC. The pipe's owner is dead.
2284 { printf 'printf "cold-%%s\\n" one\n'; sleep 6; \
2285 printf 'printf "cold-%%s\\n" two\n'; sleep 2.5; printf '\034'; } | \
2286 SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" PATH="$HPATH" timeout 60 \
2287 "$MUX" "$HHOST" > "$OUT.h1" 2> "$OUT.h1.err" &
2288 H1PID=$!
2289 # The bounded wait, with a message of its own, because of what it is FOR:
2290 # a client blocked on an announce that is never coming has no other
2291 # symptom. It cannot be left to the suite's timeout or the trap — those
2292 # report a dead process, not which line stopped arriving.
2293 wait_for "$OUT.h1" "cold-one" 25 || {
2294 echo "e2e FAIL: cold handoff: no marker in 25s — the attach never converged."
2295 echo " A client that blocks here read no announce line off the ssh pipe."
2296 cat "$OUT.h1" "$OUT.h1.err" 2>/dev/null; exit 1; }
2297
2298 # Exactly one ssh, and its pid, from the log rather than from ps.
2299 HSHIMS=$(wc -l < "$SSHIM_PIDLOG")
2300 [ "$HSHIMS" -eq 1 ] || {
2301 echo "e2e FAIL: cold handoff ran $HSHIMS ssh invocations, want exactly 1"
2302 cat "$SSHIM_PIDLOG"; exit 1; }
2303 HSHIMPID=$(head -1 "$SSHIM_PIDLOG")
2304 wait_pid_gone "$HSHIMPID" "cold handoff: QUIC took over, so ssh must be gone"
2305 # ...and the session did not go with it.
2306 wait_for "$OUT.h1" "cold-two" 20 || {
2307 echo "e2e FAIL: cold handoff: the session stopped converging once ssh was gone,"
2308 echo " so the bytes were riding the pipe rather than QUIC"
2309 cat "$OUT.h1" "$OUT.h1.err" 2>/dev/null; exit 1; }
2310 set +e
2311 wait "$H1PID"
2312 RC=$?
2313 set -e
2314 [ "$RC" -eq 0 ] || {
2315 echo "e2e FAIL: cold handoff client exited $RC (want 0; 124 means it hung)"
2316 cat "$OUT.h1" "$OUT.h1.err" 2>/dev/null; exit 1; }
2317
2318 # The daemon `muxd endpoint` started, by the pid its own up-line reported —
2319 # the only handle this suite has on a process that is nobody's child. Same
2320 # accepted gap as the M13 blocks: a failure above leaves it untracked, and
2321 # the trap's stop-by-socket is what covers that.
2322 grep -q '^muxd endpoint: starting' "$OUT.h1.err" || {
2323 echo "e2e FAIL: cold handoff printed no endpoint starting line"
2324 cat "$OUT.h1.err"; exit 1; }
2325 HAPID=$(sed -n 's/.* pid=\([0-9]*\).*/\1/p' "$OUT.h1.err" | head -1)
2326 [ -n "$HAPID" ] || {
2327 echo "e2e FAIL: endpoint up-line carries no pid"; cat "$OUT.h1.err"; exit 1; }
2328 kill -0 "$HAPID" || { echo "e2e FAIL: the handoff daemon is not alive"; exit 1; }
2329
2330 # The cache: present, private (it holds the key), and naming a port that is
2331 # really being listened on. The last one is the assertion that matters —
2332 # a remembered number proves nothing, a remembered number that maps to the
2333 # daemon's actual UDP socket proves the announce was true. /proc/net/udp
2334 # for the same reasons the M8 block uses it, and 00000000 because the lazy
2335 # bind takes 0.0.0.0.
2336 [ -f "$HCACHE" ] || {
2337 echo "e2e FAIL: cold handoff left no cache at $HCACHE"
2338 ls -la "$XDG_CACHE_HOME/mux/hosts" 2>/dev/null; exit 1; }
2339 HCPERM=$(stat -c %a "$HCACHE")
2340 [ "$HCPERM" = "600" ] || {
2341 echo "e2e FAIL: handoff cache perms $HCPERM, want 600 (the file holds a key)"; exit 1; }
2342 HCDPERM=$(stat -c %a "$XDG_CACHE_HOME/mux/hosts")
2343 [ "$HCDPERM" = "700" ] || {
2344 echo "e2e FAIL: handoff cache dir perms $HCDPERM, want 700"; exit 1; }
2345 HPORT=$(sed -n 's/^endpoint \([0-9][0-9]*\) [0-9a-f]*$/\1/p' "$HCACHE")
2346 [ -n "$HPORT" ] || {
2347 echo "e2e FAIL: the cache is not an announce line:"; cat -v "$HCACHE"; exit 1; }
2348 HHEXUDP=$(printf '00000000:%04X' "$HPORT")
2349 grep -qi " $HHEXUDP " /proc/net/udp || {
2350 echo "e2e FAIL: the cached port $HPORT names no UDP listener ($HHEXUDP);"
2351 echo " the announce the client believed was not the daemon's"
2352 exit 1; }
2353 assert_converged "$OUT.h1" "$SOCK16" "cold handoff"
2354 ok "cold handoff: ssh fetches the coordinates, QUIC carries the session"
2355
2356 # (b) WARM: the same daemon, the cache from (a). No ssh may run at all —
2357 # the whole point of remembering coordinates — so the pidlog count is
2358 # snapshotted and must not move. The transport is also silent: the M13
2359 # contract that a warm attach prints no spawn progress, extended to the
2360 # handoff's own line, which has nothing to report when nothing fell back.
2361 HSHIMS_B=$(wc -l < "$SSHIM_PIDLOG")
2362 set +e
2363 { printf 'printf "warm-%%s\\n" ok\n'; sleep 2.5; printf '\034'; } | \
2364 SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" PATH="$HPATH" timeout 40 \
2365 "$MUX" "$HHOST" > "$OUT.h2" 2> "$OUT.h2.err"
2366 RC=$?
2367 set -e
2368 [ "$RC" -eq 0 ] || {
2369 echo "e2e FAIL: warm handoff client exited $RC (want 0)"
2370 cat "$OUT.h2" "$OUT.h2.err" 2>/dev/null; exit 1; }
2371 grep -q "warm-ok" "$OUT.h2" || {
2372 echo "e2e FAIL: warm handoff served no session"
2373 cat "$OUT.h2" "$OUT.h2.err" 2>/dev/null; exit 1; }
2374 HSHIMS_B2=$(wc -l < "$SSHIM_PIDLOG")
2375 [ "$HSHIMS_B2" -eq "$HSHIMS_B" ] || {
2376 echo "e2e FAIL: warm handoff spawned ssh: pidlog went $HSHIMS_B -> $HSHIMS_B2"
2377 echo " (want unchanged — a cache hit dials QUIC and never shells out)"
2378 exit 1; }
2379 grep -q 'starting' "$OUT.h2.err" && {
2380 echo "e2e FAIL: warm handoff printed spawn progress"; cat "$OUT.h2.err"; exit 1; }
2381 grep -q 'unreachable, attaching over ssh' "$OUT.h2.err" && {
2382 echo "e2e FAIL: warm handoff printed the fallback line"; cat "$OUT.h2.err"; exit 1; }
2383 assert_converged "$OUT.h2" "$SOCK16" "warm handoff"
2384 ok "warm handoff: the cache dials QUIC, ssh never runs (pidlog still $HSHIMS_B2)"
2385
2386 # (c) STALE CACHE: a well-formed announce naming a port nothing holds. The
2387 # invariant under test is that a stale cache costs TIME and never
2388 # correctness: the dial fails, the cold path refetches, QUIC succeeds on
2389 # the fresh coordinates, and the user is told nothing — a fallback line
2390 # here would be reporting a failure that did not happen.
2391 #
2392 # The poison is written by hand because shell cannot import handoff.zig.
2393 # The grammar is `endpoint <port> <64 hex>\n`; the key is STOLEN from the
2394 # real cache above and only the port is swapped, so the line differs from a
2395 # genuine one in exactly the way the scenario is about.
2396 #
2397 # It costs a full deadline (~2s): a refused loopback port is not fast —
2398 # the ICMP lands on a sendto inside the QUIC drain and is discarded there
2399 # (measured, M14 Task 4) — so every wait below is budgeted for it.
2400 #
2401 # Which is also why the detach byte is six seconds out rather than the
2402 # suite's usual two and a half. The stdin script runs on its own clock,
2403 # in PARALLEL with an attach that spends a deadline before the session
2404 # even exists, and `waitReady` answers Ctrl-\ DURING a handshake (M8 pins
2405 # exactly that). A detach landing inside the dial window aborts the
2406 # attach: no session, no error, exit 0 — a scenario that passes its exit
2407 # check having tested nothing. Measured here at 2172ms to the marker, so
2408 # six seconds is margin, not superstition.
2409 HHEXKEY=$(sed -n 's/^endpoint [0-9][0-9]* \([0-9a-f]*\)$/\1/p' "$HCACHE")
2410 [ -n "$HHEXKEY" ] || { echo "e2e FAIL: could not read the cached key"; exit 1; }
2411 printf 'endpoint %s %s\n' "$HDEADPORT" "$HHEXKEY" > "$HCACHE"
2412 chmod 600 "$HCACHE"
2413 HSHIMS_C=$(wc -l < "$SSHIM_PIDLOG")
2414 set +e
2415 { printf 'printf "heal-%%s\\n" ok\n'; sleep 6; printf '\034'; } | \
2416 SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" PATH="$HPATH" timeout 40 \
2417 "$MUX" "$HHOST" > "$OUT.h3" 2> "$OUT.h3.err"
2418 RC=$?
2419 set -e
2420 [ "$RC" -eq 0 ] || {
2421 echo "e2e FAIL: stale-cache handoff client exited $RC (want 0)"
2422 cat "$OUT.h3" "$OUT.h3.err" 2>/dev/null; exit 1; }
2423 grep -q "heal-ok" "$OUT.h3" || {
2424 echo "e2e FAIL: stale-cache handoff served no session"
2425 cat "$OUT.h3" "$OUT.h3.err" 2>/dev/null; exit 1; }
2426 # The absent-grep, and the control that proves it can fire is scenario (d)
2427 # below: the same pattern, asserted PRESENT, against a build where the
2428 # fallback really happened.
2429 grep -q 'unreachable, attaching over ssh' "$OUT.h3.err" && {
2430 echo "e2e FAIL: a stale cache printed the fallback line; it costs time, not correctness"
2431 cat "$OUT.h3.err"; exit 1; }
2432 HPORT_C=$(sed -n 's/^endpoint \([0-9][0-9]*\) [0-9a-f]*$/\1/p' "$HCACHE")
2433 [ "$HPORT_C" = "$HPORT" ] || {
2434 echo "e2e FAIL: the cache did not heal: holds $HPORT_C, want the live $HPORT"
2435 cat -v "$HCACHE"; exit 1; }
2436 HSHIMS_C2=$(wc -l < "$SSHIM_PIDLOG")
2437 [ "$((HSHIMS_C2 - HSHIMS_C))" -eq 1 ] || {
2438 echo "e2e FAIL: the healing attach ran $((HSHIMS_C2 - HSHIMS_C)) ssh invocations, want 1"
2439 exit 1; }
2440 assert_converged "$OUT.h3" "$SOCK16" "stale-cache self-heal"
2441 ok "a stale cache self-heals: one refetch, no fallback line, the real port cached"
2442
2443 # (d) FALLBACK LINE, by key mismatch. A second daemon holding a key nothing
2444 # else on this box has; `muxd endpoint` announces the DEFAULT key (that is
2445 # what the endpoint process resolves) together with this daemon's real
2446 # port, so the coordinates are perfectly well-formed and the handshake can
2447 # never complete. PSK auth is mutual and a listener does not answer a peer
2448 # it cannot authenticate, so the client gets silence and spends the whole
2449 # deadline — which is exactly the case the fallback line exists for.
2450 #
2451 # The host is the literal `127.0.0.1`, and it has to be: a fake name would
2452 # fail at DNS instantly, skip the deadline, and leave this scenario pinning
2453 # nothing. Its cache file is `127.0.0.1`, distinct from (a)'s.
2454 #
2455 # This scenario costs ~2s of wall time by design. That is the price of
2456 # pinning the budget, and it is the only place the suite pays it.
2457 #
2458 # Six seconds to the detach byte, for the reason spelled out in (c): the
2459 # stdin script's clock runs in parallel with a deadline-long attach, and a
2460 # Ctrl-\ that lands inside the dial is answered by `waitReady` as an abort
2461 # — which would leave this scenario asserting a fallback line that a
2462 # correct build never got as far as printing.
2463 head -c 32 /dev/urandom > "$HKEY"
2464 chmod 600 "$HKEY"
2465 "$MUXD" run --sock "$SOCK17" --shell /bin/sh \
2466 --quic "127.0.0.1:$HQPORT" --key "$HKEY" --quic-idle-ms 15000 &
2467 HDPID=$!
2468 i=0
2469 while [ ! -S "$SOCK17" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
2470 [ -S "$SOCK17" ] || { echo "e2e FAIL: key-mismatch daemon never bound"; exit 1; }
2471 HSHIMS_D=$(wc -l < "$SSHIM_PIDLOG")
2472 HT0=$(date +%s%N)
2473 set +e
2474 { printf 'printf "fallback-%%s\\n" ok\n'; sleep 6; printf '\034'; } | \
2475 SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN2" PATH="$HPATH" timeout 40 \
2476 "$MUX" "127.0.0.1" > "$OUT.h4" 2> "$OUT.h4.err" &
2477 H4PID=$!
2478 set -e
2479 # Timed to the LINE rather than to the client's exit: the line is printed
2480 # the moment the budget runs out, so the elapsed time here IS the deadline
2481 # the dial spent, with no session sleeps folded into it.
2482 # Only the stderr is printed on failure, deliberately: the complaint is
2483 # about a line that is missing from it, and the capture beside it is a
2484 # screenful of escape sequences that would bury the answer.
2485 wait_for "$OUT.h4.err" "unreachable, attaching over ssh" 15 || {
2486 echo "e2e FAIL: key mismatch printed no fallback line in 15s; its stderr was:"
2487 cat "$OUT.h4.err" 2>/dev/null; exit 1; }
2488 HT1=$(date +%s%N)
2489 HMS=$(( (HT1 - HT0) / 1000000 ))
2490 # The exact line, with the port the daemon really holds: a fallback that
2491 # named the wrong coordinates would be a different bug wearing this one's
2492 # message.
2493 grep -q "^mux: quic://127.0.0.1:$HQPORT unreachable, attaching over ssh$" "$OUT.h4.err" || {
2494 echo "e2e FAIL: the fallback line is not the pinned one (want port $HQPORT):"
2495 cat "$OUT.h4.err"; exit 1; }
2496 # Bounded on BOTH sides. The floor is what proves the dial actually
2497 # happened — an implementation that never tried would fall back instantly —
2498 # and it sits below the 2000ms budget only by the slack the measurement
2499 # itself needs. The ceiling is what proves the budget bounds anything at
2500 # all: without it a hang reads as a pass that took a while.
2501 [ "$HMS" -ge 1500 ] || {
2502 echo "e2e FAIL: the fallback came after ${HMS}ms, too fast to have spent the"
2503 echo " 2000ms QUIC budget — the dial cannot have happened"
2504 exit 1; }
2505 [ "$HMS" -lt 10000 ] || {
2506 echo "e2e FAIL: the fallback took ${HMS}ms; the deadline is meant to bound it"
2507 exit 1; }
2508 set +e
2509 wait "$H4PID"
2510 RC=$?
2511 set -e
2512 [ "$RC" -eq 0 ] || {
2513 echo "e2e FAIL: key-mismatch client exited $RC (want 0)"
2514 cat "$OUT.h4" "$OUT.h4.err" 2>/dev/null; exit 1; }
2515 # ...and the session it fell back to is a real one, over the pipe that
2516 # carried the announce.
2517 grep -q "fallback-ok" "$OUT.h4" || {
2518 echo "e2e FAIL: key-mismatch attach printed the line but served no session"
2519 cat "$OUT.h4" "$OUT.h4.err" 2>/dev/null; exit 1; }
2520 HSHIMS_D2=$(wc -l < "$SSHIM_PIDLOG")
2521 [ "$((HSHIMS_D2 - HSHIMS_D))" -eq 1 ] || {
2522 echo "e2e FAIL: the fallback attach ran $((HSHIMS_D2 - HSHIMS_D)) ssh invocations, want 1"
2523 exit 1; }
2524 assert_converged "$OUT.h4" "$SOCK17" "handoff fallback to ssh"
2525 ok "a key mismatch falls back to the ssh pipe: one deadline (${HMS}ms), one line"
2526
2527 # (e) ANNOUNCE-NONE: a remote that cannot produce coordinates at all. The
2528 # lever is a config home that is a FILE, so the default key can be neither
2529 # created nor found — `muxd endpoint` says so on stderr (ssh carries it to
2530 # the user) and announces `endpoint none`.
2531 #
2532 # Against that, no deadline may be paid: no coordinates were ever in play,
2533 # so there is nothing to dial and nothing to report as unreachable. Its own
2534 # host, uncached by construction — reusing (a)'s would hit the cache and
2535 # never reach the announce at all.
2536 : > "$HCFGBAD"
2537 HSHIMS_E=$(wc -l < "$SSHIM_PIDLOG")
2538 HT2=$(date +%s%N)
2539 set +e
2540 { printf 'printf "none-%%s\\n" ok\n'; sleep 2.5; printf '\034'; } | \
2541 SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" XDG_CONFIG_HOME="$HCFGBAD" \
2542 PATH="$HPATH" timeout 40 \
2543 "$MUX" "mux-e2e-none@127.0.0.1" > "$OUT.h5" 2> "$OUT.h5.err" &
2544 H5PID=$!
2545 set -e
2546 wait_for "$OUT.h5" "none-ok" 20 || {
2547 echo "e2e FAIL: announce-none handoff never served a session"
2548 cat "$OUT.h5" "$OUT.h5.err" 2>/dev/null; exit 1; }
2549 HT3=$(date +%s%N)
2550 HMS_E=$(( (HT3 - HT2) / 1000000 ))
2551 set +e
2552 wait "$H5PID"
2553 RC=$?
2554 set -e
2555 [ "$RC" -eq 0 ] || {
2556 echo "e2e FAIL: announce-none client exited $RC (want 0)"
2557 cat "$OUT.h5" "$OUT.h5.err" 2>/dev/null; exit 1; }
2558 # The remote said why, in one line, on the stderr ssh already carries.
2559 grep -q '^muxd endpoint: .*staying on ssh' "$OUT.h5.err" || {
2560 echo "e2e FAIL: announce-none said nothing about why it stayed on ssh"
2561 cat "$OUT.h5.err"; exit 1; }
2562 # ...and said nothing about a fallback, because nothing fell back: the
2563 # client never held coordinates to fail with.
2564 grep -q 'unreachable, attaching over ssh' "$OUT.h5.err" && {
2565 echo "e2e FAIL: announce-none printed the fallback line; no dial was ever possible"
2566 cat "$OUT.h5.err"; exit 1; }
2567 HSHIMS_E2=$(wc -l < "$SSHIM_PIDLOG")
2568 [ "$((HSHIMS_E2 - HSHIMS_E))" -eq 1 ] || {
2569 echo "e2e FAIL: announce-none ran $((HSHIMS_E2 - HSHIMS_E)) ssh invocations, want 1"
2570 exit 1; }
2571 # No budget spent, measured to the first marker rather than to exit. The
2572 # bound is the 2000ms deadline with room to spare for an attach that has
2573 # to spawn a shell through two processes: anything at or above the budget
2574 # means a dial happened that the announce said was impossible.
2575 [ "$HMS_E" -lt 1500 ] || {
2576 echo "e2e FAIL: announce-none took ${HMS_E}ms to converge; a QUIC budget was"
2577 echo " spent on coordinates the remote said it did not have"
2578 exit 1; }
2579 assert_converged "$OUT.h5" "$SOCK16" "announce-none stays on ssh"
2580 ok "announce-none stays on ssh: no deadline paid (${HMS_E}ms), no fallback line"
2581
2582 # Both daemons down by the sanctioned verb, and OBSERVED dead by pid — the
2583 # M13 teardown shape, including the labelled RC capture: a stop that fails
2584 # exits 1, which under `set -e` would abort with no line of its own and the
2585 # trap would then remove the evidence.
2586 set +e
2587 "$MUXD" stop --sock "$SOCK16" 2> "$OUT.stop"
2588 RC_STOP=$?
2589 set -e
2590 [ "$RC_STOP" = "0" ] || {
2591 echo "e2e FAIL: handoff daemon stop exited $RC_STOP, want 0"; cat "$OUT.stop"; exit 1; }
2592 grep -q '^muxd: stopped' "$OUT.stop" || {
2593 echo "e2e FAIL: handoff daemon stop did not report stopped"; cat "$OUT.stop"; exit 1; }
2594 [ ! -S "$SOCK16" ] || {
2595 echo "e2e FAIL: stop left the handoff daemon's socket"; ls -l "$SOCK16"; exit 1; }
2596 wait_pid_gone "$HAPID" "handoff daemon: stop reported stopped"
2597 HAPID=""
2598 set +e
2599 "$MUXD" stop --sock "$SOCK17" 2> "$OUT.stop"
2600 RC_STOP=$?
2601 set -e
2602 [ "$RC_STOP" = "0" ] || {
2603 echo "e2e FAIL: key-mismatch daemon stop exited $RC_STOP, want 0"; cat "$OUT.stop"; exit 1; }
2604 wait_pid_gone "$HDPID" "key-mismatch daemon: stop reported stopped"
2605 HDPID=""
2606 rm -f "$OUT.h1" "$OUT.h1.err" "$OUT.h2" "$OUT.h2.err" "$OUT.h3" "$OUT.h3.err" \
2607 "$OUT.h4" "$OUT.h4.err" "$OUT.h5" "$OUT.h5.err" "$OUT.stop" "$HKEY" "$HCFGBAD"
2608 rm -rf "$SSHIM_DIR" "$HRUN" "$HRUN2"
2609
2180 # The pins. Literals, not variables set from counting something else — 2610 # The pins. Literals, not variables set from counting something else —
2181 # "assert the literal, never the constant the code under test reads" 2611 # "assert the literal, never the constant the code under test reads"
2182 # (decisions.md, M10). 15 scenario checkpoints; 28 convergence points. 2612 # (decisions.md, M10). 20 scenario checkpoints; 33 convergence points.
2183 # Anyone adding a scenario updates these by hand, on purpose. 2613 # Anyone adding a scenario updates these by hand, on purpose.
2184 [ "$OK_COUNT" = "15" ] || { 2614 [ "$OK_COUNT" = "20" ] || {
2185 echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 15 —" 2615 echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 20 —"
2186 echo " a scenario was added (update the pin) or silently lost" 2616 echo " a scenario was added (update the pin) or silently lost"
2187 exit 1 2617 exit 1
2188 } 2618 }
2189 [ "$CONV_COUNT" = "28" ] || { 2619 [ "$CONV_COUNT" = "33" ] || {
2190 echo "e2e FAIL: $CONV_COUNT convergence points ran, the pin says 28" 2620 echo "e2e FAIL: $CONV_COUNT convergence points ran, the pin says 33"
2191 exit 1 2621 exit 1
2192 } 2622 }
2193 echo "e2e OK (15 scenarios, 28 convergence points)" 2623 echo "e2e OK (20 scenarios, 33 convergence points)"