a73x

a824b1a2

test: 14_upgrade moves to a file of its own

a73x   2026-08-26 19:18

Commit message
test: 14_upgrade moves to a file of its own

`muxd upgrade`, and what `muxd stop` waits for — 5 scenarios, 597 lines. The legs move unedited; the runner
sources the file where they used to stand.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test/e2e.sh
Old New
@@ -102,7 +102,7 @@ done
102 E2E_DIR=$(dirname "$0") 102 E2E_DIR=$(dirname "$0")
103 . "$E2E_DIR/e2e_lib.sh" 103 . "$E2E_DIR/e2e_lib.sh"
104 104
105 E2E_GROUPS='01_boot 02_predict 03_side 04_handoff 05_session 06_web 07_wallcli 08_mouse 09_wallhist 10_agent 11_select 12_panes 13_birth' 105 E2E_GROUPS='01_boot 02_predict 03_side 04_handoff 05_session 06_web 07_wallcli 08_mouse 09_wallhist 10_agent 11_select 12_panes 13_birth 14_upgrade'
106 # The groups that cannot stand alone, and what they need. E2E_ONLY refuses 106 # The groups that cannot stand alone, and what they need. E2E_ONLY refuses
107 # these by name rather than running them: a group whose fixtures are 107 # these by name rather than running them: a group whose fixtures are
108 # missing does not fail, it passes having tested something else — and a 108 # missing does not fail, it passes having tested something else — and a
@@ -145,600 +145,6 @@ for _g in $E2E_GROUPS; do
145 . "$E2E_DIR/e2e_$_g.sh" 145 . "$E2E_DIR/e2e_$_g.sh"
146 done 146 done
147 147
148 # `muxd upgrade` — four daemons, because each one is a different daemon
149 # LIFECYCLE and no two can share a process: the same-binary leg's daemon
150 # ends up running a second image, the rollback leg's is born with the abort
151 # armed in its environment, the agent leg's holds an `-A` client, and the
152 # QUIC leg's owns a UDP socket. They run one after another, so the last of
153 # them is the only leg here that needs a port.
154 SOCK69="${TMPDIR:-/tmp}/muxd-e2e-upgrade-$$.sock"
155 defer_sock "$SOCK69"
156 SOCK70="${TMPDIR:-/tmp}/muxd-e2e-uproll-$$.sock"
157 defer_sock "$SOCK70"
158 SOCK71="${TMPDIR:-/tmp}/muxd-e2e-upagent-$$.sock"
159 defer_sock "$SOCK71"
160 SOCK72="${TMPDIR:-/tmp}/muxd-e2e-upquic-$$.sock"
161 defer_sock "$SOCK72"
162 SOCK73="${TMPDIR:-/tmp}/muxd-e2e-stopgone-$$.sock"
163 defer_sock "$SOCK73"
164 # The next 4000-wide band DOWN from the hydrate leg's 6000: every base from
165 # 11000 up is taken, and the 61000+ tail above the ephemeral range is spoken
166 # for by the two browser ports. A collision here reads as this leg's daemon
167 # failing to bind, the same verdict every other band gives.
168 UPQPORT=$(( 2000 + ($$ % 4000) ))
169 UPKEY="${TMPDIR:-/tmp}/mux-e2e-upkey-$$"
170 defer_rm "$UPKEY"
171 # A HOME of its own for the session shell. The title assertion below is
172 # about a title the SESSION set, and a distribution's rc file repaints the
173 # window title from PROMPT_COMMAND on every prompt (shellint.bash_init says
174 # so of Arch's) — read out of the developer's own dotfiles, this leg would
175 # assert on whatever their prompt happens to spell.
176 UPHOME="${TMPDIR:-/tmp}/mux-e2e-uphome-$$"
177 defer_rm "$UPHOME"
178 # The candidate binary the refusal leg offers: a copy, so its exec bit can
179 # be taken away without touching the one every other scenario runs.
180 UPBIN="${TMPDIR:-/tmp}/mux-e2e-upcand-$$"
181 defer_rm "$UPBIN"
182 UPAGENT="${TMPDIR:-/tmp}/mux-e2e-upagent-$$.sock"
183 defer_rm "$UPAGENT"
184 UPAGKEY="${TMPDIR:-/tmp}/mux-e2e-upagkey-$$"
185 defer_rm "$UPAGKEY" "$UPAGKEY.pub"
186
187 # --- muxd upgrade: the daemon becomes the new binary, holding everything ---
188 #
189 # `muxd upgrade` execs the candidate OVER the running daemon: same pid, same
190 # children, same descriptors. Nothing on the wire can carry the proof — a
191 # client reconnecting to a freshly started daemon sees exactly the same
192 # snapshot — so the witness is the session shell's own pid, put on the grid
193 # before the exec and read off it after. A restart cannot fake that: a new
194 # daemon forks a new shell, and a new shell has a new pid.
195 #
196 # One daemon, one attached client, and the whole story in order: what the
197 # session had before, two refusals that must change nothing, the exec, and
198 # then every one of those things again on the far side.
199 #
200 # bash with marks on, because two of the claims are the mechanism's:
201 # `muxa run` answers under `marks` before the exec and must answer under
202 # `marks` after it. The FIRST one after is the assertion that matters —
203 # the manifest carries a return watermark, and one carried into the new
204 # seq space is a watermark from the future that no return can pass, which
205 # measured as the first await after every upgrade timing out at 30s while
206 # the shell had already answered.
207 UPVER=$("$MUXD" --version | awk '{print $2}')
208 case "$UPVER" in
209 ?*.?*) ;;
210 *) echo "e2e FAIL: upgrade: muxd --version gave no version to refuse with: [$UPVER]"
211 exit 1;;
212 esac
213 mkdir -p "$UPHOME"
214 MUX_SHELL_INTEGRATION=1 HOME="$UPHOME" "$MUXD" run --sock "$SOCK69" \
215 --shell /bin/bash > "$OUT.up.d" 2>&1 &
216 D69PID=$!
217 defer_kill "$D69PID"
218 wait_sock "$SOCK69" "$OUT.up.d" "upgrade daemon never bound"
219
220 # Marks BEFORE the exec, so "marks after" is a comparison rather than a
221 # hope: a daemon that never had them would fail this line first.
222 timeout 20 "$MUXA" run --sock "$SOCK69" --timeout 8000 'echo up-marks-pre' > "$OUT.upm1" 2>&1 || {
223 echo "e2e FAIL: upgrade: muxa run failed before the upgrade even started:"
224 cat "$OUT.upm1"; exit 1; }
225 grep -qF '"mechanism":"marks"' "$OUT.upm1" || {
226 echo "e2e FAIL: upgrade: the session had no marks to lose:"
227 cat "$OUT.upm1"; exit 1; }
228
229 # The attached client, on a real pty, held across the exec. Its script
230 # stops on `expect up-resumed`, which nothing in the script produces — the
231 # suite's own post-upgrade `muxa run` does, below. That is the rendezvous:
232 # the fixture reads the master the whole time it waits, so the capture is
233 # continuous across the tear, and the needle can only arrive through the
234 # connection the client re-dialled.
235 #
236 # `unset PROMPT_COMMAND; PROMPT_COMMAND=_mux_precmd` keeps mux's own precmd
237 # (shellint's `precmd_fn`, which is where the marks come from) and drops
238 # every other member. A distribution rc appends a title-setting member
239 # there, and it repaints the title on every prompt — the title this leg
240 # sets would be gone by the next one. `unset` first because a scalar
241 # assignment onto bash's array spelling lands on element 0 and leaves the
242 # rest.
243 timeout 150 "$PTYCLIENT" --cols 100 --rows 30 \
244 --out "$OUT.upcap" --err "$OUT.upcap.err" -- \
245 "$MUX" --sock "$SOCK69" > "$OUT.uppc" 2>&1 <<'EOF' &
246 settle 1000 25000
247 send unset PROMPT_COMMAND; PROMPT_COMMAND=_mux_precmd\n
248 settle 800 25000
249 send printf '\\033]0;uptitle\\007'\n
250 expect \x1b]0;uptitle\x07 25000
251 send printf 'up-%s\n' pre\n
252 expect up-pre 25000
253 send echo shpid=$$\n
254 expect shpid= 25000
255 settle 800 25000
256 expect \x1b]0;uptitle\x07 90000
257 expect up-resumed 60000
258 settle 800 25000
259 send printf 'up-%s\n' typed\n
260 expect up-typed 25000
261 send \x1cd
262 waitexit 15000
263 EOF
264 UPPCPID=$!
265 defer_kill "$UPPCPID"
266
267 # Daemon truth for the pid, and the number this leg is built around. The
268 # echoed command line spells `$$`, so only the shell's own answer has
269 # digits after the `=` — grepping for a bare `shpid=` would match the echo.
270 wait_grid "$SOCK69" "shpid=[0-9]" "upgrade: the session shell never printed its pid"
271 UPSHPID=$(dump_session "$SOCK69" | sed -n 's/.*shpid=\([0-9][0-9]*\).*/\1/p' | head -1)
272 [ -n "$UPSHPID" ] || {
273 echo "e2e FAIL: upgrade: no shell pid read off the grid — the witness would be vacuous"
274 dump_session "$SOCK69"; exit 1; }
275
276 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst1" 2>&1 || {
277 echo "e2e FAIL: upgrade: stats did not answer before the upgrade"; exit 1; }
278 UPATT1=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst1")
279 UPSNAP1=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst1")
280 [ -n "$UPATT1" ] && [ -n "$UPSNAP1" ] || {
281 echo "e2e FAIL: upgrade: no counters read out of stats; the carry pin would be vacuous:"
282 cat "$OUT.upst1"; exit 1; }
283
284 # 1. Refused: same version, no flag. The reason names BOTH versions, which
285 # is the difference between a policy an operator can act on and a "no".
286 set +e
287 "$MUXD" upgrade --sock "$SOCK69" > "$OUT.upref1" 2>&1
288 UPRC=$?
289 set -e
290 [ "$UPRC" -eq 1 ] || {
291 echo "e2e FAIL: upgrade: a same-version upgrade exited $UPRC, want 1:"
292 cat "$OUT.upref1"; exit 1; }
293 grep -qF "refused: version: $UPVER is not newer than $UPVER" "$OUT.upref1" || {
294 echo "e2e FAIL: upgrade: the refusal did not name both versions:"
295 cat "$OUT.upref1"; exit 1; }
296
297 # 2. Refused: a candidate that is not executable. `muxd upgrade` offers its
298 # OWN path (/proc/self/exe), so the only way to spell this refusal is to
299 # take the exec bit off a copy AFTER it has started — and the ordering is
300 # made deterministic rather than hoped for. The daemon is held under
301 # SIGSTOP so it cannot read the request until the chmod has landed;
302 # `readlink /proc/PID/exe` is the witness that the copy has already
303 # exec'd, so the chmod cannot beat it and turn the leg into an exec
304 # failure that asserts nothing.
305 cp "$MUXD_ELF" "$UPBIN"
306 chmod 755 "$UPBIN"
307 kill -STOP "$D69PID"
308 "$UPBIN" upgrade --sock "$SOCK69" --allow-same-version > "$OUT.upref2" 2>&1 &
309 UPXPID=$!
310 defer_kill "$UPXPID"
311 _i=0
312 while [ "$(readlink "/proc/$UPXPID/exe" 2>/dev/null)" != "$UPBIN" ]; do
313 _i=$((_i + 1))
314 [ "$_i" -lt $(( 100 * TIME_SCALE )) ] || {
315 kill -CONT "$D69PID" 2>/dev/null || true
316 echo "e2e FAIL: upgrade: the candidate copy never exec'd; the refusal below"
317 echo " would have been about a binary that never asked"
318 exit 1; }
319 sleep 0.05
320 done
321 chmod 000 "$UPBIN"
322 kill -CONT "$D69PID"
323 set +e
324 wait "$UPXPID"
325 UPRC=$?
326 set -e
327 chmod 755 "$UPBIN"
328 [ "$UPRC" -eq 1 ] || {
329 echo "e2e FAIL: upgrade: an unexecutable candidate exited $UPRC, want 1:"
330 cat "$OUT.upref2"; exit 1; }
331 grep -qF "refused: path: not executable" "$OUT.upref2" || {
332 echo "e2e FAIL: upgrade: the refusal did not name the path check:"
333 cat "$OUT.upref2"; exit 1; }
334
335 # ...and two refusals changed NOTHING. A refusal that had torn a client
336 # down, or written the manifest and left the fd table half open, would
337 # still have printed the words above.
338 timeout 20 "$MUXA" status --sock "$SOCK69" > "$OUT.upsta" 2>&1 || {
339 echo "e2e FAIL: upgrade: the daemon stopped answering after refusing:"
340 cat "$OUT.upsta"; exit 1; }
341 wait_grid "$SOCK69" "shpid=$UPSHPID" "upgrade: a refusal cost the session its shell"
342
343 # 3. The exec itself.
344 set +e
345 "$MUXD" upgrade --sock "$SOCK69" --allow-same-version > "$OUT.upok" 2>&1
346 UPRC=$?
347 set -e
348 [ "$UPRC" -eq 0 ] || {
349 echo "e2e FAIL: upgrade: the upgrade exited $UPRC, want 0:"
350 cat "$OUT.upok" "$OUT.up.d"; exit 1; }
351 grep -qF "muxd: upgraded to $UPVER" "$OUT.upok" || {
352 echo "e2e FAIL: upgrade: no verdict naming the version it landed on:"
353 cat "$OUT.upok"; exit 1; }
354
355 # The pid is the feature. Not "a daemon is answering" — the same process,
356 # now running a different image, which /proc/PID/cmdline says in one word:
357 # a daemon that had restarted would carry the original `run --sock` argv.
358 kill -0 "$D69PID" 2>/dev/null || {
359 echo "e2e FAIL: upgrade: the daemon pid $D69PID is gone — that is a restart, not an upgrade"
360 cat "$OUT.up.d"; exit 1; }
361 tr '\0' ' ' < "/proc/$(real_pid "$D69PID")/cmdline" > "$OUT.upst2" 2>/dev/null || true
362 grep -qF -- "--resume-fd" "$OUT.upst2" || {
363 echo "e2e FAIL: upgrade: pid $D69PID is not running the resumed argv; it holds:"
364 cat "$OUT.upst2"; exit 1; }
365
366 # Counters CARRY, and it is asserted HERE rather than at the end for a
367 # reason about blame: the re-attach poll below counts `attaches` up from
368 # the reading taken before the exec, so a daemon that had zeroed its
369 # counters fails THAT poll — reported as a client that never came back,
370 # which is the wrong diagnosis of the wrong bug. `-ge`, because the
371 # re-attach this reading races has not necessarily landed yet.
372 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst2" 2>&1 || {
373 echo "e2e FAIL: upgrade: stats did not answer after the exec"; cat "$OUT.up.d"; exit 1; }
374 UPATT2=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst2")
375 UPSNAP2=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst2")
376 [ -n "$UPATT2" ] && [ -n "$UPSNAP2" ] || {
377 echo "e2e FAIL: upgrade: no counters read out of stats after the exec:"
378 cat "$OUT.upst2"; exit 1; }
379 [ "$UPATT2" -ge "$UPATT1" ] || {
380 echo "e2e FAIL: upgrade: attaches went $UPATT1 -> $UPATT2 across the exec; the"
381 echo " counters were reset, not carried, and an upgrade now looks like"
382 echo " a restart to anything sampling this daemon"
383 exit 1; }
384 [ "$UPSNAP2" -ge "$UPSNAP1" ] || {
385 echo "e2e FAIL: upgrade: snapshots went $UPSNAP1 -> $UPSNAP2 across the exec; the"
386 echo " counters were reset, not carried"
387 exit 1; }
388
389 # The client came back on its own. Counted with `attaches=`, not with
390 # `clients=`: the client is re-dialled within a millisecond of the exec, so
391 # a poll that only looked for clients=1 would be satisfied by the reading it
392 # took before the tear and would pass on a daemon that never dropped anyone.
393 _i=0
394 while : ; do
395 _upst=$("$MUXD" stats --sock "$SOCK69" 2>/dev/null || true)
396 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
397 case "$_upst" in
398 *"session 0 clients=1"*)
399 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
400 esac
401 _i=$((_i + 1))
402 [ "$_i" -lt $(( 200 * TIME_SCALE )) ] || {
403 echo "e2e FAIL: upgrade: the attached client never re-attached after the exec;"
404 echo " stats holds: $_upst"
405 cat "$OUT.up.d"; exit 1; }
406 sleep 0.1
407 done
408
409 # The grid crossed with the process: the marker the client typed, and the
410 # pid of the shell that is still running under the new image.
411 wait_grid "$SOCK69" "up-pre" "upgrade: the pre-upgrade marker did not survive the exec"
412 wait_grid "$SOCK69" "shpid=$UPSHPID" "upgrade: the session is a NEW shell, not the one that crossed"
413
414 # The FIRST muxa run after the exec, and it is also what releases the
415 # attached client's `expect up-resumed` above — one command, two claims.
416 set +e
417 timeout 40 "$MUXA" run --sock "$SOCK69" --timeout 20000 'echo up-resumed' > "$OUT.upm2" 2>&1
418 UPRC=$?
419 set -e
420 [ "$UPRC" -eq 0 ] || {
421 echo "e2e FAIL: upgrade: the FIRST muxa run after the exec exited $UPRC — a carried"
422 echo " return watermark is a watermark from the future and no return passes it:"
423 cat "$OUT.upm2"; exit 1; }
424 grep -qF '"mechanism":"marks"' "$OUT.upm2" || {
425 echo "e2e FAIL: upgrade: the first await after the exec fell off marks:"
426 cat "$OUT.upm2"; exit 1; }
427 grep -qF '"exit_code":0' "$OUT.upm2" || {
428 echo "e2e FAIL: upgrade: no real exit code came back after the exec:"
429 cat "$OUT.upm2"; exit 1; }
430 # The client's own verdict: the title came back on the re-attach, the
431 # marker arrived through the new image, and a keystroke typed AFTER the
432 # exec reached the shell.
433 set +e
434 wait "$UPPCPID"
435 UPRC=$?
436 set -e
437 UPPCPID=""
438 [ "$UPRC" -eq 0 ] || {
439 echo "e2e FAIL: upgrade: the attached client exited $UPRC across the exec:"
440 cat "$OUT.uppc" "$OUT.upcap.err"; exit 1; }
441
442 # A NONZERO exit code too, because `"exit_code":0` is also what a mechanism
443 # that guesses would say. After the client has gone, not beside it: two
444 # writers typing into one shell's pty interleave, and the leg would be
445 # racing itself. Spelled as a CHILD, not as `exit 7`, which is the session
446 # shell's own exit — the session would end, the daemon with it, and the
447 # counter reading below would be taken from nothing. Not `(exit 7)` either:
448 # a subshell sets `_mux_ran` in the subshell, so shellint's `precmd_fn`
449 # sees an untouched prompt in the parent and emits no `D;` mark at all
450 # (measured: the await ran its whole budget out).
451 timeout 40 "$MUXA" run --sock "$SOCK69" --timeout 20000 'sh -c "exit 7"' > "$OUT.upm3" 2>&1 || true
452 grep -qF '"exit_code":7' "$OUT.upm3" || {
453 echo "e2e FAIL: upgrade: the shell's own exit code did not survive the exec:"
454 cat "$OUT.upm3"; exit 1; }
455
456 # ...and the re-attach was COUNTED onto them rather than replacing them.
457 # The carry was asserted above with `-ge`; this is the other half, taken
458 # once the client's own snapshot has certainly been served.
459 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst2" 2>&1
460 UPATT2=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst2")
461 UPSNAP2=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst2")
462 [ -n "$UPATT2" ] && [ -n "$UPSNAP2" ] || {
463 echo "e2e FAIL: upgrade: no counters read out of stats after the client left:"
464 cat "$OUT.upst2"; exit 1; }
465 [ "$UPATT2" -gt "$UPATT1" ] || {
466 echo "e2e FAIL: upgrade: attaches is still $UPATT2 — the re-attach after the exec"
467 echo " was never counted"
468 exit 1; }
469 [ "$UPSNAP2" -gt "$UPSNAP1" ] || {
470 echo "e2e FAIL: upgrade: snapshots is still $UPSNAP2 — the re-attach was served"
471 echo " without a snapshot, which no fresh epoch allows"
472 exit 1; }
473
474 assert_stopped "$SOCK69" "$D69PID" "upgrade" "$OUT.upstop"
475 D69PID=""
476 rm -f "$UPBIN"
477 rm -rf "$UPHOME"
478 ok "muxd upgrade keeps the shell, its pid, its title, its marks and its counters"
479
480 # --- ...and a candidate that cannot adopt hands the daemon back -----------
481 #
482 # The rollback: the new image fails partway through adoption and execs the
483 # OLD binary — whose path the manifest carries — with the same manifest fd.
484 # Armed through the ENVIRONMENT rather than the flag it shares a name with,
485 # and that is not a convenience: `execUpgrade` builds a fixed argv
486 # (`run --resume-fd N`), so nothing typed here can put `--resume-fail-at`
487 # in front of the candidate. The daemon's environment is what crosses an
488 # exec, so the abort is armed on the daemon before any upgrade asks.
489 #
490 # The daemon's own log is the only thing that can say a rollback HAPPENED —
491 # `muxd upgrade` prints its verdict the moment the daemon accepts, and the
492 # process that would have corrected it is gone. Without that line every
493 # assertion below would pass just as well on an upgrade that simply worked.
494 mkdir -p "$UPHOME"
495 MUX_SHELL_INTEGRATION=1 MUX_RESUME_FAIL_AT=session HOME="$UPHOME" \
496 start_daemon "$SOCK70" "$OUT.url.d" "rollback daemon never bound" --shell /bin/bash
497 D70PID=$DPID
498
499 timeout 20 "$MUXA" run --sock "$SOCK70" --timeout 8000 'echo rollpid=$$' > "$OUT.urlm1" 2>&1 || {
500 echo "e2e FAIL: rollback: muxa run failed before the upgrade:"
501 cat "$OUT.urlm1"; exit 1; }
502 wait_grid "$SOCK70" "rollpid=[0-9]" "rollback: the session shell never printed its pid"
503 UPROLLPID=$(dump_session "$SOCK70" | sed -n 's/.*rollpid=\([0-9][0-9]*\).*/\1/p' | head -1)
504 [ -n "$UPROLLPID" ] || {
505 echo "e2e FAIL: rollback: no shell pid read off the grid"; dump_session "$SOCK70"; exit 1; }
506
507 set +e
508 "$MUXD" upgrade --sock "$SOCK70" --allow-same-version > "$OUT.urlup" 2>&1
509 UPRC=$?
510 set -e
511 [ "$UPRC" -eq 0 ] || {
512 echo "e2e FAIL: rollback: upgrade exited $UPRC — the rolled-back daemon must serve"
513 echo " well enough for the confirm to pass:"
514 cat "$OUT.urlup" "$OUT.url.d"; exit 1; }
515 wait_for "$OUT.url.d" "adoption failed at session" 15 || {
516 echo "e2e FAIL: rollback: the candidate adopted instead of aborting — the abort"
517 echo " was not armed, and nothing below tests a rollback:"
518 cat "$OUT.url.d"; exit 1; }
519 grep -qF "exec'ing" "$OUT.url.d" || {
520 echo "e2e FAIL: rollback: the abort never named the binary it went back to:"
521 cat "$OUT.url.d"; exit 1; }
522
523 kill -0 "$D70PID" 2>/dev/null || {
524 echo "e2e FAIL: rollback: the daemon pid $D70PID is gone; a failed adoption took"
525 echo " the sessions with it"
526 cat "$OUT.url.d"; exit 1; }
527 wait_grid "$SOCK70" "rollpid=$UPROLLPID" "rollback: the session lost its shell to a failed adoption"
528 timeout 40 "$MUXA" run --sock "$SOCK70" --timeout 20000 'echo roll-post' > "$OUT.urlm2" 2>&1 || {
529 echo "e2e FAIL: rollback: the daemon that came back cannot run a command:"
530 cat "$OUT.urlm2" "$OUT.url.d"; exit 1; }
531 grep -qF '"mechanism":"marks"' "$OUT.urlm2" || {
532 echo "e2e FAIL: rollback: the re-adopted session lost its marks:"
533 cat "$OUT.urlm2"; exit 1; }
534
535 assert_stopped "$SOCK70" "$D70PID" "rollback" "$OUT.urlstop"
536 D70PID=""
537 rm -rf "$UPHOME"
538 ok "a candidate that cannot adopt execs the old binary back, session and marks intact"
539
540 # --- ...and the session's agent socket crosses the exec -------------------
541 #
542 # The per-session agent listener is a descriptor the daemon owns and the
543 # shell reaches by path ($SSH_AUTH_SOCK, named after the daemon's pid). It
544 # has to cross the exec on both counts: the fd by inheritance, the path
545 # because the pid does not change. Nothing else in this suite would notice
546 # if it did not — the agent channels themselves are deliberately NOT
547 # carried, so the only thing that can say the listener survived is a real
548 # `ssh-add -l` answering through a re-dialled `-A` client.
549 #
550 # Its own agent rather than the forwarding legs' AGENT48: that one is left
551 # under SIGSTOP by the mute-offerer leg, and a mute agent is exactly what
552 # this leg cannot tell from a broken listener.
553 start_daemon "$SOCK71" "$OUT.uag.d" "agent-upgrade daemon never bound" --shell /bin/sh
554 D71PID=$DPID
555
556 ssh-agent -a "$UPAGENT" > "$OUT.uagenv" 2>&1
557 UPAGPID=$(sed -n 's/.*SSH_AGENT_PID=\([0-9]*\).*/\1/p' "$OUT.uagenv")
558 defer_kill "$UPAGPID"
559 [ -n "$UPAGPID" ] || {
560 echo "e2e FAIL: agent-upgrade: ssh-agent printed no pid for the trap to hold it by:"
561 cat "$OUT.uagenv"; exit 1; }
562 ssh-keygen -q -t ed25519 -N '' -C mux-e2e-upgrade -f "$UPAGKEY"
563 SSH_AUTH_SOCK="$UPAGENT" ssh-add "$UPAGKEY" > /dev/null 2>&1
564 UPFP=$(ssh-keygen -lf "$UPAGKEY" | awk '{print $2}')
565 case "$UPFP" in
566 SHA256:?*) ;;
567 *) echo "e2e FAIL: agent-upgrade: ssh-keygen -lf gave no fingerprint: [$UPFP]"; exit 1;;
568 esac
569
570 pipe_mux "$OUT.uag" "$OUT.uag.err" env SSH_AUTH_SOCK="$UPAGENT" timeout 90 \
571 "$MUX" -A --sock "$SOCK71"
572 UPATT1=$("$MUXD" stats --sock "$SOCK71" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
573 [ -n "$UPATT1" ] || { echo "e2e FAIL: agent-upgrade: no attaches counter to compare against"; exit 1; }
574 # `agpre=` rather than the fingerprint alone: the shell echoes what is
575 # typed, and `$?` is not expanded in the echo, so only ssh-add's own status
576 # can put a digit there. The fingerprint is asserted beside it because a
577 # status of 0 says an agent answered, not that it was the client's.
578 pipe_send 'ssh-add -l; echo agpre=$?\n'
579 await_out "$OUT.uag" "agpre=0" "agent-upgrade: ssh-add never answered before the upgrade"
580 # Counted by OCCURRENCE, not by matching line: this capture is an escape
581 # stream whose one "line" is the whole session, so `grep -c` would answer 1
582 # for both answers and the comparison after the exec would be vacuous.
583 UPFP1=$(grep -aoF -- "$UPFP" "$OUT.uag" | wc -l)
584 [ "$UPFP1" -ge 1 ] || {
585 echo "e2e FAIL: agent-upgrade: the client's key never came back before the upgrade"
586 echo " (wanted $UPFP)"; exit 1; }
587
588 set +e
589 "$MUXD" upgrade --sock "$SOCK71" --allow-same-version > "$OUT.uagup" 2>&1
590 UPRC=$?
591 set -e
592 [ "$UPRC" -eq 0 ] || {
593 echo "e2e FAIL: agent-upgrade: upgrade exited $UPRC:"
594 cat "$OUT.uagup" "$OUT.uag.d"; exit 1; }
595 _i=0
596 while : ; do
597 _upst=$("$MUXD" stats --sock "$SOCK71" 2>/dev/null || true)
598 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
599 case "$_upst" in
600 *"session 0 clients=1"*)
601 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
602 esac
603 _i=$((_i + 1))
604 [ "$_i" -lt $(( 200 * TIME_SCALE )) ] || {
605 echo "e2e FAIL: agent-upgrade: the -A client never re-attached; stats holds: $_upst"
606 cat "$OUT.uag.d"; exit 1; }
607 sleep 0.1
608 done
609
610 # The same question a second time, and `agdone=` is a needle the first
611 # answer cannot satisfy.
612 pipe_send 'ssh-add -l; echo agdone=$?\n'
613 await_out "$OUT.uag" "agdone=0" "agent-upgrade: ssh-add stopped answering across the exec"
614 UPFP2=$(grep -aoF -- "$UPFP" "$OUT.uag" | wc -l)
615 [ "$UPFP2" -gt "$UPFP1" ] || {
616 echo "e2e FAIL: agent-upgrade: ssh-add exited 0 but the key came back $UPFP1 time(s)"
617 echo " before the exec and $UPFP2 after — no NEW answer crossed"; exit 1; }
618 pipe_detach "agent-upgrade client"
619
620 assert_stopped "$SOCK71" "$D71PID" "agent-upgrade" "$OUT.uagstop"
621 D71PID=""
622 softkill "$UPAGPID" || true
623 wait_pid_gone "$UPAGPID" "agent-upgrade: ssh-agent killed by tracked pid"
624 UPAGPID=""
625 ok "the session's agent socket crosses the exec: ssh-add -l answers again"
626
627 # --- ...and a QUIC client is serving again in a breath, not after a timeout
628 #
629 # Unix clients need no goodbye — their fds are close-on-exec and the exec is
630 # their EOF. A QUIC peer has no such thing: silence is indistinguishable from
631 # a bad network, and a client that waited it out would sit on a blank wall
632 # for the whole idle timeout. So the daemon sends CONNECTION_CLOSE to every
633 # connection before it execs, and what that buys is the number asserted
634 # here: back in service in a fraction of `--quic-idle-ms`, not after it.
635 head -c 32 /dev/urandom > "$UPKEY"
636 chmod 600 "$UPKEY"
637 start_daemon "$SOCK72" "$OUT.uqc.d" "quic-upgrade daemon never bound" --shell /bin/sh \
638 --quic "127.0.0.1:$UPQPORT" --key "$UPKEY" --quic-idle-ms 15000
639 D72PID=$DPID
640
641 pipe_mux "$OUT.uqc" "$OUT.uqc.err" timeout 90 "$MUX" "quic://127.0.0.1:$UPQPORT" \
642 --key "$UPKEY" --quic-idle-ms 15000
643 pipe_send 'printf "uq-%%s\\n" pre\n'
644 await_out "$OUT.uqc" "uq-pre" "quic-upgrade: the QUIC client never got its pre-upgrade marker"
645
646 UPATT1=$("$MUXD" stats --sock "$SOCK72" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
647 [ -n "$UPATT1" ] || { echo "e2e FAIL: quic-upgrade: no attaches counter to compare against"; exit 1; }
648 UPT0=$(date +%s%N)
649 set +e
650 "$MUXD" upgrade --sock "$SOCK72" --allow-same-version > "$OUT.uqup" 2>&1
651 UPRC=$?
652 set -e
653 [ "$UPRC" -eq 0 ] || {
654 echo "e2e FAIL: quic-upgrade: upgrade exited $UPRC:"
655 cat "$OUT.uqup" "$OUT.uqc.d"; exit 1; }
656 _i=0
657 while : ; do
658 _upst=$("$MUXD" stats --sock "$SOCK72" 2>/dev/null || true)
659 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
660 case "$_upst" in
661 *"session 0 clients=1"*)
662 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
663 esac
664 _i=$((_i + 1))
665 [ "$_i" -lt $(( 400 * TIME_SCALE )) ] || {
666 echo "e2e FAIL: quic-upgrade: the QUIC client never re-attached; stats holds: $_upst"
667 cat "$OUT.uqc.d"; exit 1; }
668 sleep 0.05
669 done
670 UPT1=$(date +%s%N)
671 UPMS=$(( (UPT1 - UPT0) / 1000000 ))
672 # Half the idle timeout, which is the only bound that means anything here:
673 # a client that had waited the connection out would arrive at 15000ms and
674 # still pass a generous absolute budget. Measured at ~60ms.
675 [ "$UPMS" -lt $(( 7500 * TIME_SCALE )) ] || {
676 echo "e2e FAIL: quic-upgrade: the QUIC client took ${UPMS}ms to come back —"
677 echo " that is the idle timeout expiring, not a CONNECTION_CLOSE"
678 exit 1; }
679
680 # ...and it is a session, not just a connection: a round trip that did not
681 # exist before the exec.
682 pipe_send 'printf "uq-%%s\\n" post\n'
683 await_out "$OUT.uqc" "uq-post" "quic-upgrade: the reconnected QUIC client cannot reach its shell"
684 pipe_detach "quic-upgrade client"
685
686 assert_stopped "$SOCK72" "$D72PID" "quic-upgrade" "$OUT.uqstop"
687 D72PID=""
688 ok "a QUIC client is served again within a breath of the exec, not after the idle timeout"
689
690 # --- muxd stop returns when the PROCESS is gone, not when the path is
691 #
692 # The unlink is the first thing a stopping daemon does; reaping its shells
693 # and deleting its dirs come after. A stop that said "stopped" at the
694 # unlink handed a scripted `muxd start`, or a supervisor's "is it down",
695 # a daemon still running — `wait_pid_gone` after every stop in this suite
696 # was that gap, papered. Shells that ignore TERM make the window a real
697 # grace rather than a race the assertion could win by luck, and two
698 # sessions make it a table's grace, not one shell's.
699 printf '#!/bin/sh\ntrap "" HUP TERM\nwhile :; do sleep 1; done\n' > "$OUT.sg.sh"
700 chmod +x "$OUT.sg.sh"
701 start_daemon "$SOCK73" "$OUT.sg.d" "stop-gone daemon never bound" --shell "$OUT.sg.sh"
702 D73PID=$DPID
703 pipe_mux "$OUT.sg" "" timeout 60 "$MUX" --sock "$SOCK73" --session second
704 sleep 0.5
705 pipe_detach "stop-gone: the second session's client"
706 [ "$("$MUXD" stats --sock "$SOCK73" | sed -n 's/.*sessions=\([0-9]*\).*/\1/p')" = "2" ] || {
707 echo "e2e FAIL: stop-gone: wanted two sessions of stubborn shells"; exit 1; }
708 D73REAL=$(real_pid "$D73PID")
709 "$MUXD" stop --sock "$SOCK73" 2> "$OUT.sg.stop" || {
710 echo "e2e FAIL: stop-gone: stop failed"; cat "$OUT.sg.stop"; exit 1; }
711 # Asked of the OS the instant stop returns — no wait, no retry.
712 if kill -0 "$D73REAL" 2>/dev/null; then
713 echo "e2e FAIL: stop said stopped, but pid $D73PID is still running"; cat "$OUT.sg.stop"; exit 1
714 fi
715 grep -q '^muxd: stopped' "$OUT.sg.stop" || {
716 echo "e2e FAIL: stop-gone: stop did not report stopped"; cat "$OUT.sg.stop"; exit 1; }
717 D73PID=""
718 ok "muxd stop returns when the process is gone, not when the socket is"
719
720 # The pins. Literals, not variables set from counting something else —
721 # "assert the literal, never the constant the code under test reads"
722 # (decisions.md, M10). The two figures live at the END of this file and
723 # nowhere else: a count repeated in prose is a count nothing gates, and
724 # both of the ones that used to stand here had drifted away from the pins
725 # they described. Adding a scenario means editing those two literals by
726 # hand, on purpose.
727 #
728 # Why most legs carry no convergence point. `assert_converged` replays one
729 # client capture into an engine and diffs it against one daemon grid, so it
730 # can only speak where BOTH exist and the grid is the subject. It says
731 # nothing about a side channel (OSC 52, a window title, pty modes, a
732 # fingerprint ssh printed, bytes counted rather than read), about a file an
733 # editor or the wall wrote, about an exit code or a status field, about
734 # WHICH of two sessions a keystroke reached, about scrollback the live grid
735 # does not hold, about the absence of bytes two agreeing empties would
736 # agree on anyway, or about a leg that never attaches at all. It also does
737 # not claim to reproduce a full-screen application's redraws byte for byte,
738 # which is why the alternate-screen legs stay out. A leg that asserts on
739 # one of those and adds a convergence point would be pinning a fact every
740 # leg above already establishes.
741
742 # The pins. Literals, not variables set from counting something else — 148 # The pins. Literals, not variables set from counting something else —
743 # "assert the literal, never the constant the code under test reads" 149 # "assert the literal, never the constant the code under test reads"
744 # (decisions.md, M10). The two figures live at the END of this file and 150 # (decisions.md, M10). The two figures live at the END of this file and
test/e2e_14_upgrade.sh
Old New
@@ -0,0 +1,597 @@
1 # shellcheck shell=sh
2 # e2e_14_upgrade.sh — sourced by test/e2e.sh after e2e_lib.sh. Scenarios run in
3 # the order they stand in; see the lib's header for what this file may
4 # assume and what it must register.
5 # `muxd upgrade` — four daemons, because each one is a different daemon
6 # LIFECYCLE and no two can share a process: the same-binary leg's daemon
7 # ends up running a second image, the rollback leg's is born with the abort
8 # armed in its environment, the agent leg's holds an `-A` client, and the
9 # QUIC leg's owns a UDP socket. They run one after another, so the last of
10 # them is the only leg here that needs a port.
11 SOCK69="${TMPDIR:-/tmp}/muxd-e2e-upgrade-$$.sock"
12 defer_sock "$SOCK69"
13 SOCK70="${TMPDIR:-/tmp}/muxd-e2e-uproll-$$.sock"
14 defer_sock "$SOCK70"
15 SOCK71="${TMPDIR:-/tmp}/muxd-e2e-upagent-$$.sock"
16 defer_sock "$SOCK71"
17 SOCK72="${TMPDIR:-/tmp}/muxd-e2e-upquic-$$.sock"
18 defer_sock "$SOCK72"
19 SOCK73="${TMPDIR:-/tmp}/muxd-e2e-stopgone-$$.sock"
20 defer_sock "$SOCK73"
21 # The next 4000-wide band DOWN from the hydrate leg's 6000: every base from
22 # 11000 up is taken, and the 61000+ tail above the ephemeral range is spoken
23 # for by the two browser ports. A collision here reads as this leg's daemon
24 # failing to bind, the same verdict every other band gives.
25 UPQPORT=$(( 2000 + ($$ % 4000) ))
26 UPKEY="${TMPDIR:-/tmp}/mux-e2e-upkey-$$"
27 defer_rm "$UPKEY"
28 # A HOME of its own for the session shell. The title assertion below is
29 # about a title the SESSION set, and a distribution's rc file repaints the
30 # window title from PROMPT_COMMAND on every prompt (shellint.bash_init says
31 # so of Arch's) — read out of the developer's own dotfiles, this leg would
32 # assert on whatever their prompt happens to spell.
33 UPHOME="${TMPDIR:-/tmp}/mux-e2e-uphome-$$"
34 defer_rm "$UPHOME"
35 # The candidate binary the refusal leg offers: a copy, so its exec bit can
36 # be taken away without touching the one every other scenario runs.
37 UPBIN="${TMPDIR:-/tmp}/mux-e2e-upcand-$$"
38 defer_rm "$UPBIN"
39 UPAGENT="${TMPDIR:-/tmp}/mux-e2e-upagent-$$.sock"
40 defer_rm "$UPAGENT"
41 UPAGKEY="${TMPDIR:-/tmp}/mux-e2e-upagkey-$$"
42 defer_rm "$UPAGKEY" "$UPAGKEY.pub"
43
44 # --- muxd upgrade: the daemon becomes the new binary, holding everything ---
45 #
46 # `muxd upgrade` execs the candidate OVER the running daemon: same pid, same
47 # children, same descriptors. Nothing on the wire can carry the proof — a
48 # client reconnecting to a freshly started daemon sees exactly the same
49 # snapshot — so the witness is the session shell's own pid, put on the grid
50 # before the exec and read off it after. A restart cannot fake that: a new
51 # daemon forks a new shell, and a new shell has a new pid.
52 #
53 # One daemon, one attached client, and the whole story in order: what the
54 # session had before, two refusals that must change nothing, the exec, and
55 # then every one of those things again on the far side.
56 #
57 # bash with marks on, because two of the claims are the mechanism's:
58 # `muxa run` answers under `marks` before the exec and must answer under
59 # `marks` after it. The FIRST one after is the assertion that matters —
60 # the manifest carries a return watermark, and one carried into the new
61 # seq space is a watermark from the future that no return can pass, which
62 # measured as the first await after every upgrade timing out at 30s while
63 # the shell had already answered.
64 UPVER=$("$MUXD" --version | awk '{print $2}')
65 case "$UPVER" in
66 ?*.?*) ;;
67 *) echo "e2e FAIL: upgrade: muxd --version gave no version to refuse with: [$UPVER]"
68 exit 1;;
69 esac
70 mkdir -p "$UPHOME"
71 MUX_SHELL_INTEGRATION=1 HOME="$UPHOME" "$MUXD" run --sock "$SOCK69" \
72 --shell /bin/bash > "$OUT.up.d" 2>&1 &
73 D69PID=$!
74 defer_kill "$D69PID"
75 wait_sock "$SOCK69" "$OUT.up.d" "upgrade daemon never bound"
76
77 # Marks BEFORE the exec, so "marks after" is a comparison rather than a
78 # hope: a daemon that never had them would fail this line first.
79 timeout 20 "$MUXA" run --sock "$SOCK69" --timeout 8000 'echo up-marks-pre' > "$OUT.upm1" 2>&1 || {
80 echo "e2e FAIL: upgrade: muxa run failed before the upgrade even started:"
81 cat "$OUT.upm1"; exit 1; }
82 grep -qF '"mechanism":"marks"' "$OUT.upm1" || {
83 echo "e2e FAIL: upgrade: the session had no marks to lose:"
84 cat "$OUT.upm1"; exit 1; }
85
86 # The attached client, on a real pty, held across the exec. Its script
87 # stops on `expect up-resumed`, which nothing in the script produces — the
88 # suite's own post-upgrade `muxa run` does, below. That is the rendezvous:
89 # the fixture reads the master the whole time it waits, so the capture is
90 # continuous across the tear, and the needle can only arrive through the
91 # connection the client re-dialled.
92 #
93 # `unset PROMPT_COMMAND; PROMPT_COMMAND=_mux_precmd` keeps mux's own precmd
94 # (shellint's `precmd_fn`, which is where the marks come from) and drops
95 # every other member. A distribution rc appends a title-setting member
96 # there, and it repaints the title on every prompt — the title this leg
97 # sets would be gone by the next one. `unset` first because a scalar
98 # assignment onto bash's array spelling lands on element 0 and leaves the
99 # rest.
100 timeout 150 "$PTYCLIENT" --cols 100 --rows 30 \
101 --out "$OUT.upcap" --err "$OUT.upcap.err" -- \
102 "$MUX" --sock "$SOCK69" > "$OUT.uppc" 2>&1 <<'EOF' &
103 settle 1000 25000
104 send unset PROMPT_COMMAND; PROMPT_COMMAND=_mux_precmd\n
105 settle 800 25000
106 send printf '\\033]0;uptitle\\007'\n
107 expect \x1b]0;uptitle\x07 25000
108 send printf 'up-%s\n' pre\n
109 expect up-pre 25000
110 send echo shpid=$$\n
111 expect shpid= 25000
112 settle 800 25000
113 expect \x1b]0;uptitle\x07 90000
114 expect up-resumed 60000
115 settle 800 25000
116 send printf 'up-%s\n' typed\n
117 expect up-typed 25000
118 send \x1cd
119 waitexit 15000
120 EOF
121 UPPCPID=$!
122 defer_kill "$UPPCPID"
123
124 # Daemon truth for the pid, and the number this leg is built around. The
125 # echoed command line spells `$$`, so only the shell's own answer has
126 # digits after the `=` — grepping for a bare `shpid=` would match the echo.
127 wait_grid "$SOCK69" "shpid=[0-9]" "upgrade: the session shell never printed its pid"
128 UPSHPID=$(dump_session "$SOCK69" | sed -n 's/.*shpid=\([0-9][0-9]*\).*/\1/p' | head -1)
129 [ -n "$UPSHPID" ] || {
130 echo "e2e FAIL: upgrade: no shell pid read off the grid — the witness would be vacuous"
131 dump_session "$SOCK69"; exit 1; }
132
133 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst1" 2>&1 || {
134 echo "e2e FAIL: upgrade: stats did not answer before the upgrade"; exit 1; }
135 UPATT1=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst1")
136 UPSNAP1=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst1")
137 [ -n "$UPATT1" ] && [ -n "$UPSNAP1" ] || {
138 echo "e2e FAIL: upgrade: no counters read out of stats; the carry pin would be vacuous:"
139 cat "$OUT.upst1"; exit 1; }
140
141 # 1. Refused: same version, no flag. The reason names BOTH versions, which
142 # is the difference between a policy an operator can act on and a "no".
143 set +e
144 "$MUXD" upgrade --sock "$SOCK69" > "$OUT.upref1" 2>&1
145 UPRC=$?
146 set -e
147 [ "$UPRC" -eq 1 ] || {
148 echo "e2e FAIL: upgrade: a same-version upgrade exited $UPRC, want 1:"
149 cat "$OUT.upref1"; exit 1; }
150 grep -qF "refused: version: $UPVER is not newer than $UPVER" "$OUT.upref1" || {
151 echo "e2e FAIL: upgrade: the refusal did not name both versions:"
152 cat "$OUT.upref1"; exit 1; }
153
154 # 2. Refused: a candidate that is not executable. `muxd upgrade` offers its
155 # OWN path (/proc/self/exe), so the only way to spell this refusal is to
156 # take the exec bit off a copy AFTER it has started — and the ordering is
157 # made deterministic rather than hoped for. The daemon is held under
158 # SIGSTOP so it cannot read the request until the chmod has landed;
159 # `readlink /proc/PID/exe` is the witness that the copy has already
160 # exec'd, so the chmod cannot beat it and turn the leg into an exec
161 # failure that asserts nothing.
162 cp "$MUXD_ELF" "$UPBIN"
163 chmod 755 "$UPBIN"
164 kill -STOP "$D69PID"
165 "$UPBIN" upgrade --sock "$SOCK69" --allow-same-version > "$OUT.upref2" 2>&1 &
166 UPXPID=$!
167 defer_kill "$UPXPID"
168 _i=0
169 while [ "$(readlink "/proc/$UPXPID/exe" 2>/dev/null)" != "$UPBIN" ]; do
170 _i=$((_i + 1))
171 [ "$_i" -lt $(( 100 * TIME_SCALE )) ] || {
172 kill -CONT "$D69PID" 2>/dev/null || true
173 echo "e2e FAIL: upgrade: the candidate copy never exec'd; the refusal below"
174 echo " would have been about a binary that never asked"
175 exit 1; }
176 sleep 0.05
177 done
178 chmod 000 "$UPBIN"
179 kill -CONT "$D69PID"
180 set +e
181 wait "$UPXPID"
182 UPRC=$?
183 set -e
184 chmod 755 "$UPBIN"
185 [ "$UPRC" -eq 1 ] || {
186 echo "e2e FAIL: upgrade: an unexecutable candidate exited $UPRC, want 1:"
187 cat "$OUT.upref2"; exit 1; }
188 grep -qF "refused: path: not executable" "$OUT.upref2" || {
189 echo "e2e FAIL: upgrade: the refusal did not name the path check:"
190 cat "$OUT.upref2"; exit 1; }
191
192 # ...and two refusals changed NOTHING. A refusal that had torn a client
193 # down, or written the manifest and left the fd table half open, would
194 # still have printed the words above.
195 timeout 20 "$MUXA" status --sock "$SOCK69" > "$OUT.upsta" 2>&1 || {
196 echo "e2e FAIL: upgrade: the daemon stopped answering after refusing:"
197 cat "$OUT.upsta"; exit 1; }
198 wait_grid "$SOCK69" "shpid=$UPSHPID" "upgrade: a refusal cost the session its shell"
199
200 # 3. The exec itself.
201 set +e
202 "$MUXD" upgrade --sock "$SOCK69" --allow-same-version > "$OUT.upok" 2>&1
203 UPRC=$?
204 set -e
205 [ "$UPRC" -eq 0 ] || {
206 echo "e2e FAIL: upgrade: the upgrade exited $UPRC, want 0:"
207 cat "$OUT.upok" "$OUT.up.d"; exit 1; }
208 grep -qF "muxd: upgraded to $UPVER" "$OUT.upok" || {
209 echo "e2e FAIL: upgrade: no verdict naming the version it landed on:"
210 cat "$OUT.upok"; exit 1; }
211
212 # The pid is the feature. Not "a daemon is answering" — the same process,
213 # now running a different image, which /proc/PID/cmdline says in one word:
214 # a daemon that had restarted would carry the original `run --sock` argv.
215 kill -0 "$D69PID" 2>/dev/null || {
216 echo "e2e FAIL: upgrade: the daemon pid $D69PID is gone — that is a restart, not an upgrade"
217 cat "$OUT.up.d"; exit 1; }
218 tr '\0' ' ' < "/proc/$(real_pid "$D69PID")/cmdline" > "$OUT.upst2" 2>/dev/null || true
219 grep -qF -- "--resume-fd" "$OUT.upst2" || {
220 echo "e2e FAIL: upgrade: pid $D69PID is not running the resumed argv; it holds:"
221 cat "$OUT.upst2"; exit 1; }
222
223 # Counters CARRY, and it is asserted HERE rather than at the end for a
224 # reason about blame: the re-attach poll below counts `attaches` up from
225 # the reading taken before the exec, so a daemon that had zeroed its
226 # counters fails THAT poll — reported as a client that never came back,
227 # which is the wrong diagnosis of the wrong bug. `-ge`, because the
228 # re-attach this reading races has not necessarily landed yet.
229 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst2" 2>&1 || {
230 echo "e2e FAIL: upgrade: stats did not answer after the exec"; cat "$OUT.up.d"; exit 1; }
231 UPATT2=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst2")
232 UPSNAP2=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst2")
233 [ -n "$UPATT2" ] && [ -n "$UPSNAP2" ] || {
234 echo "e2e FAIL: upgrade: no counters read out of stats after the exec:"
235 cat "$OUT.upst2"; exit 1; }
236 [ "$UPATT2" -ge "$UPATT1" ] || {
237 echo "e2e FAIL: upgrade: attaches went $UPATT1 -> $UPATT2 across the exec; the"
238 echo " counters were reset, not carried, and an upgrade now looks like"
239 echo " a restart to anything sampling this daemon"
240 exit 1; }
241 [ "$UPSNAP2" -ge "$UPSNAP1" ] || {
242 echo "e2e FAIL: upgrade: snapshots went $UPSNAP1 -> $UPSNAP2 across the exec; the"
243 echo " counters were reset, not carried"
244 exit 1; }
245
246 # The client came back on its own. Counted with `attaches=`, not with
247 # `clients=`: the client is re-dialled within a millisecond of the exec, so
248 # a poll that only looked for clients=1 would be satisfied by the reading it
249 # took before the tear and would pass on a daemon that never dropped anyone.
250 _i=0
251 while : ; do
252 _upst=$("$MUXD" stats --sock "$SOCK69" 2>/dev/null || true)
253 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
254 case "$_upst" in
255 *"session 0 clients=1"*)
256 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
257 esac
258 _i=$((_i + 1))
259 [ "$_i" -lt $(( 200 * TIME_SCALE )) ] || {
260 echo "e2e FAIL: upgrade: the attached client never re-attached after the exec;"
261 echo " stats holds: $_upst"
262 cat "$OUT.up.d"; exit 1; }
263 sleep 0.1
264 done
265
266 # The grid crossed with the process: the marker the client typed, and the
267 # pid of the shell that is still running under the new image.
268 wait_grid "$SOCK69" "up-pre" "upgrade: the pre-upgrade marker did not survive the exec"
269 wait_grid "$SOCK69" "shpid=$UPSHPID" "upgrade: the session is a NEW shell, not the one that crossed"
270
271 # The FIRST muxa run after the exec, and it is also what releases the
272 # attached client's `expect up-resumed` above — one command, two claims.
273 set +e
274 timeout 40 "$MUXA" run --sock "$SOCK69" --timeout 20000 'echo up-resumed' > "$OUT.upm2" 2>&1
275 UPRC=$?
276 set -e
277 [ "$UPRC" -eq 0 ] || {
278 echo "e2e FAIL: upgrade: the FIRST muxa run after the exec exited $UPRC — a carried"
279 echo " return watermark is a watermark from the future and no return passes it:"
280 cat "$OUT.upm2"; exit 1; }
281 grep -qF '"mechanism":"marks"' "$OUT.upm2" || {
282 echo "e2e FAIL: upgrade: the first await after the exec fell off marks:"
283 cat "$OUT.upm2"; exit 1; }
284 grep -qF '"exit_code":0' "$OUT.upm2" || {
285 echo "e2e FAIL: upgrade: no real exit code came back after the exec:"
286 cat "$OUT.upm2"; exit 1; }
287 # The client's own verdict: the title came back on the re-attach, the
288 # marker arrived through the new image, and a keystroke typed AFTER the
289 # exec reached the shell.
290 set +e
291 wait "$UPPCPID"
292 UPRC=$?
293 set -e
294 UPPCPID=""
295 [ "$UPRC" -eq 0 ] || {
296 echo "e2e FAIL: upgrade: the attached client exited $UPRC across the exec:"
297 cat "$OUT.uppc" "$OUT.upcap.err"; exit 1; }
298
299 # A NONZERO exit code too, because `"exit_code":0` is also what a mechanism
300 # that guesses would say. After the client has gone, not beside it: two
301 # writers typing into one shell's pty interleave, and the leg would be
302 # racing itself. Spelled as a CHILD, not as `exit 7`, which is the session
303 # shell's own exit — the session would end, the daemon with it, and the
304 # counter reading below would be taken from nothing. Not `(exit 7)` either:
305 # a subshell sets `_mux_ran` in the subshell, so shellint's `precmd_fn`
306 # sees an untouched prompt in the parent and emits no `D;` mark at all
307 # (measured: the await ran its whole budget out).
308 timeout 40 "$MUXA" run --sock "$SOCK69" --timeout 20000 'sh -c "exit 7"' > "$OUT.upm3" 2>&1 || true
309 grep -qF '"exit_code":7' "$OUT.upm3" || {
310 echo "e2e FAIL: upgrade: the shell's own exit code did not survive the exec:"
311 cat "$OUT.upm3"; exit 1; }
312
313 # ...and the re-attach was COUNTED onto them rather than replacing them.
314 # The carry was asserted above with `-ge`; this is the other half, taken
315 # once the client's own snapshot has certainly been served.
316 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst2" 2>&1
317 UPATT2=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst2")
318 UPSNAP2=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst2")
319 [ -n "$UPATT2" ] && [ -n "$UPSNAP2" ] || {
320 echo "e2e FAIL: upgrade: no counters read out of stats after the client left:"
321 cat "$OUT.upst2"; exit 1; }
322 [ "$UPATT2" -gt "$UPATT1" ] || {
323 echo "e2e FAIL: upgrade: attaches is still $UPATT2 — the re-attach after the exec"
324 echo " was never counted"
325 exit 1; }
326 [ "$UPSNAP2" -gt "$UPSNAP1" ] || {
327 echo "e2e FAIL: upgrade: snapshots is still $UPSNAP2 — the re-attach was served"
328 echo " without a snapshot, which no fresh epoch allows"
329 exit 1; }
330
331 assert_stopped "$SOCK69" "$D69PID" "upgrade" "$OUT.upstop"
332 D69PID=""
333 rm -f "$UPBIN"
334 rm -rf "$UPHOME"
335 ok "muxd upgrade keeps the shell, its pid, its title, its marks and its counters"
336
337 # --- ...and a candidate that cannot adopt hands the daemon back -----------
338 #
339 # The rollback: the new image fails partway through adoption and execs the
340 # OLD binary — whose path the manifest carries — with the same manifest fd.
341 # Armed through the ENVIRONMENT rather than the flag it shares a name with,
342 # and that is not a convenience: `execUpgrade` builds a fixed argv
343 # (`run --resume-fd N`), so nothing typed here can put `--resume-fail-at`
344 # in front of the candidate. The daemon's environment is what crosses an
345 # exec, so the abort is armed on the daemon before any upgrade asks.
346 #
347 # The daemon's own log is the only thing that can say a rollback HAPPENED —
348 # `muxd upgrade` prints its verdict the moment the daemon accepts, and the
349 # process that would have corrected it is gone. Without that line every
350 # assertion below would pass just as well on an upgrade that simply worked.
351 mkdir -p "$UPHOME"
352 MUX_SHELL_INTEGRATION=1 MUX_RESUME_FAIL_AT=session HOME="$UPHOME" \
353 start_daemon "$SOCK70" "$OUT.url.d" "rollback daemon never bound" --shell /bin/bash
354 D70PID=$DPID
355
356 timeout 20 "$MUXA" run --sock "$SOCK70" --timeout 8000 'echo rollpid=$$' > "$OUT.urlm1" 2>&1 || {
357 echo "e2e FAIL: rollback: muxa run failed before the upgrade:"
358 cat "$OUT.urlm1"; exit 1; }
359 wait_grid "$SOCK70" "rollpid=[0-9]" "rollback: the session shell never printed its pid"
360 UPROLLPID=$(dump_session "$SOCK70" | sed -n 's/.*rollpid=\([0-9][0-9]*\).*/\1/p' | head -1)
361 [ -n "$UPROLLPID" ] || {
362 echo "e2e FAIL: rollback: no shell pid read off the grid"; dump_session "$SOCK70"; exit 1; }
363
364 set +e
365 "$MUXD" upgrade --sock "$SOCK70" --allow-same-version > "$OUT.urlup" 2>&1
366 UPRC=$?
367 set -e
368 [ "$UPRC" -eq 0 ] || {
369 echo "e2e FAIL: rollback: upgrade exited $UPRC — the rolled-back daemon must serve"
370 echo " well enough for the confirm to pass:"
371 cat "$OUT.urlup" "$OUT.url.d"; exit 1; }
372 wait_for "$OUT.url.d" "adoption failed at session" 15 || {
373 echo "e2e FAIL: rollback: the candidate adopted instead of aborting — the abort"
374 echo " was not armed, and nothing below tests a rollback:"
375 cat "$OUT.url.d"; exit 1; }
376 grep -qF "exec'ing" "$OUT.url.d" || {
377 echo "e2e FAIL: rollback: the abort never named the binary it went back to:"
378 cat "$OUT.url.d"; exit 1; }
379
380 kill -0 "$D70PID" 2>/dev/null || {
381 echo "e2e FAIL: rollback: the daemon pid $D70PID is gone; a failed adoption took"
382 echo " the sessions with it"
383 cat "$OUT.url.d"; exit 1; }
384 wait_grid "$SOCK70" "rollpid=$UPROLLPID" "rollback: the session lost its shell to a failed adoption"
385 timeout 40 "$MUXA" run --sock "$SOCK70" --timeout 20000 'echo roll-post' > "$OUT.urlm2" 2>&1 || {
386 echo "e2e FAIL: rollback: the daemon that came back cannot run a command:"
387 cat "$OUT.urlm2" "$OUT.url.d"; exit 1; }
388 grep -qF '"mechanism":"marks"' "$OUT.urlm2" || {
389 echo "e2e FAIL: rollback: the re-adopted session lost its marks:"
390 cat "$OUT.urlm2"; exit 1; }
391
392 assert_stopped "$SOCK70" "$D70PID" "rollback" "$OUT.urlstop"
393 D70PID=""
394 rm -rf "$UPHOME"
395 ok "a candidate that cannot adopt execs the old binary back, session and marks intact"
396
397 # --- ...and the session's agent socket crosses the exec -------------------
398 #
399 # The per-session agent listener is a descriptor the daemon owns and the
400 # shell reaches by path ($SSH_AUTH_SOCK, named after the daemon's pid). It
401 # has to cross the exec on both counts: the fd by inheritance, the path
402 # because the pid does not change. Nothing else in this suite would notice
403 # if it did not — the agent channels themselves are deliberately NOT
404 # carried, so the only thing that can say the listener survived is a real
405 # `ssh-add -l` answering through a re-dialled `-A` client.
406 #
407 # Its own agent rather than the forwarding legs' AGENT48: that one is left
408 # under SIGSTOP by the mute-offerer leg, and a mute agent is exactly what
409 # this leg cannot tell from a broken listener.
410 start_daemon "$SOCK71" "$OUT.uag.d" "agent-upgrade daemon never bound" --shell /bin/sh
411 D71PID=$DPID
412
413 ssh-agent -a "$UPAGENT" > "$OUT.uagenv" 2>&1
414 UPAGPID=$(sed -n 's/.*SSH_AGENT_PID=\([0-9]*\).*/\1/p' "$OUT.uagenv")
415 defer_kill "$UPAGPID"
416 [ -n "$UPAGPID" ] || {
417 echo "e2e FAIL: agent-upgrade: ssh-agent printed no pid for the trap to hold it by:"
418 cat "$OUT.uagenv"; exit 1; }
419 ssh-keygen -q -t ed25519 -N '' -C mux-e2e-upgrade -f "$UPAGKEY"
420 SSH_AUTH_SOCK="$UPAGENT" ssh-add "$UPAGKEY" > /dev/null 2>&1
421 UPFP=$(ssh-keygen -lf "$UPAGKEY" | awk '{print $2}')
422 case "$UPFP" in
423 SHA256:?*) ;;
424 *) echo "e2e FAIL: agent-upgrade: ssh-keygen -lf gave no fingerprint: [$UPFP]"; exit 1;;
425 esac
426
427 pipe_mux "$OUT.uag" "$OUT.uag.err" env SSH_AUTH_SOCK="$UPAGENT" timeout 90 \
428 "$MUX" -A --sock "$SOCK71"
429 UPATT1=$("$MUXD" stats --sock "$SOCK71" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
430 [ -n "$UPATT1" ] || { echo "e2e FAIL: agent-upgrade: no attaches counter to compare against"; exit 1; }
431 # `agpre=` rather than the fingerprint alone: the shell echoes what is
432 # typed, and `$?` is not expanded in the echo, so only ssh-add's own status
433 # can put a digit there. The fingerprint is asserted beside it because a
434 # status of 0 says an agent answered, not that it was the client's.
435 pipe_send 'ssh-add -l; echo agpre=$?\n'
436 await_out "$OUT.uag" "agpre=0" "agent-upgrade: ssh-add never answered before the upgrade"
437 # Counted by OCCURRENCE, not by matching line: this capture is an escape
438 # stream whose one "line" is the whole session, so `grep -c` would answer 1
439 # for both answers and the comparison after the exec would be vacuous.
440 UPFP1=$(grep -aoF -- "$UPFP" "$OUT.uag" | wc -l)
441 [ "$UPFP1" -ge 1 ] || {
442 echo "e2e FAIL: agent-upgrade: the client's key never came back before the upgrade"
443 echo " (wanted $UPFP)"; exit 1; }
444
445 set +e
446 "$MUXD" upgrade --sock "$SOCK71" --allow-same-version > "$OUT.uagup" 2>&1
447 UPRC=$?
448 set -e
449 [ "$UPRC" -eq 0 ] || {
450 echo "e2e FAIL: agent-upgrade: upgrade exited $UPRC:"
451 cat "$OUT.uagup" "$OUT.uag.d"; exit 1; }
452 _i=0
453 while : ; do
454 _upst=$("$MUXD" stats --sock "$SOCK71" 2>/dev/null || true)
455 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
456 case "$_upst" in
457 *"session 0 clients=1"*)
458 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
459 esac
460 _i=$((_i + 1))
461 [ "$_i" -lt $(( 200 * TIME_SCALE )) ] || {
462 echo "e2e FAIL: agent-upgrade: the -A client never re-attached; stats holds: $_upst"
463 cat "$OUT.uag.d"; exit 1; }
464 sleep 0.1
465 done
466
467 # The same question a second time, and `agdone=` is a needle the first
468 # answer cannot satisfy.
469 pipe_send 'ssh-add -l; echo agdone=$?\n'
470 await_out "$OUT.uag" "agdone=0" "agent-upgrade: ssh-add stopped answering across the exec"
471 UPFP2=$(grep -aoF -- "$UPFP" "$OUT.uag" | wc -l)
472 [ "$UPFP2" -gt "$UPFP1" ] || {
473 echo "e2e FAIL: agent-upgrade: ssh-add exited 0 but the key came back $UPFP1 time(s)"
474 echo " before the exec and $UPFP2 after — no NEW answer crossed"; exit 1; }
475 pipe_detach "agent-upgrade client"
476
477 assert_stopped "$SOCK71" "$D71PID" "agent-upgrade" "$OUT.uagstop"
478 D71PID=""
479 softkill "$UPAGPID" || true
480 wait_pid_gone "$UPAGPID" "agent-upgrade: ssh-agent killed by tracked pid"
481 UPAGPID=""
482 ok "the session's agent socket crosses the exec: ssh-add -l answers again"
483
484 # --- ...and a QUIC client is serving again in a breath, not after a timeout
485 #
486 # Unix clients need no goodbye — their fds are close-on-exec and the exec is
487 # their EOF. A QUIC peer has no such thing: silence is indistinguishable from
488 # a bad network, and a client that waited it out would sit on a blank wall
489 # for the whole idle timeout. So the daemon sends CONNECTION_CLOSE to every
490 # connection before it execs, and what that buys is the number asserted
491 # here: back in service in a fraction of `--quic-idle-ms`, not after it.
492 head -c 32 /dev/urandom > "$UPKEY"
493 chmod 600 "$UPKEY"
494 start_daemon "$SOCK72" "$OUT.uqc.d" "quic-upgrade daemon never bound" --shell /bin/sh \
495 --quic "127.0.0.1:$UPQPORT" --key "$UPKEY" --quic-idle-ms 15000
496 D72PID=$DPID
497
498 pipe_mux "$OUT.uqc" "$OUT.uqc.err" timeout 90 "$MUX" "quic://127.0.0.1:$UPQPORT" \
499 --key "$UPKEY" --quic-idle-ms 15000
500 pipe_send 'printf "uq-%%s\\n" pre\n'
501 await_out "$OUT.uqc" "uq-pre" "quic-upgrade: the QUIC client never got its pre-upgrade marker"
502
503 UPATT1=$("$MUXD" stats --sock "$SOCK72" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
504 [ -n "$UPATT1" ] || { echo "e2e FAIL: quic-upgrade: no attaches counter to compare against"; exit 1; }
505 UPT0=$(date +%s%N)
506 set +e
507 "$MUXD" upgrade --sock "$SOCK72" --allow-same-version > "$OUT.uqup" 2>&1
508 UPRC=$?
509 set -e
510 [ "$UPRC" -eq 0 ] || {
511 echo "e2e FAIL: quic-upgrade: upgrade exited $UPRC:"
512 cat "$OUT.uqup" "$OUT.uqc.d"; exit 1; }
513 _i=0
514 while : ; do
515 _upst=$("$MUXD" stats --sock "$SOCK72" 2>/dev/null || true)
516 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
517 case "$_upst" in
518 *"session 0 clients=1"*)
519 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
520 esac
521 _i=$((_i + 1))
522 [ "$_i" -lt $(( 400 * TIME_SCALE )) ] || {
523 echo "e2e FAIL: quic-upgrade: the QUIC client never re-attached; stats holds: $_upst"
524 cat "$OUT.uqc.d"; exit 1; }
525 sleep 0.05
526 done
527 UPT1=$(date +%s%N)
528 UPMS=$(( (UPT1 - UPT0) / 1000000 ))
529 # Half the idle timeout, which is the only bound that means anything here:
530 # a client that had waited the connection out would arrive at 15000ms and
531 # still pass a generous absolute budget. Measured at ~60ms.
532 [ "$UPMS" -lt $(( 7500 * TIME_SCALE )) ] || {
533 echo "e2e FAIL: quic-upgrade: the QUIC client took ${UPMS}ms to come back —"
534 echo " that is the idle timeout expiring, not a CONNECTION_CLOSE"
535 exit 1; }
536
537 # ...and it is a session, not just a connection: a round trip that did not
538 # exist before the exec.
539 pipe_send 'printf "uq-%%s\\n" post\n'
540 await_out "$OUT.uqc" "uq-post" "quic-upgrade: the reconnected QUIC client cannot reach its shell"
541 pipe_detach "quic-upgrade client"
542
543 assert_stopped "$SOCK72" "$D72PID" "quic-upgrade" "$OUT.uqstop"
544 D72PID=""
545 ok "a QUIC client is served again within a breath of the exec, not after the idle timeout"
546
547 # --- muxd stop returns when the PROCESS is gone, not when the path is
548 #
549 # The unlink is the first thing a stopping daemon does; reaping its shells
550 # and deleting its dirs come after. A stop that said "stopped" at the
551 # unlink handed a scripted `muxd start`, or a supervisor's "is it down",
552 # a daemon still running — `wait_pid_gone` after every stop in this suite
553 # was that gap, papered. Shells that ignore TERM make the window a real
554 # grace rather than a race the assertion could win by luck, and two
555 # sessions make it a table's grace, not one shell's.
556 printf '#!/bin/sh\ntrap "" HUP TERM\nwhile :; do sleep 1; done\n' > "$OUT.sg.sh"
557 chmod +x "$OUT.sg.sh"
558 start_daemon "$SOCK73" "$OUT.sg.d" "stop-gone daemon never bound" --shell "$OUT.sg.sh"
559 D73PID=$DPID
560 pipe_mux "$OUT.sg" "" timeout 60 "$MUX" --sock "$SOCK73" --session second
561 sleep 0.5
562 pipe_detach "stop-gone: the second session's client"
563 [ "$("$MUXD" stats --sock "$SOCK73" | sed -n 's/.*sessions=\([0-9]*\).*/\1/p')" = "2" ] || {
564 echo "e2e FAIL: stop-gone: wanted two sessions of stubborn shells"; exit 1; }
565 D73REAL=$(real_pid "$D73PID")
566 "$MUXD" stop --sock "$SOCK73" 2> "$OUT.sg.stop" || {
567 echo "e2e FAIL: stop-gone: stop failed"; cat "$OUT.sg.stop"; exit 1; }
568 # Asked of the OS the instant stop returns — no wait, no retry.
569 if kill -0 "$D73REAL" 2>/dev/null; then
570 echo "e2e FAIL: stop said stopped, but pid $D73PID is still running"; cat "$OUT.sg.stop"; exit 1
571 fi
572 grep -q '^muxd: stopped' "$OUT.sg.stop" || {
573 echo "e2e FAIL: stop-gone: stop did not report stopped"; cat "$OUT.sg.stop"; exit 1; }
574 D73PID=""
575 ok "muxd stop returns when the process is gone, not when the socket is"
576
577 # The pins. Literals, not variables set from counting something else —
578 # "assert the literal, never the constant the code under test reads"
579 # (decisions.md, M10). The two figures live at the END of this file and
580 # nowhere else: a count repeated in prose is a count nothing gates, and
581 # both of the ones that used to stand here had drifted away from the pins
582 # they described. Adding a scenario means editing those two literals by
583 # hand, on purpose.
584 #
585 # Why most legs carry no convergence point. `assert_converged` replays one
586 # client capture into an engine and diffs it against one daemon grid, so it
587 # can only speak where BOTH exist and the grid is the subject. It says
588 # nothing about a side channel (OSC 52, a window title, pty modes, a
589 # fingerprint ssh printed, bytes counted rather than read), about a file an
590 # editor or the wall wrote, about an exit code or a status field, about
591 # WHICH of two sessions a keystroke reached, about scrollback the live grid
592 # does not hold, about the absence of bytes two agreeing empties would
593 # agree on anyway, or about a leg that never attaches at all. It also does
594 # not claim to reproduce a full-screen application's redraws byte for byte,
595 # which is why the alternate-screen legs stay out. A leg that asserts on
596 # one of those and adds a convergence point would be pinning a fact every
597 # leg above already establishes.