a73x

a6ad1dca

test: the upgrade keeps the shell, its pid, its title, its marks, its agent

a73x   2026-08-26 15:17

Commit message
test: the upgrade keeps the shell, its pid, its title, its marks, its agent

Four legs at the end of the suite, one per daemon lifecycle the exec has
to survive.

The same-binary leg is the gate: a session whose shell pid is on the grid,
two refusals that must change nothing, `muxd upgrade`, and then every one
of those things again on the far side — including the FIRST `muxa run`
after the exec, which is the assertion the carried return watermark
failed. The shell's pid is the witness because nothing on the wire carries
one: a client reconnecting to a freshly started daemon sees the same
snapshot, but a new daemon forks a new shell.

The `path: not executable` refusal is spelled the only way `muxd upgrade`
allows — it offers /proc/self/exe, so the exec bit comes off a copy that
has already started. The daemon is held under SIGSTOP so it cannot read
the request before the chmod lands, and `readlink /proc/PID/exe` is what
says the copy exec'd; without both, the leg is a race that asserts an exec
failure half the time.

The rollback is armed through MUX_RESUME_FAIL_AT in the daemon's
environment, because a real upgrade builds a fixed argv and no flag typed
here can reach the candidate. The daemon's log is the only thing that can
say a rollback happened at all: `muxd upgrade` prints its verdict the
moment the daemon accepts, and the image that would have corrected it is
gone.

The agent leg is the only cover for the per-session listener fd crossing
the exec, and the QUIC leg's stopwatch is bounded at half the idle timeout
— the one budget that tells CONNECTION_CLOSE from waiting a connection
out.

/bin/bash becomes a suite requirement for the fourth's reason: marks exist
only in a shell mux injects OSC 133 into, and on /bin/sh those legs would
go green having proved nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test/e2e.sh
Old New
@@ -77,6 +77,19 @@ for _agtool in ssh-agent ssh-add ssh-keygen; do
77 exit 1; } 77 exit 1; }
78 done 78 done
79 79
80 # And the fifth, required for the fourth's reason: the upgrade legs assert
81 # that `muxa` still answers under the `marks` mechanism after the exec, and
82 # marks exist only in a shell mux injects OSC 133 into (shellint's
83 # `bash_init`). Run against /bin/sh those legs would still go green having
84 # proved nothing about the one piece of session state the manifest carries
85 # that nothing else can rebuild.
86 [ -x /bin/bash ] || {
87 echo "e2e FAIL: this suite needs /bin/bash (the upgrade legs check that OSC 133"
88 echo " marks survive the exec, and mux injects them into bash and zsh"
89 echo " only); install bash, or lose the only check that an upgraded"
90 echo " daemon can still report a command's real exit code"
91 exit 1; }
92
80 SOCK="${TMPDIR:-/tmp}/muxd-e2e-$$.sock" 93 SOCK="${TMPDIR:-/tmp}/muxd-e2e-$$.sock"
81 OUT="${TMPDIR:-/tmp}/mux-e2e-out-$$" 94 OUT="${TMPDIR:-/tmp}/mux-e2e-out-$$"
82 # M10: hermetic XDG homes. Key-default scenarios must see OUR key or none, 95 # M10: hermetic XDG homes. Key-default scenarios must see OUR key or none,
@@ -429,6 +442,40 @@ D67PID=""
429 W5PID="" 442 W5PID=""
430 D68PID="" 443 D68PID=""
431 W6PID="" 444 W6PID=""
445 # `muxd upgrade` — four daemons, because each one is a different daemon
446 # LIFECYCLE and no two can share a process: the same-binary leg's daemon
447 # ends up running a second image, the rollback leg's is born with the abort
448 # armed in its environment, the agent leg's holds an `-A` client, and the
449 # QUIC leg's owns a UDP socket. They run one after another, so the last of
450 # them is the only leg here that needs a port.
451 SOCK69="${TMPDIR:-/tmp}/muxd-e2e-upgrade-$$.sock"
452 SOCK70="${TMPDIR:-/tmp}/muxd-e2e-uproll-$$.sock"
453 SOCK71="${TMPDIR:-/tmp}/muxd-e2e-upagent-$$.sock"
454 SOCK72="${TMPDIR:-/tmp}/muxd-e2e-upquic-$$.sock"
455 # The next 4000-wide band DOWN from the hydrate leg's 6000: every base from
456 # 11000 up is taken, and the 61000+ tail above the ephemeral range is spoken
457 # for by the two browser ports. A collision here reads as this leg's daemon
458 # failing to bind, the same verdict every other band gives.
459 UPQPORT=$(( 2000 + ($$ % 4000) ))
460 UPKEY="${TMPDIR:-/tmp}/mux-e2e-upkey-$$"
461 # A HOME of its own for the session shell. The title assertion below is
462 # about a title the SESSION set, and a distribution's rc file repaints the
463 # window title from PROMPT_COMMAND on every prompt (shellint.bash_init says
464 # so of Arch's) — read out of the developer's own dotfiles, this leg would
465 # assert on whatever their prompt happens to spell.
466 UPHOME="${TMPDIR:-/tmp}/mux-e2e-uphome-$$"
467 # The candidate binary the refusal leg offers: a copy, so its exec bit can
468 # be taken away without touching the one every other scenario runs.
469 UPBIN="${TMPDIR:-/tmp}/mux-e2e-upcand-$$"
470 UPAGENT="${TMPDIR:-/tmp}/mux-e2e-upagent-$$.sock"
471 UPAGKEY="${TMPDIR:-/tmp}/mux-e2e-upagkey-$$"
472 D69PID=""
473 D70PID=""
474 D71PID=""
475 D72PID=""
476 UPAGPID=""
477 UPPCPID=""
478 UPCPID=""
432 D54PID="" 479 D54PID=""
433 D55PID="" 480 D55PID=""
434 D56PID="" 481 D56PID=""
@@ -1388,6 +1435,17 @@ cleanup() {
1388 [ -n "${D64PID:-}" ] && kill "$D64PID" 2>/dev/null || true 1435 [ -n "${D64PID:-}" ] && kill "$D64PID" 2>/dev/null || true
1389 [ -n "${D65PID:-}" ] && kill "$D65PID" 2>/dev/null || true 1436 [ -n "${D65PID:-}" ] && kill "$D65PID" 2>/dev/null || true
1390 [ -n "${D66PID:-}" ] && kill "$D66PID" 2>/dev/null || true 1437 [ -n "${D66PID:-}" ] && kill "$D66PID" 2>/dev/null || true
1438 # The upgrade legs' daemons. CONT first for AGENT48PID's reason turned
1439 # on a daemon: the non-executable-candidate refusal holds this one under
1440 # SIGSTOP while it takes the exec bit off the candidate, and a run that
1441 # dies inside that window leaves a process no SIGTERM can reach.
1442 [ -n "${D69PID:-}" ] && kill -CONT "$D69PID" 2>/dev/null || true
1443 [ -n "${D69PID:-}" ] && kill "$D69PID" 2>/dev/null || true
1444 [ -n "${D70PID:-}" ] && kill "$D70PID" 2>/dev/null || true
1445 [ -n "${D71PID:-}" ] && kill "$D71PID" 2>/dev/null || true
1446 [ -n "${D72PID:-}" ] && kill "$D72PID" 2>/dev/null || true
1447 [ -n "${UPPCPID:-}" ] && kill "$UPPCPID" 2>/dev/null || true
1448 [ -n "${UPCPID:-}" ] && kill "$UPCPID" 2>/dev/null || true
1391 # The ssh-agents the forwarding legs start. Not mux processes and so not 1449 # The ssh-agents the forwarding legs start. Not mux processes and so not
1392 # the leak sweep's business, but they are daemons this file forked: left 1450 # the leak sweep's business, but they are daemons this file forked: left
1393 # alive they outlive the suite holding a private key, which is the one 1451 # alive they outlive the suite holding a private key, which is the one
@@ -1398,6 +1456,7 @@ cleanup() {
1398 [ -n "$AGENT48PID" ] && kill "$AGENT48PID" 2>/dev/null || true 1456 [ -n "$AGENT48PID" ] && kill "$AGENT48PID" 2>/dev/null || true
1399 [ -n "$AGENT49APID" ] && kill "$AGENT49APID" 2>/dev/null || true 1457 [ -n "$AGENT49APID" ] && kill "$AGENT49APID" 2>/dev/null || true
1400 [ -n "$AGENT49BPID" ] && kill "$AGENT49BPID" 2>/dev/null || true 1458 [ -n "$AGENT49BPID" ] && kill "$AGENT49BPID" 2>/dev/null || true
1459 [ -n "${UPAGPID:-}" ] && kill "$UPAGPID" 2>/dev/null || true
1401 # The stops still precede the socket rm below, like SOCK14-17 above: 1460 # The stops still precede the socket rm below, like SOCK14-17 above:
1402 # unlinking a socket first would leave a live daemon nothing could reach 1461 # unlinking a socket first would leave a live daemon nothing could reach
1403 # by path. 1462 # by path.
@@ -1444,6 +1503,10 @@ cleanup() {
1444 [ -S "$SOCK66" ] && "$MUXD" stop --sock "$SOCK66" 2>/dev/null || true 1503 [ -S "$SOCK66" ] && "$MUXD" stop --sock "$SOCK66" 2>/dev/null || true
1445 [ -S "$SOCK67" ] && "$MUXD" stop --sock "$SOCK67" 2>/dev/null || true 1504 [ -S "$SOCK67" ] && "$MUXD" stop --sock "$SOCK67" 2>/dev/null || true
1446 [ -S "$SOCK68" ] && "$MUXD" stop --sock "$SOCK68" 2>/dev/null || true 1505 [ -S "$SOCK68" ] && "$MUXD" stop --sock "$SOCK68" 2>/dev/null || true
1506 [ -S "$SOCK69" ] && "$MUXD" stop --sock "$SOCK69" 2>/dev/null || true
1507 [ -S "$SOCK70" ] && "$MUXD" stop --sock "$SOCK70" 2>/dev/null || true
1508 [ -S "$SOCK71" ] && "$MUXD" stop --sock "$SOCK71" 2>/dev/null || true
1509 [ -S "$SOCK72" ] && "$MUXD" stop --sock "$SOCK72" 2>/dev/null || true
1447 1510
1448 # ---- the leak sweep (hygiene kit, 6a) ---- 1511 # ---- the leak sweep (hygiene kit, 6a) ----
1449 # Here rather than at the bottom of the file, which `set -e` reaches only 1512 # Here rather than at the bottom of the file, which `set -e` reaches only
@@ -1461,7 +1524,8 @@ cleanup() {
1461 "$D38PID" "$D39PID" "$D40PID" "$D41PID" "$D42PID" "$D43PID" "$D54PID" \ 1524 "$D38PID" "$D39PID" "$D40PID" "$D41PID" "$D42PID" "$D43PID" "$D54PID" \
1462 "$D55PID" "$D56PID" "$D57PID" "$D58PID" "$D59PID" \ 1525 "$D55PID" "$D56PID" "$D57PID" "$D58PID" "$D59PID" \
1463 "$D60PID" "$D61PID" "$D62PID" "$D63PID" "$D64PID" "$D65PID" \ 1526 "$D60PID" "$D61PID" "$D62PID" "$D63PID" "$D64PID" "$D65PID" \
1464 "$D66PID" "$D67PID" "$D68PID" 1527 "$D66PID" "$D67PID" "$D68PID" "$D69PID" "$D70PID" \
1528 "$D71PID" "$D72PID"
1465 _leak=0 1529 _leak=0
1466 leak_sweep "$_rc" || _leak=1 1530 leak_sweep "$_rc" || _leak=1
1467 1531
@@ -1569,7 +1633,21 @@ cleanup() {
1569 "$OUT.wgghost" "$OUT.wgstop" "$OUT.wgsta0" "$OUT.wgexit" \ 1633 "$OUT.wgghost" "$OUT.wgstop" "$OUT.wgsta0" "$OUT.wgexit" \
1570 "$OUT.wgws3" "$OUT.wgws3.err" "$SOCK67" "$WGKEY" \ 1634 "$OUT.wgws3" "$OUT.wgws3.err" "$SOCK67" "$WGKEY" \
1571 "$OUT.sp.d" "$OUT.sph" "$OUT.spws" "$OUT.spws.err" \ 1635 "$OUT.sp.d" "$OUT.sph" "$OUT.spws" "$OUT.spws.err" \
1572 "$OUT.spghost" "$OUT.spstop" "$SOCK68" 1636 "$OUT.spghost" "$OUT.spstop" "$SOCK68" \
1637 "$OUT.up.d" "$OUT.upcap" "$OUT.upcap.err" "$OUT.uppc" \
1638 "$OUT.upm1" "$OUT.upm2" "$OUT.upm3" "$OUT.upst1" "$OUT.upst2" \
1639 "$OUT.upref1" "$OUT.upref2" "$OUT.upok" "$OUT.upsta" "$OUT.upstop" \
1640 "$SOCK69" "$UPBIN" \
1641 "$OUT.url.d" "$OUT.urlm1" "$OUT.urlm2" "$OUT.urlup" "$OUT.urlstop" \
1642 "$SOCK70" \
1643 "$OUT.uag.d" "$OUT.uag" "$OUT.uag.err" "$OUT.uag.in" "$OUT.uagenv" \
1644 "$OUT.uagup" "$OUT.uagstop" "$SOCK71" \
1645 "$UPAGENT" "$UPAGKEY" "$UPAGKEY.pub" \
1646 "$OUT.uqc.d" "$OUT.uqc" "$OUT.uqc.err" "$OUT.uqc.in" "$OUT.uqup" \
1647 "$OUT.uqsta" "$OUT.uqstop" "$SOCK72" "$UPKEY"
1648 # The upgrade leg's private HOME, for the reason it has one: a session
1649 # shell that read the developer's rc files would set its own title.
1650 rm -rf "$UPHOME"
1573 # ...and the state homes those legs read their walls back out of, plus 1651 # ...and the state homes those legs read their walls back out of, plus
1574 # the spin leg's shim dir (its dial log lives inside it). 1652 # the spin leg's shim dir (its dial log lives inside it).
1575 rm -rf "$HYSTATE" "$WGSTATE" "$SPSTATE" "$SPDIR" 1653 rm -rf "$HYSTATE" "$WGSTATE" "$SPSTATE" "$SPDIR"
@@ -9019,8 +9097,510 @@ rm -rf "$SPSTATE" "$SPDIR"
9019 ok "a refusal the birth cannot fix backs off instead of spinning" 9097 ok "a refusal the birth cannot fix backs off instead of spinning"
9020 9098
9021 9099
9022 [ "$OK_COUNT" = "77" ] || { 9100 # --- muxd upgrade: the daemon becomes the new binary, holding everything ---
9023 echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 77 —" 9101 #
9102 # `muxd upgrade` execs the candidate OVER the running daemon: same pid, same
9103 # children, same descriptors. Nothing on the wire can carry the proof — a
9104 # client reconnecting to a freshly started daemon sees exactly the same
9105 # snapshot — so the witness is the session shell's own pid, put on the grid
9106 # before the exec and read off it after. A restart cannot fake that: a new
9107 # daemon forks a new shell, and a new shell has a new pid.
9108 #
9109 # One daemon, one attached client, and the whole story in order: what the
9110 # session had before, two refusals that must change nothing, the exec, and
9111 # then every one of those things again on the far side.
9112 #
9113 # bash with marks on, because two of the claims are the mechanism's:
9114 # `muxa run` answers under `marks` before the exec and must answer under
9115 # `marks` after it. The FIRST one after is the assertion that matters —
9116 # the manifest carries a return watermark, and one carried into the new
9117 # seq space is a watermark from the future that no return can pass, which
9118 # measured as the first await after every upgrade timing out at 30s while
9119 # the shell had already answered.
9120 UPVER=$("$MUXD" --version | awk '{print $2}')
9121 case "$UPVER" in
9122 ?*.?*) ;;
9123 *) echo "e2e FAIL: upgrade: muxd --version gave no version to refuse with: [$UPVER]"
9124 exit 1;;
9125 esac
9126 mkdir -p "$UPHOME"
9127 MUX_SHELL_INTEGRATION=1 HOME="$UPHOME" "$MUXD" run --sock "$SOCK69" \
9128 --shell /bin/bash > "$OUT.up.d" 2>&1 &
9129 D69PID=$!
9130 wait_sock "$SOCK69" "$OUT.up.d" "upgrade daemon never bound"
9131
9132 # Marks BEFORE the exec, so "marks after" is a comparison rather than a
9133 # hope: a daemon that never had them would fail this line first.
9134 timeout 20 "$MUXA" run --sock "$SOCK69" --timeout 8000 'echo up-marks-pre' > "$OUT.upm1" 2>&1 || {
9135 echo "e2e FAIL: upgrade: muxa run failed before the upgrade even started:"
9136 cat "$OUT.upm1"; exit 1; }
9137 grep -qF '"mechanism":"marks"' "$OUT.upm1" || {
9138 echo "e2e FAIL: upgrade: the session had no marks to lose:"
9139 cat "$OUT.upm1"; exit 1; }
9140
9141 # The attached client, on a real pty, held across the exec. Its script
9142 # stops on `expect up-resumed`, which nothing in the script produces — the
9143 # suite's own post-upgrade `muxa run` does, below. That is the rendezvous:
9144 # the fixture reads the master the whole time it waits, so the capture is
9145 # continuous across the tear, and the needle can only arrive through the
9146 # connection the client re-dialled.
9147 #
9148 # `unset PROMPT_COMMAND; PROMPT_COMMAND=_mux_precmd` keeps mux's own precmd
9149 # (shellint's `precmd_fn`, which is where the marks come from) and drops
9150 # every other member. A distribution rc appends a title-setting member
9151 # there, and it repaints the title on every prompt — the title this leg
9152 # sets would be gone by the next one. `unset` first because a scalar
9153 # assignment onto bash's array spelling lands on element 0 and leaves the
9154 # rest.
9155 timeout 150 "$PTYCLIENT" --cols 100 --rows 30 \
9156 --out "$OUT.upcap" --err "$OUT.upcap.err" -- \
9157 "$MUX" --sock "$SOCK69" > "$OUT.uppc" 2>&1 <<'EOF' &
9158 settle 1000 25000
9159 send unset PROMPT_COMMAND; PROMPT_COMMAND=_mux_precmd\n
9160 settle 800 25000
9161 send printf '\\033]0;uptitle\\007'\n
9162 expect \x1b]0;uptitle\x07 25000
9163 send printf 'up-%s\n' pre\n
9164 expect up-pre 25000
9165 send echo shpid=$$\n
9166 expect shpid= 25000
9167 settle 800 25000
9168 expect \x1b]0;uptitle\x07 90000
9169 expect up-resumed 60000
9170 settle 800 25000
9171 send printf 'up-%s\n' typed\n
9172 expect up-typed 25000
9173 send \x1cd
9174 waitexit 15000
9175 EOF
9176 UPPCPID=$!
9177
9178 # Daemon truth for the pid, and the number this leg is built around. The
9179 # echoed command line spells `$$`, so only the shell's own answer has
9180 # digits after the `=` — grepping for a bare `shpid=` would match the echo.
9181 wait_grid "$SOCK69" "shpid=[0-9]" "upgrade: the session shell never printed its pid"
9182 UPSHPID=$(dump_session "$SOCK69" | sed -n 's/.*shpid=\([0-9][0-9]*\).*/\1/p' | head -1)
9183 [ -n "$UPSHPID" ] || {
9184 echo "e2e FAIL: upgrade: no shell pid read off the grid — the witness would be vacuous"
9185 dump_session "$SOCK69"; exit 1; }
9186
9187 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst1" 2>&1 || {
9188 echo "e2e FAIL: upgrade: stats did not answer before the upgrade"; exit 1; }
9189 UPATT1=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst1")
9190 UPSNAP1=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst1")
9191 [ -n "$UPATT1" ] && [ -n "$UPSNAP1" ] || {
9192 echo "e2e FAIL: upgrade: no counters read out of stats; the carry pin would be vacuous:"
9193 cat "$OUT.upst1"; exit 1; }
9194
9195 # 1. Refused: same version, no flag. The reason names BOTH versions, which
9196 # is the difference between a policy an operator can act on and a "no".
9197 set +e
9198 "$MUXD" upgrade --sock "$SOCK69" > "$OUT.upref1" 2>&1
9199 UPRC=$?
9200 set -e
9201 [ "$UPRC" -eq 1 ] || {
9202 echo "e2e FAIL: upgrade: a same-version upgrade exited $UPRC, want 1:"
9203 cat "$OUT.upref1"; exit 1; }
9204 grep -qF "refused: version: $UPVER is not newer than $UPVER" "$OUT.upref1" || {
9205 echo "e2e FAIL: upgrade: the refusal did not name both versions:"
9206 cat "$OUT.upref1"; exit 1; }
9207
9208 # 2. Refused: a candidate that is not executable. `muxd upgrade` offers its
9209 # OWN path (/proc/self/exe), so the only way to spell this refusal is to
9210 # take the exec bit off a copy AFTER it has started — and the ordering is
9211 # made deterministic rather than hoped for. The daemon is held under
9212 # SIGSTOP so it cannot read the request until the chmod has landed;
9213 # `readlink /proc/PID/exe` is the witness that the copy has already
9214 # exec'd, so the chmod cannot beat it and turn the leg into an exec
9215 # failure that asserts nothing.
9216 cp "$MUXD" "$UPBIN"
9217 chmod 755 "$UPBIN"
9218 kill -STOP "$D69PID"
9219 "$UPBIN" upgrade --sock "$SOCK69" --allow-same-version > "$OUT.upref2" 2>&1 &
9220 UPXPID=$!
9221 _i=0
9222 while [ "$(readlink "/proc/$UPXPID/exe" 2>/dev/null)" != "$UPBIN" ]; do
9223 _i=$((_i + 1))
9224 [ "$_i" -lt $(( 100 * TIME_SCALE )) ] || {
9225 kill -CONT "$D69PID" 2>/dev/null || true
9226 echo "e2e FAIL: upgrade: the candidate copy never exec'd; the refusal below"
9227 echo " would have been about a binary that never asked"
9228 exit 1; }
9229 sleep 0.05
9230 done
9231 chmod 000 "$UPBIN"
9232 kill -CONT "$D69PID"
9233 set +e
9234 wait "$UPXPID"
9235 UPRC=$?
9236 set -e
9237 chmod 755 "$UPBIN"
9238 [ "$UPRC" -eq 1 ] || {
9239 echo "e2e FAIL: upgrade: an unexecutable candidate exited $UPRC, want 1:"
9240 cat "$OUT.upref2"; exit 1; }
9241 grep -qF "refused: path: not executable" "$OUT.upref2" || {
9242 echo "e2e FAIL: upgrade: the refusal did not name the path check:"
9243 cat "$OUT.upref2"; exit 1; }
9244
9245 # ...and two refusals changed NOTHING. A refusal that had torn a client
9246 # down, or written the manifest and left the fd table half open, would
9247 # still have printed the words above.
9248 timeout 20 "$MUXA" status --sock "$SOCK69" > "$OUT.upsta" 2>&1 || {
9249 echo "e2e FAIL: upgrade: the daemon stopped answering after refusing:"
9250 cat "$OUT.upsta"; exit 1; }
9251 wait_grid "$SOCK69" "shpid=$UPSHPID" "upgrade: a refusal cost the session its shell"
9252
9253 # 3. The exec itself.
9254 set +e
9255 "$MUXD" upgrade --sock "$SOCK69" --allow-same-version > "$OUT.upok" 2>&1
9256 UPRC=$?
9257 set -e
9258 [ "$UPRC" -eq 0 ] || {
9259 echo "e2e FAIL: upgrade: the upgrade exited $UPRC, want 0:"
9260 cat "$OUT.upok" "$OUT.up.d"; exit 1; }
9261 grep -qF "muxd: upgraded to $UPVER" "$OUT.upok" || {
9262 echo "e2e FAIL: upgrade: no verdict naming the version it landed on:"
9263 cat "$OUT.upok"; exit 1; }
9264
9265 # The pid is the feature. Not "a daemon is answering" — the same process,
9266 # now running a different image, which /proc/PID/cmdline says in one word:
9267 # a daemon that had restarted would carry the original `run --sock` argv.
9268 kill -0 "$D69PID" 2>/dev/null || {
9269 echo "e2e FAIL: upgrade: the daemon pid $D69PID is gone — that is a restart, not an upgrade"
9270 cat "$OUT.up.d"; exit 1; }
9271 tr '\0' ' ' < "/proc/$D69PID/cmdline" > "$OUT.upst2" 2>/dev/null || true
9272 grep -qF -- "--resume-fd" "$OUT.upst2" || {
9273 echo "e2e FAIL: upgrade: pid $D69PID is not running the resumed argv; it holds:"
9274 cat "$OUT.upst2"; exit 1; }
9275
9276 # Counters CARRY, and it is asserted HERE rather than at the end for a
9277 # reason about blame: the re-attach poll below counts `attaches` up from
9278 # the reading taken before the exec, so a daemon that had zeroed its
9279 # counters fails THAT poll — reported as a client that never came back,
9280 # which is the wrong diagnosis of the wrong bug. `-ge`, because the
9281 # re-attach this reading races has not necessarily landed yet.
9282 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst2" 2>&1 || {
9283 echo "e2e FAIL: upgrade: stats did not answer after the exec"; cat "$OUT.up.d"; exit 1; }
9284 UPATT2=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst2")
9285 UPSNAP2=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst2")
9286 [ -n "$UPATT2" ] && [ -n "$UPSNAP2" ] || {
9287 echo "e2e FAIL: upgrade: no counters read out of stats after the exec:"
9288 cat "$OUT.upst2"; exit 1; }
9289 [ "$UPATT2" -ge "$UPATT1" ] || {
9290 echo "e2e FAIL: upgrade: attaches went $UPATT1 -> $UPATT2 across the exec; the"
9291 echo " counters were reset, not carried, and an upgrade now looks like"
9292 echo " a restart to anything sampling this daemon"
9293 exit 1; }
9294 [ "$UPSNAP2" -ge "$UPSNAP1" ] || {
9295 echo "e2e FAIL: upgrade: snapshots went $UPSNAP1 -> $UPSNAP2 across the exec; the"
9296 echo " counters were reset, not carried"
9297 exit 1; }
9298
9299 # The client came back on its own. Counted with `attaches=`, not with
9300 # `clients=`: the client is re-dialled within a millisecond of the exec, so
9301 # a poll that only looked for clients=1 would be satisfied by the reading it
9302 # took before the tear and would pass on a daemon that never dropped anyone.
9303 _i=0
9304 while : ; do
9305 _upst=$("$MUXD" stats --sock "$SOCK69" 2>/dev/null || true)
9306 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
9307 case "$_upst" in
9308 *"session 0 clients=1"*)
9309 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
9310 esac
9311 _i=$((_i + 1))
9312 [ "$_i" -lt $(( 200 * TIME_SCALE )) ] || {
9313 echo "e2e FAIL: upgrade: the attached client never re-attached after the exec;"
9314 echo " stats holds: $_upst"
9315 cat "$OUT.up.d"; exit 1; }
9316 sleep 0.1
9317 done
9318
9319 # The grid crossed with the process: the marker the client typed, and the
9320 # pid of the shell that is still running under the new image.
9321 wait_grid "$SOCK69" "up-pre" "upgrade: the pre-upgrade marker did not survive the exec"
9322 wait_grid "$SOCK69" "shpid=$UPSHPID" "upgrade: the session is a NEW shell, not the one that crossed"
9323
9324 # The FIRST muxa run after the exec, and it is also what releases the
9325 # attached client's `expect up-resumed` above — one command, two claims.
9326 set +e
9327 timeout 40 "$MUXA" run --sock "$SOCK69" --timeout 20000 'echo up-resumed' > "$OUT.upm2" 2>&1
9328 UPRC=$?
9329 set -e
9330 [ "$UPRC" -eq 0 ] || {
9331 echo "e2e FAIL: upgrade: the FIRST muxa run after the exec exited $UPRC — a carried"
9332 echo " return watermark is a watermark from the future and no return passes it:"
9333 cat "$OUT.upm2"; exit 1; }
9334 grep -qF '"mechanism":"marks"' "$OUT.upm2" || {
9335 echo "e2e FAIL: upgrade: the first await after the exec fell off marks:"
9336 cat "$OUT.upm2"; exit 1; }
9337 grep -qF '"exit_code":0' "$OUT.upm2" || {
9338 echo "e2e FAIL: upgrade: no real exit code came back after the exec:"
9339 cat "$OUT.upm2"; exit 1; }
9340 # The client's own verdict: the title came back on the re-attach, the
9341 # marker arrived through the new image, and a keystroke typed AFTER the
9342 # exec reached the shell.
9343 set +e
9344 wait "$UPPCPID"
9345 UPRC=$?
9346 set -e
9347 UPPCPID=""
9348 [ "$UPRC" -eq 0 ] || {
9349 echo "e2e FAIL: upgrade: the attached client exited $UPRC across the exec:"
9350 cat "$OUT.uppc" "$OUT.upcap.err"; exit 1; }
9351
9352 # A NONZERO exit code too, because `"exit_code":0` is also what a mechanism
9353 # that guesses would say. After the client has gone, not beside it: two
9354 # writers typing into one shell's pty interleave, and the leg would be
9355 # racing itself. Spelled as a CHILD, not as `exit 7`, which is the session
9356 # shell's own exit — the session would end, the daemon with it, and the
9357 # counter reading below would be taken from nothing. Not `(exit 7)` either:
9358 # a subshell sets `_mux_ran` in the subshell, so shellint's `precmd_fn`
9359 # sees an untouched prompt in the parent and emits no `D;` mark at all
9360 # (measured: the await ran its whole budget out).
9361 timeout 40 "$MUXA" run --sock "$SOCK69" --timeout 20000 'sh -c "exit 7"' > "$OUT.upm3" 2>&1 || true
9362 grep -qF '"exit_code":7' "$OUT.upm3" || {
9363 echo "e2e FAIL: upgrade: the shell's own exit code did not survive the exec:"
9364 cat "$OUT.upm3"; exit 1; }
9365
9366 # ...and the re-attach was COUNTED onto them rather than replacing them.
9367 # The carry was asserted above with `-ge`; this is the other half, taken
9368 # once the client's own snapshot has certainly been served.
9369 "$MUXD" stats --sock "$SOCK69" > "$OUT.upst2" 2>&1
9370 UPATT2=$(sed -n 's/.*attaches=\([0-9]*\).*/\1/p' "$OUT.upst2")
9371 UPSNAP2=$(sed -n 's/^snapshots=\([0-9]*\).*/\1/p' "$OUT.upst2")
9372 [ -n "$UPATT2" ] && [ -n "$UPSNAP2" ] || {
9373 echo "e2e FAIL: upgrade: no counters read out of stats after the client left:"
9374 cat "$OUT.upst2"; exit 1; }
9375 [ "$UPATT2" -gt "$UPATT1" ] || {
9376 echo "e2e FAIL: upgrade: attaches is still $UPATT2 — the re-attach after the exec"
9377 echo " was never counted"
9378 exit 1; }
9379 [ "$UPSNAP2" -gt "$UPSNAP1" ] || {
9380 echo "e2e FAIL: upgrade: snapshots is still $UPSNAP2 — the re-attach was served"
9381 echo " without a snapshot, which no fresh epoch allows"
9382 exit 1; }
9383
9384 assert_stopped "$SOCK69" "$D69PID" "upgrade" "$OUT.upstop"
9385 D69PID=""
9386 rm -f "$UPBIN"
9387 rm -rf "$UPHOME"
9388 ok "muxd upgrade keeps the shell, its pid, its title, its marks and its counters"
9389
9390 # --- ...and a candidate that cannot adopt hands the daemon back -----------
9391 #
9392 # The rollback: the new image fails partway through adoption and execs the
9393 # OLD binary — whose path the manifest carries — with the same manifest fd.
9394 # Armed through the ENVIRONMENT rather than the flag it shares a name with,
9395 # and that is not a convenience: `execUpgrade` builds a fixed argv
9396 # (`run --resume-fd N`), so nothing typed here can put `--resume-fail-at`
9397 # in front of the candidate. The daemon's environment is what crosses an
9398 # exec, so the abort is armed on the daemon before any upgrade asks.
9399 #
9400 # The daemon's own log is the only thing that can say a rollback HAPPENED —
9401 # `muxd upgrade` prints its verdict the moment the daemon accepts, and the
9402 # process that would have corrected it is gone. Without that line every
9403 # assertion below would pass just as well on an upgrade that simply worked.
9404 mkdir -p "$UPHOME"
9405 MUX_SHELL_INTEGRATION=1 MUX_RESUME_FAIL_AT=session HOME="$UPHOME" \
9406 "$MUXD" run --sock "$SOCK70" --shell /bin/bash > "$OUT.url.d" 2>&1 &
9407 D70PID=$!
9408 wait_sock "$SOCK70" "$OUT.url.d" "rollback daemon never bound"
9409
9410 timeout 20 "$MUXA" run --sock "$SOCK70" --timeout 8000 'echo rollpid=$$' > "$OUT.urlm1" 2>&1 || {
9411 echo "e2e FAIL: rollback: muxa run failed before the upgrade:"
9412 cat "$OUT.urlm1"; exit 1; }
9413 wait_grid "$SOCK70" "rollpid=[0-9]" "rollback: the session shell never printed its pid"
9414 UPROLLPID=$(dump_session "$SOCK70" | sed -n 's/.*rollpid=\([0-9][0-9]*\).*/\1/p' | head -1)
9415 [ -n "$UPROLLPID" ] || {
9416 echo "e2e FAIL: rollback: no shell pid read off the grid"; dump_session "$SOCK70"; exit 1; }
9417
9418 set +e
9419 "$MUXD" upgrade --sock "$SOCK70" --allow-same-version > "$OUT.urlup" 2>&1
9420 UPRC=$?
9421 set -e
9422 [ "$UPRC" -eq 0 ] || {
9423 echo "e2e FAIL: rollback: upgrade exited $UPRC — the rolled-back daemon must serve"
9424 echo " well enough for the confirm to pass:"
9425 cat "$OUT.urlup" "$OUT.url.d"; exit 1; }
9426 wait_for "$OUT.url.d" "adoption failed at session" 15 || {
9427 echo "e2e FAIL: rollback: the candidate adopted instead of aborting — the abort"
9428 echo " was not armed, and nothing below tests a rollback:"
9429 cat "$OUT.url.d"; exit 1; }
9430 grep -qF "exec'ing" "$OUT.url.d" || {
9431 echo "e2e FAIL: rollback: the abort never named the binary it went back to:"
9432 cat "$OUT.url.d"; exit 1; }
9433
9434 kill -0 "$D70PID" 2>/dev/null || {
9435 echo "e2e FAIL: rollback: the daemon pid $D70PID is gone; a failed adoption took"
9436 echo " the sessions with it"
9437 cat "$OUT.url.d"; exit 1; }
9438 wait_grid "$SOCK70" "rollpid=$UPROLLPID" "rollback: the session lost its shell to a failed adoption"
9439 timeout 40 "$MUXA" run --sock "$SOCK70" --timeout 20000 'echo roll-post' > "$OUT.urlm2" 2>&1 || {
9440 echo "e2e FAIL: rollback: the daemon that came back cannot run a command:"
9441 cat "$OUT.urlm2" "$OUT.url.d"; exit 1; }
9442 grep -qF '"mechanism":"marks"' "$OUT.urlm2" || {
9443 echo "e2e FAIL: rollback: the re-adopted session lost its marks:"
9444 cat "$OUT.urlm2"; exit 1; }
9445
9446 assert_stopped "$SOCK70" "$D70PID" "rollback" "$OUT.urlstop"
9447 D70PID=""
9448 rm -rf "$UPHOME"
9449 ok "a candidate that cannot adopt execs the old binary back, session and marks intact"
9450
9451 # --- ...and the session's agent socket crosses the exec -------------------
9452 #
9453 # The per-session agent listener is a descriptor the daemon owns and the
9454 # shell reaches by path ($SSH_AUTH_SOCK, named after the daemon's pid). It
9455 # has to cross the exec on both counts: the fd by inheritance, the path
9456 # because the pid does not change. Nothing else in this suite would notice
9457 # if it did not — the agent channels themselves are deliberately NOT
9458 # carried, so the only thing that can say the listener survived is a real
9459 # `ssh-add -l` answering through a re-dialled `-A` client.
9460 #
9461 # Its own agent rather than the forwarding legs' AGENT48: that one is left
9462 # under SIGSTOP by the mute-offerer leg, and a mute agent is exactly what
9463 # this leg cannot tell from a broken listener.
9464 "$MUXD" run --sock "$SOCK71" --shell /bin/sh > "$OUT.uag.d" 2>&1 &
9465 D71PID=$!
9466 wait_sock "$SOCK71" "$OUT.uag.d" "agent-upgrade daemon never bound"
9467
9468 ssh-agent -a "$UPAGENT" > "$OUT.uagenv" 2>&1
9469 UPAGPID=$(sed -n 's/.*SSH_AGENT_PID=\([0-9]*\).*/\1/p' "$OUT.uagenv")
9470 [ -n "$UPAGPID" ] || {
9471 echo "e2e FAIL: agent-upgrade: ssh-agent printed no pid for the trap to hold it by:"
9472 cat "$OUT.uagenv"; exit 1; }
9473 ssh-keygen -q -t ed25519 -N '' -C mux-e2e-upgrade -f "$UPAGKEY"
9474 SSH_AUTH_SOCK="$UPAGENT" ssh-add "$UPAGKEY" > /dev/null 2>&1
9475 UPFP=$(ssh-keygen -lf "$UPAGKEY" | awk '{print $2}')
9476 case "$UPFP" in
9477 SHA256:?*) ;;
9478 *) echo "e2e FAIL: agent-upgrade: ssh-keygen -lf gave no fingerprint: [$UPFP]"; exit 1;;
9479 esac
9480
9481 pipe_mux "$OUT.uag" "$OUT.uag.err" env SSH_AUTH_SOCK="$UPAGENT" timeout 90 \
9482 "$MUX" -A --sock "$SOCK71"
9483 UPATT1=$("$MUXD" stats --sock "$SOCK71" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
9484 [ -n "$UPATT1" ] || { echo "e2e FAIL: agent-upgrade: no attaches counter to compare against"; exit 1; }
9485 # `agpre=` rather than the fingerprint alone: the shell echoes what is
9486 # typed, and `$?` is not expanded in the echo, so only ssh-add's own status
9487 # can put a digit there. The fingerprint is asserted beside it because a
9488 # status of 0 says an agent answered, not that it was the client's.
9489 pipe_send 'ssh-add -l; echo agpre=$?\n'
9490 await_out "$OUT.uag" "agpre=0" "agent-upgrade: ssh-add never answered before the upgrade"
9491 # Counted by OCCURRENCE, not by matching line: this capture is an escape
9492 # stream whose one "line" is the whole session, so `grep -c` would answer 1
9493 # for both answers and the comparison after the exec would be vacuous.
9494 UPFP1=$(grep -aoF -- "$UPFP" "$OUT.uag" | wc -l)
9495 [ "$UPFP1" -ge 1 ] || {
9496 echo "e2e FAIL: agent-upgrade: the client's key never came back before the upgrade"
9497 echo " (wanted $UPFP)"; exit 1; }
9498
9499 set +e
9500 "$MUXD" upgrade --sock "$SOCK71" --allow-same-version > "$OUT.uagup" 2>&1
9501 UPRC=$?
9502 set -e
9503 [ "$UPRC" -eq 0 ] || {
9504 echo "e2e FAIL: agent-upgrade: upgrade exited $UPRC:"
9505 cat "$OUT.uagup" "$OUT.uag.d"; exit 1; }
9506 _i=0
9507 while : ; do
9508 _upst=$("$MUXD" stats --sock "$SOCK71" 2>/dev/null || true)
9509 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
9510 case "$_upst" in
9511 *"session 0 clients=1"*)
9512 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
9513 esac
9514 _i=$((_i + 1))
9515 [ "$_i" -lt $(( 200 * TIME_SCALE )) ] || {
9516 echo "e2e FAIL: agent-upgrade: the -A client never re-attached; stats holds: $_upst"
9517 cat "$OUT.uag.d"; exit 1; }
9518 sleep 0.1
9519 done
9520
9521 # The same question a second time, and `agdone=` is a needle the first
9522 # answer cannot satisfy.
9523 pipe_send 'ssh-add -l; echo agdone=$?\n'
9524 await_out "$OUT.uag" "agdone=0" "agent-upgrade: ssh-add stopped answering across the exec"
9525 UPFP2=$(grep -aoF -- "$UPFP" "$OUT.uag" | wc -l)
9526 [ "$UPFP2" -gt "$UPFP1" ] || {
9527 echo "e2e FAIL: agent-upgrade: ssh-add exited 0 but the key came back $UPFP1 time(s)"
9528 echo " before the exec and $UPFP2 after — no NEW answer crossed"; exit 1; }
9529 pipe_detach "agent-upgrade client"
9530
9531 assert_stopped "$SOCK71" "$D71PID" "agent-upgrade" "$OUT.uagstop"
9532 D71PID=""
9533 kill "$UPAGPID" 2>/dev/null || true
9534 wait_pid_gone "$UPAGPID" "agent-upgrade: ssh-agent killed by tracked pid"
9535 UPAGPID=""
9536 ok "the session's agent socket crosses the exec: ssh-add -l answers again"
9537
9538 # --- ...and a QUIC client is serving again in a breath, not after a timeout
9539 #
9540 # Unix clients need no goodbye — their fds are close-on-exec and the exec is
9541 # their EOF. A QUIC peer has no such thing: silence is indistinguishable from
9542 # a bad network, and a client that waited it out would sit on a blank wall
9543 # for the whole idle timeout. So the daemon sends CONNECTION_CLOSE to every
9544 # connection before it execs, and what that buys is the number asserted
9545 # here: back in service in a fraction of `--quic-idle-ms`, not after it.
9546 head -c 32 /dev/urandom > "$UPKEY"
9547 chmod 600 "$UPKEY"
9548 "$MUXD" run --sock "$SOCK72" --shell /bin/sh \
9549 --quic "127.0.0.1:$UPQPORT" --key "$UPKEY" --quic-idle-ms 15000 > "$OUT.uqc.d" 2>&1 &
9550 D72PID=$!
9551 wait_sock "$SOCK72" "$OUT.uqc.d" "quic-upgrade daemon never bound"
9552
9553 pipe_mux "$OUT.uqc" "$OUT.uqc.err" timeout 90 "$MUX" "quic://127.0.0.1:$UPQPORT" \
9554 --key "$UPKEY" --quic-idle-ms 15000
9555 pipe_send 'printf "uq-%%s\\n" pre\n'
9556 await_out "$OUT.uqc" "uq-pre" "quic-upgrade: the QUIC client never got its pre-upgrade marker"
9557
9558 UPATT1=$("$MUXD" stats --sock "$SOCK72" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
9559 [ -n "$UPATT1" ] || { echo "e2e FAIL: quic-upgrade: no attaches counter to compare against"; exit 1; }
9560 UPT0=$(date +%s%N)
9561 set +e
9562 "$MUXD" upgrade --sock "$SOCK72" --allow-same-version > "$OUT.uqup" 2>&1
9563 UPRC=$?
9564 set -e
9565 [ "$UPRC" -eq 0 ] || {
9566 echo "e2e FAIL: quic-upgrade: upgrade exited $UPRC:"
9567 cat "$OUT.uqup" "$OUT.uqc.d"; exit 1; }
9568 _i=0
9569 while : ; do
9570 _upst=$("$MUXD" stats --sock "$SOCK72" 2>/dev/null || true)
9571 _upatt=$(printf '%s' "$_upst" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
9572 case "$_upst" in
9573 *"session 0 clients=1"*)
9574 if [ -n "$_upatt" ] && [ "$_upatt" -gt "$UPATT1" ]; then break; fi ;;
9575 esac
9576 _i=$((_i + 1))
9577 [ "$_i" -lt $(( 400 * TIME_SCALE )) ] || {
9578 echo "e2e FAIL: quic-upgrade: the QUIC client never re-attached; stats holds: $_upst"
9579 cat "$OUT.uqc.d"; exit 1; }
9580 sleep 0.05
9581 done
9582 UPT1=$(date +%s%N)
9583 UPMS=$(( (UPT1 - UPT0) / 1000000 ))
9584 # Half the idle timeout, which is the only bound that means anything here:
9585 # a client that had waited the connection out would arrive at 15000ms and
9586 # still pass a generous absolute budget. Measured at ~60ms.
9587 [ "$UPMS" -lt $(( 7500 * TIME_SCALE )) ] || {
9588 echo "e2e FAIL: quic-upgrade: the QUIC client took ${UPMS}ms to come back —"
9589 echo " that is the idle timeout expiring, not a CONNECTION_CLOSE"
9590 exit 1; }
9591
9592 # ...and it is a session, not just a connection: a round trip that did not
9593 # exist before the exec.
9594 pipe_send 'printf "uq-%%s\\n" post\n'
9595 await_out "$OUT.uqc" "uq-post" "quic-upgrade: the reconnected QUIC client cannot reach its shell"
9596 pipe_detach "quic-upgrade client"
9597
9598 assert_stopped "$SOCK72" "$D72PID" "quic-upgrade" "$OUT.uqstop"
9599 D72PID=""
9600 ok "a QUIC client is served again within a breath of the exec, not after the idle timeout"
9601
9602 [ "$OK_COUNT" = "81" ] || {
9603 echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 81 —"
9024 echo " a scenario was added (update the pin) or silently lost" 9604 echo " a scenario was added (update the pin) or silently lost"
9025 exit 1 9605 exit 1
9026 } 9606 }