a73x

158acfcc

test: the OS oracle answers on Darwin, and the harness spells time and paths through it

a73x   2026-09-03 20:59

Commit message
test: the OS oracle answers on Darwin, and the harness spells time and paths through it

os_oracle.sh becomes one `case "$(uname)"`: the Linux block moves into the
`*)` arm unchanged and a Darwin arm goes above it, where lsof is the fd
oracle and ps answers the process questions because there is no /proc.
Every spelling in that arm was measured on the port box (macOS 26.6, lsof
4.91) rather than guessed:

  - an unlinked mkstemp carrier keeps its old path in lsof, canonicalized
    under /private and with no marker of its own, so the NAME cannot tell a
    carrier from the suite's own captures under the same /tmp/mux- prefix.
    What can is that the path no longer names anything;
  - a pty master is /dev/ptmx, through openpty(3) as well as posix_openpt;
  - a unix socket answers its bound path and no inode, so this arm matches
    the path inside that pid's own fd table where Linux matches the inode;
  - lsof prints a connected UDP socket as LOCAL->REMOTE, so the arrow and
    what follows is cut before the local-address comparison the Linux arm
    already insisted on;
  - a process that exec'd /bin/sh reports /bin/bash, because Apple ships
    /bin/sh as a stub in front of the bash image — true of a copy of
    /bin/sh too, so it is the image and not the name that differs.

Two new helpers, because a group file must never learn which arm it is on:
now_ms prints epoch milliseconds and real_path canonicalizes, where
`date +%s%N` and `readlink -f` are GNU and a group file that spelled either
would be a Linux group file. Every `date +%s%N` bracket loses its /1000000
to match, and every `readlink -f` becomes real_path. udp_local_bound keeps
the /proc/net/udp hex argument on both OSes and the Darwin arm converts it
in shell arithmetic, which forks nothing — it is polled in a loop.

The three legs that time themselves from INSIDE a mux session cannot call a
helper, so os_oracle.sh also hands out the TEXT of a now_ms through
now_ms_snippet, and the harness's own now_ms is eval'd from that same text
so the two cannot drift. It stays a per-OS `date`/`gdate` rather than an
interpreter because the throughput gate's tightest ceiling is 10ms around a
6ms leg, where a python3 start measured 21.6ms would be most of the reading.

oracle_selftest moves to test/oracle_selftest.sh, sourced by e2e_lib.sh and
runnable on its own (`sh test/oracle_selftest.sh`, answering ORACLE_OK) so
that a new OS arm can be graded on a box where nothing else in the harness
works yet. Its pid_exe assertion moves off the subject shell and onto that
shell's `sleep` child, which is an ordinary binary on both OSes — asking it
about a shell would have graded Apple's stub and not the helper.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SakwJEwD9dXBoRP5kWbemW

test/e2e_01_boot.sh
Old New
@@ -677,16 +677,16 @@ mkfifo "$QFIFO"
677 ( sleep 0.3; printf '\034'; sleep 20 ) > "$QFIFO" & 677 ( sleep 0.3; printf '\034'; sleep 20 ) > "$QFIFO" &
678 QWPID=$! 678 QWPID=$!
679 defer_kill "$QWPID" 679 defer_kill "$QWPID"
680 QT0=$(date +%s%N) 680 QT0=$(now_ms)
681 set +e 681 set +e
682 timeout 30 "$MUX" "quic://127.0.0.1:$QPORT" --key "$QKEY.wrong" \ 682 timeout 30 "$MUX" "quic://127.0.0.1:$QPORT" --key "$QKEY.wrong" \
683 < "$QFIFO" > "$OUT.qa" 2>&1 683 < "$QFIFO" > "$OUT.qa" 2>&1
684 RC=$? 684 RC=$?
685 set -e 685 set -e
686 QT1=$(date +%s%N) 686 QT1=$(now_ms)
687 softkill "$QWPID" || true 687 softkill "$QWPID" || true
688 rm -f "$QFIFO" 688 rm -f "$QFIFO"
689 QMS=$(( (QT1 - QT0) / 1000000 )) 689 QMS=$(( QT1 - QT0 ))
690 [ "$RC" -eq 0 ] || { 690 [ "$RC" -eq 0 ] || {
691 echo "e2e FAIL: aborted quic handshake exited $RC (want 0; 124 means Ctrl-\ went unheard)" 691 echo "e2e FAIL: aborted quic handshake exited $RC (want 0; 124 means Ctrl-\ went unheard)"
692 cat "$OUT.qa"; exit 1; 692 cat "$OUT.qa"; exit 1;
test/e2e_03_side.sh
Old New
@@ -197,7 +197,7 @@ defer_kill "$PAPID"
197 # a logical path reached through a symlinked directory would fail a correct 197 # a logical path reached through a symlinked directory would fail a correct
198 # spawn unless this side resolves too. 198 # spawn unless this side resolves too.
199 PAEXE=$(pid_exe "$PAPID") 199 PAEXE=$(pid_exe "$PAPID")
200 PAWANT=$(readlink -f "$MUX_ELF") 200 PAWANT=$(real_path "$MUX_ELF")
201 [ "$PAEXE" = "$PAWANT" ] || { 201 [ "$PAEXE" = "$PAWANT" ] || {
202 echo "e2e FAIL: the auto-started daemon is running $PAEXE, not the build's $PAWANT" 202 echo "e2e FAIL: the auto-started daemon is running $PAEXE, not the build's $PAWANT"
203 echo " (an auto-start that resolves a NAME grades whatever is installed)" 203 echo " (an auto-start that resolves a NAME grades whatever is installed)"
test/e2e_04_handoff.sh
Old New
@@ -281,15 +281,15 @@ HHEXKEY=$(sed -n 's/^endpoint [0-9][0-9]* \([0-9a-f]*\)$/\1/p' "$HCACHE")
281 printf 'endpoint %s %s\n' "$HDEADPORT" "$HHEXKEY" > "$HCACHE" 281 printf 'endpoint %s %s\n' "$HDEADPORT" "$HHEXKEY" > "$HCACHE"
282 chmod 600 "$HCACHE" 282 chmod 600 "$HCACHE"
283 HSHIMS_C=$(wc -l < "$SSHIM_PIDLOG") 283 HSHIMS_C=$(wc -l < "$SSHIM_PIDLOG")
284 HT4=$(date +%s%N) 284 HT4=$(now_ms)
285 pipe_mux "$OUT.h3" "$OUT.h3.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" PATH="$HPATH" timeout 40 \ 285 pipe_mux "$OUT.h3" "$OUT.h3.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" PATH="$HPATH" timeout 40 \
286 "$MUX" "$HHOST" 286 "$MUX" "$HHOST"
287 pipe_send 'printf "heal-%%s\\n" ok\n' 287 pipe_send 'printf "heal-%%s\\n" ok\n'
288 wait_for "$OUT.h3" "heal-ok" 30 || { 288 wait_for "$OUT.h3" "heal-ok" 30 || {
289 echo "e2e FAIL: stale-cache handoff never served a session" 289 echo "e2e FAIL: stale-cache handoff never served a session"
290 cat "$OUT.h3" "$OUT.h3.err" 2>/dev/null; exit 1; } 290 cat "$OUT.h3" "$OUT.h3.err" 2>/dev/null; exit 1; }
291 HT5=$(date +%s%N) 291 HT5=$(now_ms)
292 HMS_C=$(( (HT5 - HT4) / 1000000 )) 292 HMS_C=$(( HT5 - HT4 ))
293 pipe_detach "stale-cache handoff client" 293 pipe_detach "stale-cache handoff client"
294 # The absent-grep, and the control that proves it can fire is scenario (d) 294 # The absent-grep, and the control that proves it can fire is scenario (d)
295 # below: the same pattern, asserted PRESENT, against a build where the 295 # below: the same pattern, asserted PRESENT, against a build where the
@@ -400,7 +400,7 @@ softkill "$HPOLLPID"
400 ok "a QUIC poller holds one client slot, not one per second" 400 ok "a QUIC poller holds one client slot, not one per second"
401 401
402 HSHIMS_D=$(wc -l < "$SSHIM_PIDLOG") 402 HSHIMS_D=$(wc -l < "$SSHIM_PIDLOG")
403 HT0=$(date +%s%N) 403 HT0=$(now_ms)
404 pipe_mux "$OUT.h4" "$OUT.h4.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN2" PATH="$HPATH" timeout 40 \ 404 pipe_mux "$OUT.h4" "$OUT.h4.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN2" PATH="$HPATH" timeout 40 \
405 "$MUX" "127.0.0.1" 405 "$MUX" "127.0.0.1"
406 pipe_send 'printf "fallback-%%s\\n" ok\n' 406 pipe_send 'printf "fallback-%%s\\n" ok\n'
@@ -413,8 +413,8 @@ pipe_send 'printf "fallback-%%s\\n" ok\n'
413 wait_for "$OUT.h4.err" "unreachable, attaching over ssh" 15 || { 413 wait_for "$OUT.h4.err" "unreachable, attaching over ssh" 15 || {
414 echo "e2e FAIL: key mismatch printed no fallback line in 15s; its stderr was:" 414 echo "e2e FAIL: key mismatch printed no fallback line in 15s; its stderr was:"
415 cat "$OUT.h4.err" 2>/dev/null; exit 1; } 415 cat "$OUT.h4.err" 2>/dev/null; exit 1; }
416 HT1=$(date +%s%N) 416 HT1=$(now_ms)
417 HMS=$(( (HT1 - HT0) / 1000000 )) 417 HMS=$(( HT1 - HT0 ))
418 # The exact line, with the port the daemon really holds: a fallback that 418 # The exact line, with the port the daemon really holds: a fallback that
419 # named the wrong coordinates would be a different bug wearing this one's 419 # named the wrong coordinates would be a different bug wearing this one's
420 # message. 420 # message.
@@ -492,7 +492,7 @@ HWMODE=$(file_mode "$HCACHE_D")
492 echo " anything looser, so this leg would attach COLD" 492 echo " anything looser, so this leg would attach COLD"
493 exit 1; } 493 exit 1; }
494 HSHIMS_W=$(wc -l < "$SSHIM_PIDLOG") 494 HSHIMS_W=$(wc -l < "$SSHIM_PIDLOG")
495 HT4=$(date +%s%N) 495 HT4=$(now_ms)
496 pipe_mux "$OUT.h6" "$OUT.h6.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN2" PATH="$HPATH" timeout 40 \ 496 pipe_mux "$OUT.h6" "$OUT.h6.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN2" PATH="$HPATH" timeout 40 \
497 "$MUX" "127.0.0.1" 497 "$MUX" "127.0.0.1"
498 pipe_send 'printf "warmmiss-%%s\\n" ok\n' 498 pipe_send 'printf "warmmiss-%%s\\n" ok\n'
@@ -500,8 +500,8 @@ pipe_send 'printf "warmmiss-%%s\\n" ok\n'
500 wait_for "$OUT.h6.err" "unreachable, attaching over ssh" 15 || { 500 wait_for "$OUT.h6.err" "unreachable, attaching over ssh" 15 || {
501 echo "e2e FAIL: the warm miss printed no fallback line in 15s; its stderr was:" 501 echo "e2e FAIL: the warm miss printed no fallback line in 15s; its stderr was:"
502 cat "$OUT.h6.err" 2>/dev/null; exit 1; } 502 cat "$OUT.h6.err" 2>/dev/null; exit 1; }
503 HT5=$(date +%s%N) 503 HT5=$(now_ms)
504 HMS_W=$(( (HT5 - HT4) / 1000000 )) 504 HMS_W=$(( HT5 - HT4 ))
505 # ONCE, and naming the port the daemon really holds. Twice would be two 505 # ONCE, and naming the port the daemon really holds. Twice would be two
506 # dials wearing one message. 506 # dials wearing one message.
507 HFB_W=$(grep -c "^mux: quic://127.0.0.1:$HQPORT unreachable, attaching over ssh$" "$OUT.h6.err" || true) 507 HFB_W=$(grep -c "^mux: quic://127.0.0.1:$HQPORT unreachable, attaching over ssh$" "$OUT.h6.err" || true)
@@ -556,7 +556,7 @@ ok "a warm miss on silent coordinates costs ONE deadline (${HMS_W}ms), not two"
556 # never reach the announce at all. 556 # never reach the announce at all.
557 : > "$HCFGBAD" 557 : > "$HCFGBAD"
558 HSHIMS_E=$(wc -l < "$SSHIM_PIDLOG") 558 HSHIMS_E=$(wc -l < "$SSHIM_PIDLOG")
559 HT2=$(date +%s%N) 559 HT2=$(now_ms)
560 pipe_mux "$OUT.h5" "$OUT.h5.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" XDG_CONFIG_HOME="$HCFGBAD" \ 560 pipe_mux "$OUT.h5" "$OUT.h5.err" env SHELL=/bin/sh XDG_RUNTIME_DIR="$HRUN" XDG_CONFIG_HOME="$HCFGBAD" \
561 PATH="$HPATH" timeout 40 \ 561 PATH="$HPATH" timeout 40 \
562 "$MUX" "mux-e2e-none@127.0.0.1" 562 "$MUX" "mux-e2e-none@127.0.0.1"
@@ -564,8 +564,8 @@ pipe_send 'printf "none-%%s\\n" ok\n'
564 wait_for "$OUT.h5" "none-ok" 20 || { 564 wait_for "$OUT.h5" "none-ok" 20 || {
565 echo "e2e FAIL: announce-none handoff never served a session" 565 echo "e2e FAIL: announce-none handoff never served a session"
566 cat "$OUT.h5" "$OUT.h5.err" 2>/dev/null; exit 1; } 566 cat "$OUT.h5" "$OUT.h5.err" 2>/dev/null; exit 1; }
567 HT3=$(date +%s%N) 567 HT3=$(now_ms)
568 HMS_E=$(( (HT3 - HT2) / 1000000 )) 568 HMS_E=$(( HT3 - HT2 ))
569 pipe_detach "announce-none client" 569 pipe_detach "announce-none client"
570 # The remote said why, in one line, on the stderr mux relays from ssh. 570 # The remote said why, in one line, on the stderr mux relays from ssh.
571 grep -q '^mux d endpoint: .*staying on ssh' "$OUT.h5.err" || { 571 grep -q '^mux d endpoint: .*staying on ssh' "$OUT.h5.err" || {
test/e2e_10_agent.sh
Old New
@@ -207,7 +207,7 @@ ok "agent forwarding: -A with no agent is a usage error, not a silent no-op"
207 assert_stopped "$SOCK48" "$D42PID" "agent forwarding: the positive leg's daemon" "$OUT.agtstop0" 207 assert_stopped "$SOCK48" "$D42PID" "agent forwarding: the positive leg's daemon" "$OUT.agtstop0"
208 start_daemon "$SOCK48" "$OUT.agtn.d" "agent-refusal daemon never bound" --shell /bin/sh 208 start_daemon "$SOCK48" "$OUT.agtn.d" "agent-refusal daemon never bound" --shell /bin/sh
209 D42PID=$DPID 209 D42PID=$DPID
210 AR0=$(date +%s%N) 210 AR0=$(now_ms)
211 set +e 211 set +e
212 hostroom agtn 212 hostroom agtn
213 # The neighbour, written down: the wall is the layout, so the daemon's own 213 # The neighbour, written down: the wall is the layout, so the daemon's own
@@ -234,8 +234,8 @@ waitexit 10000
234 EOF 234 EOF
235 RC=$? 235 RC=$?
236 set -e 236 set -e
237 AR1=$(date +%s%N) 237 AR1=$(now_ms)
238 AMS=$(( (AR1 - AR0) / 1000000 )) 238 AMS=$(( AR1 - AR0 ))
239 rc0 "agent-refusal: the leg exited $RC:" "$OUT.agtn.log" 239 rc0 "agent-refusal: the leg exited $RC:" "$OUT.agtn.log"
240 # Quoted heredoc, unlike the leg above: `$?` has to reach the SESSION's 240 # Quoted heredoc, unlike the leg above: `$?` has to reach the SESSION's
241 # shell as two characters, not be spent by this one. 241 # shell as two characters, not be spent by this one.
test/e2e_11_select.sh
Old New
@@ -374,14 +374,24 @@ defer_kill "$AGENT48PID"
374 cat "$OUT.agtmute.env"; exit 1; } 374 cat "$OUT.agtmute.env"; exit 1; }
375 kill -STOP "$AGENT48PID" 375 kill -STOP "$AGENT48PID"
376 set +e 376 set +e
377 SSH_AUTH_SOCK="$AGENT48" XDG_STATE_HOME="$AGSTATE" timeout 60 "$PTYCLIENT" --cols 100 --rows 30 \ 377 # Assembled rather than written as one heredoc, because ONE of its lines
378 --out "$OUT.agtmute" --err "$OUT.agtmute.err" \ 378 # needs a value from THIS shell: the session times its own ssh-add, and the
379 -- "$MUX" -A --sock "$SOCK48" > "$OUT.agtmute.log" 2>&1 <<'EOF' 379 # clock that does it is `date +%s%N` on one OS and `gdate +%s%N` on the
380 # other, so the text comes from now_ms_snippet. Both halves stay QUOTED
381 # heredocs — the lines around the seam spell `$?`, `$R` and a backticked
382 # `d` that all have to reach the SESSION as characters rather than be spent
383 # here — and the seam itself is a printf, which expands exactly the one
384 # thing it is given and nothing else on the line.
385 {
386 cat <<'EOF'
380 expect \x1b[?1049h 15000 387 expect \x1b[?1049h 15000
381 settle 400 15000 388 settle 400 15000
382 send timeout 8 ssh-add -l; echo mu""te1=$?\n 389 send timeout 8 ssh-add -l; echo mu""te1=$?\n
383 expect mute1= 20000 390 expect mute1= 20000
384 send S=$(date +%s%N); ssh-add -l; R=$?; echo mu""te2=$R ms=$(( ($(date +%s%N)-S)/1000000 ))\n 391 EOF
392 printf 'send %s; S=$(now_ms); ssh-add -l; R=$?; echo mu""te2=$R ms=$(( $(now_ms) - S ))\\n\n' \
393 "$(now_ms_snippet)"
394 cat <<'EOF'
385 expect mute2= 15000 395 expect mute2= 15000
386 settle 400 15000 396 settle 400 15000
387 send exit\n 397 send exit\n
@@ -396,6 +406,10 @@ settle 300 15000
396 send \x1cd 406 send \x1cd
397 waitexit 10000 407 waitexit 10000
398 EOF 408 EOF
409 } > "$OUT.agtmute.script"
410 SSH_AUTH_SOCK="$AGENT48" XDG_STATE_HOME="$AGSTATE" timeout 60 "$PTYCLIENT" --cols 100 --rows 30 \
411 --out "$OUT.agtmute" --err "$OUT.agtmute.err" \
412 -- "$MUX" -A --sock "$SOCK48" > "$OUT.agtmute.log" 2>&1 < "$OUT.agtmute.script"
399 RC=$? 413 RC=$?
400 set -e 414 set -e
401 # The agent is resumed before it is killed: a stopped process ignores 415 # The agent is resumed before it is killed: a stopped process ignores
test/e2e_14_upgrade.sh
Old New
@@ -38,7 +38,7 @@ defer_rm "$UPHOME"
38 # which answers with the resolved image — a $TMPDIR reached through a 38 # which answers with the resolved image — a $TMPDIR reached through a
39 # symlink would never equal a logical path and the wait would time out on a 39 # symlink would never equal a logical path and the wait would time out on a
40 # copy that had in fact exec'd. 40 # copy that had in fact exec'd.
41 UPBIN=$(readlink -f "${TMPDIR:-/tmp}/mux-e2e-upcand-$$") 41 UPBIN=$(real_path "${TMPDIR:-/tmp}/mux-e2e-upcand-$$")
42 defer_rm "$UPBIN" 42 defer_rm "$UPBIN"
43 UPAGENT="${TMPDIR:-/tmp}/mux-e2e-upagent-$$.sock" 43 UPAGENT="${TMPDIR:-/tmp}/mux-e2e-upagent-$$.sock"
44 defer_rm "$UPAGENT" 44 defer_rm "$UPAGENT"
@@ -545,7 +545,7 @@ await_out "$OUT.uqc" "uq-pre" "quic-upgrade: the QUIC client never got its pre-u
545 545
546 UPATT1=$("$MUX" d stats --sock "$SOCK72" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p') 546 UPATT1=$("$MUX" d stats --sock "$SOCK72" | sed -n 's/.*attaches=\([0-9]*\).*/\1/p')
547 [ -n "$UPATT1" ] || { echo "e2e FAIL: quic-upgrade: no attaches counter to compare against"; exit 1; } 547 [ -n "$UPATT1" ] || { echo "e2e FAIL: quic-upgrade: no attaches counter to compare against"; exit 1; }
548 UPT0=$(date +%s%N) 548 UPT0=$(now_ms)
549 set +e 549 set +e
550 "$MUX" d upgrade --sock "$SOCK72" --allow-same-version > "$OUT.uqup" 2>&1 550 "$MUX" d upgrade --sock "$SOCK72" --allow-same-version > "$OUT.uqup" 2>&1
551 UPRC=$? 551 UPRC=$?
@@ -567,8 +567,8 @@ while : ; do
567 cat "$OUT.uqc.d"; exit 1; } 567 cat "$OUT.uqc.d"; exit 1; }
568 sleep 0.05 568 sleep 0.05
569 done 569 done
570 UPT1=$(date +%s%N) 570 UPT1=$(now_ms)
571 UPMS=$(( (UPT1 - UPT0) / 1000000 )) 571 UPMS=$(( UPT1 - UPT0 ))
572 # Half the idle timeout, which is the only bound that means anything here: 572 # Half the idle timeout, which is the only bound that means anything here:
573 # a client that had waited the connection out would arrive at 15000ms and 573 # a client that had waited the connection out would arrive at 15000ms and
574 # still pass a generous absolute budget. Measured at ~60ms. 574 # still pass a generous absolute budget. Measured at ~60ms.
test/e2e_16_push.sh
Old New
@@ -23,12 +23,12 @@ PUSHINST="$PUSHROOT/.local/bin/mux"
23 23
24 mkdir -p "$PUSHROOT/.local/bin" "$PUSHROOT/run" "$PUSHROOT/state" \ 24 mkdir -p "$PUSHROOT/.local/bin" "$PUSHROOT/run" "$PUSHROOT/state" \
25 "$PUSHBARE" "$PUSHIM_DIR/fakebin" 25 "$PUSHBARE" "$PUSHIM_DIR/fakebin"
26 # The path the daemon's image is compared against, resolved ONCE and the 26 # The path the daemon's image is compared against, resolved ONCE and
27 # same way the oracle resolves the link: pid_exe is `readlink -f`, so a 27 # through the oracle, which is where "resolved" is spelled per OS: pid_exe
28 # $TMPDIR reached through a symlink would fail a correct push if only one 28 # answers the canonical image, so a $TMPDIR reached through a symlink would
29 # side were canonical. After the mkdir, because `readlink -f` needs every 29 # fail a correct push if only one side were canonical. After the mkdir,
30 # component but the last to exist. 30 # because real_path needs every component but the last to exist.
31 PUSHWANT=$(readlink -f "$PUSHINST") 31 PUSHWANT=$(real_path "$PUSHINST")
32 32
33 # The shim IS ssh as far as the pusher can tell (e2e_04 explains that 33 # The shim IS ssh as far as the pusher can tell (e2e_04 explains that
34 # contract), and here it also IS the remote box: it swaps in the fake box's 34 # contract), and here it also IS the remote box: it swaps in the fake box's
test/e2e_lib.sh
Old New
@@ -345,6 +345,11 @@ softkill() {
345 # ask in one place. That file is trap-free on purpose: the registry and the 345 # ask in one place. That file is trap-free on purpose: the registry and the
346 # EXIT trap below are this file's, and it must not bring a second of either. 346 # EXIT trap below are this file's, and it must not bring a second of either.
347 . "$(dirname "$0")/os_oracle.sh" 347 . "$(dirname "$0")/os_oracle.sh"
348 # And the oracle's own pin, which moved out of this file for the same
349 # reason: it has to be runnable on a box where nothing else here works
350 # yet, which is the first thing a new OS arm needs. It defines
351 # oracle_selftest and nothing else on this path.
352 . "$(dirname "$0")/oracle_selftest.sh"
348 353
349 # Poll until nothing answers on a socket path (2s). Keyed off the daemon's 354 # Poll until nothing answers on a socket path (2s). Keyed off the daemon's
350 # own liveness rather than a fixed sleep, same reasoning as wait_for. 355 # own liveness rather than a fixed sleep, same reasoning as wait_for.
@@ -1429,146 +1434,3 @@ cleanup() {
1429 fi 1434 fi
1430 } 1435 }
1431 trap cleanup EXIT INT TERM 1436 trap cleanup EXIT INT TERM
1432
1433 # ---- the oracle's own pin -----------------------------------------------
1434 # The runner calls this once, before the first group. Every helper above is
1435 # a question a pin asks INSTEAD of asking the daemon, so a helper that
1436 # quietly stopped answering would not fail a pin loudly — it would make the
1437 # pin agree with anything, and the suite would go green having tested
1438 # nothing. This asks each helper something the box can independently
1439 # confirm.
1440 #
1441 # Off-origin on every dimension the helpers could accidentally hold
1442 # constant, because a fixture that pins N=1 or offset=0 is blind to N and
1443 # to the offset (CLAUDE.md): the subject is a child that is not pid 1, it
1444 # holds MORE than the three fds any fixture would, it has an argv worth
1445 # losing, and it has a child of its own. The socket arm is asked in both
1446 # directions — a listener that HOLDS the path and a shell that does not —
1447 # because a `pid_holds_unix_sock` that answered yes to everything would
1448 # pass the one pin that reads it just as happily as a correct one.
1449 #
1450 # The `sh -c` body ends in `:` on purpose. With nothing after it, a shell
1451 # execs the last command in place rather than forking it — bash measured
1452 # doing exactly that — and the subject would be `sleep`, wearing sleep's
1453 # comm, exe and argv and having no child at all. That is a different
1454 # process from the one this means to ask about.
1455 oracle_selftest() {
1456 _osock="$OUT.oracle.sock"
1457 rm -f "$_osock"
1458 printf 'hello' > "$OUT.oracle.probe"
1459 # python3 rather than sh: nothing in POSIX sh can bind a unix socket,
1460 # and the runner has already refused a box without python3.
1461 python3 -c 'import socket,sys,time
1462 s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(300)' \
1463 "$_osock" &
1464 _olpid=$!
1465 sh -c 'exec 5>/dev/null 6>/dev/null; sleep 300; :' &
1466 _opid=$!
1467 # Registered, not merely killed at the bottom: an assertion below that
1468 # fails exits through the trap, and an unregistered `sleep 300` would
1469 # outlive the run by five minutes — ten of them under `make soak`.
1470 defer_kill "$_olpid" "$_opid"
1471 _oi=0
1472 while [ ! -S "$_osock" ] && [ "$_oi" -lt $(( 50 * TIME_SCALE )) ]; do
1473 sleep 0.1; _oi=$((_oi + 1))
1474 done
1475 [ -S "$_osock" ] || {
1476 echo "e2e FAIL: oracle: the listener bound no socket at $_osock"; exit 1; }
1477
1478 # The subject is only the process this means to ask about once it has
1479 # opened fds 5 and 6 and forked `sleep`; until then `pid_fd_count` and
1480 # `pid_children` would be graded against a shell still starting up, and
1481 # the self-test would fail for a reason that is nothing to do with the
1482 # helpers. A child is the LAST of those steps, so waiting for one waits
1483 # for all of them.
1484 wait_until 50 "oracle: the subject shell never forked its sleep" \
1485 '[ -n "$(pid_children "$_opid")" ]'
1486
1487 pid_alive "$_opid" || {
1488 echo "e2e FAIL: oracle: pid_alive says a live shell is dead"; exit 1; }
1489 [ "$(pid_comm "$_opid")" = sh ] || {
1490 echo "e2e FAIL: oracle: pid_comm of an sh is '$(pid_comm "$_opid")'"; exit 1; }
1491 pid_args "$_opid" | grep -q 'sleep 300' || {
1492 echo "e2e FAIL: oracle: pid_args lost the argv: [$(pid_args "$_opid")]"; exit 1; }
1493 [ "$(pid_fd_count "$_opid")" -ge 5 ] || {
1494 echo "e2e FAIL: oracle: pid_fd_count is $(pid_fd_count "$_opid") for a shell"
1495 echo " holding fd 5 and fd 6 on top of the usual three"; exit 1; }
1496 [ "$(pid_exe "$_opid")" = "$(readlink -f "$(command -v sh)")" ] || {
1497 echo "e2e FAIL: oracle: pid_exe is '$(pid_exe "$_opid")', want the resolved"
1498 echo " $(readlink -f "$(command -v sh)")"; exit 1; }
1499 # head -1 because this is a pid to kill as well as an answer to check,
1500 # and the registry and kill(1) below both take one word.
1501 _okid=$(pid_children "$_opid" | head -1)
1502 [ -n "$_okid" ] || {
1503 echo "e2e FAIL: oracle: pid_children names no child of a shell running sleep"
1504 exit 1; }
1505 defer_kill "$_okid"
1506 # Through a variable and a case rather than straight into test(1): a
1507 # helper that answered with nothing would otherwise abort the run with
1508 # a bare `[: -gt: unary operator expected` and no e2e FAIL line saying
1509 # which helper went quiet.
1510 _orss=$(pid_rss_kb "$_opid")
1511 case "$_orss" in ''|*[!0-9]*) _orss=0 ;; esac
1512 [ "$_orss" -gt 0 ] || {
1513 echo "e2e FAIL: oracle: pid_rss_kb is '$(pid_rss_kb "$_opid")'"; exit 1; }
1514 pid_holds_unix_sock "$_olpid" "$_osock" || {
1515 echo "e2e FAIL: oracle: pid_holds_unix_sock cannot see $_osock among the"
1516 echo " fds of the pid $_olpid that bound it"; exit 1; }
1517 # The `socket` KIND off a real fd: the listener bound one, so its count
1518 # is at least one. Only that KIND is asserted here. `carrier` and
1519 # `ptymaster` would each need a process holding an anonymous memory file
1520 # or a pty master, which is a daemon — and e2e_14_upgrade already asks
1521 # both of a real one, where a wrong answer fails that pin.
1522 _ofdk=$(pid_holds_fd_kind "$_olpid" socket)
1523 case "$_ofdk" in ''|*[!0-9]*) _ofdk=0 ;; esac
1524 [ "$_ofdk" -ge 1 ] || {
1525 echo "e2e FAIL: oracle: pid_holds_fd_kind says $_olpid holds"
1526 echo " '$(pid_holds_fd_kind "$_olpid" socket)' fds of kind socket, and it"
1527 echo " bound $_osock"; exit 1; }
1528 # `! cmd || {}` rather than `cmd && {}`, for the reason spelled out
1529 # below: an AND-list whose left side fails is a failing command under
1530 # `set -e`, so the passing case would end the run.
1531 ! pid_holds_fd_kind "$_opid" nosuchkind 2>/dev/null || {
1532 echo "e2e FAIL: oracle: pid_holds_fd_kind accepted a kind it does not know,"
1533 echo " so a typo in a group file would count zero and pass"; exit 1; }
1534 ! pid_holds_unix_sock "$_opid" "$_osock" || {
1535 echo "e2e FAIL: oracle: pid_holds_unix_sock says the shell $_opid holds"
1536 echo " $_osock, which only $_olpid ever opened"; exit 1; }
1537 # `! cmd || {}` rather than `cmd && {}`, for wait_sock's reason: an
1538 # AND-list whose left side fails is a failing command under `set -e`,
1539 # so the passing case would end the run.
1540 ! udp_local_bound 'ZZZZZZZZ:ZZZZ' || {
1541 echo "e2e FAIL: oracle: udp_local_bound found a socket at an address no"
1542 echo " kernel can spell"; exit 1; }
1543 [ -n "$(udp_table)" ] || {
1544 echo "e2e FAIL: oracle: udp_table is empty, so a bind diagnostic would"
1545 echo " print nothing"; exit 1; }
1546 # Against what the probe file IS, never against a second spelling of the
1547 # same command: comparing file_mode with `stat -c %a` would agree with
1548 # itself on any OS and would have to be ported alongside the helper. A
1549 # chmod the kernel performed, five bytes written, and the published
1550 # SHA-256 of "hello" are answers no arm of the oracle can define away.
1551 chmod 640 "$OUT.oracle.probe"
1552 [ "$(file_mode "$OUT.oracle.probe")" = "640" ] || {
1553 echo "e2e FAIL: oracle: file_mode says '$(file_mode "$OUT.oracle.probe")' for a"
1554 echo " file the kernel just chmod'd to 640"; exit 1; }
1555 [ "$(file_size "$OUT.oracle.probe")" = "5" ] || {
1556 echo "e2e FAIL: oracle: file_size says '$(file_size "$OUT.oracle.probe")' for five bytes"
1557 exit 1; }
1558 [ "$(sha256_of "$OUT.oracle.probe")" = \
1559 "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824" ] || {
1560 echo "e2e FAIL: oracle: sha256_of hashes 'hello' to"
1561 echo " '$(sha256_of "$OUT.oracle.probe")'"; exit 1; }
1562
1563 kill "$_olpid" "$_opid" "$_okid" 2>/dev/null || true
1564 wait "$_olpid" 2>/dev/null || true
1565 wait "$_opid" 2>/dev/null || true
1566 _oi=0
1567 while pid_alive "$_opid" && [ "$_oi" -lt $(( 50 * TIME_SCALE )) ]; do
1568 sleep 0.05; _oi=$((_oi + 1))
1569 done
1570 ! pid_alive "$_opid" || {
1571 echo "e2e FAIL: oracle: pid_alive says a killed shell still lives"; exit 1; }
1572 rm -f "$_osock" "$OUT.oracle.probe"
1573 ok "oracle: the OS answers the helpers by name"
1574 }
test/oracle_selftest.sh
Old New
@@ -0,0 +1,231 @@
1 # shellcheck shell=sh
2 # oracle_selftest.sh — the OS oracle's own pin, sourced and also runnable.
3 #
4 # It lives beside test/os_oracle.sh rather than inside test/e2e_lib.sh so
5 # that a new OS arm can be graded on a box where nothing else in the
6 # harness works yet:
7 #
8 # sh test/oracle_selftest.sh
9 #
10 # needs a shell, python3 and the oracle's own tools, and answers ORACLE_OK
11 # or an `e2e FAIL:` line. Sourced into the suite it is the same function,
12 # called by test/e2e.sh with the lib's registry, counter and time scale
13 # already up. It is one function either way, never a copy: a self-test with
14 # two spellings would be the second one that goes stale.
15 #
16 # Trap-free when sourced, for os_oracle.sh's reason: e2e_lib.sh arms the
17 # EXIT trap this suite cleans up through, and a second one here would
18 # replace it. The standalone arm at the bottom arms its own, because on
19 # that path there is no other.
20 #
21 # The runner calls this once, before the first group. Every helper in
22 # os_oracle.sh is a question a pin asks INSTEAD of asking the daemon, so a
23 # helper that quietly stopped answering would not fail a pin loudly — it
24 # would make the pin agree with anything, and the suite would go green
25 # having tested nothing. This asks each helper something the box can
26 # independently confirm.
27 #
28 # Off-origin on every dimension the helpers could accidentally hold
29 # constant, because a fixture that pins N=1 or offset=0 is blind to N and
30 # to the offset (CLAUDE.md): the subject is a child that is not pid 1, it
31 # holds MORE than the three fds any fixture would, it has an argv worth
32 # losing, and it has a child of its own. The socket arm is asked in both
33 # directions — a listener that HOLDS the path and a shell that does not —
34 # because a `pid_holds_unix_sock` that answered yes to everything would
35 # pass the one pin that reads it just as happily as a correct one.
36 #
37 # The `sh -c` body ends in `:` on purpose. With nothing after it, a shell
38 # execs the last command in place rather than forking it — bash measured
39 # doing exactly that — and the subject would be `sleep`, wearing sleep's
40 # comm, exe and argv and having no child at all. That is a different
41 # process from the one this means to ask about.
42 oracle_selftest() {
43 _osock="$OUT.oracle.sock"
44 rm -f "$_osock"
45 printf 'hello' > "$OUT.oracle.probe"
46 # python3 rather than sh: nothing in POSIX sh can bind a unix socket,
47 # and the runner has already refused a box without python3.
48 python3 -c 'import socket,sys,time
49 s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(300)' \
50 "$_osock" &
51 _olpid=$!
52 sh -c 'exec 5>/dev/null 6>/dev/null; sleep 300; :' &
53 _opid=$!
54 # Registered, not merely killed at the bottom: an assertion below that
55 # fails exits through the trap, and an unregistered `sleep 300` would
56 # outlive the run by five minutes — ten of them under `make soak`.
57 defer_kill "$_olpid" "$_opid"
58 _oi=0
59 while [ ! -S "$_osock" ] && [ "$_oi" -lt $(( 50 * TIME_SCALE )) ]; do
60 sleep 0.1; _oi=$((_oi + 1))
61 done
62 [ -S "$_osock" ] || {
63 echo "e2e FAIL: oracle: the listener bound no socket at $_osock"; exit 1; }
64
65 # The subject is only the process this means to ask about once it has
66 # opened fds 5 and 6 and forked `sleep`; until then `pid_fd_count` and
67 # `pid_children` would be graded against a shell still starting up, and
68 # the self-test would fail for a reason that is nothing to do with the
69 # helpers. A child is the LAST of those steps, so waiting for one waits
70 # for all of them.
71 wait_until 50 "oracle: the subject shell never forked its sleep" \
72 '[ -n "$(pid_children "$_opid")" ]'
73
74 pid_alive "$_opid" || {
75 echo "e2e FAIL: oracle: pid_alive says a live shell is dead"; exit 1; }
76 # `sh` on both OSes, measured: Linux reads /proc/PID/comm, which is the
77 # last component of the path execve was handed, and Darwin's ps prints
78 # the bare name for a binary under /bin. It is only the IMAGE behind
79 # that name that differs between them, which is pid_exe's business
80 # below and not this line's.
81 [ "$(pid_comm "$_opid")" = sh ] || {
82 echo "e2e FAIL: oracle: pid_comm of an sh is '$(pid_comm "$_opid")'"; exit 1; }
83 pid_args "$_opid" | grep -q 'sleep 300' || {
84 echo "e2e FAIL: oracle: pid_args lost the argv: [$(pid_args "$_opid")]"; exit 1; }
85 [ "$(pid_fd_count "$_opid")" -ge 5 ] || {
86 echo "e2e FAIL: oracle: pid_fd_count is $(pid_fd_count "$_opid") for a shell"
87 echo " holding fd 5 and fd 6 on top of the usual three"; exit 1; }
88 # head -1 because this is a pid to kill as well as an answer to check,
89 # and the registry and kill(1) below both take one word.
90 _okid=$(pid_children "$_opid" | head -1)
91 [ -n "$_okid" ] || {
92 echo "e2e FAIL: oracle: pid_children names no child of a shell running sleep"
93 exit 1; }
94 defer_kill "$_okid"
95 # pid_exe is asked about the CHILD and not the shell, and the shell is
96 # the reason: macOS ships /bin/sh as a stub in front of the bash image,
97 # so the running image of a process this file started as `sh` is
98 # /bin/bash there — measured, and true of a copy of /bin/sh as well, so
99 # it is the image and not the name that differs. `sleep` is an ordinary
100 # binary on both OSes and its own path is what either arm answers, so
101 # the assertion stays exact rather than being weakened to fit. Through
102 # real_path on both sides because a distribution that puts /bin on a
103 # symlink to /usr/bin, and a Darwin lsof that canonicalizes /tmp to
104 # /private/tmp, both answer the resolved path and not the spelling.
105 [ "$(pid_exe "$_okid")" = "$(real_path "$(command -v sleep)")" ] || {
106 echo "e2e FAIL: oracle: pid_exe of the subject's sleep is '$(pid_exe "$_okid")',"
107 echo " want the resolved $(real_path "$(command -v sleep)")"; exit 1; }
108 # Through a variable and a case rather than straight into test(1): a
109 # helper that answered with nothing would otherwise abort the run with
110 # a bare `[: -gt: unary operator expected` and no e2e FAIL line saying
111 # which helper went quiet.
112 _orss=$(pid_rss_kb "$_opid")
113 case "$_orss" in ''|*[!0-9]*) _orss=0 ;; esac
114 [ "$_orss" -gt 0 ] || {
115 echo "e2e FAIL: oracle: pid_rss_kb is '$(pid_rss_kb "$_opid")'"; exit 1; }
116 pid_holds_unix_sock "$_olpid" "$_osock" || {
117 echo "e2e FAIL: oracle: pid_holds_unix_sock cannot see $_osock among the"
118 echo " fds of the pid $_olpid that bound it"; exit 1; }
119 # The `socket` KIND off a real fd: the listener bound one, so its count
120 # is at least one. Only that KIND is asserted here. `carrier` and
121 # `ptymaster` would each need a process holding an anonymous memory file
122 # or a pty master, which is a daemon — and e2e_14_upgrade already asks
123 # both of a real one, where a wrong answer fails that pin.
124 _ofdk=$(pid_holds_fd_kind "$_olpid" socket)
125 case "$_ofdk" in ''|*[!0-9]*) _ofdk=0 ;; esac
126 [ "$_ofdk" -ge 1 ] || {
127 echo "e2e FAIL: oracle: pid_holds_fd_kind says $_olpid holds"
128 echo " '$(pid_holds_fd_kind "$_olpid" socket)' fds of kind socket, and it"
129 echo " bound $_osock"; exit 1; }
130 # `! cmd || {}` rather than `cmd && {}`, for the reason spelled out
131 # below: an AND-list whose left side fails is a failing command under
132 # `set -e`, so the passing case would end the run.
133 ! pid_holds_fd_kind "$_opid" nosuchkind 2>/dev/null || {
134 echo "e2e FAIL: oracle: pid_holds_fd_kind accepted a kind it does not know,"
135 echo " so a typo in a group file would count zero and pass"; exit 1; }
136 ! pid_holds_unix_sock "$_opid" "$_osock" || {
137 echo "e2e FAIL: oracle: pid_holds_unix_sock says the shell $_opid holds"
138 echo " $_osock, which only $_olpid ever opened"; exit 1; }
139 # `! cmd || {}` rather than `cmd && {}`, for wait_sock's reason: an
140 # AND-list whose left side fails is a failing command under `set -e`,
141 # so the passing case would end the run.
142 ! udp_local_bound 'ZZZZZZZZ:ZZZZ' || {
143 echo "e2e FAIL: oracle: udp_local_bound found a socket at an address no"
144 echo " kernel can spell"; exit 1; }
145 [ -n "$(udp_table)" ] || {
146 echo "e2e FAIL: oracle: udp_table is empty, so a bind diagnostic would"
147 echo " print nothing"; exit 1; }
148 # Against what the probe file IS, never against a second spelling of the
149 # same command: comparing file_mode with `stat -c %a` would agree with
150 # itself on any OS and would have to be ported alongside the helper. A
151 # chmod the kernel performed, five bytes written, and the published
152 # SHA-256 of "hello" are answers no arm of the oracle can define away.
153 chmod 640 "$OUT.oracle.probe"
154 [ "$(file_mode "$OUT.oracle.probe")" = "640" ] || {
155 echo "e2e FAIL: oracle: file_mode says '$(file_mode "$OUT.oracle.probe")' for a"
156 echo " file the kernel just chmod'd to 640"; exit 1; }
157 [ "$(file_size "$OUT.oracle.probe")" = "5" ] || {
158 echo "e2e FAIL: oracle: file_size says '$(file_size "$OUT.oracle.probe")' for five bytes"
159 exit 1; }
160 [ "$(sha256_of "$OUT.oracle.probe")" = \
161 "2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824" ] || {
162 echo "e2e FAIL: oracle: sha256_of hashes 'hello' to"
163 echo " '$(sha256_of "$OUT.oracle.probe")'"; exit 1; }
164
165 kill "$_olpid" "$_opid" "$_okid" 2>/dev/null || true
166 wait "$_olpid" 2>/dev/null || true
167 wait "$_opid" 2>/dev/null || true
168 _oi=0
169 while pid_alive "$_opid" && [ "$_oi" -lt $(( 50 * TIME_SCALE )) ]; do
170 sleep 0.05; _oi=$((_oi + 1))
171 done
172 ! pid_alive "$_opid" || {
173 echo "e2e FAIL: oracle: pid_alive says a killed shell still lives"; exit 1; }
174 rm -f "$_osock" "$OUT.oracle.probe"
175 ok "oracle: the OS answers the helpers by name"
176 # One fixed word, on top of ok()'s sentence: a port is graded over ssh
177 # and the run that grades it has one line to look for.
178 echo "ORACLE_OK"
179 }
180
181 # ---- the standalone arm -------------------------------------------------
182 # `$0` is this file only when it was RUN. Sourced from e2e_lib.sh it is the
183 # runner's path, and sourced by hand it is the shell's, so neither takes
184 # this branch and neither gets a second EXIT trap.
185 #
186 # The four names oracle_selftest reads out of the suite are spelled here in
187 # their smallest honest form. They are not a copy of the lib's: standalone
188 # there is no cleanup registry to share, no scenario count to keep and no
189 # traced run to scale timeouts for, so each is the whole of that duty on
190 # this path rather than a reimplementation of the suite's.
191 case "$0" in
192 *oracle_selftest.sh)
193 set -e
194 . "$(dirname "$0")/os_oracle.sh"
195 : "${TIME_SCALE:=1}"
196 : "${OUT:=${TMPDIR:-/tmp}/mux-oracle-$$}"
197 _SELF_KILL=""
198 defer_kill() {
199 for _dk in "$@"; do
200 if [ -n "$_dk" ]; then _SELF_KILL="$_dk $_SELF_KILL"; fi
201 done
202 return 0
203 }
204 ok() { echo "e2e OK: $1"; }
205 wait_until() {
206 _i=0
207 while [ "$_i" -lt $(( $1 * TIME_SCALE )) ]; do
208 # First command of an AND-OR list, which set -e exempts: a
209 # predicate that is false on this tick is the normal case.
210 eval "$3" && return 0
211 sleep 0.1; _i=$((_i + 1))
212 done
213 echo "e2e FAIL: $2"; exit 1
214 }
215 # Children first, and through the oracle's own pid_children: the subject
216 # is a shell whose `sleep 300` is registered only after the assertions
217 # that come before it, so a run that fails early leaves a grandchild
218 # that no list names. It inherits this script's stdout, and an ssh
219 # reading that pipe waits on the fd rather than on the process — so the
220 # leak does not merely outlive the run, it hangs whoever asked for it,
221 # which is exactly how a port gets graded from another box.
222 trap 'for _k in $_SELF_KILL; do
223 for _kc in $(pid_children "$_k" 2>/dev/null); do
224 kill "$_kc" 2>/dev/null || true
225 done
226 kill "$_k" 2>/dev/null || true
227 done
228 rm -f "$OUT".oracle.*' EXIT INT TERM
229 oracle_selftest
230 ;;
231 esac
test/os_oracle.sh
Old New
@@ -1,65 +1,241 @@
1 # shellcheck shell=sh 1 # shellcheck shell=sh
2 # os_oracle.sh — the OS oracle, sourced and never run. 2 # os_oracle.sh — the OS oracle, sourced and never run.
3 # 3 #
4 # Sourced by test/e2e_lib.sh, where these helpers used to live, and by 4 # Sourced by test/e2e_lib.sh, where these helpers used to live, by
5 # test/soak.sh, which reads two of them between runs. It is deliberately 5 # test/oracle_selftest.sh, which is their own pin, and by test/soak.sh,
6 # trap-free and state-free: both readers bring a cleanup registry and an 6 # which reads two of them between runs. It is deliberately trap-free and
7 # EXIT trap of their own, and a second trap here would replace one of 7 # state-free: its readers bring a cleanup registry and an EXIT trap of
8 # theirs. soak.sh used to carry copies of the two it needs, and a copy is a 8 # their own, and a second trap here would replace one of theirs. soak.sh
9 # second spelling to port the day a second OS arrives. 9 # used to carry copies of the two it needs, and a copy is a second spelling
10 # to port the day a second OS arrives.
10 # 11 #
11 # "Ask the OS about the OS, not the daemon" (CLAUDE.md). Every pin that 12 # "Ask the OS about the OS, not the daemon" (CLAUDE.md). Every pin that
12 # reads a pid, an fd table, a bound UDP port or a file's mode asks through 13 # reads a pid, an fd table, a bound UDP port or a file's mode asks through
13 # these names, so the SPELLING of the question lives in one place per OS 14 # these names, so the SPELLING of the question lives in one place per OS
14 # and the question itself lives in the group file. The Linux arm is /proc, 15 # and the question itself lives in the group file. Each prints its answer
15 # GNU stat and sha256sum; a second OS adds a `case "$(uname)"` arm here and 16 # on stdout and returns 1 when the OS will not say, so a caller may treat
16 # nothing in a group file changes. Each prints its answer on stdout and 17 # "no answer" and "the wrong answer" as the same failure.
17 # returns 1 when the OS will not say, so a caller may treat "no answer"
18 # and "the wrong answer" as the same failure.
19 # 18 #
20 # oracle_selftest in e2e_lib.sh is these helpers' own pin. A helper that 19 # There are two arms below and a group file must never learn which one it
21 # stopped answering — a missing binary, a /proc the sandbox will not show — 20 # is on. That is why udp_local_bound still takes the Linux hex spelling on
22 # would otherwise turn every pin that reads it into a check that passes 21 # Darwin and converts it here, and why now_ms and real_path exist at all:
23 # without running, which is the one failure a green tree cannot show. 22 # `date +%s%N` and `readlink -f` are GNU, and a group file that spelled
24 pid_alive() { kill -0 "$1" 2>/dev/null || [ -d "/proc/$1" ]; } 23 # either would be a Linux group file.
25 pid_exe() { readlink -f "/proc/$1/exe" 2>/dev/null; } 24 #
26 pid_comm() { cat "/proc/$1/comm" 2>/dev/null; } 25 # oracle_selftest (test/oracle_selftest.sh) is these helpers' own pin. A
27 pid_args() { tr '\0' ' ' < "/proc/$1/cmdline" 2>/dev/null; } 26 # helper that stopped answering — a missing binary, a /proc the sandbox
28 pid_children() { ps -o pid= --ppid "$1" 2>/dev/null | tr -d ' '; } 27 # will not show — would otherwise turn every pin that reads it into a check
29 pid_fd_count() { find "/proc/$1/fd" -mindepth 1 2>/dev/null | wc -l | tr -d ' '; } 28 # that passes without running, which is the one failure a green tree cannot
30 pid_fd_targets() { readlink "/proc/$1"/fd/* 2>/dev/null; } 29 # show.
31 # pid_holds_fd_kind PID KIND — how many of PID's open fds are of KIND, which 30
32 # is one of `socket`, `carrier` (the upgrade manifest's anonymous memory 31 case "$(uname)" in
33 # file) or `ptymaster`. Prints the count. These are the three KINDS an 32 Darwin)
34 # upgrade pin asks about, and naming them is the point: a second OS's arm 33 # No /proc here, so lsof is the fd oracle and ps answers the process
35 # counts the same three things under whatever that OS calls them, and the 34 # questions. Both read the kernel through libproc, so this arm still
36 # group file keeps asking the same question. `|| true` because grep -c 35 # asks the OS about the OS and never a daemon. lsof's -F mode prints
37 # prints 0 AND exits 1 when nothing matches, and callers run under `set -e`. 36 # one FIELD per line — `f` an fd, `t` its type, `n` its name — which is
38 pid_holds_fd_kind() { 37 # what the helpers below parse, and -a ANDs its selection flags where
39 case "$2" in 38 # lsof would otherwise OR them.
40 socket) _fdk='socket:' ;; 39
41 carrier) _fdk='memfd:' ;; 40 # _fd_rows PID — one `TYPE NAME` line per NUMBERED fd. lsof reports the
42 ptymaster) _fdk='/dev/ptmx' ;; 41 # cwd, the executable and the root directory through the same stream
43 *) echo "pid_holds_fd_kind: no such fd kind '$2'" >&2; return 1 ;; 42 # with non-numeric fd ids (`fcwd`, `ftxt`, `frtd`), and no question
44 esac 43 # below asks about those. The TYPE has to come along because an lsof
45 pid_fd_targets "$1" | grep -c -F "$_fdk" || true 44 # name alone does not say what an fd is: a unix socket and a regular
46 } 45 # file both read as a path, where a /proc symlink target says which.
47 # By INODE, never by name: a process running the right program while 46 _fd_rows() {
48 # binding something else entirely would satisfy a name check. The kernel's 47 lsof -p "$1" -Ftn 2>/dev/null | awk '
49 # listening inode for PATH, found among that pid's open fds, cannot. 48 /^f/ { fd = substr($0, 2); ty = "" }
50 pid_holds_unix_sock() { 49 /^t/ { ty = substr($0, 2) }
51 _ino=$(awk -v p="$2" '$NF == p {print $7}' /proc/net/unix | head -1) 50 /^n/ { if (fd ~ /^[0-9]+$/) print ty " " substr($0, 2) }'
52 [ -n "$_ino" ] && pid_fd_targets "$1" | grep -qx "socket:\[$_ino\]" 51 }
53 } 52 pid_alive() { kill -0 "$1" 2>/dev/null; }
54 pid_rss_kb() { awk '/VmRSS/{print $2}' "/proc/$1/status" 2>/dev/null || echo 0; } 53 # The FIRST txt vnode is the executable; the ones after it are dyld and
55 # Matched on the LOCAL address ($2) rather than anywhere on the line: the 54 # the dylibs it mapped. Two things this answers that a caller has to
56 # remote address of a client dialling this port is the same hex, and a 55 # know (both measured on macOS 26.6, lsof 4.91): lsof canonicalizes, so
57 # whole-line match read a reconnecting client as a bound listener. 56 # a binary opened as /tmp/x reads /private/tmp/x, which is exactly what
58 udp_local_bound() { awk -v h="$1" '$2==h{f=1} END{exit !f}' /proc/net/udp; } 57 # real_path answers for the same path; and a process that exec'd
59 udp_table() { cat /proc/net/udp 2>/dev/null; } 58 # /bin/sh reports /bin/bash, because Apple ships /bin/sh as a stub in
60 file_mode() { stat -c %a "$1"; } 59 # front of the bash image. No binary mux runs is such a stub, so the
61 file_size() { stat -c %s "$1"; } 60 # pins that read this get back the path they asked about — but the
62 sha256_of() { sha256sum "$1" | cut -d' ' -f1; } 61 # oracle's own self-test asks about a `sleep` for that reason.
62 pid_exe() { lsof -p "$1" -a -d txt -Fn 2>/dev/null | sed -n 's/^n//p' | head -1; }
63 # ps prints the full path for a binary outside the system directories
64 # and the bare name for one inside them, so the last path component is
65 # taken either way. Through a variable rather than basename(1): an
66 # empty answer must be no answer, and `basename ""` prints a line.
67 pid_comm() {
68 _pc=$(ps -o comm= -p "$1" 2>/dev/null) || return 1
69 [ -n "$_pc" ] || return 1
70 printf '%s\n' "${_pc##*/}"
71 }
72 pid_args() { ps -o args= -p "$1" 2>/dev/null; }
73 pid_children() { pgrep -P "$1" 2>/dev/null; }
74 pid_fd_count() { lsof -p "$1" -Ff 2>/dev/null | grep -c '^f[0-9]' || true; }
75 pid_fd_targets() { _fd_rows "$1"; }
76 pid_holds_fd_kind() {
77 case "$2" in
78 # Every socket family, which is what the Linux arm's `socket:`
79 # prefix counts. PIPE is not among them, because `pipe:[N]` is
80 # not on that side either.
81 socket) _fd_rows "$1" | grep -c '^\(unix\|IPv4\|IPv6\|sock\) ' || true ;;
82 # The upgrade manifest is an unlinked mkstemp file here, not a
83 # memfd (server_os_macos.anonFd). lsof still prints the path it
84 # had before the unlink, canonicalized and with no marker of
85 # its own, so the NAME cannot tell a carrier from an ordinary
86 # open file — the suite's own captures live under /tmp with the
87 # same `mux-` prefix and would be counted as carriers. What
88 # does tell them apart is that a carrier's path names nothing
89 # any more, which is the whole of what "anonymous" means here.
90 carrier)
91 _fd_rows "$1" | sed -n 's/^REG //p' | {
92 _fdn=0
93 while IFS= read -r _fdp; do
94 [ -e "$_fdp" ] || _fdn=$((_fdn + 1))
95 done
96 echo "$_fdn"
97 } ;;
98 ptymaster) _fd_rows "$1" | grep -c '^CHR /dev/ptmx$' || true ;;
99 *) echo "pid_holds_fd_kind: no such fd kind '$2'" >&2; return 1 ;;
100 esac
101 }
102 # By PATH, where the Linux arm matches the listening inode: lsof answers
103 # a unix socket's bound path and no inode, so there is no inode to
104 # match. It is still this pid's own fd table being read and not the
105 # filesystem, so a process that merely runs the right program does not
106 # satisfy it; what it cannot rule out, and Linux can, is a second
107 # socket bound at the same path. -F because a path holds regex
108 # metacharacters and this is a literal comparison.
109 pid_holds_unix_sock() { _fd_rows "$1" | grep -qxF "unix $2"; }
110 pid_rss_kb() { ps -o rss= -p "$1" 2>/dev/null | tr -d ' '; }
111 # _udp_dotted HEX — the Linux argument spelling turned into the dotted
112 # form lsof prints. Anything that is not eight hex digits, a colon and
113 # four more is no address at all and gets no answer, which is what the
114 # self-test asks for when it hands this a spelling no kernel can hold.
115 # Shell arithmetic reads a 0x constant by POSIX, so this forks nothing:
116 # udp_local_bound is polled in a loop and an interpreter start per tick
117 # would be most of the wait.
118 _udp_dotted() {
119 # Eight hex digits, a colon and four more. Asked as "no byte outside
120 # the alphabet, one colon, and these two lengths" rather than as one
121 # bracket pattern built from a variable, because a case pattern that
122 # came out of an expansion is not treated as a pattern by every
123 # shell — zsh needs an option set for it, and this file is sourced
124 # by whatever the caller runs.
125 case "$1" in
126 *[!0-9A-Fa-f:]* | *:*:* | *[!0-9A-Fa-f]) return 1 ;;
127 esac
128 _uh=${1%:*}
129 _ut=${1#*:}
130 [ "${#_uh}" -eq 8 ] && [ "${#_ut}" -eq 4 ] || return 1
131 _up=$(( 0x$_ut ))
132 # A wildcard bind reads `*` to lsof and not 0.0.0.0, and that is the
133 # spelling the handoff group hands over for the daemon's lazy bind.
134 if [ "$_uh" = 00000000 ]; then echo "*:$_up"; return 0; fi
135 # /proc/net/udp writes the local address as the host-order u32 in
136 # hex, which on a little-endian box puts the last octet first, so
137 # the four bytes come back out in reverse.
138 _u1=${_uh%??????}
139 _ur=${_uh#??}; _u2=${_ur%????}
140 _ur=${_uh#????}; _u3=${_ur%??}
141 _u4=${_uh#??????}
142 echo "$(( 0x$_u4 )).$(( 0x$_u3 )).$(( 0x$_u2 )).$(( 0x$_u1 )):$_up"
143 }
144 # Matched on the LOCAL address for the Linux arm's reason: lsof prints
145 # a connected socket as LOCAL->REMOTE, so a client dialling this port
146 # answers a match that reads anywhere on the line. The arrow and
147 # everything after it goes before the comparison.
148 udp_local_bound() {
149 _ua=$(_udp_dotted "$1") || return 1
150 lsof -iUDP -P -n -Fn 2>/dev/null | sed -n 's/^n//p' | sed 's/->.*$//' |
151 grep -qxF "$_ua"
152 }
153 udp_table() { lsof -iUDP -P -n 2>/dev/null; }
154 file_mode() { stat -f %Lp "$1"; }
155 file_size() { stat -f %z "$1"; }
156 sha256_of() { shasum -a 256 "$1" | cut -d' ' -f1; }
157 # GNU coreutils under their g-names, which is a dependency this harness
158 # already has on Darwin: a bare Mac has no `timeout` either, and the
159 # gtimeout fallback at the bottom of this file is how that is met.
160 # Taking them here rather than reimplementing BSD spellings keeps the
161 # two arms asking one question — greadlink -f resolves a path whose last
162 # component does not exist yet, which e2e_14_upgrade hands it and a
163 # realpath(3) would refuse.
164 real_path() { greadlink -f "$1"; }
165 # gdate and not the system date, even though macOS 26's /bin/date does
166 # answer %N (measured, and older ones do not). A `date +%s%N` here would
167 # make the suite's clock depend on how new the OS is, and its failure on
168 # an older one is a literal N inside an arithmetic expansion partway
169 # through a run. gdate is the same GNU date the other arm calls and is
170 # already on any box this harness runs on, for gtimeout's sake.
171 #
172 # An interpreter instead of either was measured and rejected: python3
173 # costs 21.6ms a call and perl 4.5 against gdate's 2.1 (2026-09-03), and
174 # the tightest thing a now_ms delta is graded against is the throughput
175 # gate's 10ms ceiling around a 6ms leg — so a python3 start would be
176 # most of the reading rather than noise around it.
177 now_ms_snippet() { echo 'now_ms() { echo $(( $(gdate +%s%N) / 1000000 )); }'; }
178 ;;
179 *)
180 pid_alive() { kill -0 "$1" 2>/dev/null || [ -d "/proc/$1" ]; }
181 pid_exe() { readlink -f "/proc/$1/exe" 2>/dev/null; }
182 pid_comm() { cat "/proc/$1/comm" 2>/dev/null; }
183 pid_args() { tr '\0' ' ' < "/proc/$1/cmdline" 2>/dev/null; }
184 pid_children() { ps -o pid= --ppid "$1" 2>/dev/null | tr -d ' '; }
185 pid_fd_count() { find "/proc/$1/fd" -mindepth 1 2>/dev/null | wc -l | tr -d ' '; }
186 pid_fd_targets() { readlink "/proc/$1"/fd/* 2>/dev/null; }
187 # pid_holds_fd_kind PID KIND — how many of PID's open fds are of KIND,
188 # which is one of `socket`, `carrier` (the upgrade manifest's anonymous
189 # memory file) or `ptymaster`. Prints the count. These are the three
190 # KINDS an upgrade pin asks about, and naming them is the point: the
191 # other arm counts the same three things under whatever that OS calls
192 # them, and the group file keeps asking the same question. `|| true`
193 # because grep -c prints 0 AND exits 1 when nothing matches, and
194 # callers run under `set -e`.
195 pid_holds_fd_kind() {
196 case "$2" in
197 socket) _fdk='socket:' ;;
198 carrier) _fdk='memfd:' ;;
199 ptymaster) _fdk='/dev/ptmx' ;;
200 *) echo "pid_holds_fd_kind: no such fd kind '$2'" >&2; return 1 ;;
201 esac
202 pid_fd_targets "$1" | grep -c -F "$_fdk" || true
203 }
204 # By INODE, never by name: a process running the right program while
205 # binding something else entirely would satisfy a name check. The
206 # kernel's listening inode for PATH, found among that pid's open fds,
207 # cannot.
208 pid_holds_unix_sock() {
209 _ino=$(awk -v p="$2" '$NF == p {print $7}' /proc/net/unix | head -1)
210 [ -n "$_ino" ] && pid_fd_targets "$1" | grep -qx "socket:\[$_ino\]"
211 }
212 pid_rss_kb() { awk '/VmRSS/{print $2}' "/proc/$1/status" 2>/dev/null || echo 0; }
213 # Matched on the LOCAL address ($2) rather than anywhere on the line:
214 # the remote address of a client dialling this port is the same hex,
215 # and a whole-line match read a reconnecting client as a bound
216 # listener.
217 udp_local_bound() { awk -v h="$1" '$2==h{f=1} END{exit !f}' /proc/net/udp; }
218 udp_table() { cat /proc/net/udp 2>/dev/null; }
219 file_mode() { stat -c %a "$1"; }
220 file_size() { stat -c %s "$1"; }
221 sha256_of() { sha256sum "$1" | cut -d' ' -f1; }
222 real_path() { readlink -f "$1"; }
223 now_ms_snippet() { echo 'now_ms() { echo $(( $(date +%s%N) / 1000000 )); }'; }
224 ;;
225 esac
226
227 # Each arm above defines now_ms_snippet — the TEXT of a now_ms — and this is
228 # where the harness's own now_ms comes from. The throughput legs and the
229 # agent-mute leg time themselves from INSIDE a mux session, so their clock
230 # has to be typed into the bytes they send rather than called, and a group
231 # file that spelled `date +%s%N` into a session would be a Linux group file
232 # however OS-blind the rest of it is.
233 #
234 # Defined THROUGH the snippet rather than beside it so the clock this shell
235 # reads and the clock it types into another shell are the same text and
236 # cannot drift into two answers on one box.
237 eval "$(now_ms_snippet)"
238
63 # GNU timeout is a binary here; a box without one names it gtimeout, and a 239 # GNU timeout is a binary here; a box without one names it gtimeout, and a
64 # group file keeps spelling `timeout` either way. 240 # group file keeps spelling `timeout` either way.
65 command -v timeout >/dev/null 2>&1 || timeout() { gtimeout "$@"; } 241 command -v timeout >/dev/null 2>&1 || timeout() { gtimeout "$@"; }
test/throughput.sh
Old New
@@ -35,6 +35,13 @@ set -eu
35 MUX="$1" 35 MUX="$1"
36 PTYCLIENT="$2" 36 PTYCLIENT="$2"
37 37
38 # The oracle, for now_ms_snippet alone. Every leg below times itself from
39 # INSIDE the session with the shell's own clock, and `date +%s%N` is GNU —
40 # so the text of that clock comes from the one place per OS that spells it,
41 # the same way soak.sh takes the two helpers it reads.
42 # shellcheck source=test/os_oracle.sh
43 . "$(dirname "$0")/os_oracle.sh"
44
38 # Lines of `yes`, and the wall-clock ceiling each leg must come in under. 45 # Lines of `yes`, and the wall-clock ceiling each leg must come in under.
39 # 46 #
40 # Every bound is overridable from the environment. All of them were 47 # Every bound is overridable from the environment. All of them were
@@ -115,7 +122,7 @@ done
115 # The `done` marker is a separate file on purpose: polling the timings file 122 # The `done` marker is a separate file on purpose: polling the timings file
116 # itself would read it after the first rep and call that the answer. 123 # itself would read it after the first rep and call that the answer.
117 "$MUX" a send --sock "$SOCK" \ 124 "$MUX" a send --sock "$SOCK" \
118 "for k in \$(seq $REPS); do A=\$(date +%s%N); yes | head -n $SOLO_LINES; B=\$(date +%s%N); echo \$(( (B-A)/1000000 )) >> $TMPD/solo; done; echo ok > $TMPD/solo.done\n" \ 125 "$(now_ms_snippet); for k in \$(seq $REPS); do A=\$(now_ms); yes | head -n $SOLO_LINES; B=\$(now_ms); echo \$(( B-A )) >> $TMPD/solo; done; echo ok > $TMPD/solo.done\n" \
119 > /dev/null 126 > /dev/null
120 i=0 127 i=0
121 while [ ! -s "$TMPD/solo.done" ] && [ "$i" -lt 60 ]; do sleep 0.5; i=$((i+1)); done 128 while [ ! -s "$TMPD/solo.done" ] && [ "$i" -lt 60 ]; do sleep 0.5; i=$((i+1)); done
@@ -147,7 +154,7 @@ echo "solo: $SOLO_LINES lines, no client: ${SOLO_MS}ms (max ${SOLO_MAX_MS}ms)
147 # an engine of its own and can regress the same way the daemon did. 154 # an engine of its own and can regress the same way the daemon did.
148 cat > "$TMPD/script" <<EOF 155 cat > "$TMPD/script" <<EOF
149 settle 500 8000 156 settle 500 8000
150 send for k in \$(seq $REPS); do A=\$(date +%s%N); yes | head -n $CLIENT_LINES; B=\$(date +%s%N); echo \$(( (B-A)/1000000 )) >> $TMPD/client; done; echo DON""E\r 157 send $(now_ms_snippet); for k in \$(seq $REPS); do A=\$(now_ms); yes | head -n $CLIENT_LINES; B=\$(now_ms); echo \$(( B-A )) >> $TMPD/client; done; echo DON""E\r
151 expect DONE 90000 158 expect DONE 90000
152 send \x1c\x1c 159 send \x1c\x1c
153 waitexit 5000 160 waitexit 5000
@@ -232,7 +239,7 @@ REPAINT_BYTES=$(wc -c < "$TMPD/big")
232 239
233 rm -f "$TMPD/repaint" "$TMPD/repaint.done" 240 rm -f "$TMPD/repaint" "$TMPD/repaint.done"
234 "$MUX" a send --sock "$SOCK" \ 241 "$MUX" a send --sock "$SOCK" \
235 "for k in \$(seq $REPS); do A=\$(date +%s%N); cat $TMPD/big; B=\$(date +%s%N); echo \$(( (B-A)/1000000 )) >> $TMPD/repaint; done; echo ok > $TMPD/repaint.done\n" \ 242 "$(now_ms_snippet); for k in \$(seq $REPS); do A=\$(now_ms); cat $TMPD/big; B=\$(now_ms); echo \$(( B-A )) >> $TMPD/repaint; done; echo ok > $TMPD/repaint.done\n" \
236 > /dev/null 243 > /dev/null
237 i=0 244 i=0
238 while [ ! -s "$TMPD/repaint.done" ] && [ "$i" -lt 120 ]; do sleep 0.5; i=$((i+1)); done 245 while [ ! -s "$TMPD/repaint.done" ] && [ "$i" -lt 120 ]; do sleep 0.5; i=$((i+1)); done