a73x

b38521e1

test: the oracle's self-test asks every helper a positive, and the clock at all

a73x   2026-09-03 21:19

Commit message
test: the oracle's self-test asks every helper a positive, and the clock at all

Three helpers were asked nothing but negatives, and one pair was asked
nothing. A pin that only ever expects "no" passes just as happily for an arm
that answers "no" to everything, which is a check that never runs rather
than a check that fails — and on Darwin the carrier body is a heuristic of
its own, so the blindness was new in substance.

The listener subject now holds one fd of every KIND before it reports it is
ready: a unix socket, a UDP socket on an ephemeral port, a carrier and a pty
master. `carrier` and `ptymaster` are asserted at least one of on the
listener and exactly zero of on the subject shell, so a helper that counts
any open file or any character device answers the second half.

The carrier is opened in the shape this OS's `server_os.anonFd` makes, and
os_oracle.sh's new carrier_kind is what names it — a memfd on Linux, an
unlinked mkstemp file on Darwin. The self-test asks the oracle rather than
reading `uname`, so every "which OS" answer stays in the one file allowed to
know one.

udp_local_bound is asked the port the listener really bound, in the
/proc/net/udp hex spelling both arms take, and then asked the SAME port at
127.0.0.2 — which is the assertion that pins the local-address rule both
arms are built around, and which nothing in the tree made before.

now_ms and now_ms_snippet were never asked at all, though six e2e brackets
and the throughput gate read them. now_ms must be the 13 digits epoch
milliseconds have, so that neither the seconds nor the nanoseconds either
side of them pass, and must move about a second across a sleep. The snippet
is run in a fresh shell, the way the throughput and agent-mute legs use it:
on Darwin it names gdate, so a Mac without coreutils now fails at this pin
instead of inside a $(( )) partway through a group file. Both are captured
with their stderr and graded out of a variable, because a clock that is not
installed exits 127 and an assignment that fails ends the run with the
shell's own "not found" and no line naming the helper.

Also from the review: Darwin pid_exe returns 1 rather than an empty success
for a pid that is gone, as the file's header promises; the socket KIND uses
an ERE rather than a GNU BRE alternation BSD grep does not guarantee; the
Darwin pid_alive says in a comment that dropping the /proc fallback leaves
kill(2) reading another user's live pid as dead; and both oracle sources in
e2e_lib.sh carry the shellcheck directive its neighbours do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SakwJEwD9dXBoRP5kWbemW
test/e2e_lib.sh
Old New
@@ -344,11 +344,13 @@ softkill() {
344 # and its copies were a second spelling of a question this repo means to 344 # and its copies were a second spelling of a question this repo means to
345 # ask in one place. That file is trap-free on purpose: the registry and the 345 # ask in one place. That file is trap-free on purpose: the registry and the
346 # EXIT trap below are this file's, and it must not bring a second of either. 346 # EXIT trap below are this file's, and it must not bring a second of either.
347 # shellcheck source=test/os_oracle.sh
347 . "$(dirname "$0")/os_oracle.sh" 348 . "$(dirname "$0")/os_oracle.sh"
348 # And the oracle's own pin, which moved out of this file for the same 349 # And the oracle's own pin, which moved out of this file for the same
349 # reason: it has to be runnable on a box where nothing else here works 350 # reason: it has to be runnable on a box where nothing else here works
350 # yet, which is the first thing a new OS arm needs. It defines 351 # yet, which is the first thing a new OS arm needs. It defines
351 # oracle_selftest and nothing else on this path. 352 # oracle_selftest and nothing else on this path.
353 # shellcheck source=test/oracle_selftest.sh
352 . "$(dirname "$0")/oracle_selftest.sh" 354 . "$(dirname "$0")/oracle_selftest.sh"
353 355
354 # Poll until nothing answers on a socket path (2s). Keyed off the daemon's 356 # Poll until nothing answers on a socket path (2s). Keyed off the daemon's
test/oracle_selftest.sh
Old New
@@ -41,13 +41,45 @@
41 # process from the one this means to ask about. 41 # process from the one this means to ask about.
42 oracle_selftest() { 42 oracle_selftest() {
43 _osock="$OUT.oracle.sock" 43 _osock="$OUT.oracle.sock"
44 rm -f "$_osock" 44 _oport_f="$OUT.oracle.port"
45 rm -f "$_osock" "$_oport_f"
45 printf 'hello' > "$OUT.oracle.probe" 46 printf 'hello' > "$OUT.oracle.probe"
46 # python3 rather than sh: nothing in POSIX sh can bind a unix socket, 47 # python3 rather than sh: nothing in POSIX sh can bind a unix socket,
47 # and the runner has already refused a box without python3. 48 # and the runner has already refused a box without python3.
48 python3 -c 'import socket,sys,time 49 #
49 s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(300)' \ 50 # It holds one fd of every KIND a pin asks about, so that each of them
50 "$_osock" & 51 # can be asked in BOTH directions. Before this the listener held only a
52 # unix socket, and `carrier`, `ptymaster` and `udp_local_bound` were
53 # asked nothing but negatives anywhere in the tree — an arm that
54 # answered 0 and false forever passed every gate on both OSes, which is
55 # a check that never runs rather than a check that fails.
56 #
57 # The carrier is opened in the local shape, which carrier_kind names:
58 # a memfd on Linux and an unlinked mkstemp file on Darwin, matching what
59 # server_os_linux.anonFd and server_os_macos.anonFd each make. Asking
60 # the oracle rather than reading `uname` here keeps every "which OS"
61 # answer in the one file that is allowed to know.
62 #
63 # The UDP port is EPHEMERAL and reported back rather than picked here:
64 # a number this file chose would be a number some other process on the
65 # box may already hold, and the pin would fail for a reason that is
66 # nothing to do with the helper.
67 python3 -c 'import os, socket, sys, tempfile, time
68 s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1)
69 u = socket.socket(socket.AF_INET, socket.SOCK_DGRAM); u.bind(("127.0.0.1", 0))
70 if sys.argv[2] == "memfd":
71 carrier = os.memfd_create("mux-upgrade")
72 else:
73 carrier, path = tempfile.mkstemp(prefix="mux-upgrade-", dir="/tmp")
74 os.unlink(path)
75 master, slave = os.openpty()
76 # Last, and to a temporary name renamed into place, so that the file
77 # existing means every fd above is open and not that some of them are.
78 with open(sys.argv[3] + ".part", "w") as f:
79 f.write("%d" % u.getsockname()[1])
80 os.rename(sys.argv[3] + ".part", sys.argv[3])
81 time.sleep(300)' \
82 "$_osock" "$(carrier_kind)" "$_oport_f" &
51 _olpid=$! 83 _olpid=$!
52 sh -c 'exec 5>/dev/null 6>/dev/null; sleep 300; :' & 84 sh -c 'exec 5>/dev/null 6>/dev/null; sleep 300; :' &
53 _opid=$! 85 _opid=$!
@@ -56,11 +88,15 @@ s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(
56 # outlive the run by five minutes — ten of them under `make soak`. 88 # outlive the run by five minutes — ten of them under `make soak`.
57 defer_kill "$_olpid" "$_opid" 89 defer_kill "$_olpid" "$_opid"
58 _oi=0 90 _oi=0
59 while [ ! -S "$_osock" ] && [ "$_oi" -lt $(( 50 * TIME_SCALE )) ]; do 91 while [ ! -f "$_oport_f" ] && [ "$_oi" -lt $(( 50 * TIME_SCALE )) ]; do
60 sleep 0.1; _oi=$((_oi + 1)) 92 sleep 0.1; _oi=$((_oi + 1))
61 done 93 done
94 [ -f "$_oport_f" ] || {
95 echo "e2e FAIL: oracle: the listener never reported its port, so one of"
96 echo " the fds the pins below read was never opened"; exit 1; }
62 [ -S "$_osock" ] || { 97 [ -S "$_osock" ] || {
63 echo "e2e FAIL: oracle: the listener bound no socket at $_osock"; exit 1; } 98 echo "e2e FAIL: oracle: the listener bound no socket at $_osock"; exit 1; }
99 _oport=$(cat "$_oport_f")
64 100
65 # The subject is only the process this means to ask about once it has 101 # The subject is only the process this means to ask about once it has
66 # opened fds 5 and 6 and forked `sleep`; until then `pid_fd_count` and 102 # opened fds 5 and 6 and forked `sleep`; until then `pid_fd_count` and
@@ -116,17 +152,32 @@ s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(
116 pid_holds_unix_sock "$_olpid" "$_osock" || { 152 pid_holds_unix_sock "$_olpid" "$_osock" || {
117 echo "e2e FAIL: oracle: pid_holds_unix_sock cannot see $_osock among the" 153 echo "e2e FAIL: oracle: pid_holds_unix_sock cannot see $_osock among the"
118 echo " fds of the pid $_olpid that bound it"; exit 1; } 154 echo " fds of the pid $_olpid that bound it"; exit 1; }
119 # The `socket` KIND off a real fd: the listener bound one, so its count 155 # Every KIND, off a real fd, in both directions: the listener holds one
120 # is at least one. Only that KIND is asserted here. `carrier` and 156 # of each and the subject shell holds none of the two that are not a
121 # `ptymaster` would each need a process holding an anonymous memory file 157 # socket. Both halves matter. e2e_14_upgrade asks `carrier` and
122 # or a pty master, which is a daemon — and e2e_14_upgrade already asks 158 # `ptymaster` too, but every one of its call sites asserts ZERO — so an
123 # both of a real one, where a wrong answer fails that pin. 159 # arm that answered 0 forever would pass that pin as happily as a
124 _ofdk=$(pid_holds_fd_kind "$_olpid" socket) 160 # correct one, and on Darwin the carrier body is a heuristic of its own
125 case "$_ofdk" in ''|*[!0-9]*) _ofdk=0 ;; esac 161 # (an unlinked regular file) rather than a name the kernel hands over.
126 [ "$_ofdk" -ge 1 ] || { 162 for _ok in socket carrier ptymaster; do
127 echo "e2e FAIL: oracle: pid_holds_fd_kind says $_olpid holds" 163 _ofdk=$(pid_holds_fd_kind "$_olpid" "$_ok")
128 echo " '$(pid_holds_fd_kind "$_olpid" socket)' fds of kind socket, and it" 164 case "$_ofdk" in ''|*[!0-9]*) _ofdk=0 ;; esac
129 echo " bound $_osock"; exit 1; } 165 [ "$_ofdk" -ge 1 ] || {
166 echo "e2e FAIL: oracle: pid_holds_fd_kind says $_olpid holds"
167 echo " '$(pid_holds_fd_kind "$_olpid" "$_ok")' fds of kind $_ok, and it"
168 echo " opened one of each before it reported its port"; exit 1; }
169 done
170 # The subject shell opened two /dev/null fds and forked a sleep, and
171 # nothing else. A helper that counted any open file as a carrier, or any
172 # character device as a pty master, would answer here.
173 for _ok in carrier ptymaster; do
174 _ofdk=$(pid_holds_fd_kind "$_opid" "$_ok")
175 case "$_ofdk" in ''|*[!0-9]*) _ofdk=-1 ;; esac
176 [ "$_ofdk" -eq 0 ] || {
177 echo "e2e FAIL: oracle: pid_holds_fd_kind says the shell $_opid holds"
178 echo " '$(pid_holds_fd_kind "$_opid" "$_ok")' fds of kind $_ok, and it"
179 echo " opened nothing but /dev/null"; exit 1; }
180 done
130 # `! cmd || {}` rather than `cmd && {}`, for the reason spelled out 181 # `! cmd || {}` rather than `cmd && {}`, for the reason spelled out
131 # below: an AND-list whose left side fails is a failing command under 182 # below: an AND-list whose left side fails is a failing command under
132 # `set -e`, so the passing case would end the run. 183 # `set -e`, so the passing case would end the run.
@@ -136,9 +187,29 @@ s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(
136 ! pid_holds_unix_sock "$_opid" "$_osock" || { 187 ! pid_holds_unix_sock "$_opid" "$_osock" || {
137 echo "e2e FAIL: oracle: pid_holds_unix_sock says the shell $_opid holds" 188 echo "e2e FAIL: oracle: pid_holds_unix_sock says the shell $_opid holds"
138 echo " $_osock, which only $_olpid ever opened"; exit 1; } 189 echo " $_osock, which only $_olpid ever opened"; exit 1; }
190 # The port the listener really bound, asked in the ONE spelling both
191 # arms take: /proc/net/udp's local address, which is the host-order u32
192 # in hex — 0100007F for 127.0.0.1 on a little-endian box, the same
193 # constant e2e_01_boot and e2e_04_handoff already print. The Darwin arm
194 # converts it to lsof's dotted form, and until this line nothing in the
195 # tree asked that conversion for an address that exists.
196 _oudp=$(printf '0100007F:%04X' "$_oport")
197 udp_local_bound "$_oudp" || {
198 echo "e2e FAIL: oracle: udp_local_bound cannot see the socket the listener"
199 echo " bound at 127.0.0.1:$_oport ($_oudp)"
200 udp_table | head -20; exit 1; }
139 # `! cmd || {}` rather than `cmd && {}`, for wait_sock's reason: an 201 # `! cmd || {}` rather than `cmd && {}`, for wait_sock's reason: an
140 # AND-list whose left side fails is a failing command under `set -e`, 202 # AND-list whose left side fails is a failing command under `set -e`,
141 # so the passing case would end the run. 203 # so the passing case would end the run.
204 #
205 # The SAME port at a different local address. This is the one that pins
206 # the local-address rule both arms are built around: a helper matching
207 # the port anywhere on its line — the remote half of a connected
208 # socket's row included — answers yes here.
209 ! udp_local_bound "$(printf '0200007F:%04X' "$_oport")" || {
210 echo "e2e FAIL: oracle: udp_local_bound reads the socket bound at"
211 echo " 127.0.0.1:$_oport as bound at 127.0.0.2 as well, so it is"
212 echo " matching the port and not the local address"; exit 1; }
142 ! udp_local_bound 'ZZZZZZZZ:ZZZZ' || { 213 ! udp_local_bound 'ZZZZZZZZ:ZZZZ' || {
143 echo "e2e FAIL: oracle: udp_local_bound found a socket at an address no" 214 echo "e2e FAIL: oracle: udp_local_bound found a socket at an address no"
144 echo " kernel can spell"; exit 1; } 215 echo " kernel can spell"; exit 1; }
@@ -162,6 +233,46 @@ s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(
162 echo "e2e FAIL: oracle: sha256_of hashes 'hello' to" 233 echo "e2e FAIL: oracle: sha256_of hashes 'hello' to"
163 echo " '$(sha256_of "$OUT.oracle.probe")'"; exit 1; } 234 echo " '$(sha256_of "$OUT.oracle.probe")'"; exit 1; }
164 235
236 # The clock, asked of the CALENDAR and then of itself. Thirteen digits is
237 # what epoch milliseconds have been since 2001 and will be until 2286, so
238 # it separates milliseconds from the seconds and the nanoseconds either
239 # side of them — the one mistake a `date +%s%N` bracket that lost or kept
240 # a /1000000 would make. Then a real second has to pass for the delta to
241 # be a second: a clock stuck at a constant would satisfy the digit count
242 # and nothing else.
243 #
244 # A band and not a number, and a generous one, because the sleep and two
245 # process starts are inside it and this runs on loaded boxes and under
246 # test/coverage.sh's tracing.
247 # Captured with its stderr and with the failure swallowed, then graded
248 # out of the variable: a now_ms whose clock is not installed exits 127,
249 # and an assignment that fails ends the run under `set -e` with the
250 # shell's own "not found" and no line saying which helper it was.
251 _onow=$(now_ms 2>&1) || _onow="$_onow (exit $?)"
252 _onowsaid="$_onow"
253 case "$_onow" in ''|*[!0-9]*) _onow='' ;; esac
254 [ "${#_onow}" -eq 13 ] || {
255 echo "e2e FAIL: oracle: now_ms answers '$_onowsaid', which is not the 13"
256 echo " digits epoch MILLISECONDS have — seconds or nanoseconds"
257 echo " would make every timing bracket read it wrong"; exit 1; }
258 sleep 1
259 _oelapsed=$(( $(now_ms) - _onow ))
260 [ "$_oelapsed" -ge 900 ] && [ "$_oelapsed" -le $(( 3000 * TIME_SCALE )) ] || {
261 echo "e2e FAIL: oracle: now_ms moved ${_oelapsed}ms across a 1s sleep"; exit 1; }
262
263 # And the TEXT of that clock, run in a shell this one did not configure,
264 # which is exactly how the throughput legs and the agent-mute leg use it.
265 # On Darwin it names gdate, so a Mac with no coreutils fails HERE rather
266 # than inside a $(( )) partway through a group file, which is where an
267 # unpinned snippet would have failed.
268 _osnip=$(sh -c "$(now_ms_snippet); now_ms" 2>&1 || true)
269 _osnipsaid="$_osnip"
270 case "$_osnip" in ''|*[!0-9]*) _osnip='' ;; esac
271 [ "${#_osnip}" -eq 13 ] || {
272 echo "e2e FAIL: oracle: now_ms_snippet run in a fresh shell answers"
273 echo " '$_osnipsaid'"
274 echo " rather than 13 digits of epoch milliseconds"; exit 1; }
275
165 kill "$_olpid" "$_opid" "$_okid" 2>/dev/null || true 276 kill "$_olpid" "$_opid" "$_okid" 2>/dev/null || true
166 wait "$_olpid" 2>/dev/null || true 277 wait "$_olpid" 2>/dev/null || true
167 wait "$_opid" 2>/dev/null || true 278 wait "$_opid" 2>/dev/null || true
@@ -171,7 +282,7 @@ s = socket.socket(socket.AF_UNIX); s.bind(sys.argv[1]); s.listen(1); time.sleep(
171 done 282 done
172 ! pid_alive "$_opid" || { 283 ! pid_alive "$_opid" || {
173 echo "e2e FAIL: oracle: pid_alive says a killed shell still lives"; exit 1; } 284 echo "e2e FAIL: oracle: pid_alive says a killed shell still lives"; exit 1; }
174 rm -f "$_osock" "$OUT.oracle.probe" 285 rm -f "$_osock" "$_oport_f" "$OUT.oracle.probe"
175 ok "oracle: the OS answers the helpers by name" 286 ok "oracle: the OS answers the helpers by name"
176 # One fixed word, on top of ok()'s sentence: a port is graded over ssh 287 # One fixed word, on top of ok()'s sentence: a port is graded over ssh
177 # and the run that grades it has one line to look for. 288 # and the run that grades it has one line to look for.
test/os_oracle.sh
Old New
@@ -49,6 +49,11 @@ Darwin)
49 /^t/ { ty = substr($0, 2) } 49 /^t/ { ty = substr($0, 2) }
50 /^n/ { if (fd ~ /^[0-9]+$/) print ty " " substr($0, 2) }' 50 /^n/ { if (fd ~ /^[0-9]+$/) print ty " " substr($0, 2) }'
51 } 51 }
52 # No `[ -d /proc/$1 ]` behind it, because there is no /proc — which
53 # leaves one behavioural difference between the arms: kill(2) on a pid
54 # this uid does not own answers EPERM, so a live process belonging to
55 # somebody else reads as dead here where Linux's directory check would
56 # still see it. Every pid this suite asks about is one it started.
52 pid_alive() { kill -0 "$1" 2>/dev/null; } 57 pid_alive() { kill -0 "$1" 2>/dev/null; }
53 # The FIRST txt vnode is the executable; the ones after it are dyld and 58 # The FIRST txt vnode is the executable; the ones after it are dyld and
54 # the dylibs it mapped. Two things this answers that a caller has to 59 # the dylibs it mapped. Two things this answers that a caller has to
@@ -59,7 +64,15 @@ Darwin)
59 # front of the bash image. No binary mux runs is such a stub, so the 64 # front of the bash image. No binary mux runs is such a stub, so the
60 # pins that read this get back the path they asked about — but the 65 # pins that read this get back the path they asked about — but the
61 # oracle's own self-test asks about a `sleep` for that reason. 66 # oracle's own self-test asks about a `sleep` for that reason.
62 pid_exe() { lsof -p "$1" -a -d txt -Fn 2>/dev/null | sed -n 's/^n//p' | head -1; } 67 # Through a variable so that no answer is a FAILURE and not an empty
68 # success: the pipeline's exit status is head's, which is 0 for a pid
69 # that is gone, and this file's header promises a 1 when the OS will not
70 # say. Linux's readlink already answers that way.
71 pid_exe() {
72 _pe=$(lsof -p "$1" -a -d txt -Fn 2>/dev/null | sed -n 's/^n//p' | head -1)
73 [ -n "$_pe" ] || return 1
74 printf '%s\n' "$_pe"
75 }
63 # ps prints the full path for a binary outside the system directories 76 # ps prints the full path for a binary outside the system directories
64 # and the bare name for one inside them, so the last path component is 77 # and the bare name for one inside them, so the last path component is
65 # taken either way. Through a variable rather than basename(1): an 78 # taken either way. Through a variable rather than basename(1): an
@@ -78,7 +91,7 @@ Darwin)
78 # Every socket family, which is what the Linux arm's `socket:` 91 # Every socket family, which is what the Linux arm's `socket:`
79 # prefix counts. PIPE is not among them, because `pipe:[N]` is 92 # prefix counts. PIPE is not among them, because `pipe:[N]` is
80 # not on that side either. 93 # not on that side either.
81 socket) _fd_rows "$1" | grep -c '^\(unix\|IPv4\|IPv6\|sock\) ' || true ;; 94 socket) _fd_rows "$1" | grep -cE '^(unix|IPv4|IPv6|sock) ' || true ;;
82 # The upgrade manifest is an unlinked mkstemp file here, not a 95 # The upgrade manifest is an unlinked mkstemp file here, not a
83 # memfd (server_os_macos.anonFd). lsof still prints the path it 96 # memfd (server_os_macos.anonFd). lsof still prints the path it
84 # had before the unlink, canonicalized and with no marker of 97 # had before the unlink, canonicalized and with no marker of
@@ -99,6 +112,10 @@ Darwin)
99 *) echo "pid_holds_fd_kind: no such fd kind '$2'" >&2; return 1 ;; 112 *) echo "pid_holds_fd_kind: no such fd kind '$2'" >&2; return 1 ;;
100 esac 113 esac
101 } 114 }
115 # What `server_os_macos.anonFd` makes, in one word, so a test that has to
116 # OPEN a carrier of the local shape can ask for it without asking which
117 # OS it is on. `unlinked` is an unlinked mkstemp file.
118 carrier_kind() { echo unlinked; }
102 # By PATH, where the Linux arm matches the listening inode: lsof answers 119 # By PATH, where the Linux arm matches the listening inode: lsof answers
103 # a unix socket's bound path and no inode, so there is no inode to 120 # a unix socket's bound path and no inode, so there is no inode to
104 # match. It is still this pid's own fd table being read and not the 121 # match. It is still this pid's own fd table being read and not the
@@ -201,6 +218,8 @@ Darwin)
201 esac 218 esac
202 pid_fd_targets "$1" | grep -c -F "$_fdk" || true 219 pid_fd_targets "$1" | grep -c -F "$_fdk" || true
203 } 220 }
221 # What `server_os_linux.anonFd` makes; see the Darwin arm's carrier_kind.
222 carrier_kind() { echo memfd; }
204 # By INODE, never by name: a process running the right program while 223 # By INODE, never by name: a process running the right program while
205 # binding something else entirely would satisfy a name check. The 224 # binding something else entirely would satisfy a name check. The
206 # kernel's listening inode for PATH, found among that pid's open fds, 225 # kernel's listening inode for PATH, found among that pid's open fds,