a73x

cf029e68

test(xversion): containerised cross-version gate, falsifiable on purpose

a73x   2026-08-15 12:22

Commit message
test(xversion): containerised cross-version gate, falsifiable on purpose

This tree's client against a previous version's daemon and the reverse,
each daemon in a container of its own. Eight legs: both directions on the
default session over both transports, both halves of the `--session`
limitation, an old muxa's empty status_req, the stats_req discriminator,
and a teardown observed rather than claimed.

It exists because the cross-version check has always been hand-rolled,
and the one run of it disagreed with the documentation — `--session`
against a pre-M18 daemon was written up as one failure and is two, split
by transport. A hand-rolled check finds that once, by luck, days late.

Both sides build static musl so ONE artifact per version runs in the
container and on the host, and `-p .xversion/` keeps it out of zig-out: a
bare `-Dtarget=` build overwrites the default prefix with binaries for
the wrong libc, and the next `make e2e` would silently run those.
--network host rather than a published port, because a rootless bridge
NATs the source address and QUIC validates the path it answers on.

The falsification path is the design, not a nicety: point XVER_OLD_BIN at
the new binaries and legs must fail. Running it caught a false pass in
this file's own first draft — the `--session`-over-QUIC leg keyed on the
client's exit code, and a daemon that HONOURS the name also leaves the
client attached until `timeout` kills it, so 124 meant both "no answer"
and "it worked". It asks whether a session called `zz` appeared instead,
which only a daemon that understood the name can produce, and stats is
the right place to ask: daemon-global, no name tail, answered by every
version. Each leg keeps its own capture, so one leg can never assert
against another's evidence.

Containers share the host kernel, so nothing here validates pty, poll or
fd semantics against a DIFFERENT kernel — most of what muxd is. That
needs VMs and is not claimed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
.gitignore
Old New
@@ -5,3 +5,6 @@ zig-out/
5 sslkeylog.log 5 sslkeylog.log
6 dist/ 6 dist/
7 .worktrees/ 7 .worktrees/
8
9 # Cross-version gate artifacts: two versions built static, per version.
10 .xversion/
test/xversion.sh
Old New
@@ -0,0 +1,324 @@
1 #!/bin/sh
2 # Cross-version compatibility gate, containerised.
3 #
4 # Runs THIS tree's client against a PREVIOUS version's daemon and the
5 # reverse, with each daemon in a container of its own. The point is to make
6 # a check that has always been hand-rolled — build two versions, start one,
7 # dial it with the other, remember what the answer is supposed to be — into
8 # something that runs the same way every time and can fail out loud.
9 #
10 # It exists because the M18 close-out ran exactly this by hand and the
11 # answer disagreed with the documentation: `--session` against a pre-M18
12 # daemon was written up as one failure and is two, split by transport (see
13 # decisions.md, M18). A hand-rolled check finds that once, by luck, days
14 # late. This one finds it on the run after someone writes the sentence.
15 #
16 # CONTAINERS, NOT VMs, AND THE LIMIT THAT COMES WITH IT: a container shares
17 # the host kernel, so nothing here validates pty, poll or fd semantics
18 # against a DIFFERENT kernel — and those are most of what muxd is. What it
19 # does buy is a stock userland (the static musl binary has to actually run
20 # somewhere that is not this box), a clean two-versions-at-once story, and
21 # a rig that does not depend on a remote machine being powered on. Kernel
22 # diversity needs VMs and is not claimed here.
23 set -u
24
25 ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
26 OLD_BIN="${XVER_OLD_BIN:-$ROOT/.xversion/old/bin}"
27 NEW_BIN="${XVER_NEW_BIN:-$ROOT/.xversion/new/bin}"
28 IMAGE="${XVER_IMAGE:-docker.io/library/alpine:latest}"
29 ENGINE="${XVER_ENGINE:-podman}"
30 # What the two sides are called in the output. Informational only — the
31 # binaries are whatever the directories hold — but a report that says
32 # "old" and "new" without saying WHICH is a report nobody can act on.
33 OLD_LABEL="${XVER_OLD_LABEL:-$OLD_BIN}"
34 NEW_LABEL="${XVER_NEW_LABEL:-$NEW_BIN}"
35
36 PASSES=0
37 FAILS=0
38 SKIPS=0
39 pass() { PASSES=$((PASSES + 1)); echo "xver PASS: $1"; }
40 fail() { FAILS=$((FAILS + 1)); echo "xver FAIL: $1"; }
41 skip() { SKIPS=$((SKIPS + 1)); echo "xver SKIP: $1"; }
42
43 # ---- preflight ---------------------------------------------------------
44 # A missing input is a refusal, never a skip: a run that quietly tested
45 # nothing and exited 0 is the failure mode this whole file exists against.
46 command -v "$ENGINE" >/dev/null 2>&1 ||
47 { echo "xver FAIL: no $ENGINE — this gate cannot run without a container engine"; exit 1; }
48 for _d in "$OLD_BIN" "$NEW_BIN"; do
49 for _b in muxd mux; do
50 [ -x "$_d/$_b" ] || {
51 echo "xver FAIL: no $_b in $_d"
52 echo " build both sides static (they run in a container AND on this host):"
53 echo " zig build -Dtarget=x86_64-linux-musl -p $ROOT/.xversion/new"
54 echo " git -C <a checkout of the older ref> ... -p $ROOT/.xversion/old"
55 exit 1
56 }
57 done
58 done
59
60 # Short paths on purpose: these directories hold unix sockets, and a socket
61 # path is capped near 108 bytes by the kernel, not by anything this script
62 # can see fail gracefully.
63 TMP="${TMPDIR:-/tmp}/mux-xver-$$"
64 RUN_OLD="$TMP/o"
65 RUN_NEW="$TMP/n"
66 mkdir -p "$RUN_OLD" "$RUN_NEW" || exit 1
67 CN_OLD="mux-xver-old-$$"
68 CN_NEW="mux-xver-new-$$"
69 PORT_OLD=$(( 40000 + ($$ % 5000) ))
70 PORT_NEW=$(( 45000 + ($$ % 5000) ))
71 KEY="$TMP/key"
72 head -c 32 /dev/urandom > "$KEY"
73 chmod 600 "$KEY"
74 cp "$KEY" "$RUN_OLD/key"
75 cp "$KEY" "$RUN_NEW/key"
76
77 cleanup() {
78 _rc=$?
79 # By name, and these names carry $$ — a pattern kill would be reaching
80 # for someone else's container.
81 "$ENGINE" rm -f "$CN_OLD" >/dev/null 2>&1
82 "$ENGINE" rm -f "$CN_NEW" >/dev/null 2>&1
83 rm -rf "$TMP"
84 exit "$_rc"
85 }
86 trap cleanup EXIT INT TERM
87
88 # start_daemon NAME BINDIR RUNDIR PORT — a daemon in a container, serving
89 # both a unix socket (through the bind mount, so the host can dial it) and
90 # QUIC (through a published UDP port).
91 #
92 # --network host rather than a published port: a rootless bridge NATs the
93 # client's source address, and QUIC validates the path it is answering on.
94 # Testing the transport through an address translation this product never
95 # runs behind would be testing pasta, not muxd. The container still has its
96 # own filesystem and process table, which is what this gate is actually
97 # asking about.
98 start_daemon() {
99 "$ENGINE" run -d --name "$1" --network host \
100 -v "$2:/mux:ro" -v "$3:/run/mux" \
101 "$IMAGE" \
102 /mux/muxd run --sock /run/mux/muxd.sock --quic "127.0.0.1:$4" \
103 --key /run/mux/key --quic-idle-ms 15000 --shell /bin/sh \
104 >/dev/null 2>&1
105 }
106
107 # Poll for the socket the container is about to create in the shared dir.
108 wait_sock() {
109 _i=0
110 while [ ! -S "$1" ] && [ "$_i" -lt 100 ]; do sleep 0.1; _i=$((_i + 1)); done
111 [ -S "$1" ]
112 }
113
114 # A marker its own shell has to EXPAND: the typed line reads
115 # `printf "x-%s\n" pin` and only the OUTPUT reads `x-pin`, so a grep that
116 # hits is the shell's work rather than an echo of our keystrokes.
117 drive() { # drive BIN MARKER TRANSPORTARGS...
118 _bin="$1"; _marker="$2"; shift 2
119 { printf 'printf "%s-%%s\\n" pin\n' "$_marker"; sleep 2; printf '\034'; } | \
120 timeout 40 "$_bin" "$@" > "$TMP/out" 2> "$TMP/err"
121 }
122
123 echo "xver: old=$OLD_LABEL"
124 echo "xver: new=$NEW_LABEL"
125 echo "xver: image=$IMAGE engine=$ENGINE ports=$PORT_OLD/$PORT_NEW"
126
127 start_daemon "$CN_OLD" "$OLD_BIN" "$RUN_OLD" "$PORT_OLD" ||
128 { echo "xver FAIL: could not start the old daemon's container"; exit 1; }
129 start_daemon "$CN_NEW" "$NEW_BIN" "$RUN_NEW" "$PORT_NEW" ||
130 { echo "xver FAIL: could not start the new daemon's container"; exit 1; }
131
132 SOCK_OLD="$RUN_OLD/muxd.sock"
133 SOCK_NEW="$RUN_NEW/muxd.sock"
134 wait_sock "$SOCK_OLD" || {
135 echo "xver FAIL: old daemon never bound in its container:"
136 "$ENGINE" logs "$CN_OLD" 2>&1 | head -20; exit 1; }
137 wait_sock "$SOCK_NEW" || {
138 echo "xver FAIL: new daemon never bound in its container:"
139 "$ENGINE" logs "$CN_NEW" 2>&1 | head -20; exit 1; }
140
141 # ---- leg 1: new client -> old daemon, default session, unix socket -----
142 # The compat claim in one line: no --session means the bare 20-byte attach
143 # that shipped, so this must be indistinguishable from an old client.
144 drive "$NEW_BIN/mux" xv1 --sock "$SOCK_OLD"
145 RC=$?
146 if [ "$RC" -ne 0 ]; then
147 fail "new client -> old daemon (socket) exited $RC [$(tr -d '\n' < "$TMP/err")]"
148 elif "$OLD_BIN/muxd" dump --sock "$SOCK_OLD" | grep -q 'xv1-pin'; then
149 pass "new client drives an old daemon's default session over a socket"
150 else
151 fail "old daemon's grid is missing the new client's output"
152 fi
153
154 # ---- leg 2: new client -> old daemon, default session, QUIC ------------
155 # The control for leg 3. Without it, "leg 3 failed" is consistent with
156 # "QUIC to an old daemon never works", which is a different bug.
157 drive "$NEW_BIN/mux" xv2 "quic://127.0.0.1:$PORT_OLD" --key "$KEY" --quic-idle-ms 15000
158 RC=$?
159 if [ "$RC" -ne 0 ]; then
160 fail "new client -> old daemon (quic) exited $RC [$(tr -d '\n' < "$TMP/err")]"
161 elif "$OLD_BIN/muxd" dump --sock "$SOCK_OLD" | grep -q 'xv2-pin'; then
162 pass "new client drives an old daemon's default session over QUIC"
163 else
164 fail "old daemon's grid is missing the new client's QUIC output"
165 fi
166
167 # ---- leg 3: the documented limitation, both halves of it ---------------
168 # `--session` against a daemon that predates it. The two transports fail
169 # DIFFERENTLY and the difference is the documentation: a socket connection
170 # is an observer until it attaches and the old serviceObserver closes on a
171 # payload it cannot parse, while a QUIC connection took a client slot at
172 # quicOnOpen and reaches handleFrame's silent `decodeAttach catch return`.
173 # Pinning both means the sentence in decisions.md cannot drift from the
174 # behaviour without this failing.
175 # The exit code alone CANNOT decide this, and an earlier draft of this
176 # file got it wrong: against a daemon that honours the name the client
177 # attaches and stays attached, so it is killed by the same `timeout` that
178 # a real hang produces — 124 either way. The falsification run (point
179 # XVER_OLD_BIN at the new binaries) is what surfaced it, which is the
180 # argument for having one. The discriminator has to be something only a
181 # daemon that UNDERSTOOD the name can produce, and that is the session
182 # itself: `stats` is daemon-global, has no name tail, and every version
183 # answers it, so "did a session called zz appear?" is askable of both.
184 # Returns 0 when the name was NOT honoured and nothing has been faulted
185 # yet — the caller's own verdict is still open. Returns 1 when this
186 # function already called `fail`, so the caller must stay quiet rather
187 # than reporting a second time on the same leg.
188 #
189 # The verdict travels as the EXIT STATUS, and the stats capture is keyed
190 # by LABEL. Both matter, and for the same reason: the earlier shape wrote
191 # one fixed path and left the caller to re-grep it, so the second
192 # transport's call overwrote the first's evidence. That was correct only
193 # because of the order the two legs happen to run in — one reorder and a
194 # leg asserts against the other leg's stats.
195 assert_name_not_honoured() { # LABEL MUXD SOCK RC ERRFILE
196 _lbl="$1"; _muxd="$2"; _sock="$3"; _rc="$4"; _err="$5"
197 _stats="$TMP/stats-$_lbl"
198 "$_muxd" stats --sock "$_sock" > "$_stats" 2>&1
199 if grep -q 'session zz' "$_stats"; then
200 fail "$_lbl: the daemon created session zz — this is not a pre-M18 daemon [$(cat "$_stats")]"
201 return 1
202 fi
203 case "$_rc" in
204 124) echo "$_lbl: no answer (client killed at its deadline)" ;;
205 0) fail "$_lbl: the client exited 0 without a session being created"; return 1 ;;
206 *) echo "$_lbl: refused (exit $_rc) [$(tr -d '\n' < "$_err")]" ;;
207 esac
208 return 0
209 }
210
211 { printf 'printf "no-%%s\\n" pin\n'; sleep 5; } | \
212 timeout 8 "$NEW_BIN/mux" --sock "$SOCK_OLD" --session zz > "$TMP/o3" 2> "$TMP/e3"
213 RC=$?
214 if assert_name_not_honoured "socket" "$OLD_BIN/muxd" "$SOCK_OLD" "$RC" "$TMP/e3"; then
215 # Reaching here means no session was created and the client did not
216 # exit 0, so the only question left is refused-vs-hung.
217 if [ "$RC" -eq 124 ]; then
218 fail "socket: --session to an old daemon HUNG; decisions.md says it is refused"
219 else
220 pass "socket: --session to an old daemon is refused, not hung (exit $RC)"
221 fi
222 fi
223
224 { printf 'printf "no-%%s\\n" pin\n'; sleep 12; } | \
225 timeout 14 "$NEW_BIN/mux" "quic://127.0.0.1:$PORT_OLD" --key "$KEY" \
226 --quic-idle-ms 15000 --session zz > "$TMP/o3q" 2> "$TMP/e3q"
227 RC=$?
228 if assert_name_not_honoured "quic" "$OLD_BIN/muxd" "$SOCK_OLD" "$RC" "$TMP/e3q"; then
229 if [ "$RC" -eq 124 ]; then
230 pass "quic: --session to an old daemon hangs, as documented (no session, no answer)"
231 else
232 # Better than documented: somebody fixed it, or the client grew the
233 # deadline banked in roadmap.md — either way the doc is now wrong.
234 fail "quic: --session to an old daemon exited $RC rather than hanging — decisions.md needs rewriting [$(tr -d '\n' < "$TMP/e3q")]"
235 fi
236 fi
237
238 # ---- leg 4: old client -> new daemon, and it must CREATE NOTHING -------
239 drive "$OLD_BIN/mux" xv4 --sock "$SOCK_NEW"
240 RC=$?
241 if [ "$RC" -ne 0 ]; then
242 fail "old client -> new daemon exited $RC [$(tr -d '\n' < "$TMP/err")]"
243 else
244 "$NEW_BIN/muxd" stats --sock "$SOCK_NEW" > "$TMP/s4" 2>&1
245 if ! "$NEW_BIN/muxd" dump --sock "$SOCK_NEW" | grep -q 'xv4-pin'; then
246 fail "new daemon's default grid is missing the old client's output"
247 elif ! grep -q 'sessions=1' "$TMP/s4"; then
248 fail "an old client's bare attach did not land in the default session [$(cat "$TMP/s4")]"
249 elif ! grep -q 'session 0 clients=' "$TMP/s4"; then
250 fail "the new daemon's default session is not named 0 [$(cat "$TMP/s4")]"
251 else
252 pass "old client drives the new daemon's default session, creating nothing"
253 fi
254 fi
255
256 # ---- leg 5: an old muxa's empty status_req still answers ---------------
257 # status_req's payload was EMPTY and now carries a name tail; an old muxa
258 # sends the empty one and it has to keep meaning "the default session".
259 if [ ! -x "$OLD_BIN/muxa" ]; then
260 skip "old muxa: the old ref predates the agent surface"
261 else
262 timeout 20 "$OLD_BIN/muxa" status --sock "$SOCK_NEW" --timeout 5000 > "$TMP/m5" 2>&1
263 RC=$?
264 if [ "$RC" -ne 0 ]; then
265 fail "old muxa status -> new daemon exited $RC [$(tr -d '\n' < "$TMP/m5")]"
266 elif grep -q '"cols"' "$TMP/m5"; then
267 pass "an old muxa's empty status_req still answers for the default session"
268 else
269 fail "old muxa got no status reply [$(tr -d '\n' < "$TMP/m5")]"
270 fi
271 fi
272
273 # ---- leg 6: the capability discriminator ------------------------------
274 # Not a compat check — a check on the thing that would MAKE the compat gap
275 # reportable. A client cannot ask "do you know about sessions?": MsgType is
276 # non-exhaustive and both of the old daemon's switch arms end in
277 # `else => {}`, so an unknown probe frame is answered with the same silence
278 # that is the bug. What an old daemon DOES answer is stats_req, and the two
279 # versions' replies differ — `sessions=` is present in one and absent in
280 # the other. That difference is the only version signal on the wire, so it
281 # is pinned here: a stats reformat that drops it would quietly retire the
282 # mitigation banked in roadmap.md before it was ever built.
283 "$OLD_BIN/muxd" stats --sock "$SOCK_OLD" > "$TMP/s6o" 2>&1
284 "$NEW_BIN/muxd" stats --sock "$SOCK_NEW" > "$TMP/s6n" 2>&1
285 if grep -q 'sessions=' "$TMP/s6o"; then
286 fail "the old daemon's stats already say 'sessions=' — it is not a version signal [$(cat "$TMP/s6o")]"
287 elif ! grep -q 'sessions=' "$TMP/s6n"; then
288 fail "the new daemon's stats do not say 'sessions=' — the banked capability probe has no signal [$(cat "$TMP/s6n")]"
289 else
290 pass "stats_req discriminates the versions (a probe the old daemon already answers)"
291 fi
292
293 # ---- teardown, by the sanctioned verb, then OBSERVED ------------------
294 "$NEW_BIN/muxd" stop --sock "$SOCK_NEW" >/dev/null 2>&1
295 "$OLD_BIN/muxd" stop --sock "$SOCK_OLD" >/dev/null 2>&1
296 _i=0
297 while [ "$_i" -lt 40 ]; do
298 _live=$("$ENGINE" ps --filter "name=$CN_OLD" --filter "name=$CN_NEW" -q 2>/dev/null | wc -l)
299 [ "$_live" -eq 0 ] && break
300 sleep 0.05; _i=$((_i + 1))
301 done
302 # Reported as observed, never as the stop command's claim. A container
303 # still up here means `muxd stop` did not end the process it was aimed at,
304 # which is a finding and not a cleanup detail.
305 _live=$("$ENGINE" ps --filter "name=$CN_OLD" --filter "name=$CN_NEW" -q 2>/dev/null | wc -l)
306 if [ "$_live" -eq 0 ]; then
307 pass "both daemons ended on muxd stop (observed: no container still running)"
308 else
309 fail "$_live container(s) still running after muxd stop"
310 fi
311
312 # The count, pinned against a literal for e2e.sh's reason: a leg that
313 # silently stops running is the failure mode no assertion inside it can
314 # catch. Skips count — a skipped leg RAN and decided not to assert.
315 TOTAL=$((PASSES + FAILS + SKIPS))
316 if [ "$TOTAL" -ne 8 ]; then
317 echo "xver FAIL: $TOTAL legs reported, want 8 — one did not run"
318 FAILS=$((FAILS + 1))
319 fi
320
321 echo
322 echo "xver: $PASSES passed, $FAILS failed, $SKIPS skipped"
323 [ "$FAILS" -eq 0 ] || exit 1
324 echo "xver OK"