a73x

bce83e89

test: the make vm gate runs real user journeys against the mux-e2e VM

a73x   2026-09-01 18:40

Commit message
test: the make vm gate runs real user journeys against the mux-e2e VM

Four journeys over real ssh to a real Ubuntu box, beside make ci and
never inside it: install like a user, the entry dial with detach and
reattach, a stop that stays stopped under wall polling, and a remote
upgrade asserted same-pid across the exec with marks surviving the
manifest. Every claim about the box is a separate ssh reading /proc or
the grid dump — the e2e ssh shims stay as the fast local pins; this is
where their assumptions get audited against a real sshd.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Makefile
Old New
@@ -2,7 +2,7 @@
2 # default zig is 0.17-dev. Override with ZIG=... if yours lives elsewhere. 2 # default zig is 0.17-dev. Override with ZIG=... if yours lives elsewhere.
3 ZIG ?= $(CURDIR)/deps/zig/zig 3 ZIG ?= $(CURDIR)/deps/zig/zig
4 4
5 .PHONY: build check ci test e2e soak bench agent throughput coverage deps clean clean-deps xversion xversion-build install release 5 .PHONY: build check ci test e2e soak bench agent throughput vm coverage deps clean clean-deps xversion xversion-build install release
6 6
7 # The QUIC stack (deps/quic) is built on demand by build.zig, so no target 7 # The QUIC stack (deps/quic) is built on demand by build.zig, so no target
8 # here needs to depend on this one. It exists to make the one-time cost 8 # here needs to depend on this one. It exists to make the one-time cost
@@ -116,6 +116,14 @@ coverage:
116 $(COVDIR)/bin/rawmode $(COVDIR)/bin/delaypipe $(COVDIR)/bin/render \ 116 $(COVDIR)/bin/rawmode $(COVDIR)/bin/delaypipe $(COVDIR)/bin/render \
117 $(COVDIR)/bin/ptyclient $(COVDIR)/bin/wsclient 117 $(COVDIR)/bin/ptyclient $(COVDIR)/bin/wsclient
118 118
119 # Real user journeys against the mux-e2e VM (test/vm.sh): a delivery-ritual
120 # gate beside ci, never inside it — ci stays hermetic and offline. Static
121 # musl because the same binary is the client here and the install there.
122 VMDIR ?= dist/vmgate
123 vm:
124 $(ZIG) build -Dtarget=x86_64-linux-musl -Doptimize=ReleaseSafe -p $(VMDIR)
125 ./test/vm.sh $(VMDIR)/bin/mux $(VMDIR)/bin/ptyclient
126
119 THRUDIR ?= dist/throughput 127 THRUDIR ?= dist/throughput
120 throughput: 128 throughput:
121 $(ZIG) build -Doptimize=ReleaseSafe -p $(THRUDIR) 129 $(ZIG) build -Doptimize=ReleaseSafe -p $(THRUDIR)
test/vm.sh
Old New
@@ -0,0 +1,304 @@
1 #!/bin/sh
2 # test/vm.sh — the `make vm` gate: real user journeys against a real box.
3 #
4 # The e2e suite's remotes are ssh SHIMS — scripts that log argv and run mux
5 # under a second HOME on this machine. Twice (2026-08-29, both askpass) a
6 # shim leg went green about a mechanism that never ran. This gate is where
7 # those assumptions get audited: a real sshd, a real network hop, a real
8 # Ubuntu, and mux spelled exactly as a user spells it. It runs beside
9 # `make ci`, never inside it — ci stays hermetic and offline.
10 #
11 # The fixture is ONE long-lived VM, like a real user's server (the spec:
12 # docs/superpowers/specs/2026-09-01-e2e-dedupe-and-vm-gate-design.md).
13 # Hermeticity comes from the scorched-earth reset below, not from
14 # re-creating the VM; VM lifecycle is a human action, never harness code.
15 #
16 # Every claim about the VM is asserted by a separate ssh reading /proc, the
17 # daemon's grid dump, or pgrep — never by trusting the daemon's own report
18 # over the connection under test ("ask the OS about the OS").
19 set -eu
20 MUX="$1" # static musl mux: the client here AND the binary installed there
21 PTYCLIENT="$2" # the real-pty client fixture, driving mux the way hands do
22
23 VM="${MUX_VM:-ubuntu@10.78.5.3}"
24
25 # BatchMode so a broken auth fails in seconds instead of prompting a harness
26 # nobody sits in front of; accept-new so the FIRST run records the box's
27 # host key in the user's own known_hosts — mux's entry dial spawns plain
28 # `ssh $VM`, which reads that file, so the trust decision has to land where
29 # plain ssh will find it.
30 SSH_OPTS="-o BatchMode=yes -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new"
31
32 vssh() {
33 # shellcheck disable=SC2086 # SSH_OPTS is words on purpose
34 timeout 30 ssh $SSH_OPTS "$VM" "$@"
35 }
36
37 OK_COUNT=0
38 ok() {
39 OK_COUNT=$((OK_COUNT + 1))
40 echo "vm OK ($OK_COUNT): $1"
41 }
42
43 # ---- preflight: refuse loudly, never skip -----------------------------
44 # A gate that can quietly not run is the one failure a green tree cannot
45 # show — the same stance e2e.sh takes on nvim and curl.
46 set +e
47 vssh true 2>/tmp/vm_preflight.err
48 RC=$?
49 set -e
50 [ "$RC" -eq 0 ] || {
51 echo "vm FAIL: cannot ssh to $VM (rc $RC):"
52 sed 's/^/ /' /tmp/vm_preflight.err
53 echo " The fixture is the eitri VM 'mux-e2e' on host charizard."
54 echo " If it is gone: recreate it PINNED to charizard (default"
55 echo " placement lands on the arm64 Mac) and re-read its underlay"
56 echo " IP. If auth broke: re-sign against ~/.config/mux-vm/vm_ca."
57 exit 1; }
58
59 ARCH=$(vssh 'uname -m')
60 [ "$ARCH" = "x86_64" ] || {
61 echo "vm FAIL: $VM is $ARCH, and this gate pushes an x86_64-musl build."
62 echo " The fleet's default placement is the arm64 Mac; recreate"
63 echo " the VM pinned to host charizard."
64 exit 1; }
65
66 # ---- the local side is a hermetic laptop ------------------------------
67 # Every XDG dir mux reads, isolated (state, runtime, config, cache — the
68 # 2026-08-29 demo leaked a QUIC cache entry through the one rig that forgot
69 # cache). Short root because a unix socket path caps at 107 bytes.
70 SCRATCH=$(mktemp -d /tmp/muxvm.XXXXXX)
71 export XDG_STATE_HOME="$SCRATCH/st" XDG_RUNTIME_DIR="$SCRATCH/rt"
72 export XDG_CONFIG_HOME="$SCRATCH/cf" XDG_CACHE_HOME="$SCRATCH/ca"
73 mkdir -p "$XDG_STATE_HOME" "$XDG_RUNTIME_DIR" "$XDG_CONFIG_HOME" "$XDG_CACHE_HOME"
74 trap 'rm -rf "$SCRATCH"' EXIT
75
76 # ---- scorched earth on the VM -----------------------------------------
77 # The reset IS the hermeticity: kill every mux, remove the binary and all
78 # state, so each run starts from the mux-less box a new user has. The two
79 # standing arrangements it (re)makes:
80 # linger — without it logind may tear down /run/user/1000 (the daemon's
81 # socket dir) between ssh sessions and reap the daemon with it;
82 # a real server user runs `enable-linger` once and so do we.
83 # PATH — mux's dials run `ssh $VM 'mux ...'`, a non-interactive bash
84 # whose PATH lacks ~/.local/bin; the line goes ABOVE bashrc's
85 # interactivity return, where Debian's ssh-aware bash reads it.
86 vssh 'sh -s' <<'RESET'
87 set -eu
88 pkill -x mux 2>/dev/null || true
89 sleep 0.3
90 pkill -9 -x mux 2>/dev/null || true
91 rm -rf ~/.local/bin/mux ~/.local/state/mux ~/.cache/mux ~/.config/mux
92 rm -rf "${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"/mux*
93 sudo -n loginctl enable-linger "$(id -un)"
94 grep -q '^export PATH="$HOME/.local/bin' ~/.bashrc 2>/dev/null || \
95 sed -i '1i export PATH="$HOME/.local/bin:$PATH"' ~/.bashrc
96 mkdir -p ~/.local/bin
97 RESET
98 ok "preflight: $VM answers, x86_64, reset to a mux-less box"
99
100 # ---- journey 1: install like a user -----------------------------------
101 set +e
102 vssh 'command -v mux' >/dev/null 2>&1
103 RC=$?
104 set -e
105 [ "$RC" -ne 0 ] || {
106 echo "vm FAIL: install: the box still answers 'command -v mux' after the reset"
107 exit 1; }
108
109 scp -q $SSH_OPTS "$MUX" "$VM":.local/bin/mux
110 WANT_VER=$("$MUX" --version)
111 GOT_VER=$(vssh 'mux --version')
112 [ "$WANT_VER" = "$GOT_VER" ] || {
113 echo "vm FAIL: install: the box answers '$GOT_VER', this build is '$WANT_VER'"
114 exit 1; }
115 ok "install: scp to ~/.local/bin and the box answers '$GOT_VER'"
116
117 # ---- journey 2: the entry dial ----------------------------------------
118 # `mux $VM` is the whole product path: plain ssh, the ONE `endpoint
119 # --start` an asked dial is allowed, the daemon auto-starting on the box,
120 # a session painting back here. The marker's $$ expands in the REMOTE
121 # shell, so the pid it paints is a pid only the VM's /proc can confirm —
122 # which is the assert no shim can fake.
123 set +e
124 timeout 90 "$PTYCLIENT" --cols 80 --rows 24 \
125 --out "$SCRATCH/j2.cap" --err "$SCRATCH/j2.err" -- "$MUX" "$VM" \
126 > "$SCRATCH/j2.log" 2>&1 <<'EOF'
127 expect \x1b[?1049h 30000
128 settle 700 20000
129 send echo vmj2=$$\n
130 expect vmj2= 20000
131 settle 600 15000
132 send \x1cd
133 waitexit 15000
134 EOF
135 RC=$?
136 set -e
137 [ "$RC" -eq 0 ] || {
138 echo "vm FAIL: entry dial: ptyclient exited $RC (did ssh or the remote start refuse?):"
139 cat "$SCRATCH/j2.log"; cat "$SCRATCH/j2.err" 2>/dev/null; exit 1; }
140
141 RPID=$(vssh 'mux d dump' | sed -n 's/.*vmj2=\([0-9][0-9]*\).*/\1/p' | head -1)
142 [ -n "$RPID" ] || {
143 echo "vm FAIL: entry dial: the marker never reached the REMOTE daemon's grid; it holds:"
144 vssh 'mux d dump' | sed 's/^/ /'
145 exit 1; }
146 vssh "test -d /proc/$RPID" || {
147 echo "vm FAIL: entry dial: the grid names shell pid $RPID but the VM has no such process"
148 exit 1; }
149 ok "entry dial: mux $VM auto-started a daemon and shell $RPID runs on the box"
150
151 DPID=$(vssh 'pgrep -x mux' | head -1)
152 NPROCS=$(vssh 'pgrep -cx mux')
153 [ "$NPROCS" = "1" ] || {
154 echo "vm FAIL: entry dial: after detach the box runs $NPROCS mux processes, want 1 (the daemon):"
155 vssh 'ps -o pid,args -C mux' | sed 's/^/ /'
156 exit 1; }
157 EXE=$(vssh "readlink /proc/$DPID/exe")
158 [ "$EXE" = "/home/ubuntu/.local/bin/mux" ] || {
159 echo "vm FAIL: entry dial: daemon $DPID execs '$EXE', not the installed image"
160 exit 1; }
161
162 # Reattach: the snapshot must carry the pre-detach marker — session state
163 # lived on the box, not in the client that left.
164 set +e
165 timeout 90 "$PTYCLIENT" --cols 80 --rows 24 \
166 --out "$SCRATCH/j2b.cap" --err "$SCRATCH/j2b.err" -- "$MUX" "$VM" \
167 > "$SCRATCH/j2b.log" 2>&1 <<'EOF'
168 expect vmj2= 30000
169 settle 600 15000
170 send \x1cd
171 waitexit 15000
172 EOF
173 RC=$?
174 set -e
175 [ "$RC" -eq 0 ] || {
176 echo "vm FAIL: reattach: ptyclient exited $RC, or the snapshot lost the marker:"
177 cat "$SCRATCH/j2b.log"; cat "$SCRATCH/j2b.err" 2>/dev/null; exit 1; }
178 ok "reattach: the snapshot still paints the first visit's marker"
179
180 # ---- journey 3: a stop stays stopped ----------------------------------
181 # The wall polls `mux d endpoint` — the bare read verb — once a second per
182 # host. The bug this pins (found on a live box): a poll that spelled
183 # --start undid every remote `mux d stop` within a second. The entry dial
184 # above already recorded $VM in the scratch hosts file, so a bare `mux`
185 # here is a user's wall watching that box.
186 vssh 'mux d stop'
187 _i=0
188 while [ "$_i" -lt 50 ]; do
189 [ "$(vssh 'pgrep -cx mux' || true)" = "0" ] && break
190 sleep 0.2; _i=$((_i + 1))
191 done
192 [ "$(vssh 'pgrep -cx mux' || true)" = "0" ] || {
193 echo "vm FAIL: stop: mux processes survive 'mux d stop' on the box:"
194 vssh 'ps -o pid,args -C mux' | sed 's/^/ /'
195 exit 1; }
196
197 grep -q "$VM" "$XDG_STATE_HOME/mux/hosts" || {
198 echo "vm FAIL: stop: the entry dial never recorded $VM in the hosts file; it holds:"
199 sed 's/^/ /' "$XDG_STATE_HOME/mux/hosts" 2>/dev/null || echo " (no file)"
200 exit 1; }
201
202 # Hold the wall open past a pipe-link poll cycle (10s), then leave. The
203 # empty wall opens the picker by itself; Esc closes it to the one-line
204 # text and the quit chord ends the wall.
205 set +e
206 timeout 90 "$PTYCLIENT" --cols 80 --rows 24 \
207 --out "$SCRATCH/j3.cap" --err "$SCRATCH/j3.err" -- "$MUX" \
208 > "$SCRATCH/j3.log" 2>&1 <<'EOF'
209 expect \x1b[?1049h 30000
210 settle 12000 20000
211 send \x1b
212 settle 400 5000
213 send \x1c\x1c
214 waitexit 15000
215 EOF
216 RC=$?
217 set -e
218 [ "$RC" -eq 0 ] || {
219 echo "vm FAIL: stop: the watching wall exited $RC:"
220 cat "$SCRATCH/j3.log"; cat "$SCRATCH/j3.err" 2>/dev/null; exit 1; }
221
222 [ "$(vssh 'pgrep -cx mux' || true)" = "0" ] || {
223 echo "vm FAIL: stop: the wall's polling RESURRECTED a daemon on the box:"
224 vssh 'ps -o pid,args -C mux' | sed 's/^/ /'
225 exit 1; }
226 grep -q 'unreachable' "$SCRATCH/j3.cap" || {
227 echo "vm FAIL: stop: the wall never painted the stopped box as unreachable"
228 exit 1; }
229 ok "stop stays stopped: 12s of wall polling started nothing on the box"
230
231 # ---- journey 4: remote upgrade, same pid across the exec --------------
232 # The daemon is started the way a shell-integration user starts it, so the
233 # manifest has OSC 133 marks to carry — the one piece of session state
234 # nothing can rebuild after the exec.
235 vssh 'MUX_SHELL_INTEGRATION=1 mux d start -d'
236 _i=0
237 while [ "$_i" -lt 50 ]; do
238 vssh 'mux d endpoint' >/dev/null 2>&1 && break
239 sleep 0.2; _i=$((_i + 1))
240 done
241
242 set +e
243 timeout 90 "$PTYCLIENT" --cols 80 --rows 24 \
244 --out "$SCRATCH/j4.cap" --err "$SCRATCH/j4.err" -- "$MUX" "$VM" \
245 > "$SCRATCH/j4.log" 2>&1 <<'EOF'
246 expect \x1b[?1049h 30000
247 settle 700 20000
248 send echo vmj4=ok\n
249 expect vmj4= 20000
250 settle 600 15000
251 send \x1cd
252 waitexit 15000
253 EOF
254 RC=$?
255 set -e
256 [ "$RC" -eq 0 ] || {
257 echo "vm FAIL: upgrade: the pre-upgrade attach exited $RC:"
258 cat "$SCRATCH/j4.log"; cat "$SCRATCH/j4.err" 2>/dev/null; exit 1; }
259 vssh 'mux d dump' | grep -q 'vmj4=ok' || {
260 echo "vm FAIL: upgrade: the pre-upgrade marker never reached the box's grid"
261 exit 1; }
262
263 PID_BEFORE=$(vssh 'pgrep -x mux' | head -1)
264 set +e
265 timeout 120 "$MUX" d upgrade "$VM" --allow-same-version > "$SCRATCH/j4.up" 2>&1
266 RC=$?
267 set -e
268 [ "$RC" -eq 0 ] || {
269 echo "vm FAIL: upgrade: 'mux d upgrade $VM' exited $RC:"
270 sed 's/^/ /' "$SCRATCH/j4.up"; exit 1; }
271
272 # The exec keeps the pid: same process, new image. Poll for the served
273 # state first — the exec is not instant.
274 _i=0
275 while [ "$_i" -lt 50 ]; do
276 vssh 'mux d endpoint' >/dev/null 2>&1 && break
277 sleep 0.2; _i=$((_i + 1))
278 done
279 PID_AFTER=$(vssh 'pgrep -x mux' | head -1)
280 [ -n "$PID_BEFORE" ] && [ "$PID_BEFORE" = "$PID_AFTER" ] || {
281 echo "vm FAIL: upgrade: daemon pid changed across the exec ($PID_BEFORE -> $PID_AFTER)"
282 exit 1; }
283 vssh 'mux d dump' | grep -q 'vmj4=ok' || {
284 echo "vm FAIL: upgrade: the session's grid did not survive the exec"
285 exit 1; }
286 ok "upgrade: same pid $PID_AFTER across the exec, session grid intact"
287
288 # marks survived the manifest: the upgraded daemon still knows real exit
289 # codes, which only OSC 133 state carried across the exec can.
290 MRUN=$(vssh 'mux a run --timeout 8000 true')
291 echo "$MRUN" | grep -q '"mechanism": *"marks"' || {
292 echo "vm FAIL: upgrade: mux a does not answer under marks after the exec; it said:"
293 echo " $MRUN"
294 exit 1; }
295 ok "upgrade: mux a still answers mechanism=marks after the exec"
296
297 # ---- the pin ----------------------------------------------------------
298 # A literal, e2e.sh-style: adding a journey means editing this by hand.
299 [ "$OK_COUNT" = "7" ] || {
300 echo "vm FAIL: $OK_COUNT checkpoints ran, the pin says 7 —"
301 echo " a journey was added (update the pin) or silently lost"
302 exit 1
303 }
304 echo "vm OK ($OK_COUNT journey checkpoints against $VM)"