a73x

a4f3ea7f

test(agent): e2e — marks exit codes, ephemeral TUI drive, settle honesty, QUIC tear/heal

a73x   2026-08-13 19:27

Commit message
test(agent): e2e — marks exit codes, ephemeral TUI drive, settle honesty, QUIC tear/heal

Nine scenarios against the two binaries, every assertion made on a PARSED
JSON object rather than a grep of the line: a field that got renamed, or a
number that became a string, fails here as `<missing>` instead of passing
as a substring that happened to still be present.

Unlike test/e2e.sh this one does not exit on the first failure. Each
scenario is bounded on its own and reports one PASS/FAIL/SKIP line, and the
count of those lines is pinned against a literal — a run that finds two
defects reports two, and a scenario that silently stops running is caught
by the count rather than by the assertions inside it that no longer run.

The four the plan named:

  * marks — a bash session's `true`/`false`/`echo` come back as exit_code
    0/1/0 with mechanism `marks`, and the output span equals the marker
    exactly rather than containing it: a prompt or an echoed command line
    leaking into the transcript is the bug that equality catches.
  * ephemeral TUI — `muxd start` has no `--` argv, so a one-line wrapper
    carries `less /etc/hostname` in through --shell (with HOME pointed at
    the tmpdir, for e2e.sh's $SHELL reason: ~/.lesskey is arbitrary code on
    the session under test). alt_screen is true, mechanism is not marks,
    the quit key is driven, the daemon's own pid goes away, and the next
    call reports that as a JSON object with no stack trace in it.
  * settle — a markless `sleep 1` comes back `settled` or `returned`,
    whichever wins the race, both honest, never `timeout`; and exit_code is
    null, because a mechanism that cannot know an exit code must not print
    a zero.
  * alt-screen guard — ordered BEFORE the drive above, which ends that
    session. `run 'true'` on a full-screen program times out and exits 3.
    A fabricated `returned` here would be the worst failure in the surface:
    an agent would read an exit code no command ever produced.

Five more that only a binary-level test can reach, all over QUIC on
loopback with a keygen'd tmp key:

  * a tear mid-await heals, and the command ran EXACTLY once — the tally is
    counted in the filesystem, not in the grid, because a grid count would
    also see the echoed command line. A client that re-sent its cmdline on
    reconnect would have run it twice, and on anything but `sleep` that is
    a second deploy.
  * a tear with no path back fails with the whole story: detail begins
    `connection lost`, within the deadline, never a hang.
  * a quiet 20s await outlives the 15s idle timeout, asserted on
    duration_ms >= 18000 — 15000 would be the idle timeout wearing a
    timeout's clothes.
  * a session that exits 5 over QUIC reports `session_ended` with code 5,
    not the connection close that followed from it.
  * a blackholed destination fails on --timeout (2s), not on the 15s
    handshake ceiling. Not a refusal: a never-listening port answers with
    ICMP and fails instantly, which proves nothing about the deadline.

The tear runs unprivileged — no iptables, no sudo. A ~35-line UDP relay
stands in front of the daemon's port and muxa dials it instead; the tear is
a control file, not a kill, because the flow has to keep being ABSORBED
after it breaks (a dead relay means ICMP port-unreachable, and a refusal is
the fast path, not the loss being modelled). The relay blackholes the flow
it is told about and keeps forwarding NEW ones, which is exactly a path
that went away and a client that came back on another one — and it removes
the timing window a kill-then-restart heal would have to hit, because the
redial's fresh source port is never the torn one. The relay's own log is
the evidence a redial happened: `flow 2` through a torn path can only be a
reconnect, observed from outside muxa.

The tear daemon runs with --quic-idle-ms 4000. The idle timeout is a
transport parameter, so the negotiated value is the min of the two ends and
that flag governs both — muxa has no idle flag of its own, and waiting out
its 15s default twice would be most of the runtime. The keepalive scenario
deliberately gets its own daemon at the default, since that number is the
thing it is pinning.

Verified by breaking it three ways, not only by watching it pass: a tear
that announces itself but blocks nothing fails on `no second flow`; a
scenario deleted from the file fails on the count pin; a forced skip prints
SKIP and still exits nonzero for the count. Two clean runs at 51s each,
identical, no leaked daemons or relays and no tmpdirs left behind.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test/agent.sh
Old New
@@ -0,0 +1,643 @@
1 #!/bin/sh
2 # End-to-end for the agent surface: muxd daemon + muxa client, binary level.
3 # Every verb muxa has prints one JSON object, so every assertion here is made
4 # against a PARSED object rather than a grep of the line — a field that got
5 # renamed, or a number that became a string, is a failure this suite can see.
6 #
7 # Unlike test/e2e.sh this one does NOT exit on the first failure: each
8 # scenario is bounded on its own and reports PASS/FAIL/SKIP, and the suite
9 # exits nonzero at the end if any failed. A run that finds two defects should
10 # report two, not the first one and a silence.
11 set -u
12
13 # Binaries: the two under test, defaulting to the build output next to this
14 # script's repo. Positional overrides keep e2e.sh's convention for a caller
15 # (build.zig, a packaging check) that wants to name them explicitly.
16 ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
17 MUXD="${1:-$ROOT/zig-out/bin/muxd}"
18 MUXA="${2:-$ROOT/zig-out/bin/muxa}"
19 [ -x "$MUXD" ] || { echo "agent FAIL: no muxd at $MUXD (run: zig build)"; exit 1; }
20 [ -x "$MUXA" ] || { echo "agent FAIL: no muxa at $MUXA (run: zig build)"; exit 1; }
21
22 # One directory for everything this run writes: sockets, keys, captures, the
23 # relay, the TUI's HOME. Removed by the trap, so a failing run leaves nothing
24 # behind but its output.
25 TMP="${TMPDIR:-/tmp}/mux-agent-$$"
26 mkdir -p "$TMP" || exit 1
27
28 # Hermetic homes, for e2e.sh's reasons: the key-bearing scenarios must see OUR
29 # key and never the developer's ~/.config/mux/key, and $SHELL is read the same
30 # way — a daemon started without --shell would otherwise run the operator's
31 # login shell and its whole rc on the session under test.
32 XDG_CONFIG_HOME="$TMP/cfg"
33 XDG_STATE_HOME="$TMP/state"
34 XDG_CACHE_HOME="$TMP/cache"
35 export XDG_CONFIG_HOME XDG_STATE_HOME XDG_CACHE_HOME
36 SHELL=/bin/sh
37 export SHELL
38 # muxa reads MUX_KEY_FILE BEFORE the default key path, so an operator who
39 # happens to have one exported would silently change which key every QUIC
40 # scenario below presents — and they would still pass, against the wrong key.
41 unset MUX_KEY_FILE
42
43 # Unix-socket daemons, one per scenario that owns its session's shell.
44 SOCK_MARKS="$TMP/marks.sock"
45 SOCK_TUI="$TMP/tui.sock"
46 SOCK_SETTLE="$TMP/settle.sock"
47 # QUIC daemons: the one behind the tearable relay, and the one the quiet-await
48 # scenario dials directly (it must keep the DEFAULT idle timeout, which is the
49 # very thing it is pinning, so it cannot share the reduced-idle daemon).
50 SOCK_TEAR="$TMP/tear.sock"
51 SOCK_QUIET="$TMP/quiet.sock"
52 # Ports in a band of their own so a concurrent test/e2e.sh (11000..46000, in
53 # 5000-wide slots) cannot collide, and per-run so two agent suites can overlap.
54 # All four are ports we BIND, which is what makes the ephemeral range safe here.
55 PORT_TEAR=$((51000 + ($$ % 900)))
56 PORT_RELAY=$((52000 + ($$ % 900)))
57 PORT_QUIET=$((53000 + ($$ % 900)))
58 PORT_SINK=$((54000 + ($$ % 900)))
59
60 KEY="$TMP/key"
61 RELAY="$TMP/relay.py"
62 RELAY_LOG="$TMP/relay.log"
63 SINK_LOG="$TMP/sink.log"
64 # The relay's two control files. Creating one is the tear; the relay removes it
65 # and narrates what it did, so the log is evidence rather than the test's own
66 # claim about what it asked for.
67 CTL_FLOW="$TMP/ctl.flow"
68 CTL_ALL="$TMP/ctl.all"
69 TUISH="$TMP/tui.sh"
70
71 # Every pid the trap may have to kill, declared before the trap is installed:
72 # under `set -u` a bare $VAR the trap reads would abort the trap itself on a
73 # failure that happened before the assignment, and the tmpdir would survive.
74 D_MARKS=""
75 D_TUI=""
76 D_SETTLE=""
77 D_TEAR=""
78 D_QUIET=""
79 RELAY_PID=""
80 SINK_PID=""
81 CLI_PID=""
82
83 cleanup() {
84 # `|| true` on every kill: without `set -e` here it is belt and braces, but
85 # this trap also runs on INT, and a half-run trap leaves a live daemon.
86 for p in "$D_MARKS" "$D_TUI" "$D_SETTLE" "$D_TEAR" "$D_QUIET" \
87 "$RELAY_PID" "$SINK_PID" "$CLI_PID"; do
88 [ -n "$p" ] && kill "$p" 2>/dev/null
89 done
90 # ...and by socket, for the window between `muxd start`'s fork and the
91 # up-line this script reads its pid from. `muxd stop` on a path nobody
92 # serves is a no-op. These must precede the rm -rf: unlinking the sockets
93 # first would leave a live daemon nothing could reach by path.
94 for s in "$SOCK_MARKS" "$SOCK_TUI" "$SOCK_SETTLE" "$SOCK_TEAR" "$SOCK_QUIET"; do
95 [ -S "$s" ] && "$MUXD" stop --sock "$s" >/dev/null 2>&1
96 done
97 rm -rf "$TMP"
98 return 0
99 }
100 trap 'cleanup' EXIT INT TERM
101
102 PASSES=0
103 FAILS=0
104 SKIPS=0
105 pass() { PASSES=$((PASSES + 1)); echo "agent PASS: $1"; }
106 fail() { FAILS=$((FAILS + 1)); echo "agent FAIL: $1"; }
107 skip() { SKIPS=$((SKIPS + 1)); echo "agent SKIP: $1"; }
108
109 # The reason a scenario function gave for stopping. Set by `why`, read by the
110 # caller: a scenario reports ONE line, and this is how the first failed
111 # assertion inside it gets into that line.
112 #
113 # It also reaps the scenario's in-flight client, because this is the ONLY path
114 # out of a QUIC scenario that skips the `wait` below it. Without this a failing
115 # tear scenario leaves a muxa still awaiting on the relay, and the scenario
116 # after it counts that stranger's flows as its own — one failure would read as
117 # two, and the second one would be fiction.
118 WHY=""
119 why() {
120 WHY="$1"
121 [ -n "$CLI_PID" ] && kill "$CLI_PID" 2>/dev/null
122 CLI_PID=""
123 return 1
124 }
125
126 # Run a scenario function: 0 passes, 1 fails with $WHY, 2 skips with $WHY.
127 # One line per scenario, which is what makes the count pin at the bottom mean
128 # "every scenario ran" rather than "some number of assertions ran".
129 run_scenario() {
130 _name="$1"
131 shift
132 WHY=""
133 "$@"
134 case $? in
135 0) pass "$_name" ;;
136 2) skip "$_name: $WHY" ;;
137 *) fail "$_name: $WHY" ;;
138 esac
139 }
140
141 # One field out of a JSON object, printed flat: `null` for JSON null, `true`
142 # /`false` for booleans (so an assertion reads as the wire spelling), and a
143 # loud sentinel for a missing key or a body that is not JSON at all. A muxa
144 # that printed a stack trace fails here as `<unparseable>`, not as a mismatch.
145 jget() {
146 python3 - "$1" "$2" <<'PY'
147 import json, sys
148 try:
149 obj = json.load(open(sys.argv[1]))
150 except Exception:
151 print("<unparseable>"); raise SystemExit(0)
152 if not isinstance(obj, dict):
153 print("<not-an-object>"); raise SystemExit(0)
154 if sys.argv[2] not in obj:
155 print("<missing>"); raise SystemExit(0)
156 v = obj[sys.argv[2]]
157 print("null" if v is None else ("true" if v is True else "false" if v is False else str(v)))
158 PY
159 }
160
161 # want FILE FIELD VALUE — assert one field, naming the whole body on a miss so
162 # a wrong answer is read in context rather than alone.
163 want() {
164 _got=$(jget "$1" "$2")
165 [ "$_got" = "$3" ] && return 0
166 why "$2=$_got, want $3 [$(tr -d '\n' < "$1")]"
167 }
168
169 # Wait until PATTERN shows up in FILE. Keyed off the process's own output
170 # rather than a fixed sleep, e2e.sh's convention.
171 wait_for() {
172 _i=0
173 while [ "$_i" -lt $((${3:-10} * 20)) ]; do
174 [ -f "$1" ] && grep -q "$2" "$1" 2>/dev/null && return 0
175 sleep 0.05
176 _i=$((_i + 1))
177 done
178 return 1
179 }
180
181 # Start a detached daemon and hand back the pid IT reported. Never a pid this
182 # script guessed from a process name: the suite kills what it started, and a
183 # name match can only ever name a bystander.
184 start_daemon() {
185 _log="$1"
186 shift
187 "$MUXD" start "$@" >"$_log" 2>&1
188 sed -n 's/^up .*pid=\([0-9]*\).*/\1/p' "$_log" | head -1
189 }
190
191 # Poll until muxa can answer on a socket: the session's shell has to have been
192 # exec'd and the daemon's listener accepted before any assertion means
193 # anything, and how long that takes is the box's business, not a constant here.
194 wait_ready() {
195 _i=0
196 while [ "$_i" -lt 100 ]; do
197 "$MUXA" status --sock "$1" --timeout 1000 >/dev/null 2>&1 && return 0
198 sleep 0.05
199 _i=$((_i + 1))
200 done
201 return 1
202 }
203
204 now_ms() { python3 -c 'import time; print(int(time.time() * 1000))'; }
205
206 echo "agent: ports ${PORT_TEAR}/${PORT_RELAY}/${PORT_QUIET}/${PORT_SINK}, tmp $TMP"
207
208 # --- 1: marks. A shell with OSC 133 injected knows its own exit codes -------
209 # The only mechanism that can report an exit code at all, so all three
210 # assertions here are really one: `mechanism=marks` is what makes the number
211 # in `exit_code` the command's rather than a guess.
212 scen_marks() {
213 [ -x /bin/bash ] || { WHY="no /bin/bash to inject marks into"; return 2; }
214 D_MARKS=$(start_daemon "$TMP/marks.log" --sock "$SOCK_MARKS" --shell /bin/bash)
215 [ -n "$D_MARKS" ] || why "daemon never printed an up-line [$(cat "$TMP/marks.log")]" || return 1
216 wait_ready "$SOCK_MARKS" || why "daemon never answered on $SOCK_MARKS" || return 1
217
218 timeout 20 "$MUXA" run --sock "$SOCK_MARKS" --timeout 8000 'true' >"$TMP/m1" 2>&1
219 _rc=$?
220 [ "$_rc" -eq 0 ] || why "run 'true' exited $_rc [$(tr -d '\n' < "$TMP/m1")]" || return 1
221 want "$TMP/m1" reason returned || return 1
222 want "$TMP/m1" mechanism marks || return 1
223 want "$TMP/m1" exit_code 0 || return 1
224
225 timeout 20 "$MUXA" run --sock "$SOCK_MARKS" --timeout 8000 'false' >"$TMP/m2" 2>&1
226 want "$TMP/m2" reason returned || return 1
227 want "$TMP/m2" mechanism marks || return 1
228 # The command failed; muxa did not. A nonzero exit_code is an ANSWER, and
229 # an agent that branches on muxa's own status must not see it as an error.
230 want "$TMP/m2" exit_code 1 || return 1
231
232 # A marker the session's shell cannot expand differently than we spell it,
233 # and one that is unique per run so a stale grid can never satisfy it.
234 _mark="out-$$"
235 timeout 20 "$MUXA" run --sock "$SOCK_MARKS" --timeout 8000 "echo $_mark" >"$TMP/m3" 2>&1
236 want "$TMP/m3" reason returned || return 1
237 want "$TMP/m3" mechanism marks || return 1
238 want "$TMP/m3" exit_code 0 || return 1
239 # Exactly the output, not "contains": the span between the two marks is
240 # the command's transcript, and a prompt or an echoed command line leaking
241 # into it is the bug this equality is here to catch.
242 want "$TMP/m3" output "$_mark" || return 1
243
244 "$MUXD" stop --sock "$SOCK_MARKS" >/dev/null 2>&1
245 D_MARKS=""
246 return 0
247 }
248 run_scenario "marks: exit codes and output come back from a bash session" scen_marks
249
250 # --- The ephemeral TUI, which two scenarios share --------------------------
251 # The spec's field specimen: a daemon whose session is not a shell at all but a
252 # throwaway full-screen program. `muxd start` has no `--` argv, and --shell
253 # execs whatever path it is given, so a one-line wrapper carries the argument.
254 # HOME points into the tmpdir for e2e.sh's $SHELL reason: ~/.lesskey and
255 # ~/.vimrc are arbitrary code on the session under test.
256 TUI_KIND=none
257 TUI_CMD=""
258 TUI_QUIT=""
259 if command -v vi >/dev/null 2>&1; then
260 TUI_KIND=vi
261 TUI_CMD="$(command -v vi) /etc/hostname"
262 TUI_QUIT=':q!\n'
263 elif command -v less >/dev/null 2>&1; then
264 TUI_KIND=less
265 TUI_CMD="$(command -v less) /etc/hostname"
266 TUI_QUIT='q'
267 fi
268 if [ "$TUI_KIND" != none ]; then
269 mkdir -p "$TMP/home"
270 {
271 echo '#!/bin/sh'
272 echo "HOME=$TMP/home; export HOME"
273 # less reads its own switches out of the environment; a developer with
274 # -F exported would make the session exit before it was ever driven.
275 echo 'LESS=; export LESS'
276 echo 'unset LESSOPEN LESSCLOSE'
277 echo "exec $TUI_CMD"
278 } > "$TUISH"
279 chmod +x "$TUISH"
280 D_TUI=$(start_daemon "$TMP/tui.log" --sock "$SOCK_TUI" --shell "$TUISH")
281 wait_ready "$SOCK_TUI" || D_TUI=""
282 fi
283
284 # --- 2: the alt-screen guard ------------------------------------------------
285 # Ordered before the drive below because that one ENDS this session. Nothing
286 # on a full-screen program's grid can mean "the command returned" — there are
287 # no marks, no prompt, and no rows to attribute — so the honest answer to
288 # `run` is that the wait timed out. A fabricated `returned` here would be the
289 # worst failure in the surface: an agent would read an exit code that no
290 # command ever produced.
291 scen_altguard() {
292 [ -n "$D_TUI" ] || { WHY="no TUI to drive (vi/less both absent or the daemon died)"; return 2; }
293 timeout 20 "$MUXA" run --sock "$SOCK_TUI" --timeout 1500 'true' >"$TMP/g1" 2>&1
294 _rc=$?
295 want "$TMP/g1" reason timeout || return 1
296 # Exit 3 is the whole point of having a code for it: `returned` and
297 # `settled` are answers and exit 0, a timeout is a question still open.
298 [ "$_rc" -eq 3 ] || why "exit $_rc, want 3 [$(tr -d '\n' < "$TMP/g1")]" || return 1
299 return 0
300 }
301 run_scenario "alt-screen: run times out rather than fabricating a return" scen_altguard
302
303 # --- 3: the ephemeral TUI, driven and quit ---------------------------------
304 scen_tui() {
305 [ -n "$D_TUI" ] || { WHY="no TUI to drive (vi/less both absent or the daemon died)"; return 2; }
306 timeout 20 "$MUXA" status --sock "$SOCK_TUI" --timeout 5000 >"$TMP/t1" 2>&1
307 want "$TMP/t1" alt_screen true || return 1
308 # Marks are a shell's doing. A program that is not a shell cannot have
309 # them, and claiming otherwise is what scenario 2 would then read.
310 _mech=$(jget "$TMP/t1" mechanism)
311 [ "$_mech" != marks ] || why "mechanism=marks on a TUI that no shell started" || return 1
312
313 timeout 10 "$MUXA" send --sock "$SOCK_TUI" -- "$TUI_QUIT" >"$TMP/t2" 2>&1
314 want "$TMP/t2" sent true || return 1
315
316 # The session ended: the pid the daemon reported for itself is gone. Not
317 # the socket, which is unlinked a moment before the process is actually
318 # down — only a pid can answer this.
319 _i=0
320 while kill -0 "$D_TUI" 2>/dev/null; do
321 _i=$((_i + 1))
322 [ "$_i" -lt 100 ] || why "the TUI quit but its daemon (pid $D_TUI) is still up 5s later" || return 1
323 sleep 0.05
324 done
325 D_TUI=""
326
327 # ...and the next call against the corpse is a JSON OBJECT, which is the
328 # contract an agent depends on: there is no reply muxa can give that an
329 # agent has to parse as prose, and no path here that panics.
330 timeout 10 "$MUXA" status --sock "$SOCK_TUI" --timeout 2000 >"$TMP/t3" 2>&1
331 _rc=$?
332 [ "$_rc" -ne 0 ] || why "status against a dead daemon exited 0 [$(tr -d '\n' < "$TMP/t3")]" || return 1
333 _err=$(jget "$TMP/t3" error)
334 case "$_err" in
335 "<unparseable>"|"<missing>"|"<not-an-object>")
336 why "no JSON error object after the session ended: [$(tr -d '\n' < "$TMP/t3")]" || return 1 ;;
337 esac
338 # A panic prints a trace and an error message; the grep is what tells the
339 # two apart when the object above happens to parse anyway.
340 ! grep -qi 'panic\|segmentation\|\.zig:[0-9]' "$TMP/t3" ||
341 why "a stack trace, not an error object: [$(cat "$TMP/t3")]" || return 1
342 return 0
343 }
344 run_scenario "ephemeral TUI: alt_screen seen, quit driven, death reported as JSON" scen_tui
345
346 # --- 4: settle, on a shell with no marks -----------------------------------
347 # `settled` and `returned` are BOTH honest here and which one arrives is a
348 # race: --settle 300 accepts 300ms of quiet as the end of the command, and the
349 # pgid probe sees the foreground group go back to the shell at the second the
350 # sleep exits. What must never come back is `timeout` — the client asked a
351 # question a markless session can answer two different ways, and "I don't
352 # know" is not one of them.
353 scen_settle() {
354 D_SETTLE=$(start_daemon "$TMP/settle.log" --sock "$SOCK_SETTLE" --shell /bin/sh)
355 [ -n "$D_SETTLE" ] || why "daemon never printed an up-line [$(cat "$TMP/settle.log")]" || return 1
356 wait_ready "$SOCK_SETTLE" || why "daemon never answered on $SOCK_SETTLE" || return 1
357
358 timeout 20 "$MUXA" run --sock "$SOCK_SETTLE" --settle 300 --timeout 10000 'sleep 1' >"$TMP/s1" 2>&1
359 _rc=$?
360 [ "$_rc" -eq 0 ] || why "run exited $_rc [$(tr -d '\n' < "$TMP/s1")]" || return 1
361 _reason=$(jget "$TMP/s1" reason)
362 case "$_reason" in
363 settled|returned) ;;
364 *) why "reason=$_reason, want settled or returned [$(tr -d '\n' < "$TMP/s1")]" || return 1 ;;
365 esac
366 # No marks means no exit code, and muxa says so with a null rather than a
367 # zero — an agent must never read "it worked" out of a mechanism that
368 # cannot know.
369 want "$TMP/s1" exit_code null || return 1
370
371 "$MUXD" stop --sock "$SOCK_SETTLE" >/dev/null 2>&1
372 D_SETTLE=""
373 return 0
374 }
375 run_scenario "settle: a markless sleep returns an answer, never a timeout" scen_settle
376
377 # --- The QUIC half ----------------------------------------------------------
378 # Everything below needs a key, a daemon holding a UDP port, and — for the two
379 # tear scenarios — a path this suite can break on purpose WITHOUT root. The
380 # relay is that path: a UDP forwarder in front of the daemon's port, which
381 # muxa dials instead. Tearing is a control file, not a signal and not a kill,
382 # because the flow has to keep being ABSORBED after it breaks: a relay that
383 # died would have the kernel answer with ICMP port-unreachable, and a refusal
384 # is the fast path, not the loss this is modelling.
385 #
386 # The tear blackholes the flow it is told about and keeps forwarding NEW ones.
387 # That is exactly a path that went away and a client that came back on another
388 # one, and it is what makes the heal deterministic: no timing window to hit,
389 # because the redial's fresh source port is never the torn one.
390 cat > "$RELAY" <<'PY'
391 import os, socket, select, sys
392
393 listen_port, target_port, ctl_flow, ctl_all = int(sys.argv[1]), int(sys.argv[2]), sys.argv[3], sys.argv[4]
394 front = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
395 front.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
396 front.bind(("127.0.0.1", listen_port))
397 backs, owner, blocked, block_all = {}, {}, set(), False
398 sys.stderr.write("relay up %d -> %d\n" % (listen_port, target_port)); sys.stderr.flush()
399
400 while True:
401 if os.path.exists(ctl_flow):
402 blocked |= set(backs.keys()); os.remove(ctl_flow)
403 sys.stderr.write("relay: tore %d flow(s)\n" % len(blocked)); sys.stderr.flush()
404 if os.path.exists(ctl_all):
405 block_all = True; os.remove(ctl_all)
406 sys.stderr.write("relay: blackholed everything\n"); sys.stderr.flush()
407 ready, _, _ = select.select([front] + list(backs.values()), [], [], 0.05)
408 for s in ready:
409 if s is front:
410 data, addr = front.recvfrom(65535)
411 if block_all or addr in blocked:
412 continue
413 b = backs.get(addr)
414 if b is None:
415 b = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
416 b.connect(("127.0.0.1", target_port))
417 backs[addr], owner[b.fileno()] = b, addr
418 sys.stderr.write("relay: flow %d\n" % len(backs)); sys.stderr.flush()
419 try:
420 b.send(data)
421 except OSError:
422 pass
423 else:
424 try:
425 data = s.recv(65535)
426 except OSError:
427 continue
428 addr = owner.get(s.fileno())
429 if addr is not None and not block_all and addr not in blocked:
430 front.sendto(data, addr)
431 PY
432
433 QUIC_WHY=""
434 if ! command -v python3 >/dev/null 2>&1; then
435 QUIC_WHY="no python3 to run the UDP relay, and the tear cannot be made without one"
436 elif ! command -v timeout >/dev/null 2>&1; then
437 QUIC_WHY="no coreutils timeout to bound the backgrounded clients"
438 elif ! "$MUXD" keygen >"$TMP/keygen.log" 2>&1; then
439 QUIC_WHY="muxd keygen failed [$(tr -d '\n' < "$TMP/keygen.log")]"
440 else
441 cp "$XDG_CONFIG_HOME/mux/key" "$KEY" 2>/dev/null || QUIC_WHY="keygen wrote no key where it said it did"
442 fi
443
444 if [ -z "$QUIC_WHY" ]; then
445 # The reduced idle is the schedule the tear scenarios wait for. It is a
446 # transport parameter, so the NEGOTIATED value is the min of the two ends
447 # and this daemon's 4s governs both — muxa has no idle flag of its own,
448 # and waiting out its 15s default twice would be most of this suite's
449 # runtime. The quiet-await scenario below deliberately does not use it.
450 D_TEAR=$(start_daemon "$TMP/tear.log" --sock "$SOCK_TEAR" --shell /bin/bash \
451 --quic "127.0.0.1:$PORT_TEAR" --key "$KEY" --quic-idle-ms 4000)
452 [ -n "$D_TEAR" ] && wait_ready "$SOCK_TEAR" || QUIC_WHY="the QUIC daemon never came up [$(cat "$TMP/tear.log")]"
453 fi
454 if [ -z "$QUIC_WHY" ]; then
455 python3 "$RELAY" "$PORT_RELAY" "$PORT_TEAR" "$CTL_FLOW" "$CTL_ALL" >"$RELAY_LOG" 2>&1 &
456 RELAY_PID=$!
457 wait_for "$RELAY_LOG" "relay up" 5 || QUIC_WHY="the relay never bound $PORT_RELAY [$(cat "$RELAY_LOG")]"
458 fi
459
460 # --- 5: a tear mid-await heals, and the command still ran exactly once ------
461 # The two claims an agent's whole reconnect story rests on. The reply that
462 # arrives after the heal carries the ORIGINAL command's return — muxa re-issued
463 # its await from the watermark it already held, so the daemon answered about
464 # the same command rather than starting a new wait — and the command ran once,
465 # which is the no-input-resend rule: a client that re-sent its cmdline on
466 # reconnect would have run it twice, and on anything but `sleep` that is a
467 # second deploy, not a second read.
468 scen_tear_heal() {
469 [ -z "$QUIC_WHY" ] || { WHY="$QUIC_WHY"; return 2; }
470 _tally="$TMP/tally"
471 rm -f "$_tally"
472 # Counted in the FILESYSTEM, not in the grid: a grid count would also see
473 # the echoed command line, and a wrapped row would make it a guess.
474 timeout 40 "$MUXA" run --quic "127.0.0.1:$PORT_RELAY" --key "$KEY" --timeout 25000 \
475 "sleep 12; echo ran >> $_tally" >"$TMP/q1" 2>&1 &
476 CLI_PID=$!
477 # Tear once the await is genuinely in flight — proved by the relay having
478 # opened the flow, not by a sleep that hopes it has.
479 wait_for "$RELAY_LOG" "relay: flow 1" 10 || why "the client never reached the relay [$(cat "$RELAY_LOG")]" || return 1
480 sleep 1
481 : > "$CTL_FLOW"
482 wait_for "$RELAY_LOG" "relay: tore" 5 || why "the relay never acted on the tear [$(cat "$RELAY_LOG")]" || return 1
483
484 wait "$CLI_PID"
485 _rc=$?
486 CLI_PID=""
487 [ "$_rc" -eq 0 ] || why "run exited $_rc after the heal [$(tr -d '\n' < "$TMP/q1")]" || return 1
488 want "$TMP/q1" reason returned || return 1
489 want "$TMP/q1" mechanism marks || return 1
490 want "$TMP/q1" exit_code 0 || return 1
491
492 # A second flow through the relay is the reconnect, observed from outside
493 # muxa. Without it the reply could only mean the tear never landed, and
494 # this scenario would be asserting nothing at all.
495 grep -q "relay: flow 2" "$RELAY_LOG" ||
496 why "no second flow: the reply came back without a redial [$(cat "$RELAY_LOG")]" || return 1
497
498 _ran=$(wc -l < "$_tally" 2>/dev/null || echo 0)
499 [ "$_ran" -eq 1 ] || why "the command ran $_ran time(s), want exactly 1" || return 1
500 return 0
501 }
502 run_scenario "quic: a tear mid-await heals, and the command ran exactly once" scen_tear_heal
503
504 # --- 6: a tear with nothing to come back to is fatal, and says so ----------
505 # The reconnect is spent once. When the redial cannot complete, the failure an
506 # agent reads must be the WHOLE story: the wait died because the path tore, and
507 # it stayed dead because the redial could not finish. An agent told only
508 # `Timeout` goes and checks its own command; an agent told `connection lost;
509 # reconnect failed: Timeout` knows to check the network.
510 scen_tear_fatal() {
511 [ -z "$QUIC_WHY" ] || { WHY="$QUIC_WHY"; return 2; }
512 # A hang is the failure mode here, so the deadline is asserted twice: the
513 # outer `timeout` makes one impossible to sit through, and the wall clock
514 # below makes one impossible to pass with.
515 # The flow number this client will be given, read off the log rather than
516 # assumed: a scenario above that failed before its redial would leave a
517 # different count, and a hardcoded 3 would then wait for a flow that never
518 # comes and report THAT as this scenario's failure.
519 _flow=$(( $(grep -c "relay: flow" "$RELAY_LOG") + 1 ))
520 _t0=$(now_ms)
521 timeout 30 "$MUXA" run --quic "127.0.0.1:$PORT_RELAY" --key "$KEY" --timeout 12000 \
522 'sleep 20' >"$TMP/q2" 2>&1 &
523 CLI_PID=$!
524 wait_for "$RELAY_LOG" "relay: flow $_flow" 10 || why "the client never opened a new flow [$(cat "$RELAY_LOG")]" || return 1
525 sleep 1
526 : > "$CTL_ALL"
527 wait_for "$RELAY_LOG" "blackholed" 5 || why "the relay never blackholed [$(cat "$RELAY_LOG")]" || return 1
528
529 wait "$CLI_PID"
530 _rc=$?
531 CLI_PID=""
532 _spent=$(( $(now_ms) - _t0 ))
533 [ "$_rc" -ne 0 ] || why "run exited 0 with the path gone [$(tr -d '\n' < "$TMP/q2")]" || return 1
534 [ "$_rc" -ne 124 ] || why "muxa hung past the outer 30s bound" || return 1
535 # --timeout plus muxa's 2s grace over the daemon's own window, plus room
536 # for the box. Anything near 30s means the deadline was not honoured.
537 [ "$_spent" -lt 20000 ] || why "took ${_spent}ms for a 12000ms timeout" || return 1
538 _detail=$(jget "$TMP/q2" detail)
539 case "$_detail" in
540 "connection lost"*) ;;
541 *) why "detail=$_detail, want 'connection lost...' [$(tr -d '\n' < "$TMP/q2")]" || return 1 ;;
542 esac
543 return 0
544 }
545 run_scenario "quic: a tear with no path back fails with the whole story" scen_tear_fatal
546
547 # --- 7: a quiet await outlives the idle timeout ----------------------------
548 # A DIFFERENT daemon, with the default 15s idle: the connection has to be kept
549 # alive by keepalives across a wait during which neither end has anything to
550 # say. The failure this pins is a timeout at ~15s reported as a lost
551 # connection — an agent would go looking for a network fault that never
552 # happened, and the honest answer (still running) would have been one field.
553 scen_keepalive() {
554 [ -z "$QUIC_WHY" ] || { WHY="$QUIC_WHY"; return 2; }
555 D_QUIET=$(start_daemon "$TMP/quiet.log" --sock "$SOCK_QUIET" --shell /bin/bash \
556 --quic "127.0.0.1:$PORT_QUIET" --key "$KEY")
557 [ -n "$D_QUIET" ] || why "daemon never printed an up-line [$(cat "$TMP/quiet.log")]" || return 1
558 wait_ready "$SOCK_QUIET" || why "daemon never answered on $SOCK_QUIET" || return 1
559
560 timeout 40 "$MUXA" await --quic "127.0.0.1:$PORT_QUIET" --key "$KEY" --timeout 20000 >"$TMP/q3" 2>&1
561 _rc=$?
562 [ "$_rc" -eq 3 ] || why "await exited $_rc, want 3 [$(tr -d '\n' < "$TMP/q3")]" || return 1
563 want "$TMP/q3" reason timeout || return 1
564 # The number is the assertion: 15000 would be the idle timeout wearing a
565 # timeout's clothes, and only a duration past it proves the keepalives ran.
566 _dur=$(jget "$TMP/q3" duration_ms)
567 [ "$_dur" -ge 18000 ] 2>/dev/null ||
568 why "duration_ms=$_dur — the wait did not survive the 15s idle timeout" || return 1
569
570 "$MUXD" stop --sock "$SOCK_QUIET" >/dev/null 2>&1
571 D_QUIET=""
572 return 0
573 }
574 run_scenario "quic: a quiet 20s await outlives the 15s idle timeout" scen_keepalive
575
576 # --- 8: the session's death beats the connection's ------------------------
577 # Both ends of this race end the wait, and only one of them is the truth. The
578 # shell exited 5; the connection then closed BECAUSE it did. Reporting the
579 # close is reporting the consequence and losing the cause, and the exit code
580 # is the one thing the agent came for.
581 scen_session_exit() {
582 [ -z "$QUIC_WHY" ] || { WHY="$QUIC_WHY"; return 2; }
583 timeout 20 "$MUXA" run --quic "127.0.0.1:$PORT_TEAR" --key "$KEY" --timeout 8000 \
584 'exit 5' >"$TMP/q4" 2>&1
585 _rc=$?
586 # An ANSWER, not a failure: the command is over and this is how.
587 [ "$_rc" -eq 0 ] || why "run exited $_rc [$(tr -d '\n' < "$TMP/q4")]" || return 1
588 want "$TMP/q4" reason session_ended || return 1
589 want "$TMP/q4" exit_code 5 || return 1
590 return 0
591 }
592 run_scenario "quic: a session that exits 5 reports 5, not a lost connection" scen_session_exit
593 # The tear daemon's session is gone with it; drop the pid so cleanup does not
594 # chase one, and let the socket backstop cover the rest.
595 D_TEAR=""
596
597 # --- 9: a destination that swallows still honours --timeout ----------------
598 # Not a refusal: a never-listening port answers with ICMP and muxa fails
599 # instantly, which proves nothing about the deadline. A UDP listener that reads
600 # and never replies makes the HANDSHAKE hang, and the only thing that can end
601 # it is muxa's own clock.
602 scen_blackhole() {
603 [ -z "$QUIC_WHY" ] || { WHY="$QUIC_WHY"; return 2; }
604 # The relay, blackholing from birth: its control file exists before it
605 # starts, so it swallows the first packet it ever sees.
606 : > "$TMP/sink.all"
607 python3 "$RELAY" "$PORT_SINK" "$PORT_TEAR" "$TMP/sink.flow" "$TMP/sink.all" >"$SINK_LOG" 2>&1 &
608 SINK_PID=$!
609 wait_for "$SINK_LOG" "relay up" 5 || why "the sink never bound $PORT_SINK [$(cat "$SINK_LOG")]" || return 1
610
611 _t0=$(now_ms)
612 timeout 20 "$MUXA" status --quic "127.0.0.1:$PORT_SINK" --key "$KEY" --timeout 2000 >"$TMP/q5" 2>&1
613 _rc=$?
614 _spent=$(( $(now_ms) - _t0 ))
615 kill "$SINK_PID" 2>/dev/null
616 SINK_PID=""
617
618 [ "$_rc" -ne 0 ] || why "status exited 0 against a blackhole [$(tr -d '\n' < "$TMP/q5")]" || return 1
619 [ "$_rc" -ne 124 ] || why "muxa hung past the outer 20s bound" || return 1
620 # The ceiling that must NOT be hit is the 15s handshake idle timeout: a
621 # muxa that ignored --timeout would land there, and this bound is under it
622 # by enough that only the flag can explain the number.
623 [ "$_spent" -lt 6000 ] || why "took ${_spent}ms for a 2000ms timeout — the flag was not honoured" || return 1
624 _err=$(jget "$TMP/q5" error)
625 case "$_err" in
626 "<unparseable>"|"<missing>"|"<not-an-object>")
627 why "no JSON error object [$(tr -d '\n' < "$TMP/q5")]" || return 1 ;;
628 esac
629 return 0
630 }
631 run_scenario "quic: a blackholed destination fails on --timeout, not on the idle ceiling" scen_blackhole
632
633 # The count, pinned against a literal for e2e.sh's reason: a scenario that
634 # silently stops running is the failure mode no assertion inside it can catch.
635 TOTAL=$((PASSES + FAILS + SKIPS))
636 if [ "$TOTAL" -ne 9 ]; then
637 echo "agent FAIL: $TOTAL scenarios reported, want 9 — one did not run"
638 FAILS=$((FAILS + 1))
639 fi
640
641 echo "agent: $PASSES passed, $FAILS failed, $SKIPS skipped"
642 [ "$FAILS" -eq 0 ] || exit 1
643 exit 0