a73x

726e0349

test: gate output throughput, so a 600x regression fails instead of feeling bad

a73x   2026-08-21 13:36

Commit message
test: gate output throughput, so a 600x regression fails instead of feeling bad

The Debug install shipped because nothing measured it. Two legs, both
`yes`: the daemon alone (20k lines, no client) and the full path with a
real client on a real pty (200k lines).

Both legs are BEST-OF-5. Noise only ever adds time, so the minimum is the
honest "how fast can this go", and a regression moves the minimum exactly
as much as it moves the mean. Single-shot readings could not carry the
bounds that matter: on an idle 16-core box one client rep in ten came in
at 61-63ms.

Solo: 6ms, hard ceiling 10ms — that leg has no spread worth the name.
Client: 45-57ms, comfort target 60ms, hard ceiling 75ms. The target is a
judgement about the product and only reports; the ceiling is set off the
measured noise floor, because a gate pinned to the noise floor gets
disabled by its third false alarm. Every reading prints, pass or fail, so
drift toward the target is visible while the gate is still green.

The client leg reads its timings off disk rather than out of the pty
capture: `mux` is a replica painting a 24-row grid, so the capture holds
coalesced screen deltas, and an intermediate rep's line scrolls away
between paints. Only the last rep is reliably on screen — one reading,
which is not a best-of.

Built ReleaseSafe into its own prefix rather than wired as a build.zig
step: a step is handed the default-optimize artifacts, and a timing
assertion against Debug measures ghostty's integrity check and nothing
this repo wrote.

`make ci` is new and is the delivery gate — check, e2e, agent,
throughput, cheapest first. Not soak, not xversion; both are asked for by
name.

Every failure arm was watched to fire: Debug binaries trip the timing
bound (3820ms), and a mutated needle trips each of the three no-op guards
— empty grid, absent prompt, and a short reading count. Two of those
mutations silently failed to apply on the first attempt and the run went
green, which is the exact failure the guards exist to catch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JDnJ6v15RvwZ2Y3zvfrWpR
Makefile
Old New
@@ -2,7 +2,7 @@
2 # default zig is 0.17-dev. Override with ZIG=... if yours lives elsewhere. 2 # default zig is 0.17-dev. Override with ZIG=... if yours lives elsewhere.
3 ZIG ?= $(HOME)/Downloads/zig-x86_64-linux-0.15.2/zig 3 ZIG ?= $(HOME)/Downloads/zig-x86_64-linux-0.15.2/zig
4 4
5 .PHONY: build test e2e soak bench agent deps clean clean-deps xversion xversion-build install release 5 .PHONY: build check ci test e2e soak bench agent throughput deps clean clean-deps xversion xversion-build install release
6 6
7 # The QUIC stack (deps/quic) is built on demand by build.zig, so no target 7 # The QUIC stack (deps/quic) is built on demand by build.zig, so no target
8 # here needs to depend on this one. It exists to make the one-time cost 8 # here needs to depend on this one. It exists to make the one-time cost
@@ -89,6 +89,33 @@ soak:
89 bench: 89 bench:
90 $(ZIG) build bench 90 $(ZIG) build bench
91 91
92 # ReleaseSafe into its own prefix, for the reason throughput.sh opens with:
93 # in the Debug tree the engine's per-mutation integrity check dominates every
94 # reading, so a timing assertion there would measure ghostty's debug wiring
95 # and never this code. That is also why there is no build.zig step — a step
96 # is handed the default-optimize artifacts, which are the wrong ones.
97 THRUDIR ?= dist/throughput
98 throughput:
99 $(ZIG) build -Doptimize=ReleaseSafe -p $(THRUDIR)
100 ./test/throughput.sh $(THRUDIR)/bin/muxd $(THRUDIR)/bin/mux \
101 $(THRUDIR)/bin/muxa $(THRUDIR)/bin/ptyclient
102
103 check:
104 $(ZIG) build check
105
106 # Everything a change must pass before delivery, cheapest gate first so a
107 # fmt slip fails in seconds rather than after the e2e suites. Sequential
108 # $(MAKE) calls rather than prerequisites: prerequisite order is not a
109 # promise make makes, and "which gate failed" is the whole output.
110 #
111 # Deliberately excludes soak (the e2e suite ten times) and xversion (needs
112 # a second checkout in XVER_OLD_WORKTREE). Both are asked for by name.
113 ci:
114 $(MAKE) check
115 $(MAKE) e2e
116 $(MAKE) agent
117 $(MAKE) throughput
118
92 agent: 119 agent:
93 $(ZIG) build agent 120 $(ZIG) build agent
94 121
test/throughput.sh
Old New
@@ -0,0 +1,151 @@
1 #!/bin/sh
2 # Output-throughput gate. Guards ONE bug class, the one that shipped: a
3 # muxd whose VT engine is doing per-mutation work nobody asked for. Built
4 # Debug, ghostty runs its page-integrity check on every page mutation —
5 # a hash map over every cell plus a fresh DebugAllocator — and this exact
6 # workload went from 7ms to 3713ms. tmux does it in 6ms. That is the
7 # distance being watched: catastrophic, not marginal.
8 #
9 # The bounds are tight, and what makes a tight bound survivable is
10 # BEST-OF-N: noise only ever ADDS time, so the minimum of N runs is the
11 # honest "how fast can this go", while a real regression moves that minimum
12 # exactly as much as it moves the mean.
13 #
14 # Two numbers on the client leg, because they answer different questions.
15 # 60ms is the COMFORT TARGET — past it the thing feels bad to use, which is
16 # a judgement about the product. It reports and does not fail. 75ms is the
17 # CEILING, set from measurement: on an idle 16-core box the best-of-5
18 # minimum ranged 45-60ms across ten runs, so a hard 60 flaked 1 run in 10
19 # with nothing wrong. 60 sits ON the noise floor, and a gate pinned to the
20 # noise floor gets disabled by the third false alarm.
21 #
22 # The solo leg gets one hard bound because it has no such spread: 6ms on
23 # every one of those runs, never 5 or 7 as a minimum.
24 #
25 # Every reading is PRINTED, pass or fail, so drift toward the target stays
26 # visible while the gate is still green.
27 #
28 # Timing is taken INSIDE the session, bracketing the pipeline with the
29 # shell's own clock. That is the number a human feels; the client's view
30 # of when bytes arrived is a different question and not this one.
31 #
32 # Only meaningful against a ReleaseSafe build — see the `throughput`
33 # target in the Makefile, which is why this is not a build.zig step.
34 set -eu
35 MUXD="$1"
36 MUX="$2"
37 MUXA="$3"
38 PTYCLIENT="$4"
39
40 # Lines of `yes`, and the wall-clock ceiling each leg must come in under.
41 SOLO_LINES=20000
42 SOLO_MAX_MS=10
43 CLIENT_LINES=200000
44 CLIENT_WARN_MS=60
45 CLIENT_MAX_MS=75
46 # Repeats per leg; the fastest one is the reading. See the header.
47 REPS=5
48
49 TMPD="$(mktemp -d "${TMPDIR:-/tmp}/muxd-throughput-XXXXXX")"
50 SOCK="$TMPD/t.sock"
51 cleanup() { kill "$DPID" 2>/dev/null || true; rm -rf "$TMPD"; }
52 trap cleanup EXIT INT TERM
53
54 "$MUXD" run --sock "$SOCK" --shell /bin/sh --cols 80 --rows 24 &
55 DPID=$!
56 i=0
57 while [ ! -S "$SOCK" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done
58 [ -S "$SOCK" ] || { echo "throughput FAIL: socket never appeared"; exit 1; }
59
60 # A sentinel rather than a sleep: `sh` has to have reached its prompt and
61 # be reading, or the flood below is typed into a void and the leg times
62 # out reporting a throughput problem it never measured.
63 # Split by quotes the session's shell strips: the needle appears in the
64 # OUTPUT and never in the echoed command line, so this cannot match itself.
65 "$MUXA" send --sock "$SOCK" 'echo REA""DY\n' > /dev/null
66 i=0
67 while [ "$i" -lt 50 ]; do
68 "$MUXA" capture --sock "$SOCK" | grep -qF 'READY' && break
69 sleep 0.1
70 i=$((i+1))
71 done
72 [ "$i" -lt 50 ] || { echo "throughput FAIL: shell never reached its prompt"; exit 1; }
73
74 # ---- leg 1: the daemon alone -------------------------------------------
75 # No client attached, so this is the engine and nothing else. It is the
76 # leg that catches the build-mode regression, because that cost is all
77 # inside feed().
78 # The `done` marker is a separate file on purpose: polling the timings file
79 # itself would read it after the first rep and call that the answer.
80 "$MUXA" send --sock "$SOCK" \
81 "for k in \$(seq $REPS); do A=\$(date +%s%N); yes | head -n $SOLO_LINES; B=\$(date +%s%N); echo \$(( (B-A)/1000000 )) >> $TMPD/solo; done; echo ok > $TMPD/solo.done\n" \
82 > /dev/null
83 i=0
84 while [ ! -s "$TMPD/solo.done" ] && [ "$i" -lt 60 ]; do sleep 0.5; i=$((i+1)); done
85 [ -s "$TMPD/solo.done" ] || {
86 echo "throughput FAIL: $REPS x $SOLO_LINES lines had not finished after 30s"
87 echo " (Debug builds take ~3.7s per rep; 30s means something worse)"
88 exit 1
89 }
90 SOLO_MS="$(sort -n "$TMPD/solo" | head -1)"
91
92 # A shell that died at startup writes no output and would sail past a
93 # timing bound it never exercised. The screen must actually be full of
94 # the flood. Byte needle: the grid arrives JSON-escaped, newlines as \n.
95 "$MUXA" capture --sock "$SOCK" | grep -qF 'y\ny\ny' || {
96 echo "throughput FAIL: the flood left no output on the grid"
97 "$MUXA" capture --sock "$SOCK"
98 exit 1
99 }
100
101 echo "solo: $SOLO_LINES lines, no client: ${SOLO_MS}ms (max ${SOLO_MAX_MS}ms) [$(tr '\n' ' ' < "$TMPD/solo")]"
102 [ "$SOLO_MS" -lt "$SOLO_MAX_MS" ] || {
103 echo "throughput FAIL: solo leg took ${SOLO_MS}ms, ceiling is ${SOLO_MAX_MS}ms"
104 echo " A Debug-built ghostty scores ~3713ms here. Check the optimize mode first."
105 exit 1
106 }
107
108 # ---- leg 2: a real client on a real pty --------------------------------
109 # The full path — engine, delta, wire, replica, paint — because `mux` runs
110 # an engine of its own and can regress the same way muxd did.
111 cat > "$TMPD/script" <<EOF
112 settle 500 8000
113 send for k in \$(seq $REPS); do A=\$(date +%s%N); yes | head -n $CLIENT_LINES; B=\$(date +%s%N); echo \$(( (B-A)/1000000 )) >> $TMPD/client; done; echo DON""E\r
114 expect DONE 90000
115 send \x1c\x1c
116 waitexit 5000
117 EOF
118 RC=0
119 "$PTYCLIENT" --cols 80 --rows 24 --out "$TMPD/out" --err "$TMPD/err" \
120 -- "$MUX" --sock "$SOCK" < "$TMPD/script" > "$TMPD/log" 2>&1 || RC=$?
121 [ "$RC" -eq 0 ] || {
122 echo "throughput FAIL: client leg exited $RC"
123 echo " (3 = the floods never completed inside 60s)"
124 cat "$TMPD/log"
125 exit 1
126 }
127 # Off disk, not out of the capture: `mux` is a replica painting a 24-row
128 # grid, so the capture holds coalesced screen deltas and an intermediate
129 # rep's line scrolls away between paints. Only the last one is ever
130 # reliably on screen, which is one reading, which is not a best-of.
131 CLIENT_ALL="$(cat "$TMPD/client" 2>/dev/null || true)"
132 CLIENT_MS="$(echo "$CLIENT_ALL" | sort -n | head -1)"
133 [ -n "$CLIENT_MS" ] || {
134 echo "throughput FAIL: client leg recorded no timing"; cat "$TMPD/out"; exit 1;
135 }
136 [ "$(echo "$CLIENT_ALL" | wc -l)" -eq "$REPS" ] || {
137 echo "throughput FAIL: wanted $REPS client readings, got: $CLIENT_ALL"
138 exit 1
139 }
140
141 echo "client: $CLIENT_LINES lines, attached: ${CLIENT_MS}ms (target ${CLIENT_WARN_MS}ms, max ${CLIENT_MAX_MS}ms) [$(echo "$CLIENT_ALL" | tr '\n' ' ')]"
142 [ "$CLIENT_MS" -lt "$CLIENT_MAX_MS" ] || {
143 echo "throughput FAIL: client leg took ${CLIENT_MS}ms, ceiling is ${CLIENT_MAX_MS}ms"
144 exit 1
145 }
146 [ "$CLIENT_MS" -lt "$CLIENT_WARN_MS" ] || {
147 echo "throughput SLOW: ${CLIENT_MS}ms is over the ${CLIENT_WARN_MS}ms comfort target"
148 echo " Not a failure. It is the number that decides whether this feels good to use."
149 }
150
151 echo "throughput OK"