ca7fac28
test: the bench bounds a keystroke's bytes, not a ratio the snapshot's size now dominates
a73x 2026-09-04 18:04
Commit message
docs/decisions.md
| Old | New | ||
|---|---|---|---|
| @@ -9068,11 +9068,17 @@ DENOMINATOR shrank 38-fold. Reading the 1% and the 43% as a regression gets | |||
| 9068 | it exactly backwards; both runs sent the same bytes, and the second one made | 9068 | it exactly backwards; both runs sent the same bytes, and the second one made |
| 9069 | a cold attach to that session sixty-five times cheaper. | 9069 | a cold attach to that session sixty-five times cheaper. |
| 9070 | 9070 | ||
| 9071 | That does leave the bench's own kill criterion sitting seven points away from | 9071 | That left the bench's own kill criterion sitting seven points away from |
| 9072 | tripping — it fails at 50% — on a change that only improved things. The | 9072 | tripping — it failed at 50% — on a change that only improved things. The |
| 9073 | criterion was written when a snapshot was expensive and a delta had to prove | 9073 | criterion was written when a snapshot was expensive and a delta had to prove |
| 9074 | it was worth having. It is worth restating against absolute bytes per | 9074 | it was worth having. It is restated the same day (user decision) as absolute |
| 9075 | keystroke before it fails for the wrong reason. | 9075 | bytes per keystroke: `delta_bytes / 120`, failing at 160. A keystroke changes |
| 9076 | one row and a delta re-sends that row whole — an 18-byte delta header, a | ||
| 9077 | 6-byte row header, and the cells — measured at 64 per keystroke on the VT | ||
| 9078 | wire and 65 on cells. A differ that re-sent the whole viewport per keystroke | ||
| 9079 | would cost about 340 on a blank 80x24 screen (23 empty rows at 8 bytes each | ||
| 9080 | plus the typed row), so 160 catches that and leaves the honest figure room. | ||
| 9081 | The bench still prints the attach snapshot's bytes beside it, for reading. | ||
| 9076 | 9082 | ||
| 9077 | `mux_core.wasm` is 19080 bytes, from 356344. That is the whole terminal | 9083 | `mux_core.wasm` is 19080 bytes, from 356344. That is the whole terminal |
| 9078 | emulator leaving the browser client: the wasm build has no ghostty dependency | 9084 | emulator leaving the browser client: the wasm build has no ghostty dependency |
test/bench.sh
| Old | New | ||
|---|---|---|---|
| @@ -1,6 +1,13 @@ | |||
| 1 | #!/bin/sh | 1 | #!/bin/sh |
| 2 | # M4 bytes-on-wire measurement: a typing-heavy workload, then compare | 2 | # M4 bytes-on-wire measurement: a typing-heavy workload, then the delta |
| 3 | # delta bytes actually sent against the measured full-snapshot equivalent. | 3 | # bytes actually sent per keystroke, against an absolute bound. |
| 4 | # | ||
| 5 | # The bound used to be a ratio of delta bytes to the full-snapshot | ||
| 6 | # equivalent. Cells on the wire (2026-09-04) made a snapshot of a mostly | ||
| 7 | # blank screen tiny — 5644 bytes of VT became 86 bytes of cells — so the | ||
| 8 | # same delta traffic went from 1% to 43% of it against a 50% gate: the | ||
| 9 | # ratio had started measuring the denominator. A keystroke's cost is a | ||
| 10 | # thing to bound on its own. | ||
| 4 | set -eu | 11 | set -eu |
| 5 | MUX="$1" | 12 | MUX="$1" |
| 6 | SOCK="${TMPDIR:-/tmp}/muxd-bench-$$.sock" | 13 | SOCK="${TMPDIR:-/tmp}/muxd-bench-$$.sock" |
| @@ -16,12 +23,13 @@ i=0 | |||
| 16 | while [ ! -S "$SOCK" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done | 23 | while [ ! -S "$SOCK" ] && [ "$i" -lt 50 ]; do sleep 0.1; i=$((i+1)); done |
| 17 | [ -S "$SOCK" ] || { echo "bench FAIL: socket never appeared"; exit 1; } | 24 | [ -S "$SOCK" ] || { echo "bench FAIL: socket never appeared"; exit 1; } |
| 18 | 25 | ||
| 19 | # Steady-state typing: 120 single characters with small gaps (no newlines, | 26 | # Steady-state typing: KEYS single characters with small gaps (no newlines, |
| 20 | # so no scroll; this is the workload the handoff's done-criterion names). | 27 | # so no scroll; this is the workload the handoff's done-criterion names). |
| 28 | KEYS=120 | ||
| 21 | { | 29 | { |
| 22 | sleep 0.5 | 30 | sleep 0.5 |
| 23 | j=0 | 31 | j=0 |
| 24 | while [ "$j" -lt 120 ]; do printf 'x'; sleep 0.05; j=$((j+1)); done | 32 | while [ "$j" -lt "$KEYS" ]; do printf 'x'; sleep 0.05; j=$((j+1)); done |
| 25 | sleep 0.5 | 33 | sleep 0.5 |
| 26 | printf '\034\034' | 34 | printf '\034\034' |
| 27 | } | "$MUX" --sock "$SOCK" > /dev/null | 35 | } | "$MUX" --sock "$SOCK" > /dev/null |
| @@ -30,18 +38,17 @@ STATS="$("$MUX" d stats --sock "$SOCK")" | |||
| 30 | echo "$STATS" | 38 | echo "$STATS" |
| 31 | 39 | ||
| 32 | DELTA=$(echo "$STATS" | sed -n 's/.*[^_]delta_bytes=\([0-9]*\).*/\1/p') | 40 | DELTA=$(echo "$STATS" | sed -n 's/.*[^_]delta_bytes=\([0-9]*\).*/\1/p') |
| 33 | EQUIV=$(echo "$STATS" | sed -n 's/.*snapshot_equiv_bytes=\([0-9]*\).*/\1/p') | 41 | SNAPBYTES=$(echo "$STATS" | sed -n 's/.*[^_]snapshot_bytes=\([0-9]*\).*/\1/p') |
| 34 | DELTAS=$(echo "$STATS" | sed -n 's/.*[^_]deltas=\([0-9]*\).*/\1/p') | 42 | DELTAS=$(echo "$STATS" | sed -n 's/.*[^_]deltas=\([0-9]*\).*/\1/p') |
| 35 | SNAPS=$(echo "$STATS" | sed -n 's/.*snapshots=\([0-9]*\).*/\1/p') | 43 | SNAPS=$(echo "$STATS" | sed -n 's/.*snapshots=\([0-9]*\).*/\1/p') |
| 36 | [ -n "$DELTA" ] && [ -n "$EQUIV" ] && [ -n "$DELTAS" ] && [ -n "$SNAPS" ] && | 44 | [ -n "$DELTA" ] && [ -n "$SNAPBYTES" ] && [ -n "$DELTAS" ] && [ -n "$SNAPS" ] || { |
| 37 | [ "$EQUIV" -gt 0 ] || { | ||
| 38 | echo "bench FAIL: could not parse stats"; exit 1; | 45 | echo "bench FAIL: could not parse stats"; exit 1; |
| 39 | } | 46 | } |
| 40 | 47 | ||
| 41 | # A delta path that sent nothing would score a perfect 0% ratio. The | 48 | # A delta path that sent nothing would score a perfect 0 per keystroke. The |
| 42 | # workload must actually have produced deltas, and exactly one snapshot | 49 | # workload must actually have produced deltas, and exactly one snapshot |
| 43 | # (the attach) — a mid-run resync means the differ fell back, and the | 50 | # (the attach) — a mid-run resync means the differ fell back, and the |
| 44 | # ratio would be measuring the fallback rather than the deltas. | 51 | # figure would be measuring the fallback rather than the deltas. |
| 45 | [ "$DELTAS" -gt 0 ] || { | 52 | [ "$DELTAS" -gt 0 ] || { |
| 46 | echo "bench FAIL: no deltas sent; the delta path is dead, not efficient" | 53 | echo "bench FAIL: no deltas sent; the delta path is dead, not efficient" |
| 47 | exit 1 | 54 | exit 1 |
| @@ -50,10 +57,18 @@ SNAPS=$(echo "$STATS" | sed -n 's/.*snapshots=\([0-9]*\).*/\1/p') | |||
| 50 | echo "bench FAIL: expected exactly 1 snapshot (the attach), got $SNAPS" | 57 | echo "bench FAIL: expected exactly 1 snapshot (the attach), got $SNAPS" |
| 51 | exit 1 | 58 | exit 1 |
| 52 | } | 59 | } |
| 53 | RATIO=$(( DELTA * 100 / EQUIV )) | 60 | # A keystroke changes one row, and a delta re-sends that row whole: the |
| 54 | echo "delta bytes: $DELTA snapshot-equivalent: $EQUIV ratio: ${RATIO}%" | 61 | # 18-byte delta header, the 6-byte row header, and the row's cells, which |
| 55 | if [ "$RATIO" -ge 50 ]; then | 62 | # on this workload grow to the prompt plus KEYS characters. Measured |
| 56 | echo "bench FAIL: deltas are not meaningfully smaller (M4 kill criterion)" | 63 | # 2026-09-04 at 64 bytes per keystroke on the VT wire and 65 on cells. |
| 64 | # The bound is what a wrong differ would cost: re-sending the whole | ||
| 65 | # viewport per keystroke is about 340 bytes on a blank 80x24 screen (23 | ||
| 66 | # empty rows at 8 bytes each plus the typed row), so 160 fails that and | ||
| 67 | # leaves the honest figure room to drift without a false alarm. | ||
| 68 | PER_KEY=$(( DELTA / KEYS )) | ||
| 69 | echo "delta bytes: $DELTA over $KEYS keystrokes = $PER_KEY per keystroke attach snapshot: $SNAPBYTES bytes" | ||
| 70 | if [ "$PER_KEY" -ge 160 ]; then | ||
| 71 | echo "bench FAIL: a keystroke costs $PER_KEY bytes, a row's worth is under 160 (M4 kill criterion)" | ||
| 57 | exit 1 | 72 | exit 1 |
| 58 | fi | 73 | fi |
| 59 | echo "bench OK" | 74 | echo "bench OK" |