a73x

da8a44a0

Input-latency bench: fold in reviewer feedback and controlled hot workload

a73x   2026-04-18 06:52

Commit message
Input-latency bench: fold in reviewer feedback and controlled hot workload

- Lock grid to 80x24 (overridable) so hot-mode sentinel survival is a
  computed guarantee rather than a race against yes's throughput.
- Hot workload is a rate-limited 500-char-line generator via pv -qL 24K;
  stresses parsing/atlas/row-rebuild without scrolling the sentinel off.
- Tighten keyboard suppression to .key events only; keep keymap/enter/
  leave/modifier updates flowing so clipboard/paste/focus keep working.
- Promote PTY termios ECHO verification from risk to main flow.
- Add WSI fallback: abort with diagnostic after >10% of first 50 samples
  time out (likely wp_presentation commit race with Mesa WSI thread).
- Add frame_counter:u64 to FrameTiming for per-stage breakdown join.
- Specify child teardown: SIGTERM -> 100ms grace -> SIGKILL.
- Tighten max_frames_per_sample to 60.
- Unify stats units to µs; drop the scroll-off bucket (no longer needed).
- Runtime note in Makefile target.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs/superpowers/specs/2026-04-18-input-latency-bench-design.md
Old New
@@ -7,7 +7,7 @@ Measure waystty's keystroke-to-display latency in a reproducible, closed-loop, a
7 The output is two headline metrics plus a per-stage breakdown: 7 The output is two headline metrics plus a per-stage breakdown:
8 8
9 - **Cold latency** — terminal is idle; press a key, measure until the echo appears on screen. Tests the best-case path. 9 - **Cold latency** — terminal is idle; press a key, measure until the echo appears on screen. Tests the best-case path.
10 - **Hot latency** — the child process is producing continuous output; press a key while output floods the PTY. Measures contended-PTY responsiveness, which is where terminals diverge in feel. 10 - **Hot latency** — the child process is producing continuous output at a controlled rate; press a key while output contends with the sentinel on the PTY. Measures contended-PTY responsiveness, which is where terminals diverge in feel.
11 11
12 ## Non-goals 12 ## Non-goals
13 13
@@ -22,19 +22,21 @@ New env var `WAYSTTY_INPUT_BENCH={cold|hot|both}` activates input-bench mode.
22 22
23 When active, waystty: 23 When active, waystty:
24 24
25 1. Spawns a minimal PTY child instead of `$SHELL`: 25 1. **Locks the grid to a known size** before spawning any child. Default 80×24, overridable via `WAYSTTY_INPUT_BENCH_COLS` / `WAYSTTY_INPUT_BENCH_ROWS`. This removes font/DPI/monitor variance so numbers are reproducible across machines and so hot-mode sentinel lifetime is a computed guarantee, not a hope.
26 2. Spawns a minimal PTY child instead of `$SHELL`:
26 - `cold` → `cat > /dev/null`. Reads and discards stdin so the PTY slave's input buffer stays drained; kernel PTY line discipline echoes each byte (ECHO flag on the slave). Cat's canonical-mode buffering is irrelevant — we measure kernel echo, not cat's reads. 27 - `cold` → `cat > /dev/null`. Reads and discards stdin so the PTY slave's input buffer stays drained; kernel PTY line discipline echoes each byte (ECHO flag on the slave). Cat's canonical-mode buffering is irrelevant — we measure kernel echo, not cat's reads.
27 - `hot` → `sh -c 'yes "spam spam spam spam spam"'`. Continuous output contends with sentinel echo bytes on the PTY. 28 - `hot` → `sh -c 'yes "$(printf "x%.0s" {1..500})" | pv -qL 24K'`. Produces 500-char lines rate-limited to 24 KB/s, ≈ 46 lines/sec. On an 80×24 grid this gives a sentinel lifetime of ≈30 frames at 60Hz — enough time for any reasonable presentation to land — while still exercising the render pipeline (100KB/s of parsing, atlas churn on long lines, per-frame row rebuild).
28 - `both` → runs cold, dumps stats, restarts child, runs hot. 29 - `both` → runs cold, prints stats, terminates the cold child, runs hot.
29 2. Disables real Wayland keyboard input (nothing is pushed to `keyboard.event_queue` from `src/wayland.zig`). Protects measurements from ambient user input or stuck modifier state. 30 3. **Child teardown**: at scenario end or shutdown, send SIGTERM; wait up to 100ms; send SIGKILL if still alive; `waitpid` to reap. Prevents stuck `pv` or `yes` from surviving the bench.
30 3. Runs the `BenchDriver` (new, in `src/bench_input.zig`) alongside the normal main loop. Driver owns the sentinel allocator, the in-flight sample, the pending-feedback map, and the stats collector. 31 4. **Drops only `.key` events** from the real Wayland keyboard in bench mode. Keymap, enter/leave, modifier updates, and repeat-timer state still flow through `src/wayland.zig` so other subsystems (clipboard, paste, focus) keep working; we just don't let ambient typing land on `keyboard.event_queue`.
31 4. On exit (N samples reached or timeout), prints stats to stderr in the existing output-bench format and exits cleanly. 32 5. Runs the `BenchDriver` (new, in `src/bench_input.zig`) alongside the normal main loop. Driver owns the sentinel allocator, the in-flight sample, the pending-feedback map, and the stats collector.
33 6. On exit (N samples reached per scenario, or abort), prints stats to stderr in the existing output-bench format and exits cleanly.
32 34
33 MAILBOX present mode is preserved throughout. The driver handles `wp_presentation_feedback.discarded` by keeping feedback listeners live on subsequent frames, since a discarded frame's sentinel is still in the grid on the next frame. 35 MAILBOX present mode is preserved throughout. The driver handles `wp_presentation_feedback.discarded` by keeping feedback listeners live on subsequent frames, since a discarded frame's sentinel is still in the grid on the next frame.
34 36
35 ## Module 1: Sentinel allocation 37 ## Module 1: Sentinel allocation
36 38
37 Each sample uses a unique codepoint from the Unicode Private Use Area U+E000…U+EFFF (4096 distinct sentinels). PUA is chosen because it never appears in normal output from `sleep infinity` or `yes`, so a grid scan for a specific codepoint cannot collide with unrelated output. If a run exceeds 4096 samples, the allocator wraps (no collision concern because only one sample is in flight at a time). 39 Each sample uses a unique codepoint from the Unicode Private Use Area U+E000…U+EFFF (4096 distinct sentinels). PUA is chosen because it never appears in normal output from `cat`, `yes`, or `pv`, so a grid scan for a specific codepoint cannot collide with unrelated output. Only one sample is in flight at a time, so wraparound at 4096 is safe.
38 40
39 PUA codepoints render as `.notdef` (tofu) for most fonts; this is fine — the grid cell carries the codepoint regardless of glyph appearance. 41 PUA codepoints render as `.notdef` (tofu) for most fonts; this is fine — the grid cell carries the codepoint regardless of glyph appearance.
40 42
@@ -48,11 +50,21 @@ if (ev.utf8_len > 0) {
48 } 50 }
49 ``` 51 ```
50 52
51 `t_inject` is captured with `std.time.Instant.now()` immediately before the push. On Linux this uses `CLOCK_MONOTONIC`, matching `wp_presentation_feedback`'s clock. 53 `t_inject` is captured with `std.time.Instant.now()` immediately before the push. On Linux this uses `CLOCK_MONOTONIC`, matching `wp_presentation_feedback`'s clock. (Leave a source comment asserting this assumption.)
52 54
53 The real `src/wayland.zig` keyboard binding remains connected so the protocol stays valid, but its callback drops events when bench mode is active. 55 The real `src/wayland.zig` keyboard binding remains connected; only `.key` events are suppressed in bench mode (see Architecture §4).
54 56
55 ## Module 3: Presentation feedback 57 ## Module 3: PTY termios
58
59 At PTY spawn, the bench explicitly validates the slave's termios:
60
61 1. `tcgetattr(slave_fd, &tio)`.
62 2. If `tio.c_lflag & ECHO` is unset, set it and `tcsetattr(slave_fd, TCSANOW, &tio)` before the child's `exec`.
63 3. Leaves canonical mode (`ICANON`) alone — the measurement is kernel-echo-based, which fires per byte regardless of canonical mode.
64
65 This removes any assumption about defaults and makes the bench portable across distros.
66
67 ## Module 4: Presentation feedback
56 68
57 Requires binding `wp_presentation_time` (global `wp_presentation`). The repo does not currently bind this protocol; this is real implementation work, not plumbing. 69 Requires binding `wp_presentation_time` (global `wp_presentation`). The repo does not currently bind this protocol; this is real implementation work, not plumbing.
58 70
@@ -64,55 +76,61 @@ Flow per frame:
64 - `discarded` — frame was superseded. Drop the feedback object but keep listeners on subsequent frames; the sentinel is still in the grid for the next frame. 76 - `discarded` — frame was superseded. Drop the feedback object but keep listeners on subsequent frames; the sentinel is still in the grid for the next frame.
65 3. The Vulkan WSI owns the Wayland `wl_surface.commit` call; `wp_presentation.feedback` attaches to the next commit on the surface. As long as the feedback request precedes `vkQueuePresentKHR`, it captures the right commit. 77 3. The Vulkan WSI owns the Wayland `wl_surface.commit` call; `wp_presentation.feedback` attaches to the next commit on the surface. As long as the feedback request precedes `vkQueuePresentKHR`, it captures the right commit.
66 78
67 ## Module 4: Pair-on-arrival matching 79 ## Module 5: Pair-on-arrival matching
68 80
69 An in-flight sample has two asynchronous completions: 81 An in-flight sample has two asynchronous completions:
70 82
71 - **Grid-side:** scan each frame's `term.snapshot()` for the sentinel codepoint. Record the first frame K where the sentinel is present. 83 - **Grid-side:** scan each frame's `term.snapshot()` for the sentinel codepoint. Record the first frame K where the sentinel is present.
72 - **Feedback-side:** when `presented` fires for frame K's feedback object, record `t_present_K`. 84 - **Feedback-side:** when `presented` fires for frame K's feedback object, record `t_present_K`.
73 85
74 The sample completes when *both* are known for the same frame K. Latency = `t_present_K - t_inject`. Store in a ring buffer (`u32` microseconds, capped at ~68 min range — more than enough). 86 The sample completes when *both* are known for the same frame K. Latency = `t_present_K - t_inject`. Store in a ring buffer (`u64` nanoseconds).
75 87
76 If frame K's feedback is `discarded`, move on to frame K+1's feedback (still looking for sentinel in K+1's grid — the sentinel cell will be there until scrolled off). 88 If frame K's feedback is `discarded`, advance to frame K+1's feedback (still looking for sentinel in K+1's grid; the sentinel cell remains until scrolled off, which the hot workload is sized to prevent within the timeout window).
77 89
78 A sample times out if neither grid nor feedback match within `max_frames_per_sample = 120` (~2s at 60Hz). Timeouts are counted as a separate stat, not folded into the latency distribution. 90 A sample times out if no match within `max_frames_per_sample = 60` (~1s at 60Hz). Timeouts are counted as a separate stat, not folded into the latency distribution.
79 91
80 Grid scan cost: scanning all cells (typical ~200×50 = 10k cells) per frame for a single codepoint is trivial (~10µs). No optimization needed. 92 Grid scan cost: for an 80×24 grid that's 1920 cells per frame — negligible.
81 93
82 ## Module 5: Sample scheduling 94 ## Module 6: Frame-K correlation
83 95
84 After a sample matches (both sides landed), schedule the next injection on the next main-loop iteration following the *next* frame. This is "one-frame floor" — prevents two injections inside the same frame — without inventing a magic millisecond number. 96 The existing `FrameTiming` struct in `src/bench_stats.zig` is positional in its ring, with no notion of absolute frame identity. To correlate a sample with the timing entry for frame K, add a monotonic `frame_counter: u64` field to `FrameTiming`, incremented once per rendered frame. The bench driver records frame K on each sample; the output stage can then join samples against timing entries for the per-stage breakdown.
85 97
86 Target sample count per scenario: **500**. At ~1 sample per refresh + echo round-trip, cold should complete in ~10-15s, hot can run longer due to contention. 98 ## Module 7: Sample scheduling & WSI fallback
87 99
88 ## Module 6: Output 100 After a sample matches, schedule the next injection on the first main-loop iteration following the *next* frame (one-frame floor — prevents two injections per frame without a magic millisecond number).
89 101
90 On exit, prints to stderr, extending the existing format: 102 Target sample count per scenario: **500**.
103
104 **WSI fallback:** If more than 10% of the first 50 samples time out, abort the bench with a diagnostic error on stderr naming `wp_presentation.feedback` commit race with the Mesa WSI thread as the likely cause, and suggesting `VK_KHR_present_wait` as an alternative coordination strategy to investigate. Prevents shipping garbage numbers silently when the feedback attachment misbehaves.
105
106 ## Module 8: Output
107
108 On exit, prints to stderr:
91 109
92 ``` 110 ```
93 === waystty input latency (500 cold, 500 hot) === 111 === waystty input latency (500 cold, 500 hot, 80x24 grid) ===
94 scenario min avg p50 p99 max (ms) timeouts 112 scenario min avg p50 p99 max (µs) timeouts
95 cold 1.2 4.8 4.1 9.2 14.1 0 113 cold 1200 4800 4100 9200 14100 0
96 hot 3.4 12.7 11.5 28.4 42.0 3 114 hot 3400 12700 11500 28400 42000 3
97 ───────────────────────────────────────── 115 ───────────────────────────────────────────────────────────
98 ``` 116 ```
99 117
100 Per-stage breakdown (for p99 samples only) reuses the existing frame-timing ring buffer at the frame K of those samples — snapshots which stage of the output path dominated the p99 tail: 118 All latencies in µs, matching the existing output-bench format.
119
120 Per-stage breakdown for p99 samples (one line per p99 sample, joined on `frame_counter`):
101 121
102 ``` 122 ```
103 p99 cold sample breakdown (µs): 123 p99 breakdown (µs, sample latency / frame K):
104 inject → pty_write_seen: 820 124 cold sample 487 (latency 9200, frame 12340):
105 pty_write → term.write: 140 125 snapshot 4, row_rebuild 120, atlas_upload 0, instance_upload 6, gpu_submit 8100
106 term.write → frame K start: 620
107 frame K total render: 6800
108 frame K submit → present: 820
109 ``` 126 ```
110 127
111 (This is an additional module, not gating. Ship it after the headline metric works.) 128 This module is not gating — ship it after the headline metric works.
112 129
113 ## Module 7: Makefile target 130 ## Module 9: Makefile target
114 131
115 ```makefile 132 ```makefile
133 # Expected runtime: ~15s cold + ~25s hot = ~40s total
116 bench-input: 134 bench-input:
117 $(ZIG) build -Doptimize=$(OPT) 135 $(ZIG) build -Doptimize=$(OPT)
118 WAYSTTY_INPUT_BENCH=both ./zig-out/bin/waystty 2>bench-input.log || true 136 WAYSTTY_INPUT_BENCH=both ./zig-out/bin/waystty 2>bench-input.log || true
@@ -124,8 +142,9 @@ Mirrors the existing `bench` / `profile` targets: `OPT` defaults to `ReleaseFast
124 142
125 ## Files changed 143 ## Files changed
126 144
127 - `src/bench_input.zig` — new. `BenchDriver` struct, sentinel allocator, in-flight sample state, pending-feedback map, stats printer. 145 - `src/bench_input.zig` — new. `BenchDriver` struct, sentinel allocator, in-flight sample state, pending-feedback map, stats printer, WSI fallback trigger.
128 - `src/main.zig` — env parsing, driver init/tick hookup, PTY child switcheroo, scenario sequencer, sentinel scan in the post-frame hook, bench-keyboard-suppression gate. 146 - `src/bench_stats.zig` — add `frame_counter: u64` to `FrameTiming`; increment on each rendered frame.
147 - `src/main.zig` — env parsing, grid-size lock, driver init/tick hookup, PTY child switcheroo, termios verification, scenario sequencer with teardown, sentinel scan in the post-frame hook, bench-keyboard `.key`-suppression gate.
129 - `src/wayland.zig` — bind `wp_presentation` global, plumb `wp_presentation_feedback` creation and event callbacks. 148 - `src/wayland.zig` — bind `wp_presentation` global, plumb `wp_presentation_feedback` creation and event callbacks.
130 - `src/renderer.zig` — expose swapchain present call site so `wp_presentation.feedback(surface)` can be called immediately before `vkQueuePresentKHR`. 149 - `src/renderer.zig` — expose swapchain present call site so `wp_presentation.feedback(surface)` can be called immediately before `vkQueuePresentKHR`.
131 - `Makefile` — `bench-input` target. 150 - `Makefile` — `bench-input` target.
@@ -135,11 +154,14 @@ Mirrors the existing `bench` / `profile` targets: `OPT` defaults to `ReleaseFast
135 - **Unit:** `BenchDriver` grid scan finds the active sentinel at arbitrary row/col positions in a synthetic grid. 154 - **Unit:** `BenchDriver` grid scan finds the active sentinel at arbitrary row/col positions in a synthetic grid.
136 - **Unit:** sentinel allocator produces 4096 distinct codepoints before wrapping. 155 - **Unit:** sentinel allocator produces 4096 distinct codepoints before wrapping.
137 - **Unit:** pair-on-arrival state machine completes when events arrive in either order; advances past `discarded`. 156 - **Unit:** pair-on-arrival state machine completes when events arrive in either order; advances past `discarded`.
157 - **Unit:** WSI-fallback trigger fires after >10% of first 50 samples time out and not before.
158 - **Manual:** grid is 80×24 at bench start regardless of window size; verify via logging.
159 - **Manual:** termios `ECHO` is set on the PTY slave after spawn; verify with a short run of `stty -a` in debug mode.
138 - **Manual:** `make bench-input` produces cold and hot numbers with cold < hot. Sanity: cold p50 on the order of one refresh interval (wait-to-vsync plus compositor latch) is expected; hot p99 noticeably larger than cold p99 confirms contention is being observed. 160 - **Manual:** `make bench-input` produces cold and hot numbers with cold < hot. Sanity: cold p50 on the order of one refresh interval (wait-to-vsync plus compositor latch) is expected; hot p99 noticeably larger than cold p99 confirms contention is being observed.
139 - **Manual:** `WAYSTTY_INPUT_BENCH=cold` against a debug build; verify via logs that each injected sentinel round-trips and matches within a few frames. 161 - **Manual:** run `WAYSTTY_INPUT_BENCH=cold` against a debug build; verify via logs that each injected sentinel round-trips and matches within a few frames.
140 162
141 ## Open implementation risks 163 ## Open implementation risks
142 164
143 - **`wp_presentation_time` binding is non-trivial.** Neither the global nor the feedback callbacks exist in the repo. Budget real time for this, including the WSI-commit coordination (feedback request must precede `vkQueuePresentKHR`). 165 - **`wp_presentation_time` binding is non-trivial.** Neither the global nor the feedback callbacks exist in the repo. Budget real time for this, including the WSI-commit coordination (feedback request must precede `vkQueuePresentKHR`).
144 - **Mesa Wayland WSI internals.** Confirm empirically that `wp_presentation.feedback(surface)` called immediately before `vkQueuePresentKHR` attaches to the driver's commit. If not, we may need `VK_KHR_present_wait` or a different coordination strategy. 166 - **Mesa Wayland WSI internals.** Confirm empirically that `wp_presentation.feedback(surface)` called immediately before `vkQueuePresentKHR` attaches to the driver's commit. If not, the Module 7 fallback surfaces it; the longer-term fix is `VK_KHR_present_wait` or a different coordination strategy.
145 - **PTY termios state.** Confirm the default termios on the spawned child's slave fd has `ECHO` set so kernel echo works without an explicit `stty echo`. If not, call `tcsetattr` on the slave before exec. 167 - **`pv` must be available.** It's the rate-limiter for the hot workload. If not installed, the hot scenario aborts with a clear install hint. Listed in Makefile comments.