da8a44a0
Input-latency bench: fold in reviewer feedback and controlled hot workload
a73x 2026-04-18 06:52
- Lock grid to 80x24 (overridable) so hot-mode sentinel survival is a computed guarantee rather than a race against yes's throughput. - Hot workload is a rate-limited 500-char-line generator via pv -qL 24K; stresses parsing/atlas/row-rebuild without scrolling the sentinel off. - Tighten keyboard suppression to .key events only; keep keymap/enter/ leave/modifier updates flowing so clipboard/paste/focus keep working. - Promote PTY termios ECHO verification from risk to main flow. - Add WSI fallback: abort with diagnostic after >10% of first 50 samples time out (likely wp_presentation commit race with Mesa WSI thread). - Add frame_counter:u64 to FrameTiming for per-stage breakdown join. - Specify child teardown: SIGTERM -> 100ms grace -> SIGKILL. - Tighten max_frames_per_sample to 60. - Unify stats units to µs; drop the scroll-off bucket (no longer needed). - Runtime note in Makefile target. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
diff --git a/docs/superpowers/specs/2026-04-18-input-latency-bench-design.md b/docs/superpowers/specs/2026-04-18-input-latency-bench-design.md index a8716bb..d3e318f 100644 --- a/docs/superpowers/specs/2026-04-18-input-latency-bench-design.md +++ b/docs/superpowers/specs/2026-04-18-input-latency-bench-design.md @@ -7,7 +7,7 @@ Measure waystty's keystroke-to-display latency in a reproducible, closed-loop, a The output is two headline metrics plus a per-stage breakdown: - **Cold latency** — terminal is idle; press a key, measure until the echo appears on screen. Tests the best-case path. - **Hot latency** — the child process is producing continuous output; press a key while output floods the PTY. Measures contended-PTY responsiveness, which is where terminals diverge in feel. - **Hot latency** — the child process is producing continuous output at a controlled rate; press a key while output contends with the sentinel on the PTY. Measures contended-PTY responsiveness, which is where terminals diverge in feel. ## Non-goals @@ -22,19 +22,21 @@ New env var `WAYSTTY_INPUT_BENCH={cold|hot|both}` activates input-bench mode. When active, waystty: 1. Spawns a minimal PTY child instead of `$SHELL`: 1. **Locks the grid to a known size** before spawning any child. Default 80×24, overridable via `WAYSTTY_INPUT_BENCH_COLS` / `WAYSTTY_INPUT_BENCH_ROWS`. This removes font/DPI/monitor variance so numbers are reproducible across machines and so hot-mode sentinel lifetime is a computed guarantee, not a hope. 2. Spawns a minimal PTY child instead of `$SHELL`: - `cold` → `cat > /dev/null`. Reads and discards stdin so the PTY slave's input buffer stays drained; kernel PTY line discipline echoes each byte (ECHO flag on the slave). Cat's canonical-mode buffering is irrelevant — we measure kernel echo, not cat's reads. - `hot` → `sh -c 'yes "spam spam spam spam spam"'`. Continuous output contends with sentinel echo bytes on the PTY. - `both` → runs cold, dumps stats, restarts child, runs hot. 2. Disables real Wayland keyboard input (nothing is pushed to `keyboard.event_queue` from `src/wayland.zig`). Protects measurements from ambient user input or stuck modifier state. 3. Runs the `BenchDriver` (new, in `src/bench_input.zig`) alongside the normal main loop. Driver owns the sentinel allocator, the in-flight sample, the pending-feedback map, and the stats collector. 4. On exit (N samples reached or timeout), prints stats to stderr in the existing output-bench format and exits cleanly. - `hot` → `sh -c 'yes "$(printf "x%.0s" {1..500})" | pv -qL 24K'`. Produces 500-char lines rate-limited to 24 KB/s, ≈ 46 lines/sec. On an 80×24 grid this gives a sentinel lifetime of ≈30 frames at 60Hz — enough time for any reasonable presentation to land — while still exercising the render pipeline (100KB/s of parsing, atlas churn on long lines, per-frame row rebuild). - `both` → runs cold, prints stats, terminates the cold child, runs hot. 3. **Child teardown**: at scenario end or shutdown, send SIGTERM; wait up to 100ms; send SIGKILL if still alive; `waitpid` to reap. Prevents stuck `pv` or `yes` from surviving the bench. 4. **Drops only `.key` events** from the real Wayland keyboard in bench mode. Keymap, enter/leave, modifier updates, and repeat-timer state still flow through `src/wayland.zig` so other subsystems (clipboard, paste, focus) keep working; we just don't let ambient typing land on `keyboard.event_queue`. 5. Runs the `BenchDriver` (new, in `src/bench_input.zig`) alongside the normal main loop. Driver owns the sentinel allocator, the in-flight sample, the pending-feedback map, and the stats collector. 6. On exit (N samples reached per scenario, or abort), prints stats to stderr in the existing output-bench format and exits cleanly. MAILBOX present mode is preserved throughout. The driver handles `wp_presentation_feedback.discarded` by keeping feedback listeners live on subsequent frames, since a discarded frame's sentinel is still in the grid on the next frame. ## Module 1: Sentinel allocation Each sample uses a unique codepoint from the Unicode Private Use Area U+E000…U+EFFF (4096 distinct sentinels). PUA is chosen because it never appears in normal output from `sleep infinity` or `yes`, so a grid scan for a specific codepoint cannot collide with unrelated output. If a run exceeds 4096 samples, the allocator wraps (no collision concern because only one sample is in flight at a time). Each sample uses a unique codepoint from the Unicode Private Use Area U+E000…U+EFFF (4096 distinct sentinels). PUA is chosen because it never appears in normal output from `cat`, `yes`, or `pv`, so a grid scan for a specific codepoint cannot collide with unrelated output. Only one sample is in flight at a time, so wraparound at 4096 is safe. PUA codepoints render as `.notdef` (tofu) for most fonts; this is fine — the grid cell carries the codepoint regardless of glyph appearance. @@ -48,11 +50,21 @@ if (ev.utf8_len > 0) { } ``` `t_inject` is captured with `std.time.Instant.now()` immediately before the push. On Linux this uses `CLOCK_MONOTONIC`, matching `wp_presentation_feedback`'s clock. `t_inject` is captured with `std.time.Instant.now()` immediately before the push. On Linux this uses `CLOCK_MONOTONIC`, matching `wp_presentation_feedback`'s clock. (Leave a source comment asserting this assumption.) The real `src/wayland.zig` keyboard binding remains connected so the protocol stays valid, but its callback drops events when bench mode is active. The real `src/wayland.zig` keyboard binding remains connected; only `.key` events are suppressed in bench mode (see Architecture §4). ## Module 3: Presentation feedback ## Module 3: PTY termios At PTY spawn, the bench explicitly validates the slave's termios: 1. `tcgetattr(slave_fd, &tio)`. 2. If `tio.c_lflag & ECHO` is unset, set it and `tcsetattr(slave_fd, TCSANOW, &tio)` before the child's `exec`. 3. Leaves canonical mode (`ICANON`) alone — the measurement is kernel-echo-based, which fires per byte regardless of canonical mode. This removes any assumption about defaults and makes the bench portable across distros. ## Module 4: Presentation feedback Requires binding `wp_presentation_time` (global `wp_presentation`). The repo does not currently bind this protocol; this is real implementation work, not plumbing. @@ -64,55 +76,61 @@ Flow per frame: - `discarded` — frame was superseded. Drop the feedback object but keep listeners on subsequent frames; the sentinel is still in the grid for the next frame. 3. The Vulkan WSI owns the Wayland `wl_surface.commit` call; `wp_presentation.feedback` attaches to the next commit on the surface. As long as the feedback request precedes `vkQueuePresentKHR`, it captures the right commit. ## Module 4: Pair-on-arrival matching ## Module 5: Pair-on-arrival matching An in-flight sample has two asynchronous completions: - **Grid-side:** scan each frame's `term.snapshot()` for the sentinel codepoint. Record the first frame K where the sentinel is present. - **Feedback-side:** when `presented` fires for frame K's feedback object, record `t_present_K`. The sample completes when *both* are known for the same frame K. Latency = `t_present_K - t_inject`. Store in a ring buffer (`u32` microseconds, capped at ~68 min range — more than enough). The sample completes when *both* are known for the same frame K. Latency = `t_present_K - t_inject`. Store in a ring buffer (`u64` nanoseconds). If frame K's feedback is `discarded`, move on to frame K+1's feedback (still looking for sentinel in K+1's grid — the sentinel cell will be there until scrolled off). If frame K's feedback is `discarded`, advance to frame K+1's feedback (still looking for sentinel in K+1's grid; the sentinel cell remains until scrolled off, which the hot workload is sized to prevent within the timeout window). A sample times out if neither grid nor feedback match within `max_frames_per_sample = 120` (~2s at 60Hz). Timeouts are counted as a separate stat, not folded into the latency distribution. A sample times out if no match within `max_frames_per_sample = 60` (~1s at 60Hz). Timeouts are counted as a separate stat, not folded into the latency distribution. Grid scan cost: scanning all cells (typical ~200×50 = 10k cells) per frame for a single codepoint is trivial (~10µs). No optimization needed. Grid scan cost: for an 80×24 grid that's 1920 cells per frame — negligible. ## Module 5: Sample scheduling ## Module 6: Frame-K correlation After a sample matches (both sides landed), schedule the next injection on the next main-loop iteration following the *next* frame. This is "one-frame floor" — prevents two injections inside the same frame — without inventing a magic millisecond number. The existing `FrameTiming` struct in `src/bench_stats.zig` is positional in its ring, with no notion of absolute frame identity. To correlate a sample with the timing entry for frame K, add a monotonic `frame_counter: u64` field to `FrameTiming`, incremented once per rendered frame. The bench driver records frame K on each sample; the output stage can then join samples against timing entries for the per-stage breakdown. Target sample count per scenario: **500**. At ~1 sample per refresh + echo round-trip, cold should complete in ~10-15s, hot can run longer due to contention. ## Module 7: Sample scheduling & WSI fallback ## Module 6: Output After a sample matches, schedule the next injection on the first main-loop iteration following the *next* frame (one-frame floor — prevents two injections per frame without a magic millisecond number). On exit, prints to stderr, extending the existing format: Target sample count per scenario: **500**. **WSI fallback:** If more than 10% of the first 50 samples time out, abort the bench with a diagnostic error on stderr naming `wp_presentation.feedback` commit race with the Mesa WSI thread as the likely cause, and suggesting `VK_KHR_present_wait` as an alternative coordination strategy to investigate. Prevents shipping garbage numbers silently when the feedback attachment misbehaves. ## Module 8: Output On exit, prints to stderr: ``` === waystty input latency (500 cold, 500 hot) === scenario min avg p50 p99 max (ms) timeouts cold 1.2 4.8 4.1 9.2 14.1 0 hot 3.4 12.7 11.5 28.4 42.0 3 ───────────────────────────────────────── === waystty input latency (500 cold, 500 hot, 80x24 grid) === scenario min avg p50 p99 max (µs) timeouts cold 1200 4800 4100 9200 14100 0 hot 3400 12700 11500 28400 42000 3 ─────────────────────────────────────────────────────────── ``` Per-stage breakdown (for p99 samples only) reuses the existing frame-timing ring buffer at the frame K of those samples — snapshots which stage of the output path dominated the p99 tail: All latencies in µs, matching the existing output-bench format. Per-stage breakdown for p99 samples (one line per p99 sample, joined on `frame_counter`): ``` p99 cold sample breakdown (µs): inject → pty_write_seen: 820 pty_write → term.write: 140 term.write → frame K start: 620 frame K total render: 6800 frame K submit → present: 820 p99 breakdown (µs, sample latency / frame K): cold sample 487 (latency 9200, frame 12340): snapshot 4, row_rebuild 120, atlas_upload 0, instance_upload 6, gpu_submit 8100 ``` (This is an additional module, not gating. Ship it after the headline metric works.) This module is not gating — ship it after the headline metric works. ## Module 7: Makefile target ## Module 9: Makefile target ```makefile # Expected runtime: ~15s cold + ~25s hot = ~40s total bench-input: $(ZIG) build -Doptimize=$(OPT) WAYSTTY_INPUT_BENCH=both ./zig-out/bin/waystty 2>bench-input.log || true @@ -124,8 +142,9 @@ Mirrors the existing `bench` / `profile` targets: `OPT` defaults to `ReleaseFast ## Files changed - `src/bench_input.zig` — new. `BenchDriver` struct, sentinel allocator, in-flight sample state, pending-feedback map, stats printer. - `src/main.zig` — env parsing, driver init/tick hookup, PTY child switcheroo, scenario sequencer, sentinel scan in the post-frame hook, bench-keyboard-suppression gate. - `src/bench_input.zig` — new. `BenchDriver` struct, sentinel allocator, in-flight sample state, pending-feedback map, stats printer, WSI fallback trigger. - `src/bench_stats.zig` — add `frame_counter: u64` to `FrameTiming`; increment on each rendered frame. - `src/main.zig` — env parsing, grid-size lock, driver init/tick hookup, PTY child switcheroo, termios verification, scenario sequencer with teardown, sentinel scan in the post-frame hook, bench-keyboard `.key`-suppression gate. - `src/wayland.zig` — bind `wp_presentation` global, plumb `wp_presentation_feedback` creation and event callbacks. - `src/renderer.zig` — expose swapchain present call site so `wp_presentation.feedback(surface)` can be called immediately before `vkQueuePresentKHR`. - `Makefile` — `bench-input` target. @@ -135,11 +154,14 @@ Mirrors the existing `bench` / `profile` targets: `OPT` defaults to `ReleaseFast - **Unit:** `BenchDriver` grid scan finds the active sentinel at arbitrary row/col positions in a synthetic grid. - **Unit:** sentinel allocator produces 4096 distinct codepoints before wrapping. - **Unit:** pair-on-arrival state machine completes when events arrive in either order; advances past `discarded`. - **Unit:** WSI-fallback trigger fires after >10% of first 50 samples time out and not before. - **Manual:** grid is 80×24 at bench start regardless of window size; verify via logging. - **Manual:** termios `ECHO` is set on the PTY slave after spawn; verify with a short run of `stty -a` in debug mode. - **Manual:** `make bench-input` produces cold and hot numbers with cold < hot. Sanity: cold p50 on the order of one refresh interval (wait-to-vsync plus compositor latch) is expected; hot p99 noticeably larger than cold p99 confirms contention is being observed. - **Manual:** `WAYSTTY_INPUT_BENCH=cold` against a debug build; verify via logs that each injected sentinel round-trips and matches within a few frames. - **Manual:** run `WAYSTTY_INPUT_BENCH=cold` against a debug build; verify via logs that each injected sentinel round-trips and matches within a few frames. ## Open implementation risks - **`wp_presentation_time` binding is non-trivial.** Neither the global nor the feedback callbacks exist in the repo. Budget real time for this, including the WSI-commit coordination (feedback request must precede `vkQueuePresentKHR`). - **Mesa Wayland WSI internals.** Confirm empirically that `wp_presentation.feedback(surface)` called immediately before `vkQueuePresentKHR` attaches to the driver's commit. If not, we may need `VK_KHR_present_wait` or a different coordination strategy. - **PTY termios state.** Confirm the default termios on the spawned child's slave fd has `ECHO` set so kernel echo works without an explicit `stty echo`. If not, call `tcsetattr` on the slave before exec. - **Mesa Wayland WSI internals.** Confirm empirically that `wp_presentation.feedback(surface)` called immediately before `vkQueuePresentKHR` attaches to the driver's commit. If not, the Module 7 fallback surfaces it; the longer-term fix is `VK_KHR_present_wait` or a different coordination strategy. - **`pv` must be available.** It's the rate-limiter for the hot workload. If not installed, the hot scenario aborts with a clear install hint. Listed in Makefile comments.