a73x

b0ade34d

Add GPU render testing design spec

a73x   2026-04-17 10:39

Commit message
Add GPU render testing design spec

Covers automated visual regression testing via Vulkan readback,
golden image comparison, and performance baseline detection.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
docs/superpowers/specs/2026-04-17-gpu-render-testing-design.md
Old New
@@ -0,0 +1,159 @@
1 # GPU Render Testing Design
2
3 Automated visual regression and performance regression testing for the Vulkan rendering pipeline.
4
5 ## Problem
6
7 There is no automated way to verify that waystty's GPU output is correct, or to detect rendering performance regressions. The frame timing instrumentation measures *how fast* frames render but not *what* they render. Smoke tests validate subsystem init but not visual output.
8
9 ## Goals
10
11 - Detect visual rendering regressions automatically (wrong colors, missing glyphs, shifted cells, broken attributes)
12 - Detect performance regressions in the rendering pipeline
13 - Keep the test infrastructure local-only, zero external dependencies beyond Zig
14 - Produce actionable failure output (diff images, timing comparisons)
15
16 ## Non-Goals
17
18 - CI/headless testing (requires real GPU + compositor)
19 - Perceptual/SSIM diffing (RMSE is sufficient for terminal content)
20 - External compositor screenshots (captures happen in-process via Vulkan readback)
21 - Multi-scale testing (tests run at current display scale; add scale variants later if needed)
22
23 ## Design
24
25 ### 1. Capture Mode (`--capture <script> <output.png>`)
26
27 A new CLI mode that renders a VT script and captures the final frame as a PNG.
28
29 **Flow:**
30
31 1. waystty starts normally (Wayland surface, Vulkan pipeline, PTY)
32 2. The child shell is replaced by `cat <script>` piped into the PTY
33 3. After PTY input is exhausted (child exits), one final frame is rendered to flush all dirty rows
34 4. `vkCmdCopyImageToBuffer` copies the swapchain image to a host-visible staging buffer after fence-wait on the last submit
35 5. Raw BGRA pixel data is written as PNG to `<output.png>`
36 6. Frame timing stats are dumped to stderr (reusing existing benchmark output)
37 7. waystty exits cleanly
38
39 **Key detail:** The readback happens after the GPU has finished the final present. This captures exactly what would have been composited.
40
41 **Code locations:**
42 - Arg parsing + quiescence detection + exit-after-capture: `src/main.zig`
43 - Vulkan readback + PNG write: `src/renderer.zig`
44
45 ### 2. Golden Image Comparison
46
47 A standalone Zig tool that compares two PNG images and reports pass/fail.
48
49 **Comparison logic:**
50 - Load actual and reference PNGs
51 - Hard fail if dimensions differ (means grid size or scale changed)
52 - Per-pixel RGBA distance, averaged across all pixels to compute RMSE
53 - Pass threshold: RMSE < 0.3% (configurable via `WAYSTTY_TEST_THRESHOLD`)
54 - On failure: write a diff image (red overlay on dimmed actual) and report the RMSE value
55
56 **Failure output:**
57 ```
58 FAIL: tests/golden/reference/bold_colors.png
59 RMSE: 1.2% (threshold: 0.3%)
60 Diff: tests/golden/output/bold_colors.diff.png
61 Actual: tests/golden/output/bold_colors.png
62 ```
63
64 **Why Zig, not ImageMagick/Python:**
65 - Zero external deps — project is pure Zig
66 - PNG encode/decode is straightforward (use zigimg or minimal vendored decoder)
67 - Same build system — `zig build test-render` builds and runs everything
68
69 **Code location:** `src/tools/imgdiff.zig` — standalone build target.
70
71 ### 3. VT Test Scripts
72
73 Curated escape sequences, one per rendering feature:
74
75 | Script | Exercises |
76 |--------|-----------|
77 | `basic_ascii.vt` | Full printable ASCII range, default colors |
78 | `bold_colors.vt` | Bold, dim, italic, underline + 16 ANSI colors |
79 | `256_colors.vt` | 256-color palette grid |
80 | `truecolor.vt` | 24-bit RGB gradients |
81 | `box_drawing.vt` | Box-drawing and block element characters |
82 | `cursor_movement.vt` | Cursor positioning, clear, scroll regions |
83 | `reverse_video.vt` | Reverse video, hidden, strikethrough attributes |
84
85 Start with 3-4 scripts initially, add more as features land.
86
87 **Directory layout:**
88 ```
89 tests/golden/
90 ├── scripts/ # VT input files (raw escape sequences)
91 │ ├── basic_ascii.vt
92 │ ├── bold_colors.vt
93 │ └── ...
94 ├── reference/ # Approved golden PNGs (checked into git)
95 │ ├── basic_ascii.png
96 │ ├── bold_colors.png
97 │ └── ...
98 └── output/ # Generated by test run (gitignored)
99 ├── basic_ascii.png
100 ├── basic_ascii.diff.png
101 └── ...
102 ```
103
104 ### 4. Performance Regression Detection
105
106 Leverages the existing FrameTimingRing. The `--capture` mode gets timing data for free.
107
108 **Baseline file comparison:**
109 - `make bench-baseline` runs the benchmark workload, writes frame timing stats (min/avg/p99/max per section) to `tests/bench/baseline.json`
110 - `make bench-check` runs the same workload, compares each metric against the baseline
111
112 **Regression threshold:** p99 must not increase by more than 20% versus baseline. Configurable via `WAYSTTY_BENCH_REGRESSION_PCT`.
113
114 **Why p99:** Averages hide spikes. A change adding occasional 5ms stalls to `gpu_submit` won't move the average but will tank perceived smoothness. p99 catches these.
115
116 **Output:**
117 ```
118 bench: snapshot p99 21us (baseline 19us) +10.5% OK
119 bench: row_rebuild p99 603us (baseline 580us) +3.9% OK
120 bench: gpu_submit p99 290us (baseline 90us) +222.2% REGRESSION
121 ```
122
123 **Baseline management:**
124 - `baseline.json` is checked into git — the "known good" perf profile for this machine
125 - After intentional perf changes, `make bench-baseline` to update
126 - Machine-specific by nature — local-only tool, so this is fine
127
128 ### 5. Makefile Targets
129
130 | Target | Action |
131 |--------|--------|
132 | `make test-render` | For each `.vt` script: run `--capture`, compare against golden, report pass/fail. Non-zero exit on failure. |
133 | `make golden-update` | Run all captures, copy output to `reference/`. Used after visual verification of intentional changes. |
134 | `make bench-baseline` | Save current perf profile to `tests/bench/baseline.json` |
135 | `make bench-check` | Run benchmark workload, compare against baseline, flag regressions |
136
137 ### 6. Dependencies
138
139 - `zigimg` (or minimal PNG encoder/decoder) added to `build.zig.zon`
140 - No other external dependencies
141
142 ## Workflow
143
144 1. Make a rendering change
145 2. `make test-render` — see if anything visually regressed
146 3. If a test fails, inspect the diff image in `tests/golden/output/`
147 4. If the change is intentional, `make golden-update` to approve new baselines
148 5. `make bench-check` — see if anything got slower
149 6. If perf changed intentionally, `make bench-baseline` to update
150
151 ## Implementation Order
152
153 1. Vulkan readback + PNG write in renderer.zig
154 2. `--capture` mode in main.zig
155 3. Image comparison tool (imgdiff.zig)
156 4. Initial VT test scripts (basic_ascii, bold_colors, box_drawing)
157 5. Makefile targets (test-render, golden-update)
158 6. Generate and commit initial golden reference images
159 7. Benchmark baseline infrastructure (bench-baseline, bench-check)