b0ade34d
Add GPU render testing design spec
a73x 2026-04-17 10:39
Covers automated visual regression testing via Vulkan readback, golden image comparison, and performance baseline detection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
diff --git a/docs/superpowers/specs/2026-04-17-gpu-render-testing-design.md b/docs/superpowers/specs/2026-04-17-gpu-render-testing-design.md new file mode 100644 index 0000000..d93c835 --- /dev/null +++ b/docs/superpowers/specs/2026-04-17-gpu-render-testing-design.md @@ -0,0 +1,159 @@ # GPU Render Testing Design Automated visual regression and performance regression testing for the Vulkan rendering pipeline. ## Problem There is no automated way to verify that waystty's GPU output is correct, or to detect rendering performance regressions. The frame timing instrumentation measures *how fast* frames render but not *what* they render. Smoke tests validate subsystem init but not visual output. ## Goals - Detect visual rendering regressions automatically (wrong colors, missing glyphs, shifted cells, broken attributes) - Detect performance regressions in the rendering pipeline - Keep the test infrastructure local-only, zero external dependencies beyond Zig - Produce actionable failure output (diff images, timing comparisons) ## Non-Goals - CI/headless testing (requires real GPU + compositor) - Perceptual/SSIM diffing (RMSE is sufficient for terminal content) - External compositor screenshots (captures happen in-process via Vulkan readback) - Multi-scale testing (tests run at current display scale; add scale variants later if needed) ## Design ### 1. Capture Mode (`--capture <script> <output.png>`) A new CLI mode that renders a VT script and captures the final frame as a PNG. **Flow:** 1. waystty starts normally (Wayland surface, Vulkan pipeline, PTY) 2. The child shell is replaced by `cat <script>` piped into the PTY 3. After PTY input is exhausted (child exits), one final frame is rendered to flush all dirty rows 4. `vkCmdCopyImageToBuffer` copies the swapchain image to a host-visible staging buffer after fence-wait on the last submit 5. Raw BGRA pixel data is written as PNG to `<output.png>` 6. Frame timing stats are dumped to stderr (reusing existing benchmark output) 7. waystty exits cleanly **Key detail:** The readback happens after the GPU has finished the final present. This captures exactly what would have been composited. **Code locations:** - Arg parsing + quiescence detection + exit-after-capture: `src/main.zig` - Vulkan readback + PNG write: `src/renderer.zig` ### 2. Golden Image Comparison A standalone Zig tool that compares two PNG images and reports pass/fail. **Comparison logic:** - Load actual and reference PNGs - Hard fail if dimensions differ (means grid size or scale changed) - Per-pixel RGBA distance, averaged across all pixels to compute RMSE - Pass threshold: RMSE < 0.3% (configurable via `WAYSTTY_TEST_THRESHOLD`) - On failure: write a diff image (red overlay on dimmed actual) and report the RMSE value **Failure output:** ``` FAIL: tests/golden/reference/bold_colors.png RMSE: 1.2% (threshold: 0.3%) Diff: tests/golden/output/bold_colors.diff.png Actual: tests/golden/output/bold_colors.png ``` **Why Zig, not ImageMagick/Python:** - Zero external deps — project is pure Zig - PNG encode/decode is straightforward (use zigimg or minimal vendored decoder) - Same build system — `zig build test-render` builds and runs everything **Code location:** `src/tools/imgdiff.zig` — standalone build target. ### 3. VT Test Scripts Curated escape sequences, one per rendering feature: | Script | Exercises | |--------|-----------| | `basic_ascii.vt` | Full printable ASCII range, default colors | | `bold_colors.vt` | Bold, dim, italic, underline + 16 ANSI colors | | `256_colors.vt` | 256-color palette grid | | `truecolor.vt` | 24-bit RGB gradients | | `box_drawing.vt` | Box-drawing and block element characters | | `cursor_movement.vt` | Cursor positioning, clear, scroll regions | | `reverse_video.vt` | Reverse video, hidden, strikethrough attributes | Start with 3-4 scripts initially, add more as features land. **Directory layout:** ``` tests/golden/ ├── scripts/ # VT input files (raw escape sequences) │ ├── basic_ascii.vt │ ├── bold_colors.vt │ └── ... ├── reference/ # Approved golden PNGs (checked into git) │ ├── basic_ascii.png │ ├── bold_colors.png │ └── ... └── output/ # Generated by test run (gitignored) ├── basic_ascii.png ├── basic_ascii.diff.png └── ... ``` ### 4. Performance Regression Detection Leverages the existing FrameTimingRing. The `--capture` mode gets timing data for free. **Baseline file comparison:** - `make bench-baseline` runs the benchmark workload, writes frame timing stats (min/avg/p99/max per section) to `tests/bench/baseline.json` - `make bench-check` runs the same workload, compares each metric against the baseline **Regression threshold:** p99 must not increase by more than 20% versus baseline. Configurable via `WAYSTTY_BENCH_REGRESSION_PCT`. **Why p99:** Averages hide spikes. A change adding occasional 5ms stalls to `gpu_submit` won't move the average but will tank perceived smoothness. p99 catches these. **Output:** ``` bench: snapshot p99 21us (baseline 19us) +10.5% OK bench: row_rebuild p99 603us (baseline 580us) +3.9% OK bench: gpu_submit p99 290us (baseline 90us) +222.2% REGRESSION ``` **Baseline management:** - `baseline.json` is checked into git — the "known good" perf profile for this machine - After intentional perf changes, `make bench-baseline` to update - Machine-specific by nature — local-only tool, so this is fine ### 5. Makefile Targets | Target | Action | |--------|--------| | `make test-render` | For each `.vt` script: run `--capture`, compare against golden, report pass/fail. Non-zero exit on failure. | | `make golden-update` | Run all captures, copy output to `reference/`. Used after visual verification of intentional changes. | | `make bench-baseline` | Save current perf profile to `tests/bench/baseline.json` | | `make bench-check` | Run benchmark workload, compare against baseline, flag regressions | ### 6. Dependencies - `zigimg` (or minimal PNG encoder/decoder) added to `build.zig.zon` - No other external dependencies ## Workflow 1. Make a rendering change 2. `make test-render` — see if anything visually regressed 3. If a test fails, inspect the diff image in `tests/golden/output/` 4. If the change is intentional, `make golden-update` to approve new baselines 5. `make bench-check` — see if anything got slower 6. If perf changed intentionally, `make bench-baseline` to update ## Implementation Order 1. Vulkan readback + PNG write in renderer.zig 2. `--capture` mode in main.zig 3. Image comparison tool (imgdiff.zig) 4. Initial VT test scripts (basic_ascii, bold_colors, box_drawing) 5. Makefile targets (test-render, golden-update) 6. Generate and commit initial golden reference images 7. Benchmark baseline infrastructure (bench-baseline, bench-check)