91570f6e

Revise GPU render testing spec per review

a73x 2026-04-17 10:54

Commit message

Revise GPU render testing spec per review

Addresses reviewer-identified issues: render to offscreen VkImage
instead of swapchain, force fixed window size and scale=1, spell out
quiescence sequencing (visibility wait + drain + settle), define RMSE
units with per-pixel max cap, vendor minimal PNG codec, add baseline
fingerprint and explicit error paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

docs/superpowers/specs/2026-04-17-gpu-render-testing-design.md

Old		New
@@ -18,7 +18,8 @@ There is no automated way to verify that waystty's GPU output is correct, or to
18	- CI/headless testing (requires real GPU + compositor)	18	- CI/headless testing (requires real GPU + compositor)
19	- Perceptual/SSIM diffing (RMSE is sufficient for terminal content)	19	- Perceptual/SSIM diffing (RMSE is sufficient for terminal content)
20	- External compositor screenshots (captures happen in-process via Vulkan readback)	20	- External compositor screenshots (captures happen in-process via Vulkan readback)
21	- Multi-scale testing (tests run at current display scale; add scale variants later if needed)	21	- Multi-scale testing (tests fix scale to 1x; scale-variant coverage is future work)
		22	- Cross-machine reproducibility (baselines are inherently local; font is pinned by system fontconfig)
22		23
23	## Design	24	## Design
24		25
@@ -26,47 +27,76 @@ There is no automated way to verify that waystty's GPU output is correct, or to
26		27
27	A new CLI mode that renders a VT script and captures the final frame as a PNG.	28	A new CLI mode that renders a VT script and captures the final frame as a PNG.
28		29
		30	Rendering target: dedicated offscreen image, not the swapchain.
		31
		32	The capture frame is rendered to a waystty-owned `VkImage` created with `TRANSFER_SRC_BIT \| COLOR_ATTACHMENT_BIT`, separate from the swapchain. This avoids two problems:
		33	- Swapchain images are created without `TRANSFER_SRC_BIT` and sit in `PRESENT_SRC_KHR` layout after present — can't be copied from without layout dance + reacquisition
		34	- Compositor behavior (damage, alpha handling) can in principle alter presented pixels
		35
		36	The offscreen image uses the same format as the swapchain (`B8G8R8A8_UNORM`) and a dedicated single-use render pass + framebuffer.
		37
		38	Fixed window configuration.
		39
		40	Capture mode forces a known, deterministic environment:
		41	- Window size: fixed 80 columns × 24 rows at scale=1 (so ~`80 * cell_width × 24 * cell_height` px)
		42	- Scale: forced to 1x; any compositor-reported scale is ignored in capture mode
		43	- Font: inherited from `src/config.zig` (Monaspace Argon @ 16px, resolved via system fontconfig). Not user-overridable without a code change; treated as pinned for local use.
		44
		45	If the compositor sizes the window differently, capture mode rejects the frame and fails with an explicit error.
		46
29	Flow:	47	Flow:
30		48
31	1. waystty starts normally (Wayland surface, Vulkan pipeline, PTY)	49	1. waystty starts normally (Wayland surface, Vulkan pipeline, PTY)
32	2. The child shell is replaced by `cat <script>` piped into the PTY	50	2. The child shell is replaced by `cat <script>` piped into the PTY
33	3. After PTY input is exhausted (child exits), one final frame is rendered to flush all dirty rows	51	3. Wait for visibility: block until first `xdg_surface.configure` + first frame callback complete. If this doesn't happen within 3 seconds, exit with error (e.g. window was spawned hidden)
34	4. `vkCmdCopyImageToBuffer` copies the swapchain image to a host-visible staging buffer after fence-wait on the last submit	52	4. Script playback: feed script bytes through the PTY; child exits at EOF
35	5. Raw BGRA pixel data is written as PNG to `<output.png>`	53	5. Drain: after child exits, keep polling the PTY read end until it returns no data for two consecutive 20ms poll cycles (drains any VT output still pending in the kernel buffer)
36	6. Frame timing stats are dumped to stderr (reusing existing benchmark output)	54	6. Settle: run the event loop for one additional 50ms tick to let the VT parser process everything
37	7. waystty exits cleanly	55	7. Render final frame to offscreen image: one synchronous render pass targeting the offscreen `VkImage`, with a fence we wait on
38		56	8. Readback: `vkCmdCopyImageToBuffer` from the offscreen image (in `TRANSFER_SRC_OPTIMAL` layout) to a host-visible staging buffer, second fence-wait
39	Key detail: The readback happens after the GPU has finished the final present. This captures exactly what would have been composited.	57	9. Write PNG: raw BGRA pixel data → PNG (BGRA→RGBA swap, sRGB-encoded bytes; PNG gets an sRGB chunk). Alpha channel is forced to 255 (opaque) — compositor surface uses `opaque_bit_khr` composite-alpha so source alpha is meaningless
		58	10. Dump timings: frame timing stats to stderr (reusing existing benchmark output)
		59	11. Exit cleanly
40		60
41	Code locations:	61	Code locations:
42	- Arg parsing + quiescence detection + exit-after-capture: `src/main.zig`	62	- Arg parsing + visibility wait + quiescence sequencing + exit-after-capture: `src/main.zig`
43	- Vulkan readback + PNG write: `src/renderer.zig`	63	- Offscreen image creation + readback + PNG write: `src/renderer.zig`
44		64
45	### 2. Golden Image Comparison	65	### 2. Golden Image Comparison
46		66
47	A standalone Zig tool that compares two PNG images and reports pass/fail.	67	A standalone Zig tool that compares two PNG images and reports pass/fail.
48		68
49	Comparison logic:	69	Comparison logic:
50	- Load actual and reference PNGs	70	- Load actual and reference PNGs (RGBA8)
51	- Hard fail if dimensions differ (means grid size or scale changed)	71	- Hard fail if dimensions differ (means grid size or scale drifted — a structural change, not a threshold miss)
52	- Per-pixel RGBA distance, averaged across all pixels to compute RMSE	72	- RMSE: for each pixel, compute Euclidean distance across RGB channels (alpha ignored; capture forces opaque). Normalize each channel to [0,1] before distance, so `distance = sqrt((dR² + dG² + dB²) / 3)` per pixel. RMSE is the root-mean-square of those per-pixel distances across the whole image
53	- Pass threshold: RMSE < 0.3% (configurable via `WAYSTTY_TEST_THRESHOLD`)	73	- Per-pixel max: track the worst single-pixel distance (same normalization)
54	- On failure: write a diff image (red overlay on dimmed actual) and report the RMSE value	74	- Pass criteria (both must hold):
		75	- RMSE ≤ `WAYSTTY_TEST_RMSE_MAX` (default 0.005, i.e. 0.5%)
		76	- Max per-pixel distance ≤ `WAYSTTY_TEST_PIXEL_MAX` (default 0.125, i.e. 32/255)
		77	- Defaults are tuned empirically during initial reference generation — re-running on the same machine without code changes should pass with margin. If it doesn't, thresholds get loosened before shipping
		78	- On failure: write a diff image and report both values
		79
		80	Diff image: side-by-side layout (`[actual \| reference \| delta]`), where `delta` is a grayscale heatmap of per-pixel RGB distance (bright = divergent). Easier to eyeball than a pure overlay.
55		81
56	Failure output:	82	Failure output:
57	```	83	```
58	FAIL: tests/golden/reference/bold_colors.png	84	FAIL: tests/golden/reference/bold_colors.png
59	RMSE: 1.2% (threshold: 0.3%)	85	RMSE: 1.2% (max 0.5%)
60	Diff: tests/golden/output/bold_colors.diff.png	86	worst pixel: 18.0% (max 12.5%)
61	Actual: tests/golden/output/bold_colors.png	87	diff: tests/golden/output/bold_colors.diff.png
		88	actual: tests/golden/output/bold_colors.png
62	```	89	```
63		90
64	Why Zig, not ImageMagick/Python:	91	Why Zig, not ImageMagick/Python:
65	- Zero external deps — project is pure Zig	92	- Zero external deps — project is pure Zig
66	- PNG encode/decode is straightforward (use zigimg or minimal vendored decoder)
67	- Same build system — `zig build test-render` builds and runs everything	93	- Same build system — `zig build test-render` builds and runs everything
68		94
69	Code location: `src/tools/imgdiff.zig` — standalone build target.	95	PNG implementation: vendored minimal RGBA8 encoder/decoder (~300 lines). zigimg would work but its breadth is overkill for a file format we only use one byte layout of. Vendor the code into `src/png.zig` to keep the dep graph small.
		96
		97	Code locations:
		98	- Image comparison executable: `src/tools/imgdiff.zig` (new build target in `build.zig`)
		99	- PNG encode/decode: `src/png.zig` (shared by renderer capture + imgdiff)
70		100
71	### 3. VT Test Scripts	101	### 3. VT Test Scripts
72		102
@@ -82,12 +112,18 @@ Curated escape sequences, one per rendering feature:
82	\| `cursor_movement.vt` \| Cursor positioning, clear, scroll regions \|	112	\| `cursor_movement.vt` \| Cursor positioning, clear, scroll regions \|
83	\| `reverse_video.vt` \| Reverse video, hidden, strikethrough attributes \|	113	\| `reverse_video.vt` \| Reverse video, hidden, strikethrough attributes \|
84		114
85	Start with 3-4 scripts initially, add more as features land.	115	Start with 3–4 scripts initially (`basic_ascii`, `bold_colors`, `box_drawing`), add more as features land.
		116
		117	Script format convention:
		118	- Raw bytes, no encoding (scripts are fed verbatim through the PTY)
		119	- Use `\r\n` line endings (PTY defaults make this the common case)
		120	- End each script with cursor-home (`\x1b[H`) so final cursor position is deterministic
		121	- No trailing "capture marker" needed — the capture sequence (child exit → drain → settle → render) handles quiescence
86		122
87	Directory layout:	123	Directory layout:
88	```	124	```
89	tests/golden/	125	tests/golden/
90	├── scripts/ # VT input files (raw escape sequences)	126	├── scripts/ # VT input files
91	│ ├── basic_ascii.vt	127	│ ├── basic_ascii.vt
92	│ ├── bold_colors.vt	128	│ ├── bold_colors.vt
93	│ └── ...	129	│ └── ...
@@ -103,15 +139,39 @@ tests/golden/
103		139
104	### 4. Performance Regression Detection	140	### 4. Performance Regression Detection
105		141
106	Leverages the existing FrameTimingRing. The `--capture` mode gets timing data for free.	142	Leverages the existing FrameTimingRing. The `--capture` mode gets timing data for free, but the capture workload is too short for stable p99s. Perf regression uses the existing `WAYSTTY_BENCH=1` scripted workload (`src/main.zig:~216`).
107		143
108	Baseline file comparison:	144	Baseline file comparison:
109	- `make bench-baseline` runs the benchmark workload, writes frame timing stats (min/avg/p99/max per section) to `tests/bench/baseline.json`	145	- `make bench-baseline` runs the benchmark workload, writes stats to `tests/bench/baseline.json`
110	- `make bench-check` runs the same workload, compares each metric against the baseline	146	- `make bench-check` runs the same workload, compares against baseline
		147
		148	Baseline schema:
		149	```json
		150	{
		151	"workload_sha": "sha256 of the exact bench shell script",
		152	"zig_version": "0.15.0",
		153	"waystty_sha": "git HEAD at capture time",
		154	"frame_count": 256,
		155	"sections": {
		156	"snapshot": {"min": 8, "avg": 98, "p99": 21, "max": 266},
		157	"row_rebuild": {"min": 109, "avg": 761, "p99": 603, "max": 1572},
		158	"atlas_upload": {"min": 0, "avg": 0, "p99": 0, "max": 0},
		159	"instance_upload": {"min": 13, "avg": 51, "p99": 19, "max": 122},
		160	"gpu_submit": {"min": 30, "avg": 73, "p99": 90, "max": 100}
		161	}
		162	}
		163	```
		164
		165	All timings in microseconds.
111		166
112	Regression threshold: p99 must not increase by more than 20% versus baseline. Configurable via `WAYSTTY_BENCH_REGRESSION_PCT`.	167	Regression detection:
		168	- Each section's p99 is compared independently
		169	- p99 must not increase by more than `WAYSTTY_BENCH_REGRESSION_PCT` (default 20%)
		170	- Any single section exceeding the threshold triggers a failure, even if other sections improved
		171	- On `workload_sha` mismatch: warn loudly (not fail) — means the bench script changed, baseline should be regenerated
		172	- On `frame_count` mismatch of more than 20%: warn (statistical power differs enough to make the comparison unreliable)
113		173
114	Why p99: Averages hide spikes. A change adding occasional 5ms stalls to `gpu_submit` won't move the average but will tank perceived smoothness. p99 catches these.	174	Why p99: Averages hide spikes. A change adding occasional 5ms stalls to `gpu_submit` won't move the average but will tank perceived smoothness.
115		175
116	Output:	176	Output:
117	```	177	```
@@ -129,31 +189,51 @@ bench: gpu_submit p99 290us (baseline 90us) +222.2% REGRESSION
129		189
130	\| Target \| Action \|	190	\| Target \| Action \|
131	\|--------\|--------\|	191	\|--------\|--------\|
132	\| `make test-render` \| For each `.vt` script: run `--capture`, compare against golden, report pass/fail. Non-zero exit on failure. \|	192	\| `make test-render` \| Run all `.vt` scripts via `--capture`, diff against goldens. Continues on failure; summarizes at end; non-zero exit if any failed. Orchestrated by a Zig tool (`zig build test-render`), not a shell loop, for consistency with the rest of the build. \|
133	\| `make golden-update` \| Run all captures, copy output to `reference/`. Used after visual verification of intentional changes. \|	193	\| `make golden-update` \| Run all captures, copy `output/` to `reference/`. Used after visual verification of intentional changes. \|
134	\| `make bench-baseline` \| Save current perf profile to `tests/bench/baseline.json` \|	194	\| `make bench-baseline` \| Save current perf profile to `tests/bench/baseline.json` \|
135	\| `make bench-check` \| Run benchmark workload, compare against baseline, flag regressions \|	195	\| `make bench-check` \| Run benchmark workload, compare against baseline, flag regressions \|
136		196
137	### 6. Dependencies	197	### 6. Error Handling
		198
		199	Capture mode is a developer tool; failure mode clarity matters. Explicit failures (stderr + non-zero exit):
		200
		201	\| Condition \| Exit code / message \|
		202	\|-----------\|---------------------\|
		203	\| Script file not found \| `capture: script not found: <path>` \|
		204	\| Output directory unwritable \| `capture: cannot write output: <path>: <errno>` \|
		205	\| Window never becomes visible within 3s \| `capture: window not visible after 3s (compositor hidden window?)` \|
		206	\| Compositor sized window wrong \| `capture: window size mismatch; expected 80x24 cells, got NxM` \|
		207	\| Vulkan readback fence times out \| `capture: GPU readback timed out (10s)` \|
		208	\| PNG write fails \| `capture: png encode failed: <reason>` \|
138		209
139	- `zigimg` (or minimal PNG encoder/decoder) added to `build.zig.zon`	210	All errors exit with status ≥ 2 (reserving 1 for generic failure). The orchestrator (`zig build test-render`) treats any non-zero exit from `--capture` as a test failure with the error message surfaced.
140	- No other external dependencies	211
		212	### 7. Dependencies
		213
		214	- No new external dependencies
		215	- Vendored minimal PNG encoder/decoder at `src/png.zig`
141		216
142	## Workflow	217	## Workflow
143		218
144	1. Make a rendering change	219	1. Make a rendering change
145	2. `make test-render` — see if anything visually regressed	220	2. `make test-render` — see if anything visually regressed
146	3. If a test fails, inspect the diff image in `tests/golden/output/`	221	3. If a test fails, inspect the diff image in `tests/golden/output/*.diff.png`
147	4. If the change is intentional, `make golden-update` to approve new baselines	222	4. If the change is intentional, `make golden-update` to approve new baselines
148	5. `make bench-check` — see if anything got slower	223	5. `make bench-check` — see if anything got slower
149	6. If perf changed intentionally, `make bench-baseline` to update	224	6. If perf changed intentionally, `make bench-baseline` to update
150		225
151	## Implementation Order	226	## Implementation Order
152		227
153	1. Vulkan readback + PNG write in renderer.zig	228	1. Offscreen render target + Vulkan readback + minimal PNG encode (`src/png.zig` + `src/renderer.zig`)
154	2. `--capture` mode in main.zig	229	2. `--capture` CLI mode with full sequencing (visibility wait → playback → drain → settle → render → readback → write) (`src/main.zig`)
155	3. Image comparison tool (imgdiff.zig)	230	3. End-to-end smoke: write one trivial `.vt`, run `--capture`, eyeball the PNG
156	4. Initial VT test scripts (basic_ascii, bold_colors, box_drawing)	231	4. Image comparison tool (`src/tools/imgdiff.zig` + PNG decode)
157	5. Makefile targets (test-render, golden-update)	232	5. Initial VT test scripts (`basic_ascii`, `bold_colors`, `box_drawing`)
158	6. Generate and commit initial golden reference images	233	6. Test orchestrator + Makefile targets (`test-render`, `golden-update`)
159	7. Benchmark baseline infrastructure (bench-baseline, bench-check)	234	7. Generate and commit initial golden reference images; tune thresholds if needed
		235	8. Benchmark baseline infrastructure (`bench-baseline`, `bench-check`) + initial `baseline.json`
		236
		237	## Open Housekeeping
		238
		239	The working tree has stray test binaries at repo root (`test_io`, `test_io2`, `test_io3`, `test_sig`, `test_timer`) that should be gitignored or relocated before `tests/` lands, to keep the test namespace clean.