a73x

1a0fc00e

Scenario runner design: agent-runnable render smoke

a73x   2026-04-19 08:37

Commit message
Scenario runner design: agent-runnable render smoke

In-process --scenario flag that plays a line-based DSL through the
real runTerminal main loop. Agents get PNG frames at checkpoints
plus per-frame timing. Fills the gap where blink and similar
feature work had no agent-executable verification.

Reviewer passes folded in: no dummy child (relaxed isChildAlive
instead), no double-render on capture, sleep-until-flip rendezvous
for timing-sensitive scenarios, tightened cursor predicates, split
exit codes for vulkan-flake vs. real regression.

Implementation decomposes into three landable plans (imgdiff
extract; parser+state pure; main-loop integration).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs/superpowers/specs/2026-04-19-scenario-runner-design.md
Old New
@@ -0,0 +1,258 @@
1 # Scenario Runner Design
2
3 ## Goal
4
5 Let agents test waystty's render / VT behavior end-to-end without a human in the loop. An agent writes a small scenario file, runs `waystty --scenario foo.scenario`, and gets back PNG frames at declared checkpoints plus a per-frame timing dump. Scenarios drive the *real* main loop — real Vulkan, real blink timer, real `frame_loop` — so features like "the cursor blinks 500 ms after DECSCUSR 5" are actually observable.
6
7 ## Background
8
9 Context that motivated this:
10
11 - The cursor-blink feature (landed 2026-04-19, commits `73e2fd1..07d3d20`) shipped without an agent-runnable smoke. The manual smoke list in that plan asked a human to observe the cursor over time. Subagents can't do that.
12 - `capture.zig` already does single-frame offscreen rendering for golden-PNG tests (`tests/golden/scripts/*.vt` → `tests/golden/reference/*.png`), but it's one-shot: render the final state, diff, done. No time axis, no way to check blink phase at t=250 ms vs t=750 ms.
13 - The bench harness (`bench-baseline` / `bench-check`) measures per-frame timing but is driven by a static workload, not a scripted scenario.
14
15 The gap is a lightweight *scenario runner* in-process — not a new process, not a compositor harness, not an IPC channel. See `memory/project_scenario_runner_direction.md` for the brainstorm summary, including paths that were explicitly rejected (nested-compositor via `cage`, driver-socket IPC, in-process module-level harness).
16
17 ## Non-goals
18
19 - **Testing wayland event handling.** Focus change, xkb keyboard, pointer, clipboard — none of these are exercisable because scenarios bypass the wayland input path entirely. A second harness flavor could add this later; not here.
20 - **Shell-in-the-loop.** Scenarios don't spawn a real interactive shell. If a scenario wants `\e[5 q` sent, it writes the bytes directly via the `bytes` directive. "Agent types a command and watches the shell run it" is a different problem (Problem 2 in the brainstorm, likely best solved by wrapping tmux).
21 - **Render-golden against the existing `.vt` + `capture.zig` harness.** Scenarios are a *superset* of that harness (they can express the same single-frame render), but we are not rewriting existing goldens. Both harnesses coexist.
22 - **Multi-frame streaming capture / event-triggered capture.** v1 is clock-based checkpoints only. Stream capture is a reasonable v2 extension — the bench stats already record per-frame data.
23 - **Scenario DSL features beyond a flat directive list.** No loops, variables, conditionals, macros, or interpolation. If a scenario needs logic, write it in Zig as a dedicated tool.
24 - **Non-ASCII bytes in the DSL beyond a small escape set.** v1 accepts `\n`, `\t`, `\r`, `\e` (0x1B), `\\`, `\"` and raw hex via `bytes-hex`. Full-Unicode string literals in the DSL are out of scope.
25
26 ## Architecture
27
28 One new CLI flag, one new module, one integration hook in the main loop. No new dependencies.
29
30 ```
31 ┌──────────────────────────────────────────────────────────────┐
32 │ waystty --scenario <path> │
33 │ │
34 │ main.zig │
35 │ ├── parse --scenario flag → ScenarioState (from scenario.zig)│
36 │ ├── spawn Pty with a long-lived no-op child │
37 │ └── runTerminal(..., scenario_state) │
38 │ │ │
39 │ main loop iteration: │
40 │ │ │
41 │ │ waitForWork(pollfds, timeout) │
42 │ │ tickBlinkPhase + pty read + events + … │
43 │ │ ▶ scenario_state.tick(now_ns) ←── NEW │
44 │ │ emits pending directives: │
45 │ │ bytes → term.write(payload) │
46 │ │ capture → renderer offscreen + PNG + diff │
47 │ │ assert → cell-region predicate │
48 │ │ if (!render_pending) continue; │
49 │ │ … existing render path … │
50 │ │ │
51 │ │ exit when scenario_state.isDone() │
52 │ │
53 └──────────────────────────────────────────────────────────────┘
54 ```
55
56 **Files:**
57
58 - *New* `src/scenario.zig` — scenario parser (line-based DSL), `ScenarioState` struct, tick function, predicate evaluator. Pure where possible; offscreen-render is delegated.
59 - *New* `src/imgdiff.zig` — pure-math RMSE + max-pixel diff, extracted from the current `src/tools/imgdiff.zig` so both the CLI tool and `ScenarioState` can call it. (Renamed from the earlier working title `scenario_diff.zig` for consistency with the existing CLI name.)
60 - *Modify* `src/main.zig` — parse new `--scenario` arg (mirrors existing `--capture` arg handling), plumb `ScenarioState` into `runTerminal`, add one tick call per main-loop iteration, adjust loop-exit condition.
61 - *Modify* `build.zig` — wire `scenario.zig` as a module; add a test runner entry for scenario self-tests.
62 - *New* `tests/scenarios/*.scenario` — scenario fixtures.
63 - *New* `tests/scenarios/golden/<scenario-name>/<label>.png` — reference frames.
64 - *New* `tests/scenarios/out/<scenario-name>/<label>.png` + `.diff.png` — failure outputs (gitignored).
65 - *Modify* `Makefile` — new `scenario`, `scenario-update` targets; `test` target calls `scenario`.
66
67 **Why extend `runTerminal` rather than add a parallel mode (like `capture.zig` does):** fidelity. Blink timer, frame-callback throttling, surface-suspend handling, DECSCUSR parsing, atlas upload — all live inside `runTerminal`. A parallel mode (`src/scenario_mode.zig run(...)`) would inevitably drift. One tick call inside the existing loop is cheap and keeps one source of truth for main-loop behavior.
68
69 **No dummy child process:** the pty itself is part of waystty's real runtime surface (selection, reported size, TIOCSWINSZ) and stays. But the `p.isChildAlive()` check in the loop guard at `src/main.zig:344` is relaxed to `(scenario_state == null and p.isChildAlive()) or (scenario_state != null and !scenario_state.?.isDone())`. A live pty with no spawned child is supported via a small `Pty.openWithoutChild()` or equivalent — pty is open, no fork, the main loop just never reads any bytes. Rejected a `sleep 3650000` child on portability and zombie-accumulation grounds.
70
71 ## Scenario file format
72
73 Line-based directives. One directive per line. Lines starting with `#` (optionally after whitespace) are comments. Blank lines ignored.
74
75 ```
76 # tests/scenarios/blink-bar.scenario
77 size 80 24
78 timeout 5000ms
79
80 # blinking bar cursor
81 bytes "\e[5 q"
82 sleep 250ms
83
84 capture on-phase # t=250ms; cursor on
85 sleep 500ms
86 capture off-phase # t=750ms; cursor off (one flip past)
87
88 assert-cell 0 0 cursor-visible-at on-phase
89 assert-cell 0 0 cursor-absent-at off-phase
90 ```
91
92 **Directives (v1):**
93
94 | Directive | Meaning |
95 |---|---|
96 | `size COLS ROWS` | Required first directive. **v1 informational only** — scenario runner pins to 80×24 (matching `capture.zig`) and rejects scenarios whose declared size differs. Making the size actually variable requires plumbing through the wayland-window-sizing path that `capture.zig` also hardcodes, which is a separate refactor. The directive exists in the DSL so scenarios don't need rewriting when that refactor happens. |
97 | `timeout MS` | Required. Hard wall-clock cap on total scenario runtime. Scenario fails if the cumulative sleep + directive execution exceeds this. |
98 | `sleep MS` | Advance the monotonic timeline by `MS` ms before the next directive executes. Runs real wall-clock sleeps (not fake clocks) so the blink timer fires naturally. Real clock = inherits real flakiness; see **Timing tolerance** below. |
99 | `sleep-until-flip` | Advance until the blink timer's next phase flip actually fires, then return. Phase-aligned rendezvous — avoids capturing a frame straddling the flip. Fails with a descriptive error if blink is not currently armed (scenario wrote no DECSCUSR to enable blink, etc.). |
100 | `bytes "STR"` | Inject bytes directly into `term.write()`. DSL escape set: `\n` `\t` `\r` `\e` `\\` `\"`. No `printf`-style `%` processing, no interpolation. |
101 | `bytes-hex 1B 5B 35 20 71` | Inject raw bytes by hex. Useful when scenarios need bytes outside the DSL escape set. |
102 | `capture LABEL` | Trigger an offscreen render + PNG write to `tests/scenarios/out/<scenario>/<label>.png`. Diff against `tests/scenarios/golden/<scenario>/<label>.png`. On mismatch: write diff heatmap to `<label>.diff.png` and mark scenario failed. Under `WAYSTTY_SCENARIO_UPDATE=1`: rewrite the golden instead of diffing. |
103 | `assert-cell R C PRED` | Cell-region predicate on the *most recent capture*. Predicates (v1): `cell-matches-golden` (workhorse — diffs the cell's pixel-rect against the corresponding rect in `golden/<scenario>/<label>.cell-R-C.png`, auto-captured on golden-update runs), `cursor-block-at` / `cursor-bar-at` / `cursor-underline-at` (explicit shape checks: centroid of bright pixels in the cell must fall within the bbox a cursor of that shape would occupy), `cell-empty` (no pixel in the cell differs from the terminal's default-bg beyond a tolerance). No lax `cursor-visible` predicate — too noise-prone at 2 px line width. |
104 | `assert-cell-at LABEL R C PRED` | Same as above but evaluates against a named prior capture instead of the most recent. |
105
106 Parser: strict. Unknown directives fail with a clear error message including the line number. No leniency — scenarios are meant to be read by humans, so typos shouldn't silently pass.
107
108 ## Data flow
109
110 ### Tick integration
111
112 `ScenarioState.tick(now_ns)` is called once per main-loop iteration, immediately after the blink phase check and before the `if (!render_pending) continue` bail:
113
114 ```
115 // inside runTerminal, per iteration:
116 try frame_loop.waitForWork(&pollfds_extra, timeout);
117 // ... tickBlinkPhase + pty read + event processing ...
118 if (scenario_state) |s| {
119 try s.tick(std.time.nanoTimestamp(), &term, &ctx, &frame_ring);
120 if (s.isDone()) window.should_close = true;
121 }
122 if (!render_pending) continue;
123 ```
124
125 **Why no `render_pending = true` after capture:** `capture` performs an offscreen render *inside* `tick()`. Setting `render_pending = true` would then fire a separate on-screen render this same iteration against the same terminal state — wasted work, and worse, it changes which frame "counts" as the logical capture (the blink timer could advance between the two renders). The offscreen render is the capture and nothing else.
126
127 `tick` implementation:
128
129 ```
130 fn tick(self: *ScenarioState, now_ns: i128, term: *Terminal, ctx: *renderer.Context, ring: *FrameTimingRing) !void {
131 while (self.cursor < self.directives.len) {
132 const d = self.directives[self.cursor];
133 if (now_ns < self.timeline_origin_ns + d.scheduled_offset_ns) return;
134 try self.execute(d, term, ctx, ring);
135 self.cursor += 1;
136 }
137 }
138 ```
139
140 `scheduled_offset_ns` is computed at parse time by walking `sleep` directives — each `sleep N` bumps the offset for everything that follows. `bytes` and `assert-cell` are zero-duration; `capture` takes a small real render cost but is not counted as timeline advance.
141
142 ### Capture execution
143
144 `capture` reuses the public surface already present in `renderer.zig`:
145
146 ```
147 pub fn createOffscreen(...) !OffscreenTarget;
148 pub fn renderToOffscreen(self: *Context, ...) !void;
149 pub fn readbackOffscreen(self: *Context, ...) !void;
150 ```
151
152 Scenario runner calls these exactly as `capture.zig` does for its single-frame case. The offscreen target is created lazily on the first `capture` directive and reused across subsequent captures in the same scenario (allocating a fresh VkImage per capture is wasteful).
153
154 PNG encode uses the existing `png.zig` module. Golden diff uses a shared library extraction of the RMSE + max-pixel math currently in `src/tools/imgdiff.zig` — move the pure-math parts into `src/imgdiff.zig` so both the CLI tool and `scenario.zig` can call them.
155
156 ### Bench data
157
158 Per-scenario timing dump is automatic. At scenario end, the `FrameTimingRing` is serialized via the existing `bench_stats.zig` JSON path to `tests/scenarios/out/<scenario>/bench.json`. Agents can assert on p50/p99/mean via a separate `bench-check`-style command in a follow-up; v1 just writes the file.
159
160 ## Timing tolerance
161
162 Scenarios rely on real wall-clock sleeps, and real clocks are noisy on loaded CI. Two mitigations:
163
164 1. **`sleep-until-flip`** (defined above) for scenarios that care about blink-phase boundaries. Rather than sleeping N ms and hoping the flip has happened, the scenario rendezvouses with the actual flip event. Implementation: `tick()` observes blink-state transitions by comparing `blink_state.blink_on` before and after each main-loop iteration. When `sleep-until-flip` is the current directive, `tick()` blocks on that transition (with a 2× `blink_period_ns` overall cap so a broken scenario doesn't hang forever).
165
166 2. **Capture retry within a tolerance window.** A `capture` directive has a small built-in tolerance: if the first diff against golden misses by less than `rmse ≤ 2× threshold`, the runner waits `floor(frame_period_ns / 2)` and re-captures once. Second miss is a failure. This paves over single-frame anti-aliasing noise without giving up real regression detection. Threshold constants live in `src/imgdiff.zig` for easy tuning.
167
168 Scenarios that can't tolerate either (e.g. "exactly this frame") aren't expressible in v1 — write a dedicated unit test.
169
170 ## Error handling
171
172 Failure modes and their contracts:
173
174 - **Parse error.** Reported as `scenario: <file>:<line>: <message>` to stderr, exit code 2. No partial execution.
175 - **Scheduled-timeline overshoot.** If `tick()` ever sees `now_ns` > `origin + cumulative_sleep + timeout`, the scenario fails with `scenario: <name>: timeout exceeded after <elapsed>ms`, exit code 3.
176 - **Capture PNG write fails.** Scenario fails at that directive with `scenario: <name>: capture <label>: <underlying IO error>`.
177 - **Golden diff mismatch.** Scenario records the failure and continues (so later assertions also run), then exits non-zero at the end. The actual frame and diff heatmap are written to `out/`.
178 - **Cell predicate fails.** Same continue-then-exit-nonzero semantics.
179 - **Vulkan error during scenario render.** Propagates up the existing bounded-wait error path (`vk_sync` returns `VkWaitTimeout` etc.). Scenario reports `scenario: <name>: render error: <tag>` and fails with a distinct exit code so CI can tell "flaky vulkan" apart from "real regression."
180
181 Exit codes:
182 - 0: all scenarios passed.
183 - 2: parse error in any scenario.
184 - 3: wall-clock timeout.
185 - 4: capture or assertion mismatch (regression).
186 - 5: Vulkan `VkWaitTimeout` / `VkAcquireTimeout` during scenario render. CI should treat 5 as "flake, retry once" distinct from 4.
187 - 6: other render / I/O error (OutOfMemory, file I/O, swapchain unrecoverable).
188
189 The runner never mutates the repo unless `WAYSTTY_SCENARIO_UPDATE=1`. With that env set, mismatches rewrite goldens and exit 0 (mirrors the existing `WAYSTTY_GOLDEN_UPDATE=1` convention).
190
191 ## Testing
192
193 Three layers:
194
195 1. **Parser unit tests** in `src/scenario.zig` inline tests:
196 - Every directive round-trips (lex + parse + serialize back via a small dumper for test readability).
197 - Unknown directive → parse error at correct line.
198 - Escape set correctness (`\e` → 0x1B, `\n` → 0x0A, `\\` → 0x5C, etc.).
199 - `bytes-hex` accepts mixed-case, rejects odd-length input and non-hex characters.
200 - Empty scenario and minimum-valid scenario both round-trip.
201
202 2. **Tick-function unit tests** in `src/scenario.zig`:
203 - Directives with `scheduled_offset_ns` in the future are held.
204 - Past-due directives all fire in order on a single tick.
205 - `isDone()` returns true only after last directive executed (not just scheduled).
206 - `tick` does not panic on empty directive list.
207
208 3. **Scenario-level integration** in `tests/scenarios/`:
209 - `smoke-hello.scenario` — writes a single character, captures, asserts cell-solid-bright. Covers the basic render path without any time axis.
210 - `blink-bar.scenario` — the DECSCUSR-5 example above. Covers time-based behavior.
211 - `shape-matrix.scenario` — cycles through `\e[1 q` through `\e[6 q`, captures each, diffs golden. Exercises DECSCUSR branches.
212 - Run via `make scenario` (new make target). CI fails on any non-zero exit.
213
214 **Not tested in v1:**
215 - Predicates beyond the four listed above.
216 - Concurrent scenarios (runner is single-scenario at a time; multi-scenario is a higher-level shell wrapper).
217 - Render timing regression (bench.json is written but not asserted against; future `scenario-bench-check`).
218
219 ## Make targets
220
221 ```
222 make scenario # run all scenarios, fail on any mismatch
223 make scenario-one NAME=foo # run a single scenario (pass-through to tool)
224 make scenario-update # WAYSTTY_SCENARIO_UPDATE=1 make scenario
225 ```
226
227 `make test` chains to `make scenario` so CI picks it up automatically.
228
229 ## Forward-looking notes
230
231 Filed here so future contributors don't need to rediscover:
232
233 - **Stream capture** (continuous per-frame PNG dump) is additive — `FrameTimingRing` already records per-frame data, extending to per-frame PNG writeback is a `--stream` flag on `capture`.
234 - **Cell predicate expansion** — `cursor-visible` / `cursor-absent` / `solid-*` is minimal. If a new predicate is needed, add it to the `Pred` enum and the evaluator, then document in the table above.
235 - **Scenario generation by agents** — scenarios are plain text, so an agent can author them, run, iterate. No need for a separate authoring tool.
236 - **Multi-output scenarios** (one scenario producing many PNGs) already work via multiple `capture` directives. No API change needed for that.
237 - **Testing wayland event paths** (the explicitly out-of-scope item) — if we need this, the likely shape is a second flavor of scenario where the tick function synthesizes `wl_keyboard`/`wl_pointer` events against the dispatcher. Same DSL, different harness variant.
238
239 ## Implementation phasing (for the plan)
240
241 Spec is one coherent feature; implementation decomposes into three independently-landable plans:
242
243 1. **`src/imgdiff.zig` extraction.** Move pure-math RMSE + max-pixel from `src/tools/imgdiff.zig` into a module, keep the CLI tool as a thin wrapper. Landable standalone; no behavior change for existing `test_render`.
244 2. **`src/scenario.zig` parser + state + tick (pure, no integration).** All helpers unit-tested with a fake clock. `isDone()`, directive scheduling, predicate evaluation — all pure. No `main.zig` touch yet.
245 3. **Main-loop integration + first fixtures.** `--scenario` flag, `runTerminal` hook, one `blink-bar.scenario` fixture with golden PNGs, `make scenario` target. This plan is the visible landing; failure of plan 1 or 2 shouldn't force rework here.
246
247 Each plan gets its own `docs/superpowers/plans/YYYY-MM-DD-*.md`. The writing-plans skill invocation after this spec approval will draft plan 1 first.
248
249 ## Files touched (rough)
250
251 - `src/main.zig` — `--scenario` arg parsing, pty dummy-child spawn, `runTerminal` gains `scenario_state: ?*ScenarioState` param, one-line `tick` call in loop, exit condition.
252 - `src/scenario.zig` — parser, state, tick, predicate evaluator (~500 LOC estimate).
253 - `src/imgdiff.zig` — new public module extracted from `src/tools/imgdiff.zig` (pure diff math; existing CLI tool wraps it).
254 - `src/tools/imgdiff.zig` — thin wrapper over new `src/imgdiff.zig`.
255 - `build.zig` — wire new modules.
256 - `Makefile` — new targets.
257 - `tests/scenarios/` — fixtures and goldens.
258 - No changes to `src/vk_sync.zig`, `src/renderer.zig`, `src/frame_loop.zig`, `src/vt.zig`, `src/wayland.zig` (except possibly tiny additions if the scenario tick needs a hook).