docs/superpowers/specs/2026-04-19-scenario-runner-design.md
Ref: Size: 20.0 KiB
# Scenario Runner Design
## Goal
Let agents test waystty's render / VT behavior end-to-end without a human in the loop. An agent writes a small scenario file, runs `waystty --scenario foo.scenario`, and gets back PNG frames at declared checkpoints plus a per-frame timing dump. Scenarios drive the *real* main loop — real Vulkan, real blink timer, real `frame_loop` — so features like "the cursor blinks 500 ms after DECSCUSR 5" are actually observable.
## Background
Context that motivated this:
- The cursor-blink feature (landed 2026-04-19, commits `73e2fd1..07d3d20`) shipped without an agent-runnable smoke. The manual smoke list in that plan asked a human to observe the cursor over time. Subagents can't do that.
- `capture.zig` already does single-frame offscreen rendering for golden-PNG tests (`tests/golden/scripts/*.vt` → `tests/golden/reference/*.png`), but it's one-shot: render the final state, diff, done. No time axis, no way to check blink phase at t=250 ms vs t=750 ms.
- The bench harness (`bench-baseline` / `bench-check`) measures per-frame timing but is driven by a static workload, not a scripted scenario.
The gap is a lightweight *scenario runner* in-process — not a new process, not a compositor harness, not an IPC channel. See `memory/project_scenario_runner_direction.md` for the brainstorm summary, including paths that were explicitly rejected (nested-compositor via `cage`, driver-socket IPC, in-process module-level harness).
## Non-goals
- **Testing wayland event handling.** Focus change, xkb keyboard, pointer, clipboard — none of these are exercisable because scenarios bypass the wayland input path entirely. A second harness flavor could add this later; not here.
- **Shell-in-the-loop.** Scenarios don't spawn a real interactive shell. If a scenario wants `\e[5 q` sent, it writes the bytes directly via the `bytes` directive. "Agent types a command and watches the shell run it" is a different problem (Problem 2 in the brainstorm, likely best solved by wrapping tmux).
- **Render-golden against the existing `.vt` + `capture.zig` harness.** Scenarios are a *superset* of that harness (they can express the same single-frame render), but we are not rewriting existing goldens. Both harnesses coexist.
- **Multi-frame streaming capture / event-triggered capture.** v1 is clock-based checkpoints only. Stream capture is a reasonable v2 extension — the bench stats already record per-frame data.
- **Scenario DSL features beyond a flat directive list.** No loops, variables, conditionals, macros, or interpolation. If a scenario needs logic, write it in Zig as a dedicated tool.
- **Non-ASCII bytes in the DSL beyond a small escape set.** v1 accepts `\n`, `\t`, `\r`, `\e` (0x1B), `\\`, `\"` and raw hex via `bytes-hex`. Full-Unicode string literals in the DSL are out of scope.
## Architecture
One new CLI flag, one new module, one integration hook in the main loop. No new dependencies.
```
┌──────────────────────────────────────────────────────────────┐
│ waystty --scenario <path> │
│ │
│ main.zig │
│ ├── parse --scenario flag → ScenarioState (from scenario.zig)│
│ ├── spawn Pty with a long-lived no-op child │
│ └── runTerminal(..., scenario_state) │
│ │ │
│ main loop iteration: │
│ │ │
│ │ waitForWork(pollfds, timeout) │
│ │ tickBlinkPhase + pty read + events + … │
│ │ ▶ scenario_state.tick(now_ns) ←── NEW │
│ │ emits pending directives: │
│ │ bytes → term.write(payload) │
│ │ capture → renderer offscreen + PNG + diff │
│ │ assert → cell-region predicate │
│ │ if (!render_pending) continue; │
│ │ … existing render path … │
│ │ │
│ │ exit when scenario_state.isDone() │
│ │
└──────────────────────────────────────────────────────────────┘
```
**Files:**
- *New* `src/scenario.zig` — scenario parser (line-based DSL), `ScenarioState` struct, tick function, predicate evaluator. Pure where possible; offscreen-render is delegated.
- *New* `src/imgdiff.zig` — pure-math RMSE + max-pixel diff, extracted from the current `src/tools/imgdiff.zig` so both the CLI tool and `ScenarioState` can call it. (Renamed from the earlier working title `scenario_diff.zig` for consistency with the existing CLI name.)
- *Modify* `src/main.zig` — parse new `--scenario` arg (mirrors existing `--capture` arg handling), plumb `ScenarioState` into `runTerminal`, add one tick call per main-loop iteration, adjust loop-exit condition.
- *Modify* `build.zig` — wire `scenario.zig` as a module; add a test runner entry for scenario self-tests.
- *New* `tests/scenarios/*.scenario` — scenario fixtures.
- *New* `tests/scenarios/golden/<scenario-name>/<label>.png` — reference frames.
- *New* `tests/scenarios/out/<scenario-name>/<label>.png` + `.diff.png` — failure outputs (gitignored).
- *Modify* `Makefile` — new `scenario`, `scenario-update` targets; `test` target calls `scenario`.
**Why extend `runTerminal` rather than add a parallel mode (like `capture.zig` does):** fidelity. Blink timer, frame-callback throttling, surface-suspend handling, DECSCUSR parsing, atlas upload — all live inside `runTerminal`. A parallel mode (`src/scenario_mode.zig run(...)`) would inevitably drift. One tick call inside the existing loop is cheap and keeps one source of truth for main-loop behavior.
**No dummy child process:** the pty itself is part of waystty's real runtime surface (selection, reported size, TIOCSWINSZ) and stays. But the `p.isChildAlive()` check in the loop guard at `src/main.zig:344` is relaxed to `(scenario_state == null and p.isChildAlive()) or (scenario_state != null and !scenario_state.?.isDone())`. A live pty with no spawned child is supported via a small `Pty.openWithoutChild()` or equivalent — pty is open, no fork, the main loop just never reads any bytes. Rejected a `sleep 3650000` child on portability and zombie-accumulation grounds.
## Scenario file format
Line-based directives. One directive per line. Lines starting with `#` (optionally after whitespace) are comments. Blank lines ignored.
```
# tests/scenarios/blink-bar.scenario
size 80 24
timeout 5000ms
# blinking bar cursor
bytes "\e[5 q"
sleep 250ms
capture on-phase # t=250ms; cursor on
sleep 500ms
capture off-phase # t=750ms; cursor off (one flip past)
assert-cell 0 0 cursor-visible-at on-phase
assert-cell 0 0 cursor-absent-at off-phase
```
**Directives (v1):**
| Directive | Meaning |
|---|---|
| `size COLS ROWS` | Required first directive. **v1 informational only** — scenario runner pins to 80×24 (matching `capture.zig`) and rejects scenarios whose declared size differs. Making the size actually variable requires plumbing through the wayland-window-sizing path that `capture.zig` also hardcodes, which is a separate refactor. The directive exists in the DSL so scenarios don't need rewriting when that refactor happens. |
| `timeout MS` | Required. Hard wall-clock cap on total scenario runtime. Scenario fails if the cumulative sleep + directive execution exceeds this. |
| `sleep MS` | Advance the monotonic timeline by `MS` ms before the next directive executes. Runs real wall-clock sleeps (not fake clocks) so the blink timer fires naturally. Real clock = inherits real flakiness; see **Timing tolerance** below. |
| `sleep-until-flip` | Advance until the blink timer's next phase flip actually fires, then return. Phase-aligned rendezvous — avoids capturing a frame straddling the flip. Fails with a descriptive error if blink is not currently armed (scenario wrote no DECSCUSR to enable blink, etc.). |
| `bytes "STR"` | Inject bytes directly into `term.write()`. DSL escape set: `\n` `\t` `\r` `\e` `\\` `\"`. No `printf`-style `%` processing, no interpolation. |
| `bytes-hex 1B 5B 35 20 71` | Inject raw bytes by hex. Useful when scenarios need bytes outside the DSL escape set. |
| `capture LABEL` | Trigger an offscreen render + PNG write to `tests/scenarios/out/<scenario>/<label>.png`. Diff against `tests/scenarios/golden/<scenario>/<label>.png`. On mismatch: write diff heatmap to `<label>.diff.png` and mark scenario failed. Under `WAYSTTY_SCENARIO_UPDATE=1`: rewrite the golden instead of diffing. |
| `assert-cell R C PRED` | Cell-region predicate on the *most recent capture*. Predicates (v1): `cell-matches-golden` (workhorse — diffs the cell's pixel-rect against the corresponding rect in `golden/<scenario>/<label>.cell-R-C.png`, auto-captured on golden-update runs), `cursor-block-at` / `cursor-bar-at` / `cursor-underline-at` (explicit shape checks: centroid of bright pixels in the cell must fall within the bbox a cursor of that shape would occupy), `cell-empty` (no pixel in the cell differs from the terminal's default-bg beyond a tolerance). No lax `cursor-visible` predicate — too noise-prone at 2 px line width. |
| `assert-cell-at LABEL R C PRED` | Same as above but evaluates against a named prior capture instead of the most recent. |
Parser: strict. Unknown directives fail with a clear error message including the line number. No leniency — scenarios are meant to be read by humans, so typos shouldn't silently pass.
## Data flow
### Tick integration
`ScenarioState.tick(now_ns)` is called once per main-loop iteration, immediately after the blink phase check and before the `if (!render_pending) continue` bail:
```
// inside runTerminal, per iteration:
try frame_loop.waitForWork(&pollfds_extra, timeout);
// ... tickBlinkPhase + pty read + event processing ...
if (scenario_state) |s| {
try s.tick(std.time.nanoTimestamp(), &term, &ctx, &frame_ring);
if (s.isDone()) window.should_close = true;
}
if (!render_pending) continue;
```
**Why no `render_pending = true` after capture:** `capture` performs an offscreen render *inside* `tick()`. Setting `render_pending = true` would then fire a separate on-screen render this same iteration against the same terminal state — wasted work, and worse, it changes which frame "counts" as the logical capture (the blink timer could advance between the two renders). The offscreen render is the capture and nothing else.
`tick` implementation:
```
fn tick(self: *ScenarioState, now_ns: i128, term: *Terminal, ctx: *renderer.Context, ring: *FrameTimingRing) !void {
while (self.cursor < self.directives.len) {
const d = self.directives[self.cursor];
if (now_ns < self.timeline_origin_ns + d.scheduled_offset_ns) return;
try self.execute(d, term, ctx, ring);
self.cursor += 1;
}
}
```
`scheduled_offset_ns` is computed at parse time by walking `sleep` directives — each `sleep N` bumps the offset for everything that follows. `bytes` and `assert-cell` are zero-duration; `capture` takes a small real render cost but is not counted as timeline advance.
### Capture execution
`capture` reuses the public surface already present in `renderer.zig`:
```
pub fn createOffscreen(...) !OffscreenTarget;
pub fn renderToOffscreen(self: *Context, ...) !void;
pub fn readbackOffscreen(self: *Context, ...) !void;
```
Scenario runner calls these exactly as `capture.zig` does for its single-frame case. The offscreen target is created lazily on the first `capture` directive and reused across subsequent captures in the same scenario (allocating a fresh VkImage per capture is wasteful).
PNG encode uses the existing `png.zig` module. Golden diff uses a shared library extraction of the RMSE + max-pixel math currently in `src/tools/imgdiff.zig` — move the pure-math parts into `src/imgdiff.zig` so both the CLI tool and `scenario.zig` can call them.
### Bench data
Per-scenario timing dump is automatic. At scenario end, the `FrameTimingRing` is serialized via the existing `bench_stats.zig` JSON path to `tests/scenarios/out/<scenario>/bench.json`. Agents can assert on p50/p99/mean via a separate `bench-check`-style command in a follow-up; v1 just writes the file.
## Timing tolerance
Scenarios rely on real wall-clock sleeps, and real clocks are noisy on loaded CI. Two mitigations:
1. **`sleep-until-flip`** (defined above) for scenarios that care about blink-phase boundaries. Rather than sleeping N ms and hoping the flip has happened, the scenario rendezvouses with the actual flip event. Implementation: `tick()` observes blink-state transitions by comparing `blink_state.blink_on` before and after each main-loop iteration. When `sleep-until-flip` is the current directive, `tick()` blocks on that transition (with a 2× `blink_period_ns` overall cap so a broken scenario doesn't hang forever).
2. **Capture retry within a tolerance window.** A `capture` directive has a small built-in tolerance: if the first diff against golden misses by less than `rmse ≤ 2× threshold`, the runner waits `floor(frame_period_ns / 2)` and re-captures once. Second miss is a failure. This paves over single-frame anti-aliasing noise without giving up real regression detection. Threshold constants live in `src/imgdiff.zig` for easy tuning.
Scenarios that can't tolerate either (e.g. "exactly this frame") aren't expressible in v1 — write a dedicated unit test.
## Error handling
Failure modes and their contracts:
- **Parse error.** Reported as `scenario: <file>:<line>: <message>` to stderr, exit code 2. No partial execution.
- **Scheduled-timeline overshoot.** If `tick()` ever sees `now_ns` > `origin + cumulative_sleep + timeout`, the scenario fails with `scenario: <name>: timeout exceeded after <elapsed>ms`, exit code 3.
- **Capture PNG write fails.** Scenario fails at that directive with `scenario: <name>: capture <label>: <underlying IO error>`.
- **Golden diff mismatch.** Scenario records the failure and continues (so later assertions also run), then exits non-zero at the end. The actual frame and diff heatmap are written to `out/`.
- **Cell predicate fails.** Same continue-then-exit-nonzero semantics.
- **Vulkan error during scenario render.** Propagates up the existing bounded-wait error path (`vk_sync` returns `VkWaitTimeout` etc.). Scenario reports `scenario: <name>: render error: <tag>` and fails with a distinct exit code so CI can tell "flaky vulkan" apart from "real regression."
Exit codes:
- 0: all scenarios passed.
- 2: parse error in any scenario.
- 3: wall-clock timeout.
- 4: capture or assertion mismatch (regression).
- 5: Vulkan `VkWaitTimeout` / `VkAcquireTimeout` during scenario render. CI should treat 5 as "flake, retry once" distinct from 4.
- 6: other render / I/O error (OutOfMemory, file I/O, swapchain unrecoverable).
The runner never mutates the repo unless `WAYSTTY_SCENARIO_UPDATE=1`. With that env set, mismatches rewrite goldens and exit 0 (mirrors the existing `WAYSTTY_GOLDEN_UPDATE=1` convention).
## Testing
Three layers:
1. **Parser unit tests** in `src/scenario.zig` inline tests:
- Every directive round-trips (lex + parse + serialize back via a small dumper for test readability).
- Unknown directive → parse error at correct line.
- Escape set correctness (`\e` → 0x1B, `\n` → 0x0A, `\\` → 0x5C, etc.).
- `bytes-hex` accepts mixed-case, rejects odd-length input and non-hex characters.
- Empty scenario and minimum-valid scenario both round-trip.
2. **Tick-function unit tests** in `src/scenario.zig`:
- Directives with `scheduled_offset_ns` in the future are held.
- Past-due directives all fire in order on a single tick.
- `isDone()` returns true only after last directive executed (not just scheduled).
- `tick` does not panic on empty directive list.
3. **Scenario-level integration** in `tests/scenarios/`:
- `smoke-hello.scenario` — writes a single character, captures, asserts cell-solid-bright. Covers the basic render path without any time axis.
- `blink-bar.scenario` — the DECSCUSR-5 example above. Covers time-based behavior.
- `shape-matrix.scenario` — cycles through `\e[1 q` through `\e[6 q`, captures each, diffs golden. Exercises DECSCUSR branches.
- Run via `make scenario` (new make target). CI fails on any non-zero exit.
**Not tested in v1:**
- Predicates beyond the four listed above.
- Concurrent scenarios (runner is single-scenario at a time; multi-scenario is a higher-level shell wrapper).
- Render timing regression (bench.json is written but not asserted against; future `scenario-bench-check`).
## Make targets
```
make scenario # run all scenarios, fail on any mismatch
make scenario-one NAME=foo # run a single scenario (pass-through to tool)
make scenario-update # WAYSTTY_SCENARIO_UPDATE=1 make scenario
```
`make test` chains to `make scenario` so CI picks it up automatically.
## Forward-looking notes
Filed here so future contributors don't need to rediscover:
- **Stream capture** (continuous per-frame PNG dump) is additive — `FrameTimingRing` already records per-frame data, extending to per-frame PNG writeback is a `--stream` flag on `capture`.
- **Cell predicate expansion** — `cursor-visible` / `cursor-absent` / `solid-*` is minimal. If a new predicate is needed, add it to the `Pred` enum and the evaluator, then document in the table above.
- **Scenario generation by agents** — scenarios are plain text, so an agent can author them, run, iterate. No need for a separate authoring tool.
- **Multi-output scenarios** (one scenario producing many PNGs) already work via multiple `capture` directives. No API change needed for that.
- **Testing wayland event paths** (the explicitly out-of-scope item) — if we need this, the likely shape is a second flavor of scenario where the tick function synthesizes `wl_keyboard`/`wl_pointer` events against the dispatcher. Same DSL, different harness variant.
## Implementation phasing (for the plan)
Spec is one coherent feature; implementation decomposes into three independently-landable plans:
1. **`src/imgdiff.zig` extraction.** Move pure-math RMSE + max-pixel from `src/tools/imgdiff.zig` into a module, keep the CLI tool as a thin wrapper. Landable standalone; no behavior change for existing `test_render`.
2. **`src/scenario.zig` parser + state + tick (pure, no integration).** All helpers unit-tested with a fake clock. `isDone()`, directive scheduling, predicate evaluation — all pure. No `main.zig` touch yet.
3. **Main-loop integration + first fixtures.** `--scenario` flag, `runTerminal` hook, one `blink-bar.scenario` fixture with golden PNGs, `make scenario` target. This plan is the visible landing; failure of plan 1 or 2 shouldn't force rework here.
Each plan gets its own `docs/superpowers/plans/YYYY-MM-DD-*.md`. The writing-plans skill invocation after this spec approval will draft plan 1 first.
## Files touched (rough)
- `src/main.zig` — `--scenario` arg parsing, pty dummy-child spawn, `runTerminal` gains `scenario_state: ?*ScenarioState` param, one-line `tick` call in loop, exit condition.
- `src/scenario.zig` — parser, state, tick, predicate evaluator (~500 LOC estimate).
- `src/imgdiff.zig` — new public module extracted from `src/tools/imgdiff.zig` (pure diff math; existing CLI tool wraps it).
- `src/tools/imgdiff.zig` — thin wrapper over new `src/imgdiff.zig`.
- `build.zig` — wire new modules.
- `Makefile` — new targets.
- `tests/scenarios/` — fixtures and goldens.
- No changes to `src/vk_sync.zig`, `src/renderer.zig`, `src/frame_loop.zig`, `src/vt.zig`, `src/wayland.zig` (except possibly tiny additions if the scenario tick needs a hook).