Recover from Vulkan fence/acquire timeout + device-lost

open   by a73x

Labels: review

[claude 2026-04-18] Diagnosed during today's debugging — observed twice within a few hours of normal daily-driver use.

## Observed symptom

Window visible and frozen on last-presented frame: keystrokes ignored, resize events ignored, no dispatch. Process stays alive indefinitely (first freeze was 11h52m at kill time; second occurred shortly after).

## Root cause

Main thread wedged in `vkWaitForFences(..., timeout=UINT64_MAX)` inside the NVIDIA 595.58.03 driver. Backtrace (identical on both occurrences, byte-for-byte in waystty frames):

```
main.runTerminal
  renderer.Context.drawCells
    vk.DeviceWrapper.waitForFences(..., timeout=18446744073709551615)
      libnvidia-glcore internal poll() ← wedged
```

The NVIDIA driver drops a fence signal for `in_flight_fence` — the GPU work likely completes but the kernel syncobj / sync fd the driver polls never gets notified. Since our wait has no timeout, the main loop blocks permanently: no input dispatch, no resize, no exit.

Not a waystty logic bug in isolation — a Vulkan-tutorial antipattern (`UINT64_MAX`) meeting a real-world driver flake.

## Offending sites (all `std.math.maxInt(u64)`)

- `src/renderer.zig:1275` — `drawClear` fence wait
- `src/renderer.zig:1282` — `drawClear` acquireNextImageKHR
- `src/renderer.zig:1478` — atlas transfer fence wait
- `src/renderer.zig:1710` — `drawCells` fence wait (wedged in both freezes observed today)
- `src/renderer.zig:1721` — `drawCells` acquireNextImageKHR
- `src/renderer.zig:1818` — capture path fence wait
- `src/renderer.zig:1824` — capture fence wait

## Proposed scope — full device-lost recovery

1. **Bounded timeouts.** ~2s for fence waits, ~100ms for acquireNextImageKHR.
2. **Classify errors**: `VK_TIMEOUT`, `VK_ERROR_OUT_OF_DATE_KHR`, `VK_ERROR_DEVICE_LOST`.
3. **On first timeout**: log, skip the frame, retry next iteration. Don't spam.
4. **On N consecutive timeouts or `VK_ERROR_DEVICE_LOST`**: tear down and recreate GPU resources —
   - Swapchain (existing path)
   - All fences (`in_flight_fence`, `atlas_transfer_fence`, `capture_fence`)
   - Pipeline + render pass (if device-lost)
   - Atlas GPU image + instance buffer (CPU-side glyph cache survives; reupload on demand)
   - The `VkDevice` itself on true device-lost
5. **State preservation**: `vt.Terminal` grid + font atlas CPU cache live in RAM and must not be touched by recovery. After recovery, mark the terminal fully dirty and re-render.
6. **User-visible signal**: one-line stderr warning when recovery fires, so a repeated driver flake becomes observable rather than invisible.

## Non-goals (v1)

- Exotic swapchain-lost paths (surface lost, exclusive full-screen, etc.) — handle only the fence-timeout and `DEVICE_LOST` cases that match observed symptom.
- Persisting scrollback across a recovery — stays in RAM, survives naturally.

## Related

- #7b6053b0 — opt-in file logging for freeze/debug diagnosis. Complementary: logging diagnoses future wedges, this ticket survives them.

## Next step

Needs a design (brainstorm → spec → plan). Recovery touches renderer ownership + GPU-resource lifetime; not a one-file change.

Comments

[claude 2026-04-18] Implementation complete on branch feat/vulkan-bounded-waits (worktree at .worktrees/vulkan-bounded-waits).

Spec: docs/superpowers/specs/2026-04-18-vulkan-bounded-waits-design.md
Plan: docs/superpowers/plans/2026-04-18-vulkan-bounded-waits.md

Eight commits (six implementation + two review-fix follow-ups):

- Add vk_sync module with bounded Vulkan wait helpers
- vk_sync: fail closed on unknown acquire result + comments
- renderer: reorder resetFences after acquire, add errdefer re-signal
- Migrate 10 Vulkan wait/acquire sites to bounded helpers
- renderer: gate fence re-signal on pre-submit errors only
- Migrate 14 *WaitIdle sites to vk_sync.waitIdleForShutdown
- Add grep gate that forbids unbounded Vulkan waits outside vk_sync
- check_unbounded_vk: comment the PATTERNS list as maintainer-update target
- main: add exponential backoff on Vulkan timeout retries

zig build test passes; grep gate clean (16 files, no violations).

Follow-up filed: 0e342b6e — audit five mid-flight waitIdleForShutdown sites for bounded semantics (resize, scale, OutOfDateKHR, uploadAtlas init).