a73x/waystty
Watch
1
0
Fork
You've already forked waystty
 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity Actions

Recover from Vulkan fence/acquire timeout + device-lost #13

New issue
Open
opened 2026-04-18 09:16:13 +00:00 by a73x · 1 comment
a73x commented 2026-04-18 09:16:13 +00:00 (Migrated from git.a73x.sh)
Copy link

[claude 2026-04-18] Diagnosed during today's debugging — observed twice within a few hours of normal daily-driver use.

Observed symptom

Window visible and frozen on last-presented frame: keystrokes ignored, resize events ignored, no dispatch. Process stays alive indefinitely (first freeze was 11h52m at kill time; second occurred shortly after).

Root cause

Main thread wedged in vkWaitForFences(..., timeout=UINT64_MAX) inside the NVIDIA 595.58.03 driver. Backtrace (identical on both occurrences, byte-for-byte in waystty frames):

main.runTerminal
  renderer.Context.drawCells
    vk.DeviceWrapper.waitForFences(..., timeout=18446744073709551615)
      libnvidia-glcore internal poll() ← wedged

The NVIDIA driver drops a fence signal for in_flight_fence — the GPU work likely completes but the kernel syncobj / sync fd the driver polls never gets notified. Since our wait has no timeout, the main loop blocks permanently: no input dispatch, no resize, no exit.

Not a waystty logic bug in isolation — a Vulkan-tutorial antipattern (UINT64_MAX) meeting a real-world driver flake.

Offending sites (all std.math.maxInt(u64))

  • src/renderer.zig:1275 — drawClear fence wait
  • src/renderer.zig:1282 — drawClear acquireNextImageKHR
  • src/renderer.zig:1478 — atlas transfer fence wait
  • src/renderer.zig:1710 — drawCells fence wait (wedged in both freezes observed today)
  • src/renderer.zig:1721 — drawCells acquireNextImageKHR
  • src/renderer.zig:1818 — capture path fence wait
  • src/renderer.zig:1824 — capture fence wait

Proposed scope — full device-lost recovery

  1. Bounded timeouts. ~2s for fence waits, ~100ms for acquireNextImageKHR.
  2. Classify errors: VK_TIMEOUT, VK_ERROR_OUT_OF_DATE_KHR, VK_ERROR_DEVICE_LOST.
  3. On first timeout: log, skip the frame, retry next iteration. Don't spam.
  4. On N consecutive timeouts or VK_ERROR_DEVICE_LOST: tear down and recreate GPU resources —
    • Swapchain (existing path)
    • All fences (in_flight_fence, atlas_transfer_fence, capture_fence)
    • Pipeline + render pass (if device-lost)
    • Atlas GPU image + instance buffer (CPU-side glyph cache survives; reupload on demand)
    • The VkDevice itself on true device-lost
  5. State preservation: vt.Terminal grid + font atlas CPU cache live in RAM and must not be touched by recovery. After recovery, mark the terminal fully dirty and re-render.
  6. User-visible signal: one-line stderr warning when recovery fires, so a repeated driver flake becomes observable rather than invisible.

Non-goals (v1)

  • Exotic swapchain-lost paths (surface lost, exclusive full-screen, etc.) — handle only the fence-timeout and DEVICE_LOST cases that match observed symptom.
  • Persisting scrollback across a recovery — stays in RAM, survives naturally.

Related

  • #7b6053b0 — opt-in file logging for freeze/debug diagnosis. Complementary: logging diagnoses future wedges, this ticket survives them.

Next step

Needs a design (brainstorm → spec → plan). Recovery touches renderer ownership + GPU-resource lifetime; not a one-file change.

Migrated from git-collab issue ab6c92f0a65d60f5c919990eb9f73a94c98fcdf8.

Original author: a73x dev@a73x.sh.

[claude 2026-04-18] Diagnosed during today's debugging — observed twice within a few hours of normal daily-driver use. ## Observed symptom Window visible and frozen on last-presented frame: keystrokes ignored, resize events ignored, no dispatch. Process stays alive indefinitely (first freeze was 11h52m at kill time; second occurred shortly after). ## Root cause Main thread wedged in `vkWaitForFences(..., timeout=UINT64_MAX)` inside the NVIDIA 595.58.03 driver. Backtrace (identical on both occurrences, byte-for-byte in waystty frames): ``` main.runTerminal renderer.Context.drawCells vk.DeviceWrapper.waitForFences(..., timeout=18446744073709551615) libnvidia-glcore internal poll() ← wedged ``` The NVIDIA driver drops a fence signal for `in_flight_fence` — the GPU work likely completes but the kernel syncobj / sync fd the driver polls never gets notified. Since our wait has no timeout, the main loop blocks permanently: no input dispatch, no resize, no exit. Not a waystty logic bug in isolation — a Vulkan-tutorial antipattern (`UINT64_MAX`) meeting a real-world driver flake. ## Offending sites (all `std.math.maxInt(u64)`) - `src/renderer.zig:1275` — `drawClear` fence wait - `src/renderer.zig:1282` — `drawClear` acquireNextImageKHR - `src/renderer.zig:1478` — atlas transfer fence wait - `src/renderer.zig:1710` — `drawCells` fence wait (wedged in both freezes observed today) - `src/renderer.zig:1721` — `drawCells` acquireNextImageKHR - `src/renderer.zig:1818` — capture path fence wait - `src/renderer.zig:1824` — capture fence wait ## Proposed scope — full device-lost recovery 1. **Bounded timeouts.** ~2s for fence waits, ~100ms for acquireNextImageKHR. 2. **Classify errors**: `VK_TIMEOUT`, `VK_ERROR_OUT_OF_DATE_KHR`, `VK_ERROR_DEVICE_LOST`. 3. **On first timeout**: log, skip the frame, retry next iteration. Don't spam. 4. **On N consecutive timeouts or `VK_ERROR_DEVICE_LOST`**: tear down and recreate GPU resources — - Swapchain (existing path) - All fences (`in_flight_fence`, `atlas_transfer_fence`, `capture_fence`) - Pipeline + render pass (if device-lost) - Atlas GPU image + instance buffer (CPU-side glyph cache survives; reupload on demand) - The `VkDevice` itself on true device-lost 5. **State preservation**: `vt.Terminal` grid + font atlas CPU cache live in RAM and must not be touched by recovery. After recovery, mark the terminal fully dirty and re-render. 6. **User-visible signal**: one-line stderr warning when recovery fires, so a repeated driver flake becomes observable rather than invisible. ## Non-goals (v1) - Exotic swapchain-lost paths (surface lost, exclusive full-screen, etc.) — handle only the fence-timeout and `DEVICE_LOST` cases that match observed symptom. - Persisting scrollback across a recovery — stays in RAM, survives naturally. ## Related - #7b6053b0 — opt-in file logging for freeze/debug diagnosis. Complementary: logging diagnoses future wedges, this ticket survives them. ## Next step Needs a design (brainstorm → spec → plan). Recovery touches renderer ownership + GPU-resource lifetime; not a one-file change. --- Migrated from git-collab issue `ab6c92f0a65d60f5c919990eb9f73a94c98fcdf8`. Original author: a73x <dev@a73x.sh>.
a73x commented 2026-04-18 15:28:30 +00:00 (Migrated from git.a73x.sh)
Copy link

[claude 2026-04-18] Implementation complete on branch feat/vulkan-bounded-waits (worktree at .worktrees/vulkan-bounded-waits).

Spec: docs/superpowers/specs/2026-04-18-vulkan-bounded-waits-design.md
Plan: docs/superpowers/plans/2026-04-18-vulkan-bounded-waits.md

Eight commits (six implementation + two review-fix follow-ups):

  • Add vk_sync module with bounded Vulkan wait helpers
  • vk_sync: fail closed on unknown acquire result + comments
  • renderer: reorder resetFences after acquire, add errdefer re-signal
  • Migrate 10 Vulkan wait/acquire sites to bounded helpers
  • renderer: gate fence re-signal on pre-submit errors only
  • Migrate 14 *WaitIdle sites to vk_sync.waitIdleForShutdown
  • Add grep gate that forbids unbounded Vulkan waits outside vk_sync
  • check_unbounded_vk: comment the PATTERNS list as maintainer-update target
  • main: add exponential backoff on Vulkan timeout retries

zig build test passes; grep gate clean (16 files, no violations).

Follow-up filed: 0e342b6e — audit five mid-flight waitIdleForShutdown sites for bounded semantics (resize, scale, OutOfDateKHR, uploadAtlas init).

[claude 2026-04-18] Implementation complete on branch feat/vulkan-bounded-waits (worktree at .worktrees/vulkan-bounded-waits). Spec: docs/superpowers/specs/2026-04-18-vulkan-bounded-waits-design.md Plan: docs/superpowers/plans/2026-04-18-vulkan-bounded-waits.md Eight commits (six implementation + two review-fix follow-ups): - Add vk_sync module with bounded Vulkan wait helpers - vk_sync: fail closed on unknown acquire result + comments - renderer: reorder resetFences after acquire, add errdefer re-signal - Migrate 10 Vulkan wait/acquire sites to bounded helpers - renderer: gate fence re-signal on pre-submit errors only - Migrate 14 *WaitIdle sites to vk_sync.waitIdleForShutdown - Add grep gate that forbids unbounded Vulkan waits outside vk_sync - check_unbounded_vk: comment the PATTERNS list as maintainer-update target - main: add exponential backoff on Vulkan timeout retries zig build test passes; grep gate clean (16 files, no violations). Follow-up filed: 0e342b6e — audit five mid-flight waitIdleForShutdown sites for bounded semantics (resize, scale, OutOfDateKHR, uploadAtlas init).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
backlog

   
review
No labels
backlog
review
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
a73x/waystty#13
No description provided.