No way to see WHY a prediction stalled: capture a profile on a latency spike

open   by a73x

Labels: backlog, observability, client, perf

[claude 2026-08-23] Reported by the operator.

## Symptom

Local session, no WAN. In vim: hit ESC, then `b` a few times. Three
underlined `bbb` sit on screen, then everything catches up at once. No
insight into what happened — was the daemon stalled, or did the overlay
simply hold a doomed prediction until its deadline?

## What exists today

Aggregate counters only. `predict.Counters` (src/predict.zig:95) tracks
made / confirmed / contradicted / expired / suppressed, and the client
prints them once at exit (`dumpPredictStats`, src/interact.zig:1383); the
wall republishes the zoomed tile's set each pass (src/wallview.zig:1514).

Not one of those carries TIME. There is no per-prediction age, no ingress ->
egress timeline, nothing that distinguishes a slow daemon from a prediction
that was never going to be confirmed. `predict.zig` is deliberately
clock-free — it takes `now_ms` from the caller (src/predict.zig:172) — so
the instrumentation belongs at the caller, not in the module.

## Leading hypothesis for the vim case (needs the tooling to confirm)

Possibly not a latency spike at all:

- ESC is refused by `predictAt` (`k.ch < 0x20`, src/predict.zig:334) — but
  `suppress()` only counts the refusal. It does NOT flush the queue and does
  NOT drop `confident`. So confidence earned while typing in insert mode
  still stands after ESC.
- Each `b` is printable, so it IS predicted at the cursor, and displayed
  because the overlay is still confident.
- In normal mode vim consumes `b` and moves the cursor; the predicted cell
  never changes to `b`. Under the "not yet is not wrong" rule
  (src/predict.zig:17) the prediction is not refuted — it stays pending
  until `expire_after_frames` (8) or `expire_after_ms` (1000,
  src/predict.zig:152-156) fires.
- Then all three expire together: "suddenly everything catches up".

If that is what is happening, the delay is the designed 1s bound, and the
real bug is a policy one (a mode-changing key should cost confidence, or
demote). The point of this issue is that TODAY WE CANNOT TELL.

## The ask

An opt-in mechanism that captures a profile when prediction delay spikes.
Sketch, not a decision:

- Threshold on a pending prediction's age (or on the interval between
  `made` and its verdict). Crossing it arms one capture.
- The record must answer "who was slow": pending age, frames judged,
  `made_seq` vs the applied seq, the verdict that ended it (confirmed /
  contradicted / expired), and the overlay's policy state at the time
  (`ctx`, `confident`, `streak`) — the last three are what separate the
  hypothesis above from a genuine stall.
- Daemon side, a matching timeline for the same keystroke: input frame
  ingress -> pty write -> pty read -> engine feed -> delta egress. Local
  means a unix socket, so any multi-hundred-ms gap is a stall inside a
  process, not on the wire.
- Bounded and off by default: a capture per spike with a rate limit, not a
  continuous trace. Write it somewhere findable — the daemon log, or under
  `$XDG_STATE_HOME/mux/`.

## Trap for whoever picks this up

The record MUST stamp the build mode. `zig build` is Debug and Debug builds
ghostty with its page-integrity check on — ~600x on output-heavy work
(CLAUDE.md, and [[mux-debug-builds-are-600x-slow]]). A profile captured in
the dev tree that does not say so will indict the wrong code.

Comments

[claude 2026-08-23] Repro correction from the operator: it is not vim. It is
the Claude Code input box WITH VIM MODE ENABLED — a TUI that repaints its own
frame per keystroke. The keystroke mechanism is the same (normal-mode `b`,
`w`, `j` are non-echoing printable bytes, predicted and displayed on
confidence earned in insert mode, retired only at expire_after_ms), but the
cost is worse: the delta for one keystroke is already a frame-sized repaint
from the TUI, and a contradiction adds `renderClipped` over the whole
viewport on top of it (src/interact.zig:1691).

Also reported: typing RAPIDLY into that box stalls. Candidate that is not a
latency spike at all — `offerPrediction` refuses any chunk where
`chunk.len != 1` (src/interact.zig:961). Fast typing coalesces at the tty, so
a burst arrives as one multi-byte read and gets zero prediction. The faster
you type, the less prediction you get.

Control not yet run: does the same rapid typing stall in Claude Code OUTSIDE
mux? If it does, prediction is a red herring and the profile should be
pointed at the TUI's own render cost.

[claude 2026-08-23] First increment landed on branch worktree-predict-local-gate
(aac3c2d): the overlay measures its own keystroke-to-confirm round trip and
hides predictions below 20ms srtt (shows again above 30ms; mosh's pair). On a
local socket nothing underlined is ever painted, so the "bbb" cannot occur.
Predictions stay queued and judged while hidden, and the stats line has a
new `local=` counter. make check + full e2e green; installed to ~/.local/bin
for the operator's hands-on.

Still open from this thread, deliberately deferred: the full-screen repaint
on expiry fires even when nothing was painted (per-cell rollback, item 3),
ESC-demotes-confidence, coalesced bursts get zero prediction, and the
profile capture this issue is actually about.