b8b8af9c

docs: M9 prediction — the record

a73x 2026-08-08 17:23

Commit message

docs: M9 prediction — the record

The verdict per leg, with the numbers beside the path they were taken on:
predicted paint 0.1ms median against a measured 150.5ms round trip, and the
control — unpredicted input on the same connection in the same run — at
154.2ms. One keystroke, two clocks, which is what makes the first number
evidence rather than an observation about how fast a local paint is.

Two notes about the criterion itself, since this milestone was the one that
had to write one. Its literal wording was unmeasurable: an "echo-off
context" paints nothing, so there is no arrival to time, and the measurable
form of the same claim is input prediction refuses. And a control that
cannot fail proves nothing, so the harness now fails leg 1 when the control
comes back fast — the floor-form philosophy applied to a control rather
than a threshold. Floor form itself is recorded as the standard this
discharges, including that M6's reattach gate still fails structurally on a
fast link and is now owed the same rewrite retroactively.

The re-entrancy defect at full weight: found by following a leaked temp
directory to an abort, root-caused by audit, and — the part that revised
the theory — proven by a mutation that fires immediately rather than
rarely. It was happening on every QUIC exchange; only the consequence was
rare. Recorded with the honesty it needs: the original transient was never
reproduced and its test name never captured, so the causal link is strong
inference and not proof, and three of the hardening items are reasoned
rather than mutation-pinned.

Also the readline finding (the everyday prompt is the adaptive tier, so
`.always` covers canonical readers only, which is why /bin/cat is the
session shell wherever that tier is tested), reconcile v2's reason to exist,
the burst/expiry ceiling that bounds where these numbers reproduce, leg 2's
convergence narrowing stated plainly against the plan's stronger wording,
and two named traps: "a write is not a keystroke" with its three sightings,
and the sixth sighting of pgrep -f matching itself — which happened while
checking for that very hazard.

Roadmap: M9 complete with a one-line verdict, prediction polish banked as a
group at the head of the queue, trial feedback still outranking all of it.
README: the status paragraph gains the 0.1ms-vs-154ms pair and drops
prediction from the not-yet list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

README.md

Old		New
@@ -110,7 +110,7 @@ Design: `docs/handoff.md`. Every decision and measurement:
110		110
111	## Status	111	## Status
112		112
113	Prototype, eight milestones in, first trial cut tagged `v0.0.1-1`. All	113	Prototype, nine milestones in, first trial cut tagged `v0.0.1-1`. All
114	founding kill criteria cleared, measured on real networks rather than	114	founding kill criteria cleared, measured on real networks rather than
115	loopback: reattach into a live full-screen `nvim` in ~5ms from one	115	loopback: reattach into a live full-screen `nvim` in ~5ms from one
116	snapshot; steady-state delta traffic ~1% of snapshot-equivalent cost; the	116	snapshot; steady-state delta traffic ~1% of snapshot-equivalent cost; the
@@ -118,5 +118,9 @@ protocol adds ~4ms to a raw ssh round trip (constant, not a multiple of
118	RTT); ten consecutive real network tears resumed hands-off, all	118	RTT); ten consecutive real network tears resumed hands-off, all
119	delta-served, on both ssh and QUIC transports; QUIC cold attach beats	119	delta-served, on both ssh and QUIC transports; QUIC cold attach beats
120	ssh-via (6.9ms vs 9.8ms clean, 234.1ms vs 235.7ms with 75ms emulated	120	ssh-via (6.9ms vs 9.8ms clean, 234.1ms vs 235.7ms with 75ms emulated
121	delay on a real path). Not yet here: daemon auto-start on attach,	121	delay on a real path); and a predicted keystroke paints in 0.1ms on a
122	predictive local echo for high-RTT links, reconnect backoff tuning.	122	150ms path where the same keystroke's authoritative echo takes 154ms —
		123	typing stops waiting for the round trip, and the two numbers come from one
		124	keystroke on one connection so the gap is not an artefact of comparing
		125	runs. Not yet here: daemon auto-start on attach, reconnect backoff tuning,
		126	prediction for backspace and for multi-byte input.

docs/decisions.md

@@ @@ -942,6 +942,304 @@ loop. @@
+  and possibly worth a dedup pass across the three arms now that there are
+  three.
+## 2026-08-08 (M9)
+Speculative local echo. The client paints a predicted glyph for a printable
+keystroke immediately, underlined, and reconciles it against the daemon's
+authoritative delta when that arrives. Echo latency stops being a function
+of the round trip where prediction applies — and is provably absent where it
+must not.
+### The verdict, per leg
+Measured on the LAN box with `netem delay 150ms` on the interface facing the
+client, round trip measured at **150.5ms** (not assumed — the baseline is
+printed beside the result).
+| | min | med | max | n |
+|---|---|---|---|---|
+| baseline (the path) | 150.4 | 150.5 | 150.5 | 10 |
+| predicted paint | 0.1 | **0.1** | 0.3 | 20 |
+| unpredicted input | 153.8 | **154.2** | 154.8 | 10 |
+| same keystroke's authoritative echo | 0.3 | 154.2 | 154.6 | 20 |
+| burst convergence | 233.4 | 233.8 | 234.2 | 10 |
+- **Leg 1 (latency) — cleared.** 0.1ms median against a 30ms threshold, at a
+  round trip three orders of magnitude larger.
+- **Leg 2 (convergence) — cleared, in the narrowed form below.** Ten
+  adversarial bursts: `made=54 confirmed=54 contradicted=0 expired=0
+  abandoned=0 pending=0`, last burst present in the daemon's grid.
+- **Leg 3 (safety) — cleared by the e2e suite**, not by the WAN harness. A
+  session that is canonical with echo off from its first instruction makes
+  **nothing**: `made=0 displayed=0`, the typed secret appears nowhere in the
+  bytes the client emits, and `pw-len-7` in the daemon's grid proves the
+  shell received all seven characters so the absence is prediction declining
+  rather than nothing having been typed.
+- **Leg 4 (fallback completeness) — cleared by unit and e2e tests.**
+  Multi-byte input, last column, scroll mode, pending resize and unknown
+  mode bits each suppress, each with its own test.
+### The control is the measurement
+Leg 1's number means nothing on its own — a fast local paint is what a
+terminal does anyway. What makes it evidence is that the **unpredicted**
+control was taken on the same connection in the same run, from the same
+kind of keystroke, and came back at 154.2ms. One keystroke, two clocks: no
+argument about warm caches, differing conditions or hardware can be made
+about the gap between them.
+Two things about that control are worth keeping.
+- **The criterion's wording was unmeasurable as written.** It asked for an
+  "echo-off context" as the control. With echo off *nothing is painted*, so
+  there is no arrival to put a clock on. The measurable form of the same
+  claim is input prediction **refuses**: two characters in one write reach
+  the client as one chunk, are refused for being multi-byte, and therefore
+  cannot appear until the daemon answers. A criterion can be unmeasurable
+  while sounding precise, and the time to find that out is while writing the
+  harness, not while reading the results.
+- **A control that cannot fail proves nothing**, so the harness fails leg 1
+  if the control comes back fast. A quick "unpredicted" number would mean
+  the input was being predicted after all and the comparison was empty —
+  which is the floor-form philosophy applied to a control rather than to a
+  threshold.
+### Floor-form thresholds: the new standard
+This discharges the rewrite banked by M8. Every threshold from here is
+stated in **absolute units with an explicit validity floor**, never as a
+multiple of RTT:
+- Each leg names the minimum RTT at which it is meaningful. Below that, the
+  harness prints **NOT EXERCISED** — never PASS. Leg 1's floor is 50ms and
+  `wan.sh` implements the check rather than describing it.
+- The failure this prevents is not hypothetical: two milestones' criteria
+  (M6's reattach gate, M8's leg 2) failed on fast hardware for the same
+  structural reason, and a criterion that fails because the link is *good*
+  is measuring the wrong thing.
+**M6's reattach gate still fails structurally on a fast link**, and did so
+again in every M9 run: "2× round-trip" is 0.4ms on a LAN, which nothing can
+meet. That is the pre-existing ruling, unrelated to M9, and it is now the
+clearest argument for applying the floor form **retroactively** to M6's
+criterion. Banked below.
+### The re-entrancy defect: found by a leaked temp directory
+The most valuable thing this milestone produced was not prediction.
+A `make test` run came back 171/172 with one test binary failed. The run
+immediately before it, of the same binaries, passed; the change between them
+was a comment. **The failing test's name was not captured** — the
+verification command grepped only the summary line — and 22 further runs,
+six of them under 8-way CPU load, never reproduced it.
+What the hunt did turn up: two M8-era QUIC tests intermittently leaked their
+`TmpDir`. A leaked temp directory means the cleanup `defer` never ran, which
+points at an **abort** rather than a clean test failure. An audit followed
+that scent to a real defect:
+- ngtcp2 is not re-entrant, and we re-entered it. `read_pkt` →
+  `recv_stream_data` callback → the daemon's frame handling → a reply queued
+  → `send` → `drain` → `writev_stream` **on the same connection**, with
+  `read_pkt` still on the stack below.
+- Monotonic timestamps move backwards within one `read_pkt`, quietly
+  corrupting loss detection. When a datagram carries a STREAM frame ahead of
+  an ACK, the nested write mutates the retransmission buffer the outer ack
+  walk is about to traverse.
+- `ngtcp2_unreachable()` calls `abort()` **unconditionally, even under
+  NDEBUG**. The symptom is a bare SIGABRT with no Zig panic banner and no
+  defers run — which is exactly what a leaked `TmpDir` looks like.
+**The fix is an invariant, not a patch:** `Listener.send` queues and never
+drains. Draining happens only where the stack is ours — after `read_pkt`
+returns, in `tick`, in `reapClosing`, and in the daemon's explicit
+`drainAll` at the end of each pump. An `assert(ngtcp2_depth == 0)` at the
+top of `drain` makes a regression a deterministic Debug failure instead of a
+one-in-a-few-hundred abort somewhere else entirely.
+**The mutation check revised the theory.** Reverting queue-only with the
+assert in place fires it *immediately* — in the unit tests and again in
+e2e, on the first QUIC exchange. The re-entrancy was not rare; it happened
+on essentially every QUIC session. Only its *consequence* was rare,
+depending on whether a datagram carried the frame ordering that turns it
+fatal. Every QUIC session before this fix was corrupting loss-detection
+state silently.
+**Honesty about the causal link.** The original 171/172 transient was never
+reproduced and its test name was never captured, so "this defect caused that
+failure" is **strong inference, not proof**. What is proven: the defect was
+real, constant, and capable of producing exactly that signature. What is
+not: that it produced *that* one.
+Two things queue-only broke that had to be fixed with it, neither in the
+audit's list — latency (a reply would have waited for the next poll cycle,
+since `tick` only services connections whose timer is due) and the
+session-full refusal (queued, then the connection reaped before it could
+leave; `reapClosing` now drains once before freeing).
+**Three hardening items in that batch are reasoned, not mutation-pinned**,
+and are recorded as such because the batch would otherwise read as
+uniformly verified: `closed`-recomputed-after-reap needs a handler that
+closes a *different* connection mid-reap; took-before-error-check needs
+ngtcp2 to commit a stream offset and then fail; the two `deinit` orderings
+need a use-after-free detector. All three are corrections to paths the tests
+do not reach.
+### Reconcile v2: judge evidence, not arrival order
+The first design read any cell that did not already hold the prediction as a
+refutation. That is wrong about the **ordinary** case, not an edge one: type
+`hello` faster than the round trip and the first frame back was built when
+the daemon had seen only `h`, so `e,l,l,o` are judged against a screen that
+predates them, all four read as contradictions, and the queue flushes.
+Prediction would have erased itself once per RTT, in every tier — the exact
+opposite of the feature.
+So a prediction carries what the cell held when it was made, and the three
+answers are distinguished: our character confirms; **the character that was
+already there means the frame has said nothing yet**, so the prediction
+waits; anything else means somebody wrote that cell, which is the only thing
+that refutes us. A refutation still flushes the whole queue — everything
+typed after a wrong prediction was typed into a screen that never existed.
+Waiting is bounded, or "no evidence yet" becomes a state a prediction sits
+in forever: eight judging frames, or a second of wall time. That is the
+phantom guard — nvim swallowing a normal-mode `j` repaints some other row
+and leaves the predicted cell untouched. The frame bound cannot catch the
+version where the application simply goes quiet (no frame ever comes back to
+trigger it), so `expire` does the same job off the client's idle path.
+The WAN run settles the risk this was banked against: `contradicted=0` and
+`expired=0` across ten adversarial bursts at 150ms RTT.
+### The burst/expiry ceiling
+A prediction is retired unanswered after `expire_after_ms` (1000ms). What it
+must survive is **the round trip plus however long a burst's later
+keystrokes queue behind its earlier ones**. Cross that and predictions
+expire mid-burst and the counters collapse.
+This is measured, not theorised: at 400ms each way the e2e burst scenario
+dropped to `confirmed=1`, because `delaypipe` delays each chunk serially and
+the later keystrokes aged past the bound while queued. At the WAN's 150ms
+the burst converges in 233.8ms against the 1000ms bound. `wan.sh` prints
+that headroom beside the burst numbers, because it is what decides whether
+the numbers reproduce on a slower path.
+### The readline finding: the tiers describe termios, not UX
+An interactive bash or zsh prompt runs at **icanon=0, echo=0** — readline
+turns both off and echoes for itself. So the everyday shell prompt is the
+`.adaptive` tier, where display must be earned; `.always` covers only
+genuinely canonical readers (`cat`, a shell's `read` builtin, dash without
+line editing).
+Two consequences. The mode bits move once or twice **per command** as
+readline hands the terminal back and forth, and since any move in the bits
+flushes the queue and un-earns display, the first two keystrokes after each
+prompt are invisible predictions. That is the conservative trade taken
+deliberately; per-context confidence memory is the banked polish.
+It also decided the test design: `/bin/cat` is the session shell wherever
+`.always` is under test, in e2e and in the WAN harness alike. A suite that
+used `/bin/sh` there would have been testing a different tier than it
+claimed.
+### Named trap: "a write is not a keystroke"
+Three sightings this milestone, all the same root cause — a `printf` or a
+loop of them arrives at the client as **one read**, which is a multi-byte
+chunk, which prediction refuses:
+. **e2e password scenario.** `printf 'hunter2\n'` is one write, so the
+   chunk was refused for being multi-byte long before the tier was
+   consulted. `made=0` held for a reason that had nothing to do with the
+   password tier — the scenario passed with echo-off canonical mapped to the
+   always-predict tier, and would have shipped guarding nothing.
+. **e2e multi-byte test.** All three chunks had non-printable lead bytes,
+   which `predictAt` refuses anyway. The untested shape was a paste of plain
+   ASCII, whose lead byte is printable — the one case the length guard is
+   the only thing catching.
+. **wan.sh burst.** Characters sent back to back gave `made=5` for 50
+   keystrokes: the bursts contained no predictions at all, and leg 2 would
+   have "passed" on an empty measurement.
+The fix is always the same — pace the keystrokes so each is its own read —
+and the lesson is that **a test which types must prove what it typed became
+separate keystrokes**, by asserting `made`.
+### Named trap, sixth sighting: `pgrep -f` matching itself
+Recorded five times before this milestone. The sixth: checking the LAN box
+for leftovers with `pgrep -f "muxd-wan-"` reported two matches, both of
+which were the diagnostic's **own** shell — its command line contained the
+literal path glob `/tmp/muxd-wan-*`. It happened *while checking for the
+very hazard it is*, which is the detail worth keeping: knowing about a trap
+is not the same as not being in it.
+The authoritative check is by `comm` plus a file listing — `pgrep -x muxd`
+and `ls` — which showed the box clean. The earlier "box clean" reports were
+correct, but rested on weaker evidence than was claimed for them.
+### Leg 2's convergence, narrowed — and why
+The plan's leg 2 says the client grid converges **byte-identical** to `muxd
+dump`. The WAN harness does not check that. It asserts convergence through
+the daemon's grid plus attribution: every prediction accounted for
+(`made = confirmed + abandoned + pending`, with `pending = 0`), and the last
+burst present in the daemon's grid.
+This is the **same narrowing, for the same reason, as M7's**: the client
+emits a stream of paints, not a grid, so comparing it to a dump means
+parsing VT inside the harness — a second terminal emulator whose own bugs
+would be indistinguishable from the ones it is meant to catch. `client.zig`
+and `server.zig` tests do make grid comparisons against a real replica
+Engine, over a socket. The full render-vs-dump harness stays banked.
+### Method notes
+- **Write the mutation first.** Five assertions across this milestone passed
+  for reasons other than the mechanism they named, and every one was caught
+  by asking "what change should break this?" *before* trusting the green.
+  Three were test weaknesses found on a mutation's first run (the
+  `paintOverlay` confidence guard, multi-byte chunks, the client's
+  `markPainted` call site); two were e2e scenarios that survived a mutation
+  of the thing they existed to protect.
+- **Mutation-check the call site separately from the rule.** A module's own
+  test cannot see whether its caller ever calls it: removing the client's
+  `markPainted` call survived until a client-side test existed for it.
+- **`predict.zig` is engine-free**, so the whole policy — tiers, promotion,
+  refutation, expiry — is exercised with no terminal, pty or daemon in the
+  picture. `reconcile` takes its grid duck-typed; `PlainGrid` adapts the
+  plain dump a client already has. The debt that created (no real-Engine
+  reconcile test) was discharged at the client wiring point, where the
+  `prev_ch` read is pinned by a test that fails twice over if it reads the
+  wrong cell.
+### Banked by M9
+- **Backspace prediction.** M9 predicts printable ASCII only; backspace is
+  the most-missed omission in ordinary typing.
+- **Underline only when late.** Underlining every prediction is honest but
+  noisy on a fast link, where the authoritative echo replaces it within
+  milliseconds. Show the distinction only once a prediction has been
+  outstanding long enough to matter.
+- **Per-context confidence memory.** Confidence is re-earned from scratch on
+  every mode change, which at a readline prompt is once or twice per
+  command. Remembering it per context would return the first two keystrokes
+  after each prompt.
+- **An input ack in the protocol.** The proper fix if spurious contradiction
+  ever reappears: the daemon telling the client which input it has seen
+  removes the guesswork from judging entirely. Not needed today —
+  `contradicted=0` at 150ms — and a protocol change is not worth spending
+  before it is.
+- **Full render-vs-dump convergence harness** (shared with M7's identical
+  banked item).
+- **Floor-form rewrite applied retroactively to M6's reattach criterion.**
+  It still fails structurally on fast links; the form that fixes it is now
+  standard and written down.
+## Open (owed by later milestones)
+- Scrollback retention *tuning*. The policy itself was decided in M1 and

docs/roadmap.md

Old		New
@@ -5,35 +5,33 @@ The forward view, one item per line, ranked. History and evidence live in
5	this file at each milestone close and whenever the queue reorders; the	5	this file at each milestone close and whenever the queue reorders; the
6	queue's order is set by the user, not by this file.	6	queue's order is set by the user, not by this file.
7		7
8	Now: `v0.0.1-1` tagged; M1–M8 complete; in field trial on real VMs.	8	Now: `v0.0.1-1` tagged; M1–M9 complete; in field trial on real VMs.
9	Trial feedback outranks everything below — what actually hurts in use is	9	Trial feedback outranks everything below — what actually hurts in use is
10	better data than any of this ranking.	10	better data than any of this ranking.
11		11
12	## M9 — prediction (speculative local echo) — queued	12	## M9 — prediction (speculative local echo) — complete
13		13
14	The client paints predicted echo for printable keystrokes immediately and	14	Verdict: cleared. A predicted keystroke paints in 0.1ms while the
15	reconciles against the daemon's authoritative delta when it arrives.	15	same keystroke's authoritative echo — and unpredicted input on the same
16	Sequencing set 2026-08-07: reconnect → QUIC → prediction. Only pays on	16	connection in the same run — takes 154ms over a 150ms path. Echo
17	high-RTT links — the protocol itself costs ~4ms over baseline, so LAN	17	latency has stopped being a function of the round trip where prediction
18	users will never see the difference (decisions.md, M6 verdict).	18	applies, and is provably absent where it must not (decisions.md, M9).
19		19
20	Scope per handoff §5: conservative, line-mode only, printable chars,	20	Prediction is an overlay on the replica, never fed into it; the daemon
21	unconfirmed text visually distinguished, instant fallback on anything	21	ships the pty's mode bits so disabled contexts are known rather than
22	unpredictable. Design sketch discussed 2026-08-08: prediction is an	22	guessed. The blocking pre-requisite — the RTT-multiple criterion form — was
23	overlay on the replica, never fed into it; daemon ships pty mode bits	23	discharged: thresholds are now stated in absolute units with an explicit
24	(echo/icanon) so disabled contexts are known, not guessed.	24	validity floor, and a harness prints NOT EXERCISED below it rather than
		25	PASS.
25		26
26	Pre-requisite, blocking: rewrite the RTT-multiple kill-criterion form	27	Two findings outlived the milestone. The everyday shell prompt is
27	— two milestones' criteria (M6 reattach gate, M8 leg 2) failed on fast	28	icanon=0/echo=0 (readline echoes for itself), so it is the adaptive tier
28	hardware for the same structural reason; a threshold needs a floor term or	29	and not the always-predict one. And the hunt for a one-in-hundreds test
29	a stated minimum RTT before M9 writes another one (decisions.md, "Banked	30	failure found ngtcp2 being re-entered on every QUIC exchange —
30	by M8").	31	constant, with a rare consequence — which is fixed and, unusually, was
		32	proven so by a mutation that fired immediately.
31		33
32	Hazard carried in: speculative buffers held across callbacks are the	34	## Trial friction, likely next — and still outranked by trial feedback
33	habitat of the UAF-that-never-crashes (decisions.md, M8 egress/teardown
34	records).
35
36	## Trial friction, likely next after (or before) M9
37		35
38	- Daemon lifecycle: attach does not auto-start `muxd run`; no install	36	- Daemon lifecycle: attach does not auto-start `muxd run`; no install
39	script; systemd user units exist in `contrib/` but are not the deployed	37	script; systemd user units exist in `contrib/` but are not the deployed
@@ -44,6 +42,12 @@ records).
44		42
45	## Banked, ranked by expected win	43	## Banked, ranked by expected win
46		44
		45	0. Prediction polish, as a group — backspace prediction (the
		46	most-missed omission in ordinary typing); underline only once a
		47	prediction has been outstanding long enough to matter, rather than
		48	always; per-context confidence memory, which would return the first two
		49	keystrokes after every readline prompt. All three are small and all
		50	three are felt on every keystroke (decisions.md, "Banked by M9").
47	1. Retry-skip via NEW_TOKEN — ~1 RTT off QUIC cold attach, which is	51	1. Retry-skip via NEW_TOKEN — ~1 RTT off QUIC cold attach, which is
48	most of QUIC's remaining margin over ssh-via. Not "skip Retry": token	52	most of QUIC's remaining margin over ssh-via. Not "skip Retry": token
49	for validated clients, RFC 9000 3x anti-amplification for the rest —	53	for validated clients, RFC 9000 3x anti-amplification for the rest —