a73x

078e9712

test: 02_predict moves to a file of its own

a73x   2026-08-26 18:48

Commit message
test: 02_predict moves to a file of its own

prediction, and the ptyclient fixture's own controls — 3 scenarios, 803 lines. The legs move unedited; the runner
sources the file where they used to stand.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test/e2e.sh
Old New
@@ -102,7 +102,7 @@ done
102 E2E_DIR=$(dirname "$0") 102 E2E_DIR=$(dirname "$0")
103 . "$E2E_DIR/e2e_lib.sh" 103 . "$E2E_DIR/e2e_lib.sh"
104 104
105 E2E_GROUPS='01_boot' 105 E2E_GROUPS='01_boot 02_predict'
106 E2E_NEEDS="" 106 E2E_NEEDS=""
107 107
108 # E2E_ONLY=<group> — run one group file and stop. For the loop a leg is in 108 # E2E_ONLY=<group> — run one group file and stop. For the loop a leg is in
@@ -140,806 +140,6 @@ for _g in $E2E_GROUPS; do
140 . "$E2E_DIR/e2e_$_g.sh" 140 . "$E2E_DIR/e2e_$_g.sh"
141 done 141 done
142 142
143 # M9 prediction: each scenario needs a session whose LINE DISCIPLINE it
144 # controls, so they cannot share the long-lived /bin/sh daemon.
145 SOCK5="${TMPDIR:-/tmp}/muxd-e2e-pred-$$.sock"
146 defer_sock "$SOCK5"
147 SOCK6="${TMPDIR:-/tmp}/muxd-e2e-pw-$$.sock"
148 defer_sock "$SOCK6"
149 SOCK7="${TMPDIR:-/tmp}/muxd-e2e-raw-$$.sock"
150 defer_sock "$SOCK7"
151 PWSH="${TMPDIR:-/tmp}/mux-e2e-pw-$$.sh"
152 defer_rm "$PWSH"
153 # M12 pty scenarios. Each needs a daemon whose grid size it owns: tp2
154 # resizes the grid twice and tp1's session is a scrollback-generating
155 # wrapper, so neither can share the long-lived /bin/sh daemon.
156 SOCK12="${TMPDIR:-/tmp}/muxd-e2e-tp2-$$.sock"
157 defer_sock "$SOCK12"
158 SOCK13="${TMPDIR:-/tmp}/muxd-e2e-tp1-$$.sock"
159 defer_sock "$SOCK13"
160 # tp1's session: a wrapper that fills the scrollback and then becomes cat.
161 TP1SH="${TMPDIR:-/tmp}/mux-e2e-tp1-$$.sh"
162 defer_rm "$TP1SH"
163
164 # ---- M9: prediction ---------------------------------------------------
165 #
166 # Every scenario here asserts COUNTERS, because the screen cannot tell a
167 # predicted glyph from an echoed one — that is the entire point of the
168 # feature — so a rendering test would pass just as happily with prediction
169 # switched off.
170 #
171 # The session shell is /bin/cat, not /bin/sh. An interactive bash prompt is
172 # icanon=0/echo=0 (readline echoes for itself), which is the ADAPTIVE tier,
173 # not the always-predict one; cat is a genuinely canonical reader and the
174 # only way to exercise `.always` here. That distinction cost a milestone's
175 # worth of confusion to learn and is why it is written down twice.
176
177 # Delay per direction. The round trip is twice this, and every assertion
178 # below about "before the daemon could have answered" is measured against
179 # it. Chosen against three constraints at once, which is why it is not a
180 # round number picked for looks:
181 #
182 # - Larger than delaypipe's built-in default of 150, and far enough from
183 # it to be told apart. The snapshot below is taken 450ms after the
184 # keystroke: the echo cannot have returned over this 600ms round trip,
185 # but WOULD have over the 300ms one the default gives. A DELAY_MS lookup
186 # that silently stopped working — leaving every scenario here measuring
187 # a path twice as fast as it claimed — fails that assertion rather than
188 # passing quietly.
189 # - Small enough that the round trip stays clear of the overlay's 1000ms
190 # expiry bound. This is a real ceiling, not a test artefact: delaypipe
191 # delays each chunk serially, so a burst's later keystrokes queue behind
192 # the earlier ones and age while they wait. At 400ms each way the burst
193 # scenario went to confirmed=1 — the predictions were expiring before
194 # their echo could arrive, exactly as the overlay is designed to do.
195 # - Leaving margin on both sides, so neither is a race under load.
196 PDELAY=300
197
198 # Own file, not the client capture: the daemon's fd survives the client's truncation and would write into the replayed stream.
199 start_daemon "$SOCK5" "$OUT.p1.d" "prediction daemon never bound" --shell /bin/cat
200 D5PID=$DPID
201
202 # 1. Line mode: the glyph is on screen before the round trip could have
203 # delivered it.
204 set +e
205 { sleep 2; printf 'z'; sleep 4; printf '\034\034'; } | \
206 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 40 "$MUX" \
207 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK5 | $DELAYPIPE" \
208 > "$OUT.p1" 2> "$OUT.p1.err" &
209 P1PID=$!
210 defer_kill "$P1PID"
211 set -e
212 # Snapshot the client output 450ms after the keystroke: well
213 # before this 600ms round trip could bring the pty own echo back, and well
214 # AFTER the 300ms round trip delaypipe default of 150 would have given.
215 sleep 2.45
216 cp "$OUT.p1" "$OUT.p1.early" 2>/dev/null || true
217 set +e
218 wait "$P1PID"
219 RC=$?
220 set -e
221 [ "$RC" -eq 0 ] || {
222 echo "e2e FAIL: line-mode prediction client exited $RC"
223 cat "$OUT.p1" "$OUT.p1.err" 2>/dev/null; exit 1; }
224
225 # The underlined glyph, in the snapshot taken before the echo could arrive.
226 # This is the whole claim of the milestone, as an effect rather than a
227 # counter: the character was on the screen while it was still in flight.
228 grep -q "$(printf '\033\[4mz')" "$OUT.p1.early" || {
229 echo "e2e FAIL: no predicted glyph 450ms after the keystroke (RTT is $((PDELAY * 2))ms)"
230 echo "--- early snapshot ---"; cat -v "$OUT.p1.early"; exit 1;
231 }
232 # ...and the daemon's own answer was NOT there yet, which is what makes the
233 # line above mean anything. A delta paints a row with ECH before its
234 # content; the prediction never does.
235 if grep -q "$(printf '\033\[80X\033\[0mz')" "$OUT.p1.early"; then
236 echo "e2e FAIL: the echo arrived within 450ms, so the path is not $((PDELAY * 2))ms;"
237 echo " either DELAY_MS was ignored (default is 150) or delaypipe is not delaying"
238 exit 1
239 fi
240 want_stat "$OUT.p1.err" contradicted 0 "line mode"
241 CONF=$(predict_stat "$OUT.p1.err" confirmed)
242 [ -n "$CONF" ] && [ "$CONF" -ge 1 ] || {
243 echo "e2e FAIL: line mode: confirmed=$CONF, want >=1"; grep "^predict " "$OUT.p1.err"; exit 1;
244 }
245 # The session really received the keystroke: cat echoed it into the grid.
246 "$MUXD" dump --sock "$SOCK5" | grep -q "z" || {
247 echo "e2e FAIL: line mode: daemon grid never saw the keystroke"; exit 1;
248 }
249 assert_converged "$OUT.p1" "$SOCK5" "line-mode prediction"
250
251 # 2. A burst outrunning the round trip. This is reconcile v2's reason to
252 # exist: under the old rule every character after the first was judged by
253 # a frame built before it was typed, read as a contradiction, and the
254 # whole queue was flushed once per round trip. If anyone regresses the
255 # three-way judgment, this scenario is what says so.
256 set +e
257 { sleep 2; for c in b u r s t; do printf '%s' "$c"; sleep 0.25; done; \
258 sleep 4; printf '\034\034'; } | \
259 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 40 "$MUX" \
260 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK5 | $DELAYPIPE" \
261 > "$OUT.pb" 2> "$OUT.pb.err"
262 RC=$?
263 set -e
264 [ "$RC" -eq 0 ] || {
265 echo "e2e FAIL: burst client exited $RC"
266 cat "$OUT.pb" "$OUT.pb.err" 2>/dev/null; exit 1; }
267
268 # Five keystrokes typed at 250ms into a 600ms round trip: three are always
269 # outstanding at once (at t=600, when the first is confirmed, the keys typed
270 # at 0, 250 and 500 are all in flight). `made` is asserted too, so two
271 # keystrokes arriving in one read (which would be suppressed as a multi-byte
272 # chunk, not predicted) fails loudly instead of quietly weakening the test.
273 #
274 # The interval is 250 and not 200 because of a bound that has to be COMPUTED
275 # rather than eyeballed. delaypipe delivers one chunk per DELAY_MS serially
276 # in EACH direction, so keystrokes closer together than DELAY_MS queue, and
277 # each one ages (DELAY_MS - interval) longer than the one before it:
278 #
279 # age at which keystroke k is confirmed = 2*DELAY_MS + k*(DELAY_MS - interval)
280 #
281 # At interval=200 the fifth keystroke lands at 600 + 4*100 = 1000ms, which is
282 # exactly predict.expire_after_ms: a coin flip, not a margin. It failed 4 runs
283 # in 20 on an idle machine. At 250 it is 600 + 4*50 = 800ms, measured at
284 # 798-799 across 12 runs. The same formula puts the SECOND keystroke at
285 # 1000ms when DELAY_MS=400, which is the confirmed=1 collapse noted above.
286 want_stat "$OUT.pb.err" made 5 "burst"
287 # `confirmed` is a LIVENESS counter and the only timing-dependent one here:
288 # the fifth confirmation can still be in flight when the detach byte lands,
289 # which is why it is a floor rather than an equality. Observed 1 run in 3
290 # under load (review runs at 715f71b, 2026-08-09), and decisions.md records
291 # the same counter coming up one short on the M9 WAN burst (made=55
292 # confirmed=54 abandoned=1, the mode-transition churn flush).
293 #
294 # `contradicted` and `expired` stay EXACT. They are the correctness claims —
295 # a prediction judged wrong, or one that aged out unjudged, is a defect
296 # whenever it happens, and nothing about timing makes either acceptable.
297 want_stat_ge "$OUT.pb.err" confirmed 4 "burst"
298 # `expired` first: an expiry runs through the same abandonAll tail as a
299 # refutation and increments BOTH counters, so contradicted >= expired always.
300 # Asserting contradicted first reports an age-out as "a prediction judged
301 # wrong" — a different defect with a different cause, and the reason this
302 # flake was first investigated as a reconcile bug.
303 want_stat "$OUT.pb.err" expired 0 "burst"
304 want_stat "$OUT.pb.err" contradicted 0 "burst"
305 "$MUXD" dump --sock "$SOCK5" | grep -q "burst" || {
306 echo "e2e FAIL: burst: daemon grid does not hold the typed text"; exit 1;
307 }
308 assert_converged "$OUT.pb" "$SOCK5" "burst"
309
310 softkill "$D5PID" || true
311 D5PID=""
312
313 # 3. A password prompt. The session is canonical with echo OFF from the
314 # moment it starts, so the whole run sits in the tier that predicts
315 # nothing at all.
316 cat > "$PWSH" <<'PWEOF'
317 #!/bin/sh
318 stty -echo
319 printf 'pw-ready\n'
320 read secret
321 printf 'pw-len-%s\n' "${#secret}"
322 # Blocks forever rather than exiting: the shell exiting would end the
323 # session, and the daemon would be gone before its grid could be read.
324 read _hold
325 PWEOF
326 chmod +x "$PWSH"
327
328 start_daemon "$SOCK6" "$OUT.pw.d" "password daemon never bound" --shell "$PWSH"
329 D6PID=$DPID
330
331 # One character per write, and NOT as a single `printf hunter2`. A whole
332 # word in one write reaches the client as a multi-byte chunk, which is
333 # refused for being multi-byte before the tier is ever consulted — so the
334 # scenario would pass with the password tier completely broken. It did:
335 # mapping echo-off canonical to the always-predict tier survived this
336 # scenario until the typing was split up.
337 set +e
338 { sleep 2; for c in h u n t e r 2; do printf '%s' "$c"; sleep 0.2; done; \
339 printf '\n'; sleep 2; printf '\034\034'; } | \
340 MUX_PREDICT_STATS=1 timeout 40 "$MUX" --sock "$SOCK6" > "$OUT.pw" 2> "$OUT.pw.err"
341 RC=$?
342 set -e
343 [ "$RC" -eq 0 ] || {
344 echo "e2e FAIL: password client exited $RC"
345 cat "$OUT.pw" "$OUT.pw.err" 2>/dev/null; exit 1; }
346
347 # Nothing was shown, and nothing was even attempted: echo-off canonical is
348 # the tier where a prediction must never be MADE, not merely never painted,
349 # because a made prediction sits in a buffer the overlay paints from.
350 want_stat "$OUT.pw.err" made 0 "password"
351 want_stat "$OUT.pw.err" displayed 0 "password"
352 # The effect, not the counter: the secret appears nowhere in the bytes the
353 # client wrote — either stream. The stats line cannot carry a secret, but
354 # checking both costs one word and the claim is about everything the client
355 # emitted, not about the half of it that happens to be the screen.
356 if grep -q "hunter2" "$OUT.pw" "$OUT.pw.err"; then
357 echo "e2e FAIL: the password appears in the client's terminal output"; exit 1
358 fi
359 # ...and the run was not vacuous: the shell really received all seven
360 # characters, so the absence above is prediction declining rather than
361 # nothing having been typed.
362 "$MUXD" dump --sock "$SOCK6" | grep -q "pw-len-7" || {
363 echo "e2e FAIL: the password never reached the shell; the absence above proves nothing"
364 "$MUXD" dump --sock "$SOCK6"; exit 1;
365 }
366 assert_converged "$OUT.pw" "$SOCK6" "password"
367
368 softkill "$D6PID" || true
369 D6PID=""
370
371 # 4. Raw mode: display is earned, then lost to a keystroke the application
372 # swallows. rawmode echoes like an editor in insert mode until it is sent
373 # 0x00, after which it consumes input and prints nothing.
374 #
375 # Through delaypipe, like leg 1, and for the same reason leg 4b below is
376 # NOT: the overlay measures its own round trip and hides everything on a
377 # path faster than `predict.local_below_ms`. This leg is about the
378 # earn/lose policy, which only a remote path lets through to the screen.
379 # Keys are 1s apart so each confirm (600ms) lands before the next key —
380 # at 0.5s `c` would find only `a` confirmed and stay hidden.
381 start_daemon "$SOCK7" "$OUT.rw.d" "rawmode daemon never bound" --shell "$RAWMODE"
382 D7PID=$DPID
383
384 set +e
385 { sleep 2; for c in a b c; do printf '%s' "$c"; sleep 1; done; \
386 printf '\000'; sleep 1; printf 'j'; sleep 2.5; printf '\034\034'; } | \
387 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 40 "$MUX" \
388 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK7 | $DELAYPIPE" \
389 > "$OUT.rw" 2> "$OUT.rw.err"
390 RC=$?
391 set -e
392 [ "$RC" -eq 0 ] || {
393 echo "e2e FAIL: rawmode client exited $RC"
394 cat "$OUT.rw" "$OUT.rw.err" 2>/dev/null; exit 1; }
395
396 # Four printable keystrokes made predictions; the 0x00 that switched modes
397 # is not printable and was refused.
398 want_stat "$OUT.rw.err" made 4 "raw mode"
399 want_stat "$OUT.rw.err" suppressed 1 "raw mode"
400 # a and b are invisible while promotion is earned; c is the first painted,
401 # and j is painted because by then display had been earned.
402 want_stat "$OUT.rw.err" displayed 2 "raw mode"
403 want_stat "$OUT.rw.err" confirmed 3 "raw mode"
404 # j was swallowed: no frame ever answered the cell it was drawn in, so the
405 # expiry bound retired it rather than leaving a phantom glyph on screen for
406 # the rest of the session.
407 want_stat "$OUT.rw.err" expired 1 "raw mode"
408 want_stat "$OUT.rw.err" contradicted 1 "raw mode"
409 want_stat "$OUT.rw.err" abandoned 1 "raw mode"
410
411 # Every prediction accounted for: made = confirmed + abandoned + pending,
412 # and the queue is empty at exit, so nothing survived unexplained.
413 RW_MADE=$(predict_stat "$OUT.rw.err" made)
414 RW_CONF=$(predict_stat "$OUT.rw.err" confirmed)
415 RW_ABND=$(predict_stat "$OUT.rw.err" abandoned)
416 [ "$((RW_MADE - RW_CONF - RW_ABND))" -eq 0 ] || {
417 echo "e2e FAIL: raw mode: $((RW_MADE - RW_CONF - RW_ABND)) predictions unaccounted for"
418 grep "^predict " "$OUT.rw.err"; exit 1;
419 }
420 # The counters say `j` expired; this says the screen agrees. An expiry that
421 # retired the prediction in the book but left its underlined glyph painted is
422 # exactly the phantom the styled half of this check exists to catch.
423 assert_converged "$OUT.rw" "$SOCK7" "raw mode"
424
425 # 4b. The same keystrokes on the bare socket: a local daemon answers in
426 # about a millisecond, and that is the one path on which a prediction
427 # can only ever be SEEN when it is wrong. Display is still earned (the
428 # counters say so) and nothing is painted — `local` is where `c` and
429 # `j` went instead of `displayed`. `j` still expires: the queue keeps
430 # judging while hidden, which is how the gate would lift again on a
431 # box that got slow. The screen check is the point: no phantom `j`,
432 # because there was never an underlined one to leave behind.
433 #
434 # A fresh daemon: the rawmode above has taken its 0x00 and echoes
435 # nothing now, and the leg needs the insert-mode half back.
436 softkill "$D7PID" || true
437 wait "$D7PID" 2>/dev/null || true
438 rm -f "$SOCK7"
439 start_daemon "$SOCK7" "$OUT.rwl.d" "rawmode local daemon never bound" --shell "$RAWMODE"
440 D7PID=$DPID
441
442 set +e
443 { sleep 2; for c in a b c; do printf '%s' "$c"; sleep 0.5; done; \
444 printf '\000'; sleep 0.5; printf 'j'; sleep 2.5; printf '\034\034'; } | \
445 MUX_PREDICT_STATS=1 timeout 40 "$MUX" --sock "$SOCK7" > "$OUT.rwl" 2> "$OUT.rwl.err"
446 RC=$?
447 set -e
448 [ "$RC" -eq 0 ] || {
449 echo "e2e FAIL: rawmode local client exited $RC"
450 cat "$OUT.rwl" "$OUT.rwl.err" 2>/dev/null; exit 1; }
451 want_stat "$OUT.rwl.err" made 4 "raw mode local"
452 want_stat "$OUT.rwl.err" confirmed 3 "raw mode local"
453 want_stat "$OUT.rwl.err" displayed 0 "raw mode local"
454 want_stat "$OUT.rwl.err" local 2 "raw mode local"
455 want_stat "$OUT.rwl.err" expired 1 "raw mode local"
456 grep -q "$(printf '\033\[4m')" "$OUT.rwl" && {
457 echo "e2e FAIL: raw mode local: an underlined prediction reached the screen"
458 cat -v "$OUT.rwl"; exit 1;
459 }
460 assert_converged "$OUT.rwl" "$SOCK7" "raw mode local"
461
462 softkill "$D7PID" || true
463 D7PID=""
464
465 # 5. A transport torn down with predictions outstanding. The overlay must
466 # come back empty: what was queued was predicted against a connection
467 # that no longer exists.
468 start_daemon "$SOCK5" "$OUT.pr.d" "reconnect daemon never bound" --shell /bin/cat
469 D5PID=$DPID
470
471 set +e
472 { sleep 2; printf 'p'; sleep 5; printf 'q'; sleep 4; printf '\034\034'; } | \
473 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 60 "$MUX" \
474 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK5 | $DELAYPIPE" \
475 > "$OUT.pr" 2> "$OUT.pr.err" &
476 PRPID=$!
477 defer_kill "$PRPID"
478 set -e
479 # Kill the transport a tenth of a second after the keystroke, while its
480 # prediction is still outstanding — the round trip is 600ms, so it cannot
481 # have been judged yet. By comm+pid: the client's own argv contains the
482 # --via string, so a pattern kill would take out the client under test.
483 sleep 2.1
484 PP=$(proxy_pid "$SOCK5")
485 [ -n "$PP" ] || {
486 echo "e2e FAIL: no proxy to tear down"
487 cat "$OUT.pr" "$OUT.pr.err" 2>/dev/null; exit 1; }
488 kill -9 "$PP"
489
490 set +e
491 wait "$PRPID"
492 RC=$?
493 set -e
494 [ "$RC" -eq 0 ] || {
495 echo "e2e FAIL: reconnect client exited $RC"
496 cat "$OUT.pr" "$OUT.pr.err" 2>/dev/null; exit 1; }
497
498 # The queue is empty at exit, by attribution rather than by inspection:
499 # nothing predicted is still outstanding, and the flush that dropped the
500 # in-flight prediction was not counted as anybody being wrong.
501 PR_MADE=$(predict_stat "$OUT.pr.err" made)
502 PR_CONF=$(predict_stat "$OUT.pr.err" confirmed)
503 PR_ABND=$(predict_stat "$OUT.pr.err" abandoned)
504 [ -n "$PR_MADE" ] || {
505 echo "e2e FAIL: reconnect: no predict stats"
506 cat "$OUT.pr" "$OUT.pr.err" 2>/dev/null; exit 1; }
507 [ "$((PR_MADE - PR_CONF - PR_ABND))" -eq 0 ] || {
508 echo "e2e FAIL: reconnect: $((PR_MADE - PR_CONF - PR_ABND)) predictions outlived the tear"
509 grep "^predict " "$OUT.pr.err"; exit 1;
510 }
511 # And it was the reconnect that dropped it, not the expiry bound catching it
512 # a second later. Without this the scenario cannot tell the two apart: the
513 # resync after a reconnect flushes as well, so the outcome looks identical
514 # either way and removing the reconnect's own flush passes unnoticed. It
515 # did, until this line.
516 want_stat "$OUT.pr.err" expired 0 "reconnect"
517 [ -n "$PR_ABND" ] && [ "$PR_ABND" -ge 1 ] || {
518 echo "e2e FAIL: reconnect: nothing was abandoned ($PR_ABND), so no prediction was"
519 echo " outstanding when the transport died and the tear proves nothing"
520 grep "^predict " "$OUT.pr.err"; exit 1;
521 }
522 # The session survived and the client resumed into it: the keystroke typed
523 # after the tear reached the shell.
524 "$MUXD" dump --sock "$SOCK5" | grep -q "q" || {
525 echo "e2e FAIL: reconnect: the post-tear keystroke never reached the session"
526 "$MUXD" dump --sock "$SOCK5"; exit 1;
527 }
528 assert_converged "$OUT.pr" "$SOCK5" "reconnect flush"
529
530 softkill "$D5PID" || true
531 D5PID=""
532
533 # ---- M12: ptyclient fixture controls ----------------------------------
534 # Before any scenario trusts the fixture, prove both directions: a
535 # roundtrip over plain /bin/cat (the pty line discipline's own echo
536 # answers — no mux anywhere, so a failure here is the FIXTURE'S), and an
537 # expect that cannot match, which must time out, exit nonzero, and say
538 # what it did see. A check that cannot fail proves nothing.
539 PCLOG="$OUT.pc.log"
540 set +e
541 timeout 30 "$PTYCLIENT" --cols 80 --rows 24 --out "$OUT.pc" --err "$OUT.pc.err" -- /bin/cat > "$PCLOG" 2>&1 <<'EOF'
542 send hello\n
543 expect hello 10000
544 send \x04
545 waitexit 10000
546 EOF
547 RC=$?
548 set -e
549 [ "$RC" -eq 0 ] || {
550 echo "e2e FAIL: ptyclient roundtrip over cat exited $RC:"; cat "$PCLOG"; exit 1; }
551 grep -q "hello" "$OUT.pc" || {
552 echo "e2e FAIL: ptyclient capture missing the pty echo"; cat -v "$OUT.pc"; exit 1; }
553 # The must-fail leg. 500ms: nothing is being waited FOR — the needle never
554 # arrives by construction — so the deadline only bounds the control's cost.
555 set +e
556 timeout 30 "$PTYCLIENT" --cols 80 --rows 24 --out "$OUT.pc2" --err "$OUT.pc2.err" -- /bin/cat > "$PCLOG.2" 2>&1 <<'EOF'
557 expect never-going-to-match 500
558 EOF
559 RC=$?
560 set -e
561 # The literal, not merely nonzero: 3 is EXIT_TIMEOUT, and the tp1/tp2
562 # scenarios read that code to tell a missed needle apart from a client that
563 # died or a script the fixture refused. A bare -ne 0 would pass just as
564 # happily on a usage error that never ran the script at all.
565 [ "$RC" -eq 3 ] || {
566 echo "e2e FAIL: ptyclient expect control exited $RC, want 3 (EXIT_TIMEOUT)"; cat "$PCLOG.2"; exit 1; }
567 grep -q "did not arrive" "$PCLOG.2" || {
568 echo "e2e FAIL: ptyclient timeout fired but never said what it saw"; cat "$PCLOG.2"; exit 1; }
569 # The stderr split, end to end: the client's predict stats must land in the
570 # .err sibling and NOT in the capture the convergence machinery diffs. Only
571 # a child that writes to BOTH streams can tell a working split from a child
572 # that simply never wrote to stderr.
573 set +e
574 timeout 30 "$PTYCLIENT" --cols 80 --rows 24 --out "$OUT.pc3" --err "$OUT.pc3.err" -- \
575 /bin/sh -c 'printf pc-stdout; printf pc-stderr 1>&2' > "$PCLOG.3" 2>&1 <<'EOF'
576 expect pc-stdout 10000
577 waitexit 10000
578 EOF
579 RC=$?
580 set -e
581 [ "$RC" -eq 0 ] || {
582 echo "e2e FAIL: ptyclient stderr-split leg exited $RC:"; cat "$PCLOG.3"; exit 1; }
583 grep -q "pc-stdout" "$OUT.pc3" || {
584 echo "e2e FAIL: ptyclient capture missing the child's stdout"; cat -v "$OUT.pc3"; exit 1; }
585 ! grep -q "pc-stderr" "$OUT.pc3" || {
586 echo "e2e FAIL: the child's stderr leaked into the capture"; cat -v "$OUT.pc3"; exit 1; }
587 grep -q "pc-stderr" "$OUT.pc3.err" || {
588 echo "e2e FAIL: the child's stderr reached neither the capture nor --err"
589 cat -v "$OUT.pc3.err"; exit 1; }
590 rm_swept "$OUT.pc" "$OUT.pc.err" "$OUT.pc2" "$OUT.pc2.err" "$OUT.pc3" "$OUT.pc3.err" \
591 "$PCLOG" "$PCLOG.2" "$PCLOG.3"
592 ok "ptyclient controls: pty echo roundtrips, impossible expect fails loudly, stderr stays off the capture"
593
594 # ---- M12 tp2: resize under a real tty (campaign row 7's branch) --------
595 # Two clients, two convergence points, because convergence is a scenario's
596 # LAST act after detach.
597 #
598 # tp2a is the suite's first attach at a size that is not the non-tty 80x24
599 # default: it pins the size a real tty reports reaching the daemon, and
600 # convergence measured against that size rather than the default. It does
601 # NOT exercise the snapshot resize prefix — an attach applies the
602 # attacher's size to the grid BEFORE the answering resync, so the prefix
603 # always equals the size the replica already has and the client's
604 # resize-on-prefix guard never fires. Verified by probe, not by reading.
605 #
606 # tp2b is the leg that catches row 7 (the prefix ignored), and it only
607 # catches it because of WHERE the damage can show. DELTA STREAMING
608 # bypasses the replica: those rows go straight to the tty, absolutely
609 # addressed and clipped to the tty, so a stale replica cannot spoil them
610 # and a wrong row is healed by the next delta that re-addresses it. The
611 # damage therefore has to survive to a FULL REPAINT, which is the paint
612 # that reads the replica back out and clips at min(replica, tty) — still
613 # on screen by then, and in a region the stale geometry cannot reach.
614 # Hence: fill the screen, put a 95-wide row on it, then winch UP to
615 # 100x30. With the prefix ignored the replica stays 90x28 and the repaint
616 # answering the resize reproduces the OLD geometry.
617 start_daemon "$SOCK12" "$OUT.tp2.d" "tp2 daemon never bound" --shell /bin/sh
618 D12PID=$DPID
619
620 # Two rules govern every script below, and breaking either one produces a
621 # capture that is honestly SHORT rather than wrong — a divergence whose
622 # diff blames the product for a scenario's mistake:
623 #
624 # A marker must not appear in the command line that produces it. The
625 # shell echoes what is typed, so `echo tp2-claim` paints the needle
626 # before it has run anything; the expect then matches the echo and the
627 # script detaches while the OUTPUT is still in flight. `printf 'tp2-%s'
628 # claim` keeps the literal off the typed line, so a hit is proof the
629 # shell ran it — the same argument-splitting trick the M5 scenario uses.
630 #
631 # The LAST thing expected must be the prompt, not the output. A shell
632 # writes its result and its next prompt as two separate writes, so a
633 # script that detaches on the result can stop one frame short of the
634 # grid the daemon holds. PS1 is set to a marker here purely so that
635 # prompt is assertable: it is the quiesce signal that would otherwise
636 # have to be a sleep.
637 set +e
638 timeout 40 "$PTYCLIENT" --cols 100 --rows 30 --out "$OUT.tp2a" --err "$OUT.tp2a.err" -- \
639 "$MUX" --sock "$SOCK12" > "$OUT.tp2a.log" 2>&1 <<'EOF'
640 expect \x1b[?1049h 10000
641 send PS1=tp2rdy@\n
642 # Twice, and the fixture's cursor is what makes that meaningful: the first
643 # hit is the ECHO of the assignment line, the second is the prompt the
644 # shell painted from it. Only the second proves PS1 took effect.
645 #
646 # Counting sentinel occurrences is safe HERE and nowhere else in tp2: this
647 # grid is nearly empty, so nothing scrolls, no repaint splits across
648 # deltas, and the two hits keep the order they were painted in. tp2b runs
649 # on a full screen and cannot assume any of that — see the hazard note
650 # there before copying this shape.
651 expect tp2rdy@ 10000
652 expect tp2rdy@ 10000
653 send printf 'tp2-%s\\n' claim\n
654 expect tp2-claim 10000
655 expect tp2rdy@ 10000
656 settle 500 15000
657 send \x1c\x1c
658 waitexit 10000
659 EOF
660 RC=$?
661 set -e
662 [ "$RC" -eq 0 ] || {
663 echo "e2e FAIL: tp2a ptyclient exited $RC:"; cat "$OUT.tp2a.log"; exit 1; }
664 assert_converged "$OUT.tp2a" "$SOCK12" "pty attach at 100x30" 100 30
665
666 # Resize UP, not down: a replica LARGER than the grid is invisible (the
667 # paint clips at min(replica, tty) either way), so only a replica left
668 # too SMALL has anywhere to show. `seq 1 60` fills the screen so growing
669 # to 30 rows has to pull history down rather than append blanks, and the
670 # 95-wide row is wider than the stale 90 and narrower than the new 100.
671 #
672 # PS1 is still tp2rdy@ from tp2a: same daemon, same shell, and the
673 # assignment outlived the client that typed it. Note where the prompt
674 # expects sit — the sentinel is re-emitted whenever the prompt ROW is
675 # repainted, including by the echo of a command typed at it, so each
676 # trailing prompt is consumed BEFORE the resize. That leaves the expect
677 # after `resize` satisfiable only by the answering snapshot's repaint.
678 #
679 # THE HAZARD, and it is worth stating precisely because three shapes of
680 # this scenario died to it: on a FULL screen the output scrolls, and the
681 # daemon can flush the new prompt row in one delta and the scrolled rows
682 # in a LATER one. Prompt-row sentinels therefore have NO reliable order
683 # relative to the output rows around them, and counting their occurrences
684 # fails in both directions. Too late: the sentinel's bytes arrive before
685 # the output needle that precedes it in the script, the needle's match
686 # consumes the cursor past them, and the next sentinel expect waits on a
687 # shell that has gone idle — a hang. Too early: a post-resize sentinel
688 # matches bytes from BEFORE the resize and the client detaches mid-repaint,
689 # leaving a capture in the old geometry. Measured under load, that second
690 # one is the dangerous one: it produces a diff byte-identical to row 7's
691 # — missing rows and a still-wrapped wide row — so a healthy build fails
692 # wearing the bug's own signature.
693 #
694 # The immune forms, and the only ones used below. UNIQUE OUTPUT MARKERS
695 # (`fill-done`, `00007`) for progress: each is printed once, by a command
696 # whose typed line cannot contain it, so its arrival is a fact about the
697 # session rather than about paint order. Occurrence-counting of a
698 # REPAINTED row — which is what a prompt sentinel is — is not a
699 # substitute, and is what the three earlier shapes of this leg died to.
700 #
701 # `expect 00007` earns its place as more than a marker: its arrival proves
702 # the daemon processed the wide row while the grid was still 90 columns,
703 # which is what puts the damage on the screen the resize will repaint from.
704 #
705 # After the resize the leg asserts TWO different things, and neither one
706 # implies the other:
707 # the 91-zero WITNESS proves the reflowed paint ARRIVED. 91 contiguous
708 # zeros cannot exist at width 90 — the row is painted wrapped there — so
709 # this needle is structurally incapable of firing early, whatever order
710 # the rows came in. It is also not the load risk an extra deadline looks
711 # like: this needle went 150/150 under 16-way load, at the same load that
712 # broke the two prompt-sentinel shapes 9 and 15 failures in 150.
713 # `settle` then proves nothing further is IN FLIGHT. The witness alone
714 # would permit a detach mid-repaint, with the rest of the screen still
715 # on its way.
716 # Dropping the witness and keeping only settle looks equivalent and is
717 # not: settle succeeds on 500ms of silence measured from its own start and
718 # never requires that a single byte arrived, so a healthy build whose
719 # repaint has not BEGUN inside that window satisfies it having observed
720 # nothing at all — leaving a pre-resize capture that diffs exactly like
721 # row 7. It would also be used outside its own precondition, which is that
722 # the session is already idle; straight after a resize the answering
723 # snapshot has not been generated yet.
724 set +e
725 timeout 40 "$PTYCLIENT" --cols 90 --rows 28 --out "$OUT.tp2b" --err "$OUT.tp2b.err" -- \
726 "$MUX" --sock "$SOCK12" > "$OUT.tp2b.log" 2>&1 <<'EOF'
727 expect \x1b[?1049h 10000
728 send seq 1 60; printf 'fill-%s\\n' done\n
729 expect fill-done 10000
730 send printf '%095d\\n' 7\n
731 expect 00007 10000
732 resize 100 30
733 expect 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 10000
734 settle 500 15000
735 send \x1c\x1c
736 waitexit 10000
737 EOF
738 RC=$?
739 set -e
740 [ "$RC" -eq 0 ] || {
741 echo "e2e FAIL: tp2b ptyclient exited $RC:"; cat "$OUT.tp2b.log"; exit 1; }
742 # Graded against the mutation (row 7: the prefix resize commented out) in
743 # this exact shape: 10 targeted runs, 10 red, plus a red full-suite run. What
744 # dies is the WITNESS, above — a replica still 90 wide cannot paint 91
745 # contiguous zeros, so the leg fails on the fixture timeout at that verb
746 # ("expect ... did not arrive within 10000ms", exit 3) and the RC check
747 # reports it in ~10s.
748 #
749 # This assertion is the backstop rather than the tripwire, and it is kept
750 # because it sees what the witness does not: before the witness existed
751 # the same build reached here and diverged with both halves of the stale
752 # geometry in one diff — a 30-row grid coming back short its top rows,
753 # because a 28-row replica has only 28 rows to repaint from, and the
754 # 95-wide row still WRAPPED at 90 into "...0" + "00007" while the daemon
755 # had rejoined it into one row at 100.
756 assert_converged "$OUT.tp2b" "$SOCK12" "pty resize mid-session" 100 30
757 rm_swept "$OUT.tp2a" "$OUT.tp2a.err" "$OUT.tp2a.log" \
758 "$OUT.tp2b" "$OUT.tp2b.err" "$OUT.tp2b.log" "$OUT.tp2.d"
759 # Closed here like every other per-scenario daemon, not left to the trap:
760 # tp1 runs below and would otherwise share the box with a daemon nobody is
761 # watching, holding a session whose shell is still alive.
762 softkill "$D12PID" || true
763 D12PID=""
764 ok "a pty client resizes: snapshot prefix applied, winch follows the tty"
765
766 # ---- M12 tp1: reconnect while scrolled (row 18 + the 412f38f pin) ------
767 # The session fills its scrollback BY ITSELF and then execs cat: no
768 # pre-tear typing means no pre-tear predictions, so the predict counters at
769 # exit belong entirely to the one post-resync keystroke — which is what
770 # lets them isolate the 412f38f fix. cat leaves the pty canonical and
771 # echoing, the `.always` tier, so a real keystroke predicts.
772 cat > "$TP1SH" <<'EOF'
773 #!/bin/sh
774 seq 1 100
775 exec /bin/cat
776 EOF
777 chmod +x "$TP1SH"
778 start_daemon "$SOCK13" "$OUT.tp1.d" "tp1 daemon never bound" --shell "$TP1SH"
779 D13PID=$DPID
780 # Attach only after seq finished: the client must be served a snapshot of
781 # the TAIL, so the scrollback page it fetches later is content it has never
782 # been sent. Observed at 80x24: the attach snapshot carries rows 78..100 and
783 # the session goes quiet there (cat prints no prompt), history_rows is 77,
784 # and page 1 is rows 54..77. Every needle below is chosen against those
785 # numbers, which is why this wait is part of the scenario and not a nicety.
786 i=0
787 until "$MUXD" dump --sock "$SOCK13" | grep -q "100"; do
788 i=$((i+1)); [ "$i" -lt 100 ] || { echo "e2e FAIL: tp1 session never finished seq"; exit 1; }
789 sleep 0.1
790 done
791
792 # MUX_PREDICT_STATS is exported into the FIXTURE's environment, not spelled
793 # as a `VAR=x cmd` prefix on the client: the fixture execve()s its child
794 # with an absolute path and no PATH search (src/pty.zig), so `env` would
795 # have to be an absolute path too, and the child inherits this environment
796 # anyway. Getting that wrong costs a client that exits 127 before the first
797 # verb, which the fixture reports as "closed the pty before ... matched".
798 #
799 # EXACTLY ONE keystroke, and that is the load-bearing choice in this script.
800 # offerKeystroke refuses any stdin chunk that is not one byte — an escape
801 # sequence, a multi-byte character, or a paste — and counts it suppressed.
802 # Consecutive `send`s have no barrier between them, so the client's next
803 # read can pick up two of them at once and predict neither: measured at
804 # idle, a three-keystroke version of this leg scored made=3 on one run and
805 # made=2/suppressed=1 on the next. Under load all three could coalesce,
806 # which would read as made=0 — the mutation's own signature. One send
807 # cannot coalesce with anything, so the counters below are exact.
808 #
809 # suppressed=0 therefore says more than "the keystroke predicted", and the
810 # rest is worth writing down because a future edit to the input path could
811 # take it away silently: Ctrl-\ is answered and returned on before
812 # offerKeystroke is ever reached (client.zig:874), so the detach chord
813 # cannot land in the counter; and Shift+PageUp has to arrive as ONE read,
814 # because the whole-chunk mem.eql at client.zig:882 is what recognizes it —
815 # split across two reads it matches nothing, falls through to the keystroke
816 # path, and is both counted suppressed and forwarded to the shell as input.
817 MUX_PREDICT_STATS=1 timeout 60 "$PTYCLIENT" --cols 80 --rows 24 \
818 --out "$OUT.tp1" --err "$OUT.tp1.err" -- \
819 "$MUX" --via "$MUXD proxy --sock $SOCK13" > "$OUT.tp1.log" 2>&1 <<'EOF' &
820 expect \x1b[0m100 10000
821 send \x1b[5;2~
822 expect 60 10000
823 expect reconnecting 20000
824 expect \x1b[0m100 20000
825 send t
826 expect \x1b[80X\x1b[0mt 20000
827 settle 500 15000
828 send \x1c\x1c
829 waitexit 15000
830 EOF
831 TP1PID=$!
832 defer_kill "$TP1PID"
833
834 # The tear goes between verb 3 (the scroll view is on screen) and verb 4;
835 # the fixture's "done 3" line is the barrier. Every needle above is unique
836 # to its phase, which is what makes each expect sound under the no-counting
837 # rule:
838 # the first "100" is the attach snapshot's last row, and it is spelled
839 # with the SGR reset the paint puts in front of every row because a bare
840 # "100" is no longer unique to it: the client asks its own terminal for
841 # mouse reporting with `\x1b[?1000h`, which lands in this capture before
842 # any row does. Measured, not feared — that is the collision that made
843 # this scenario fail when the wheel landed;
844 # "60" is in the history page (54..77) and in NOTHING before it. The
845 # snapshot's rows are 78..100, and the escapes around them cannot spell
846 # it either — but that is a fact about THIS session, not about the paint
847 # code: both paint paths end with a CUP carrying the cursor's real
848 # column (client.zig:968, :991), and a cursor at column 60 would put a
849 # "60" in the stream. Here it cannot be: the session is cat in canonical
850 # mode with one keystroke typed after the tear, so that column is 1 or 2
851 # for the whole scenario, and the row half of a CUP tops out at 24;
852 # "reconnecting" is the banner, painted once, on the tear;
853 # the SECOND "\x1b[0m100" can only be the post-resync repaint: the scroll view
854 # holds 54..77, and the banner carries no digits but its own position;
855 # the last needle is STRUCTURAL, and it has to be. In canonical mode the
856 # only thing a single keystroke produces is the line discipline's echo of
857 # the very glyph the prediction just painted, so no content needle can
858 # tell the daemon's answer from the client's own guess. `\x1b[80X` can:
859 # it is emitted by exactly one paint path in the client, the one that
860 # paints a DELTA (client.zig, paintDeltaClipped), so its arrival means a
861 # frame came back. The prediction paints the same glyph underlined and
862 # without an erase ("\x1b[24;1H\x1b[4mt\x1b[0m"), and the resync repaint
863 # erases the whole screen with \x1b[2J instead — neither can forge this.
864 # The two post-tear deadlines are 20000 because a reconnect is allowed to
865 # retry: the daemon may not have reaped the dead client's slot yet, and the
866 # client tolerates that for reconnect_grace_ms (5s) with a backoff that
867 # reaches 2s. That is a budget for a documented retry loop, not a deadline
868 # tuned until a flake stopped.
869 wait_for "$OUT.tp1.log" "done 3" 20 || {
870 echo "e2e FAIL: tp1 never reached the scroll view:"; cat "$OUT.tp1.log"; exit 1; }
871 TP1PROXY=$(proxy_pid "$SOCK13")
872 [ -n "$TP1PROXY" ] || { echo "e2e FAIL: tp1: no proxy to tear"; exit 1; }
873 kill -9 "$TP1PROXY"
874 # The daemon must still be SERVING, which is what `kill -0` cannot tell us:
875 # it is our own backgrounded child, so a dead one is a zombie until the
876 # shell reaps it, and signalling a zombie succeeds. A dump answers only if
877 # the process is alive and its accept loop is running.
878 "$MUXD" dump --sock "$SOCK13" >/dev/null 2>&1 || {
879 echo "e2e FAIL: tp1 tear killed the daemon, not the proxy"; exit 1; }
880
881 set +e
882 wait "$TP1PID"
883 RC=$?
884 TP1PID=""
885 set -e
886 [ "$RC" -eq 0 ] || {
887 echo "e2e FAIL: tp1 ptyclient exited $RC:"; cat "$OUT.tp1.log"
888 cat -v "$OUT.tp1.err" 2>/dev/null; exit 1; }
889
890 # Verb 5 is row 18's catch: a resync that fails to leave the scroll view
891 # suppresses every live paint, so the second "100" never arrives.
892 #
893 # The counters are the 412f38f pin, and they are EXACT rather than floors.
894 # One keystroke in the whole scenario, typed after the resync onto a blank
895 # cell at a live cursor, in a session whose termios never moves: nothing
896 # here is timing-dependent, so every number is a fact.
897 # made=1, suppressed=0 is the fix working. With it reverted the overlay
898 # still believes it is scrolled and predictAt refuses on that bit alone —
899 # made=0, suppressed=1, with nothing else in the run to hide it.
900 # displayed=1 says the glyph reached the SCREEN, which `made` does not:
901 # the overlay queues while unconfident and paints nothing, so a client
902 # that lost the `.always` tier would still count the prediction.
903 # confirmed=1 and contradicted=0 say the daemon's own echo agreed with
904 # the guess, which is what makes the prediction a speedup and not a lie.
905 want_stat "$OUT.tp1.err" made 1 "pty scroll reconnect"
906 want_stat "$OUT.tp1.err" displayed 1 "pty scroll reconnect"
907 want_stat "$OUT.tp1.err" confirmed 1 "pty scroll reconnect"
908 want_stat "$OUT.tp1.err" contradicted 0 "pty scroll reconnect"
909 want_stat "$OUT.tp1.err" suppressed 0 "pty scroll reconnect"
910 assert_converged "$OUT.tp1" "$SOCK13" "pty scroll reconnect"
911
912 # The doctored control, extended to a pty capture: this capture's byte shape
913 # (alt screen, banner paints, a history page) exists nowhere else in the
914 # corpus, so it earns its own cannot-fail check.
915 #
916 # It CANNOT be the plain append the non-pty control uses. render replays
917 # only up to the LAST alt-screen exit, and a tty client's capture ends with
918 # one — so appended bytes land after the grid under test and change nothing.
919 # Measured: the doctored capture rendered byte-identical to the original,
920 # i.e. a control that could never fire. Dropping that trailing 8-byte exit
921 # puts the append back INSIDE the alt screen (a stream with no alt-exit is
922 # replayed whole), which corrupts exactly one row of the real grid — row 10,
923 # "87" in this session. The tail is asserted rather than assumed, because if
924 # the client's teardown ever stops ending there, the truncation would silently
925 # go back to being a no-op appended after the grid.
926 [ "$(tail -c 8 "$OUT.tp1" | od -An -tx1 | tr -d ' \n')" = "1b5b3f313034396c" ] || {
927 echo "e2e FAIL: tp1 capture does not end with the alt-screen exit;"
928 echo " the doctored control below would not be doctoring the grid"
929 tail -c 16 "$OUT.tp1" | cat -v; exit 1; }
930 head -c "$(( $(wc -c < "$OUT.tp1") - 8 ))" "$OUT.tp1" > "$OUT.tp1.doc"
931 printf '\033[10;1Hpty-doctor-glyphs' >> "$OUT.tp1.doc"
932 if converged_quiet "$OUT.tp1.doc" "$SOCK13"; then
933 echo "e2e FAIL: convergence control did not fire on a doctored pty capture"; exit 1
934 fi
935 rm_swept "$OUT.tp1.doc" "$OUT.tp1.doc.render" "$OUT.tp1.doc.dump" \
936 "$OUT.tp1.doc.render.n" "$OUT.tp1.doc.dump.n" "$OUT.tp1.doc.diff" \
937 "$OUT.tp1.doc.rvt" "$OUT.tp1.doc.dvt"
938 softkill "$D13PID" || true
939 D13PID=""
940 rm_swept "$OUT.tp1" "$OUT.tp1.err" "$OUT.tp1.log" "$OUT.tp1.d" "$TP1SH"
941 ok "reconnect while scrolled: view restored, prediction resumed"
942
943 # tp1 runs its fixture in the BACKGROUND — the tear has to happen while the 143 # tp1 runs its fixture in the BACKGROUND — the tear has to happen while the
944 # script is mid-flight — so its pid is tracked and killed like a daemon's. 144 # script is mid-flight — so its pid is tracked and killed like a daemon's.
945 # M13 auto-start + stop. Two paths: the proxy arc's daemon is spawned BY 145 # M13 auto-start + stop. Two paths: the proxy arc's daemon is spawned BY
test/e2e_02_predict.sh
Old New
@@ -0,0 +1,803 @@
1 # shellcheck shell=sh
2 # e2e_02_predict.sh — sourced by test/e2e.sh after e2e_lib.sh. Scenarios run in
3 # the order they stand in; see the lib's header for what this file may
4 # assume and what it must register.
5 # M9 prediction: each scenario needs a session whose LINE DISCIPLINE it
6 # controls, so they cannot share the long-lived /bin/sh daemon.
7 SOCK5="${TMPDIR:-/tmp}/muxd-e2e-pred-$$.sock"
8 defer_sock "$SOCK5"
9 SOCK6="${TMPDIR:-/tmp}/muxd-e2e-pw-$$.sock"
10 defer_sock "$SOCK6"
11 SOCK7="${TMPDIR:-/tmp}/muxd-e2e-raw-$$.sock"
12 defer_sock "$SOCK7"
13 PWSH="${TMPDIR:-/tmp}/mux-e2e-pw-$$.sh"
14 defer_rm "$PWSH"
15 # M12 pty scenarios. Each needs a daemon whose grid size it owns: tp2
16 # resizes the grid twice and tp1's session is a scrollback-generating
17 # wrapper, so neither can share the long-lived /bin/sh daemon.
18 SOCK12="${TMPDIR:-/tmp}/muxd-e2e-tp2-$$.sock"
19 defer_sock "$SOCK12"
20 SOCK13="${TMPDIR:-/tmp}/muxd-e2e-tp1-$$.sock"
21 defer_sock "$SOCK13"
22 # tp1's session: a wrapper that fills the scrollback and then becomes cat.
23 TP1SH="${TMPDIR:-/tmp}/mux-e2e-tp1-$$.sh"
24 defer_rm "$TP1SH"
25
26 # ---- M9: prediction ---------------------------------------------------
27 #
28 # Every scenario here asserts COUNTERS, because the screen cannot tell a
29 # predicted glyph from an echoed one — that is the entire point of the
30 # feature — so a rendering test would pass just as happily with prediction
31 # switched off.
32 #
33 # The session shell is /bin/cat, not /bin/sh. An interactive bash prompt is
34 # icanon=0/echo=0 (readline echoes for itself), which is the ADAPTIVE tier,
35 # not the always-predict one; cat is a genuinely canonical reader and the
36 # only way to exercise `.always` here. That distinction cost a milestone's
37 # worth of confusion to learn and is why it is written down twice.
38
39 # Delay per direction. The round trip is twice this, and every assertion
40 # below about "before the daemon could have answered" is measured against
41 # it. Chosen against three constraints at once, which is why it is not a
42 # round number picked for looks:
43 #
44 # - Larger than delaypipe's built-in default of 150, and far enough from
45 # it to be told apart. The snapshot below is taken 450ms after the
46 # keystroke: the echo cannot have returned over this 600ms round trip,
47 # but WOULD have over the 300ms one the default gives. A DELAY_MS lookup
48 # that silently stopped working — leaving every scenario here measuring
49 # a path twice as fast as it claimed — fails that assertion rather than
50 # passing quietly.
51 # - Small enough that the round trip stays clear of the overlay's 1000ms
52 # expiry bound. This is a real ceiling, not a test artefact: delaypipe
53 # delays each chunk serially, so a burst's later keystrokes queue behind
54 # the earlier ones and age while they wait. At 400ms each way the burst
55 # scenario went to confirmed=1 — the predictions were expiring before
56 # their echo could arrive, exactly as the overlay is designed to do.
57 # - Leaving margin on both sides, so neither is a race under load.
58 PDELAY=300
59
60 # Own file, not the client capture: the daemon's fd survives the client's truncation and would write into the replayed stream.
61 start_daemon "$SOCK5" "$OUT.p1.d" "prediction daemon never bound" --shell /bin/cat
62 D5PID=$DPID
63
64 # 1. Line mode: the glyph is on screen before the round trip could have
65 # delivered it.
66 set +e
67 { sleep 2; printf 'z'; sleep 4; printf '\034\034'; } | \
68 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 40 "$MUX" \
69 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK5 | $DELAYPIPE" \
70 > "$OUT.p1" 2> "$OUT.p1.err" &
71 P1PID=$!
72 defer_kill "$P1PID"
73 set -e
74 # Snapshot the client output 450ms after the keystroke: well
75 # before this 600ms round trip could bring the pty own echo back, and well
76 # AFTER the 300ms round trip delaypipe default of 150 would have given.
77 sleep 2.45
78 cp "$OUT.p1" "$OUT.p1.early" 2>/dev/null || true
79 set +e
80 wait "$P1PID"
81 RC=$?
82 set -e
83 [ "$RC" -eq 0 ] || {
84 echo "e2e FAIL: line-mode prediction client exited $RC"
85 cat "$OUT.p1" "$OUT.p1.err" 2>/dev/null; exit 1; }
86
87 # The underlined glyph, in the snapshot taken before the echo could arrive.
88 # This is the whole claim of the milestone, as an effect rather than a
89 # counter: the character was on the screen while it was still in flight.
90 grep -q "$(printf '\033\[4mz')" "$OUT.p1.early" || {
91 echo "e2e FAIL: no predicted glyph 450ms after the keystroke (RTT is $((PDELAY * 2))ms)"
92 echo "--- early snapshot ---"; cat -v "$OUT.p1.early"; exit 1;
93 }
94 # ...and the daemon's own answer was NOT there yet, which is what makes the
95 # line above mean anything. A delta paints a row with ECH before its
96 # content; the prediction never does.
97 if grep -q "$(printf '\033\[80X\033\[0mz')" "$OUT.p1.early"; then
98 echo "e2e FAIL: the echo arrived within 450ms, so the path is not $((PDELAY * 2))ms;"
99 echo " either DELAY_MS was ignored (default is 150) or delaypipe is not delaying"
100 exit 1
101 fi
102 want_stat "$OUT.p1.err" contradicted 0 "line mode"
103 CONF=$(predict_stat "$OUT.p1.err" confirmed)
104 [ -n "$CONF" ] && [ "$CONF" -ge 1 ] || {
105 echo "e2e FAIL: line mode: confirmed=$CONF, want >=1"; grep "^predict " "$OUT.p1.err"; exit 1;
106 }
107 # The session really received the keystroke: cat echoed it into the grid.
108 "$MUXD" dump --sock "$SOCK5" | grep -q "z" || {
109 echo "e2e FAIL: line mode: daemon grid never saw the keystroke"; exit 1;
110 }
111 assert_converged "$OUT.p1" "$SOCK5" "line-mode prediction"
112
113 # 2. A burst outrunning the round trip. This is reconcile v2's reason to
114 # exist: under the old rule every character after the first was judged by
115 # a frame built before it was typed, read as a contradiction, and the
116 # whole queue was flushed once per round trip. If anyone regresses the
117 # three-way judgment, this scenario is what says so.
118 set +e
119 { sleep 2; for c in b u r s t; do printf '%s' "$c"; sleep 0.25; done; \
120 sleep 4; printf '\034\034'; } | \
121 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 40 "$MUX" \
122 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK5 | $DELAYPIPE" \
123 > "$OUT.pb" 2> "$OUT.pb.err"
124 RC=$?
125 set -e
126 [ "$RC" -eq 0 ] || {
127 echo "e2e FAIL: burst client exited $RC"
128 cat "$OUT.pb" "$OUT.pb.err" 2>/dev/null; exit 1; }
129
130 # Five keystrokes typed at 250ms into a 600ms round trip: three are always
131 # outstanding at once (at t=600, when the first is confirmed, the keys typed
132 # at 0, 250 and 500 are all in flight). `made` is asserted too, so two
133 # keystrokes arriving in one read (which would be suppressed as a multi-byte
134 # chunk, not predicted) fails loudly instead of quietly weakening the test.
135 #
136 # The interval is 250 and not 200 because of a bound that has to be COMPUTED
137 # rather than eyeballed. delaypipe delivers one chunk per DELAY_MS serially
138 # in EACH direction, so keystrokes closer together than DELAY_MS queue, and
139 # each one ages (DELAY_MS - interval) longer than the one before it:
140 #
141 # age at which keystroke k is confirmed = 2*DELAY_MS + k*(DELAY_MS - interval)
142 #
143 # At interval=200 the fifth keystroke lands at 600 + 4*100 = 1000ms, which is
144 # exactly predict.expire_after_ms: a coin flip, not a margin. It failed 4 runs
145 # in 20 on an idle machine. At 250 it is 600 + 4*50 = 800ms, measured at
146 # 798-799 across 12 runs. The same formula puts the SECOND keystroke at
147 # 1000ms when DELAY_MS=400, which is the confirmed=1 collapse noted above.
148 want_stat "$OUT.pb.err" made 5 "burst"
149 # `confirmed` is a LIVENESS counter and the only timing-dependent one here:
150 # the fifth confirmation can still be in flight when the detach byte lands,
151 # which is why it is a floor rather than an equality. Observed 1 run in 3
152 # under load (review runs at 715f71b, 2026-08-09), and decisions.md records
153 # the same counter coming up one short on the M9 WAN burst (made=55
154 # confirmed=54 abandoned=1, the mode-transition churn flush).
155 #
156 # `contradicted` and `expired` stay EXACT. They are the correctness claims —
157 # a prediction judged wrong, or one that aged out unjudged, is a defect
158 # whenever it happens, and nothing about timing makes either acceptable.
159 want_stat_ge "$OUT.pb.err" confirmed 4 "burst"
160 # `expired` first: an expiry runs through the same abandonAll tail as a
161 # refutation and increments BOTH counters, so contradicted >= expired always.
162 # Asserting contradicted first reports an age-out as "a prediction judged
163 # wrong" — a different defect with a different cause, and the reason this
164 # flake was first investigated as a reconcile bug.
165 want_stat "$OUT.pb.err" expired 0 "burst"
166 want_stat "$OUT.pb.err" contradicted 0 "burst"
167 "$MUXD" dump --sock "$SOCK5" | grep -q "burst" || {
168 echo "e2e FAIL: burst: daemon grid does not hold the typed text"; exit 1;
169 }
170 assert_converged "$OUT.pb" "$SOCK5" "burst"
171
172 softkill "$D5PID" || true
173 D5PID=""
174
175 # 3. A password prompt. The session is canonical with echo OFF from the
176 # moment it starts, so the whole run sits in the tier that predicts
177 # nothing at all.
178 cat > "$PWSH" <<'PWEOF'
179 #!/bin/sh
180 stty -echo
181 printf 'pw-ready\n'
182 read secret
183 printf 'pw-len-%s\n' "${#secret}"
184 # Blocks forever rather than exiting: the shell exiting would end the
185 # session, and the daemon would be gone before its grid could be read.
186 read _hold
187 PWEOF
188 chmod +x "$PWSH"
189
190 start_daemon "$SOCK6" "$OUT.pw.d" "password daemon never bound" --shell "$PWSH"
191 D6PID=$DPID
192
193 # One character per write, and NOT as a single `printf hunter2`. A whole
194 # word in one write reaches the client as a multi-byte chunk, which is
195 # refused for being multi-byte before the tier is ever consulted — so the
196 # scenario would pass with the password tier completely broken. It did:
197 # mapping echo-off canonical to the always-predict tier survived this
198 # scenario until the typing was split up.
199 set +e
200 { sleep 2; for c in h u n t e r 2; do printf '%s' "$c"; sleep 0.2; done; \
201 printf '\n'; sleep 2; printf '\034\034'; } | \
202 MUX_PREDICT_STATS=1 timeout 40 "$MUX" --sock "$SOCK6" > "$OUT.pw" 2> "$OUT.pw.err"
203 RC=$?
204 set -e
205 [ "$RC" -eq 0 ] || {
206 echo "e2e FAIL: password client exited $RC"
207 cat "$OUT.pw" "$OUT.pw.err" 2>/dev/null; exit 1; }
208
209 # Nothing was shown, and nothing was even attempted: echo-off canonical is
210 # the tier where a prediction must never be MADE, not merely never painted,
211 # because a made prediction sits in a buffer the overlay paints from.
212 want_stat "$OUT.pw.err" made 0 "password"
213 want_stat "$OUT.pw.err" displayed 0 "password"
214 # The effect, not the counter: the secret appears nowhere in the bytes the
215 # client wrote — either stream. The stats line cannot carry a secret, but
216 # checking both costs one word and the claim is about everything the client
217 # emitted, not about the half of it that happens to be the screen.
218 if grep -q "hunter2" "$OUT.pw" "$OUT.pw.err"; then
219 echo "e2e FAIL: the password appears in the client's terminal output"; exit 1
220 fi
221 # ...and the run was not vacuous: the shell really received all seven
222 # characters, so the absence above is prediction declining rather than
223 # nothing having been typed.
224 "$MUXD" dump --sock "$SOCK6" | grep -q "pw-len-7" || {
225 echo "e2e FAIL: the password never reached the shell; the absence above proves nothing"
226 "$MUXD" dump --sock "$SOCK6"; exit 1;
227 }
228 assert_converged "$OUT.pw" "$SOCK6" "password"
229
230 softkill "$D6PID" || true
231 D6PID=""
232
233 # 4. Raw mode: display is earned, then lost to a keystroke the application
234 # swallows. rawmode echoes like an editor in insert mode until it is sent
235 # 0x00, after which it consumes input and prints nothing.
236 #
237 # Through delaypipe, like leg 1, and for the same reason leg 4b below is
238 # NOT: the overlay measures its own round trip and hides everything on a
239 # path faster than `predict.local_below_ms`. This leg is about the
240 # earn/lose policy, which only a remote path lets through to the screen.
241 # Keys are 1s apart so each confirm (600ms) lands before the next key —
242 # at 0.5s `c` would find only `a` confirmed and stay hidden.
243 start_daemon "$SOCK7" "$OUT.rw.d" "rawmode daemon never bound" --shell "$RAWMODE"
244 D7PID=$DPID
245
246 set +e
247 { sleep 2; for c in a b c; do printf '%s' "$c"; sleep 1; done; \
248 printf '\000'; sleep 1; printf 'j'; sleep 2.5; printf '\034\034'; } | \
249 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 40 "$MUX" \
250 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK7 | $DELAYPIPE" \
251 > "$OUT.rw" 2> "$OUT.rw.err"
252 RC=$?
253 set -e
254 [ "$RC" -eq 0 ] || {
255 echo "e2e FAIL: rawmode client exited $RC"
256 cat "$OUT.rw" "$OUT.rw.err" 2>/dev/null; exit 1; }
257
258 # Four printable keystrokes made predictions; the 0x00 that switched modes
259 # is not printable and was refused.
260 want_stat "$OUT.rw.err" made 4 "raw mode"
261 want_stat "$OUT.rw.err" suppressed 1 "raw mode"
262 # a and b are invisible while promotion is earned; c is the first painted,
263 # and j is painted because by then display had been earned.
264 want_stat "$OUT.rw.err" displayed 2 "raw mode"
265 want_stat "$OUT.rw.err" confirmed 3 "raw mode"
266 # j was swallowed: no frame ever answered the cell it was drawn in, so the
267 # expiry bound retired it rather than leaving a phantom glyph on screen for
268 # the rest of the session.
269 want_stat "$OUT.rw.err" expired 1 "raw mode"
270 want_stat "$OUT.rw.err" contradicted 1 "raw mode"
271 want_stat "$OUT.rw.err" abandoned 1 "raw mode"
272
273 # Every prediction accounted for: made = confirmed + abandoned + pending,
274 # and the queue is empty at exit, so nothing survived unexplained.
275 RW_MADE=$(predict_stat "$OUT.rw.err" made)
276 RW_CONF=$(predict_stat "$OUT.rw.err" confirmed)
277 RW_ABND=$(predict_stat "$OUT.rw.err" abandoned)
278 [ "$((RW_MADE - RW_CONF - RW_ABND))" -eq 0 ] || {
279 echo "e2e FAIL: raw mode: $((RW_MADE - RW_CONF - RW_ABND)) predictions unaccounted for"
280 grep "^predict " "$OUT.rw.err"; exit 1;
281 }
282 # The counters say `j` expired; this says the screen agrees. An expiry that
283 # retired the prediction in the book but left its underlined glyph painted is
284 # exactly the phantom the styled half of this check exists to catch.
285 assert_converged "$OUT.rw" "$SOCK7" "raw mode"
286
287 # 4b. The same keystrokes on the bare socket: a local daemon answers in
288 # about a millisecond, and that is the one path on which a prediction
289 # can only ever be SEEN when it is wrong. Display is still earned (the
290 # counters say so) and nothing is painted — `local` is where `c` and
291 # `j` went instead of `displayed`. `j` still expires: the queue keeps
292 # judging while hidden, which is how the gate would lift again on a
293 # box that got slow. The screen check is the point: no phantom `j`,
294 # because there was never an underlined one to leave behind.
295 #
296 # A fresh daemon: the rawmode above has taken its 0x00 and echoes
297 # nothing now, and the leg needs the insert-mode half back.
298 softkill "$D7PID" || true
299 wait "$D7PID" 2>/dev/null || true
300 rm -f "$SOCK7"
301 start_daemon "$SOCK7" "$OUT.rwl.d" "rawmode local daemon never bound" --shell "$RAWMODE"
302 D7PID=$DPID
303
304 set +e
305 { sleep 2; for c in a b c; do printf '%s' "$c"; sleep 0.5; done; \
306 printf '\000'; sleep 0.5; printf 'j'; sleep 2.5; printf '\034\034'; } | \
307 MUX_PREDICT_STATS=1 timeout 40 "$MUX" --sock "$SOCK7" > "$OUT.rwl" 2> "$OUT.rwl.err"
308 RC=$?
309 set -e
310 [ "$RC" -eq 0 ] || {
311 echo "e2e FAIL: rawmode local client exited $RC"
312 cat "$OUT.rwl" "$OUT.rwl.err" 2>/dev/null; exit 1; }
313 want_stat "$OUT.rwl.err" made 4 "raw mode local"
314 want_stat "$OUT.rwl.err" confirmed 3 "raw mode local"
315 want_stat "$OUT.rwl.err" displayed 0 "raw mode local"
316 want_stat "$OUT.rwl.err" local 2 "raw mode local"
317 want_stat "$OUT.rwl.err" expired 1 "raw mode local"
318 grep -q "$(printf '\033\[4m')" "$OUT.rwl" && {
319 echo "e2e FAIL: raw mode local: an underlined prediction reached the screen"
320 cat -v "$OUT.rwl"; exit 1;
321 }
322 assert_converged "$OUT.rwl" "$SOCK7" "raw mode local"
323
324 softkill "$D7PID" || true
325 D7PID=""
326
327 # 5. A transport torn down with predictions outstanding. The overlay must
328 # come back empty: what was queued was predicted against a connection
329 # that no longer exists.
330 start_daemon "$SOCK5" "$OUT.pr.d" "reconnect daemon never bound" --shell /bin/cat
331 D5PID=$DPID
332
333 set +e
334 { sleep 2; printf 'p'; sleep 5; printf 'q'; sleep 4; printf '\034\034'; } | \
335 DELAY_MS="$PDELAY" MUX_PREDICT_STATS=1 timeout 60 "$MUX" \
336 --via "$DELAYPIPE | $MUXD proxy --sock $SOCK5 | $DELAYPIPE" \
337 > "$OUT.pr" 2> "$OUT.pr.err" &
338 PRPID=$!
339 defer_kill "$PRPID"
340 set -e
341 # Kill the transport a tenth of a second after the keystroke, while its
342 # prediction is still outstanding — the round trip is 600ms, so it cannot
343 # have been judged yet. By comm+pid: the client's own argv contains the
344 # --via string, so a pattern kill would take out the client under test.
345 sleep 2.1
346 PP=$(proxy_pid "$SOCK5")
347 [ -n "$PP" ] || {
348 echo "e2e FAIL: no proxy to tear down"
349 cat "$OUT.pr" "$OUT.pr.err" 2>/dev/null; exit 1; }
350 kill -9 "$PP"
351
352 set +e
353 wait "$PRPID"
354 RC=$?
355 set -e
356 [ "$RC" -eq 0 ] || {
357 echo "e2e FAIL: reconnect client exited $RC"
358 cat "$OUT.pr" "$OUT.pr.err" 2>/dev/null; exit 1; }
359
360 # The queue is empty at exit, by attribution rather than by inspection:
361 # nothing predicted is still outstanding, and the flush that dropped the
362 # in-flight prediction was not counted as anybody being wrong.
363 PR_MADE=$(predict_stat "$OUT.pr.err" made)
364 PR_CONF=$(predict_stat "$OUT.pr.err" confirmed)
365 PR_ABND=$(predict_stat "$OUT.pr.err" abandoned)
366 [ -n "$PR_MADE" ] || {
367 echo "e2e FAIL: reconnect: no predict stats"
368 cat "$OUT.pr" "$OUT.pr.err" 2>/dev/null; exit 1; }
369 [ "$((PR_MADE - PR_CONF - PR_ABND))" -eq 0 ] || {
370 echo "e2e FAIL: reconnect: $((PR_MADE - PR_CONF - PR_ABND)) predictions outlived the tear"
371 grep "^predict " "$OUT.pr.err"; exit 1;
372 }
373 # And it was the reconnect that dropped it, not the expiry bound catching it
374 # a second later. Without this the scenario cannot tell the two apart: the
375 # resync after a reconnect flushes as well, so the outcome looks identical
376 # either way and removing the reconnect's own flush passes unnoticed. It
377 # did, until this line.
378 want_stat "$OUT.pr.err" expired 0 "reconnect"
379 [ -n "$PR_ABND" ] && [ "$PR_ABND" -ge 1 ] || {
380 echo "e2e FAIL: reconnect: nothing was abandoned ($PR_ABND), so no prediction was"
381 echo " outstanding when the transport died and the tear proves nothing"
382 grep "^predict " "$OUT.pr.err"; exit 1;
383 }
384 # The session survived and the client resumed into it: the keystroke typed
385 # after the tear reached the shell.
386 "$MUXD" dump --sock "$SOCK5" | grep -q "q" || {
387 echo "e2e FAIL: reconnect: the post-tear keystroke never reached the session"
388 "$MUXD" dump --sock "$SOCK5"; exit 1;
389 }
390 assert_converged "$OUT.pr" "$SOCK5" "reconnect flush"
391
392 softkill "$D5PID" || true
393 D5PID=""
394
395 # ---- M12: ptyclient fixture controls ----------------------------------
396 # Before any scenario trusts the fixture, prove both directions: a
397 # roundtrip over plain /bin/cat (the pty line discipline's own echo
398 # answers — no mux anywhere, so a failure here is the FIXTURE'S), and an
399 # expect that cannot match, which must time out, exit nonzero, and say
400 # what it did see. A check that cannot fail proves nothing.
401 PCLOG="$OUT.pc.log"
402 set +e
403 timeout 30 "$PTYCLIENT" --cols 80 --rows 24 --out "$OUT.pc" --err "$OUT.pc.err" -- /bin/cat > "$PCLOG" 2>&1 <<'EOF'
404 send hello\n
405 expect hello 10000
406 send \x04
407 waitexit 10000
408 EOF
409 RC=$?
410 set -e
411 [ "$RC" -eq 0 ] || {
412 echo "e2e FAIL: ptyclient roundtrip over cat exited $RC:"; cat "$PCLOG"; exit 1; }
413 grep -q "hello" "$OUT.pc" || {
414 echo "e2e FAIL: ptyclient capture missing the pty echo"; cat -v "$OUT.pc"; exit 1; }
415 # The must-fail leg. 500ms: nothing is being waited FOR — the needle never
416 # arrives by construction — so the deadline only bounds the control's cost.
417 set +e
418 timeout 30 "$PTYCLIENT" --cols 80 --rows 24 --out "$OUT.pc2" --err "$OUT.pc2.err" -- /bin/cat > "$PCLOG.2" 2>&1 <<'EOF'
419 expect never-going-to-match 500
420 EOF
421 RC=$?
422 set -e
423 # The literal, not merely nonzero: 3 is EXIT_TIMEOUT, and the tp1/tp2
424 # scenarios read that code to tell a missed needle apart from a client that
425 # died or a script the fixture refused. A bare -ne 0 would pass just as
426 # happily on a usage error that never ran the script at all.
427 [ "$RC" -eq 3 ] || {
428 echo "e2e FAIL: ptyclient expect control exited $RC, want 3 (EXIT_TIMEOUT)"; cat "$PCLOG.2"; exit 1; }
429 grep -q "did not arrive" "$PCLOG.2" || {
430 echo "e2e FAIL: ptyclient timeout fired but never said what it saw"; cat "$PCLOG.2"; exit 1; }
431 # The stderr split, end to end: the client's predict stats must land in the
432 # .err sibling and NOT in the capture the convergence machinery diffs. Only
433 # a child that writes to BOTH streams can tell a working split from a child
434 # that simply never wrote to stderr.
435 set +e
436 timeout 30 "$PTYCLIENT" --cols 80 --rows 24 --out "$OUT.pc3" --err "$OUT.pc3.err" -- \
437 /bin/sh -c 'printf pc-stdout; printf pc-stderr 1>&2' > "$PCLOG.3" 2>&1 <<'EOF'
438 expect pc-stdout 10000
439 waitexit 10000
440 EOF
441 RC=$?
442 set -e
443 [ "$RC" -eq 0 ] || {
444 echo "e2e FAIL: ptyclient stderr-split leg exited $RC:"; cat "$PCLOG.3"; exit 1; }
445 grep -q "pc-stdout" "$OUT.pc3" || {
446 echo "e2e FAIL: ptyclient capture missing the child's stdout"; cat -v "$OUT.pc3"; exit 1; }
447 ! grep -q "pc-stderr" "$OUT.pc3" || {
448 echo "e2e FAIL: the child's stderr leaked into the capture"; cat -v "$OUT.pc3"; exit 1; }
449 grep -q "pc-stderr" "$OUT.pc3.err" || {
450 echo "e2e FAIL: the child's stderr reached neither the capture nor --err"
451 cat -v "$OUT.pc3.err"; exit 1; }
452 rm_swept "$OUT.pc" "$OUT.pc.err" "$OUT.pc2" "$OUT.pc2.err" "$OUT.pc3" "$OUT.pc3.err" \
453 "$PCLOG" "$PCLOG.2" "$PCLOG.3"
454 ok "ptyclient controls: pty echo roundtrips, impossible expect fails loudly, stderr stays off the capture"
455
456 # ---- M12 tp2: resize under a real tty (campaign row 7's branch) --------
457 # Two clients, two convergence points, because convergence is a scenario's
458 # LAST act after detach.
459 #
460 # tp2a is the suite's first attach at a size that is not the non-tty 80x24
461 # default: it pins the size a real tty reports reaching the daemon, and
462 # convergence measured against that size rather than the default. It does
463 # NOT exercise the snapshot resize prefix — an attach applies the
464 # attacher's size to the grid BEFORE the answering resync, so the prefix
465 # always equals the size the replica already has and the client's
466 # resize-on-prefix guard never fires. Verified by probe, not by reading.
467 #
468 # tp2b is the leg that catches row 7 (the prefix ignored), and it only
469 # catches it because of WHERE the damage can show. DELTA STREAMING
470 # bypasses the replica: those rows go straight to the tty, absolutely
471 # addressed and clipped to the tty, so a stale replica cannot spoil them
472 # and a wrong row is healed by the next delta that re-addresses it. The
473 # damage therefore has to survive to a FULL REPAINT, which is the paint
474 # that reads the replica back out and clips at min(replica, tty) — still
475 # on screen by then, and in a region the stale geometry cannot reach.
476 # Hence: fill the screen, put a 95-wide row on it, then winch UP to
477 # 100x30. With the prefix ignored the replica stays 90x28 and the repaint
478 # answering the resize reproduces the OLD geometry.
479 start_daemon "$SOCK12" "$OUT.tp2.d" "tp2 daemon never bound" --shell /bin/sh
480 D12PID=$DPID
481
482 # Two rules govern every script below, and breaking either one produces a
483 # capture that is honestly SHORT rather than wrong — a divergence whose
484 # diff blames the product for a scenario's mistake:
485 #
486 # A marker must not appear in the command line that produces it. The
487 # shell echoes what is typed, so `echo tp2-claim` paints the needle
488 # before it has run anything; the expect then matches the echo and the
489 # script detaches while the OUTPUT is still in flight. `printf 'tp2-%s'
490 # claim` keeps the literal off the typed line, so a hit is proof the
491 # shell ran it — the same argument-splitting trick the M5 scenario uses.
492 #
493 # The LAST thing expected must be the prompt, not the output. A shell
494 # writes its result and its next prompt as two separate writes, so a
495 # script that detaches on the result can stop one frame short of the
496 # grid the daemon holds. PS1 is set to a marker here purely so that
497 # prompt is assertable: it is the quiesce signal that would otherwise
498 # have to be a sleep.
499 set +e
500 timeout 40 "$PTYCLIENT" --cols 100 --rows 30 --out "$OUT.tp2a" --err "$OUT.tp2a.err" -- \
501 "$MUX" --sock "$SOCK12" > "$OUT.tp2a.log" 2>&1 <<'EOF'
502 expect \x1b[?1049h 10000
503 send PS1=tp2rdy@\n
504 # Twice, and the fixture's cursor is what makes that meaningful: the first
505 # hit is the ECHO of the assignment line, the second is the prompt the
506 # shell painted from it. Only the second proves PS1 took effect.
507 #
508 # Counting sentinel occurrences is safe HERE and nowhere else in tp2: this
509 # grid is nearly empty, so nothing scrolls, no repaint splits across
510 # deltas, and the two hits keep the order they were painted in. tp2b runs
511 # on a full screen and cannot assume any of that — see the hazard note
512 # there before copying this shape.
513 expect tp2rdy@ 10000
514 expect tp2rdy@ 10000
515 send printf 'tp2-%s\\n' claim\n
516 expect tp2-claim 10000
517 expect tp2rdy@ 10000
518 settle 500 15000
519 send \x1c\x1c
520 waitexit 10000
521 EOF
522 RC=$?
523 set -e
524 [ "$RC" -eq 0 ] || {
525 echo "e2e FAIL: tp2a ptyclient exited $RC:"; cat "$OUT.tp2a.log"; exit 1; }
526 assert_converged "$OUT.tp2a" "$SOCK12" "pty attach at 100x30" 100 30
527
528 # Resize UP, not down: a replica LARGER than the grid is invisible (the
529 # paint clips at min(replica, tty) either way), so only a replica left
530 # too SMALL has anywhere to show. `seq 1 60` fills the screen so growing
531 # to 30 rows has to pull history down rather than append blanks, and the
532 # 95-wide row is wider than the stale 90 and narrower than the new 100.
533 #
534 # PS1 is still tp2rdy@ from tp2a: same daemon, same shell, and the
535 # assignment outlived the client that typed it. Note where the prompt
536 # expects sit — the sentinel is re-emitted whenever the prompt ROW is
537 # repainted, including by the echo of a command typed at it, so each
538 # trailing prompt is consumed BEFORE the resize. That leaves the expect
539 # after `resize` satisfiable only by the answering snapshot's repaint.
540 #
541 # THE HAZARD, and it is worth stating precisely because three shapes of
542 # this scenario died to it: on a FULL screen the output scrolls, and the
543 # daemon can flush the new prompt row in one delta and the scrolled rows
544 # in a LATER one. Prompt-row sentinels therefore have NO reliable order
545 # relative to the output rows around them, and counting their occurrences
546 # fails in both directions. Too late: the sentinel's bytes arrive before
547 # the output needle that precedes it in the script, the needle's match
548 # consumes the cursor past them, and the next sentinel expect waits on a
549 # shell that has gone idle — a hang. Too early: a post-resize sentinel
550 # matches bytes from BEFORE the resize and the client detaches mid-repaint,
551 # leaving a capture in the old geometry. Measured under load, that second
552 # one is the dangerous one: it produces a diff byte-identical to row 7's
553 # — missing rows and a still-wrapped wide row — so a healthy build fails
554 # wearing the bug's own signature.
555 #
556 # The immune forms, and the only ones used below. UNIQUE OUTPUT MARKERS
557 # (`fill-done`, `00007`) for progress: each is printed once, by a command
558 # whose typed line cannot contain it, so its arrival is a fact about the
559 # session rather than about paint order. Occurrence-counting of a
560 # REPAINTED row — which is what a prompt sentinel is — is not a
561 # substitute, and is what the three earlier shapes of this leg died to.
562 #
563 # `expect 00007` earns its place as more than a marker: its arrival proves
564 # the daemon processed the wide row while the grid was still 90 columns,
565 # which is what puts the damage on the screen the resize will repaint from.
566 #
567 # After the resize the leg asserts TWO different things, and neither one
568 # implies the other:
569 # the 91-zero WITNESS proves the reflowed paint ARRIVED. 91 contiguous
570 # zeros cannot exist at width 90 — the row is painted wrapped there — so
571 # this needle is structurally incapable of firing early, whatever order
572 # the rows came in. It is also not the load risk an extra deadline looks
573 # like: this needle went 150/150 under 16-way load, at the same load that
574 # broke the two prompt-sentinel shapes 9 and 15 failures in 150.
575 # `settle` then proves nothing further is IN FLIGHT. The witness alone
576 # would permit a detach mid-repaint, with the rest of the screen still
577 # on its way.
578 # Dropping the witness and keeping only settle looks equivalent and is
579 # not: settle succeeds on 500ms of silence measured from its own start and
580 # never requires that a single byte arrived, so a healthy build whose
581 # repaint has not BEGUN inside that window satisfies it having observed
582 # nothing at all — leaving a pre-resize capture that diffs exactly like
583 # row 7. It would also be used outside its own precondition, which is that
584 # the session is already idle; straight after a resize the answering
585 # snapshot has not been generated yet.
586 set +e
587 timeout 40 "$PTYCLIENT" --cols 90 --rows 28 --out "$OUT.tp2b" --err "$OUT.tp2b.err" -- \
588 "$MUX" --sock "$SOCK12" > "$OUT.tp2b.log" 2>&1 <<'EOF'
589 expect \x1b[?1049h 10000
590 send seq 1 60; printf 'fill-%s\\n' done\n
591 expect fill-done 10000
592 send printf '%095d\\n' 7\n
593 expect 00007 10000
594 resize 100 30
595 expect 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 10000
596 settle 500 15000
597 send \x1c\x1c
598 waitexit 10000
599 EOF
600 RC=$?
601 set -e
602 [ "$RC" -eq 0 ] || {
603 echo "e2e FAIL: tp2b ptyclient exited $RC:"; cat "$OUT.tp2b.log"; exit 1; }
604 # Graded against the mutation (row 7: the prefix resize commented out) in
605 # this exact shape: 10 targeted runs, 10 red, plus a red full-suite run. What
606 # dies is the WITNESS, above — a replica still 90 wide cannot paint 91
607 # contiguous zeros, so the leg fails on the fixture timeout at that verb
608 # ("expect ... did not arrive within 10000ms", exit 3) and the RC check
609 # reports it in ~10s.
610 #
611 # This assertion is the backstop rather than the tripwire, and it is kept
612 # because it sees what the witness does not: before the witness existed
613 # the same build reached here and diverged with both halves of the stale
614 # geometry in one diff — a 30-row grid coming back short its top rows,
615 # because a 28-row replica has only 28 rows to repaint from, and the
616 # 95-wide row still WRAPPED at 90 into "...0" + "00007" while the daemon
617 # had rejoined it into one row at 100.
618 assert_converged "$OUT.tp2b" "$SOCK12" "pty resize mid-session" 100 30
619 rm_swept "$OUT.tp2a" "$OUT.tp2a.err" "$OUT.tp2a.log" \
620 "$OUT.tp2b" "$OUT.tp2b.err" "$OUT.tp2b.log" "$OUT.tp2.d"
621 # Closed here like every other per-scenario daemon, not left to the trap:
622 # tp1 runs below and would otherwise share the box with a daemon nobody is
623 # watching, holding a session whose shell is still alive.
624 softkill "$D12PID" || true
625 D12PID=""
626 ok "a pty client resizes: snapshot prefix applied, winch follows the tty"
627
628 # ---- M12 tp1: reconnect while scrolled (row 18 + the 412f38f pin) ------
629 # The session fills its scrollback BY ITSELF and then execs cat: no
630 # pre-tear typing means no pre-tear predictions, so the predict counters at
631 # exit belong entirely to the one post-resync keystroke — which is what
632 # lets them isolate the 412f38f fix. cat leaves the pty canonical and
633 # echoing, the `.always` tier, so a real keystroke predicts.
634 cat > "$TP1SH" <<'EOF'
635 #!/bin/sh
636 seq 1 100
637 exec /bin/cat
638 EOF
639 chmod +x "$TP1SH"
640 start_daemon "$SOCK13" "$OUT.tp1.d" "tp1 daemon never bound" --shell "$TP1SH"
641 D13PID=$DPID
642 # Attach only after seq finished: the client must be served a snapshot of
643 # the TAIL, so the scrollback page it fetches later is content it has never
644 # been sent. Observed at 80x24: the attach snapshot carries rows 78..100 and
645 # the session goes quiet there (cat prints no prompt), history_rows is 77,
646 # and page 1 is rows 54..77. Every needle below is chosen against those
647 # numbers, which is why this wait is part of the scenario and not a nicety.
648 i=0
649 until "$MUXD" dump --sock "$SOCK13" | grep -q "100"; do
650 i=$((i+1)); [ "$i" -lt 100 ] || { echo "e2e FAIL: tp1 session never finished seq"; exit 1; }
651 sleep 0.1
652 done
653
654 # MUX_PREDICT_STATS is exported into the FIXTURE's environment, not spelled
655 # as a `VAR=x cmd` prefix on the client: the fixture execve()s its child
656 # with an absolute path and no PATH search (src/pty.zig), so `env` would
657 # have to be an absolute path too, and the child inherits this environment
658 # anyway. Getting that wrong costs a client that exits 127 before the first
659 # verb, which the fixture reports as "closed the pty before ... matched".
660 #
661 # EXACTLY ONE keystroke, and that is the load-bearing choice in this script.
662 # offerKeystroke refuses any stdin chunk that is not one byte — an escape
663 # sequence, a multi-byte character, or a paste — and counts it suppressed.
664 # Consecutive `send`s have no barrier between them, so the client's next
665 # read can pick up two of them at once and predict neither: measured at
666 # idle, a three-keystroke version of this leg scored made=3 on one run and
667 # made=2/suppressed=1 on the next. Under load all three could coalesce,
668 # which would read as made=0 — the mutation's own signature. One send
669 # cannot coalesce with anything, so the counters below are exact.
670 #
671 # suppressed=0 therefore says more than "the keystroke predicted", and the
672 # rest is worth writing down because a future edit to the input path could
673 # take it away silently: Ctrl-\ is answered and returned on before
674 # offerKeystroke is ever reached (client.zig:874), so the detach chord
675 # cannot land in the counter; and Shift+PageUp has to arrive as ONE read,
676 # because the whole-chunk mem.eql at client.zig:882 is what recognizes it —
677 # split across two reads it matches nothing, falls through to the keystroke
678 # path, and is both counted suppressed and forwarded to the shell as input.
679 MUX_PREDICT_STATS=1 timeout 60 "$PTYCLIENT" --cols 80 --rows 24 \
680 --out "$OUT.tp1" --err "$OUT.tp1.err" -- \
681 "$MUX" --via "$MUXD proxy --sock $SOCK13" > "$OUT.tp1.log" 2>&1 <<'EOF' &
682 expect \x1b[0m100 10000
683 send \x1b[5;2~
684 expect 60 10000
685 expect reconnecting 20000
686 expect \x1b[0m100 20000
687 send t
688 expect \x1b[80X\x1b[0mt 20000
689 settle 500 15000
690 send \x1c\x1c
691 waitexit 15000
692 EOF
693 TP1PID=$!
694 defer_kill "$TP1PID"
695
696 # The tear goes between verb 3 (the scroll view is on screen) and verb 4;
697 # the fixture's "done 3" line is the barrier. Every needle above is unique
698 # to its phase, which is what makes each expect sound under the no-counting
699 # rule:
700 # the first "100" is the attach snapshot's last row, and it is spelled
701 # with the SGR reset the paint puts in front of every row because a bare
702 # "100" is no longer unique to it: the client asks its own terminal for
703 # mouse reporting with `\x1b[?1000h`, which lands in this capture before
704 # any row does. Measured, not feared — that is the collision that made
705 # this scenario fail when the wheel landed;
706 # "60" is in the history page (54..77) and in NOTHING before it. The
707 # snapshot's rows are 78..100, and the escapes around them cannot spell
708 # it either — but that is a fact about THIS session, not about the paint
709 # code: both paint paths end with a CUP carrying the cursor's real
710 # column (client.zig:968, :991), and a cursor at column 60 would put a
711 # "60" in the stream. Here it cannot be: the session is cat in canonical
712 # mode with one keystroke typed after the tear, so that column is 1 or 2
713 # for the whole scenario, and the row half of a CUP tops out at 24;
714 # "reconnecting" is the banner, painted once, on the tear;
715 # the SECOND "\x1b[0m100" can only be the post-resync repaint: the scroll view
716 # holds 54..77, and the banner carries no digits but its own position;
717 # the last needle is STRUCTURAL, and it has to be. In canonical mode the
718 # only thing a single keystroke produces is the line discipline's echo of
719 # the very glyph the prediction just painted, so no content needle can
720 # tell the daemon's answer from the client's own guess. `\x1b[80X` can:
721 # it is emitted by exactly one paint path in the client, the one that
722 # paints a DELTA (client.zig, paintDeltaClipped), so its arrival means a
723 # frame came back. The prediction paints the same glyph underlined and
724 # without an erase ("\x1b[24;1H\x1b[4mt\x1b[0m"), and the resync repaint
725 # erases the whole screen with \x1b[2J instead — neither can forge this.
726 # The two post-tear deadlines are 20000 because a reconnect is allowed to
727 # retry: the daemon may not have reaped the dead client's slot yet, and the
728 # client tolerates that for reconnect_grace_ms (5s) with a backoff that
729 # reaches 2s. That is a budget for a documented retry loop, not a deadline
730 # tuned until a flake stopped.
731 wait_for "$OUT.tp1.log" "done 3" 20 || {
732 echo "e2e FAIL: tp1 never reached the scroll view:"; cat "$OUT.tp1.log"; exit 1; }
733 TP1PROXY=$(proxy_pid "$SOCK13")
734 [ -n "$TP1PROXY" ] || { echo "e2e FAIL: tp1: no proxy to tear"; exit 1; }
735 kill -9 "$TP1PROXY"
736 # The daemon must still be SERVING, which is what `kill -0` cannot tell us:
737 # it is our own backgrounded child, so a dead one is a zombie until the
738 # shell reaps it, and signalling a zombie succeeds. A dump answers only if
739 # the process is alive and its accept loop is running.
740 "$MUXD" dump --sock "$SOCK13" >/dev/null 2>&1 || {
741 echo "e2e FAIL: tp1 tear killed the daemon, not the proxy"; exit 1; }
742
743 set +e
744 wait "$TP1PID"
745 RC=$?
746 TP1PID=""
747 set -e
748 [ "$RC" -eq 0 ] || {
749 echo "e2e FAIL: tp1 ptyclient exited $RC:"; cat "$OUT.tp1.log"
750 cat -v "$OUT.tp1.err" 2>/dev/null; exit 1; }
751
752 # Verb 5 is row 18's catch: a resync that fails to leave the scroll view
753 # suppresses every live paint, so the second "100" never arrives.
754 #
755 # The counters are the 412f38f pin, and they are EXACT rather than floors.
756 # One keystroke in the whole scenario, typed after the resync onto a blank
757 # cell at a live cursor, in a session whose termios never moves: nothing
758 # here is timing-dependent, so every number is a fact.
759 # made=1, suppressed=0 is the fix working. With it reverted the overlay
760 # still believes it is scrolled and predictAt refuses on that bit alone —
761 # made=0, suppressed=1, with nothing else in the run to hide it.
762 # displayed=1 says the glyph reached the SCREEN, which `made` does not:
763 # the overlay queues while unconfident and paints nothing, so a client
764 # that lost the `.always` tier would still count the prediction.
765 # confirmed=1 and contradicted=0 say the daemon's own echo agreed with
766 # the guess, which is what makes the prediction a speedup and not a lie.
767 want_stat "$OUT.tp1.err" made 1 "pty scroll reconnect"
768 want_stat "$OUT.tp1.err" displayed 1 "pty scroll reconnect"
769 want_stat "$OUT.tp1.err" confirmed 1 "pty scroll reconnect"
770 want_stat "$OUT.tp1.err" contradicted 0 "pty scroll reconnect"
771 want_stat "$OUT.tp1.err" suppressed 0 "pty scroll reconnect"
772 assert_converged "$OUT.tp1" "$SOCK13" "pty scroll reconnect"
773
774 # The doctored control, extended to a pty capture: this capture's byte shape
775 # (alt screen, banner paints, a history page) exists nowhere else in the
776 # corpus, so it earns its own cannot-fail check.
777 #
778 # It CANNOT be the plain append the non-pty control uses. render replays
779 # only up to the LAST alt-screen exit, and a tty client's capture ends with
780 # one — so appended bytes land after the grid under test and change nothing.
781 # Measured: the doctored capture rendered byte-identical to the original,
782 # i.e. a control that could never fire. Dropping that trailing 8-byte exit
783 # puts the append back INSIDE the alt screen (a stream with no alt-exit is
784 # replayed whole), which corrupts exactly one row of the real grid — row 10,
785 # "87" in this session. The tail is asserted rather than assumed, because if
786 # the client's teardown ever stops ending there, the truncation would silently
787 # go back to being a no-op appended after the grid.
788 [ "$(tail -c 8 "$OUT.tp1" | od -An -tx1 | tr -d ' \n')" = "1b5b3f313034396c" ] || {
789 echo "e2e FAIL: tp1 capture does not end with the alt-screen exit;"
790 echo " the doctored control below would not be doctoring the grid"
791 tail -c 16 "$OUT.tp1" | cat -v; exit 1; }
792 head -c "$(( $(wc -c < "$OUT.tp1") - 8 ))" "$OUT.tp1" > "$OUT.tp1.doc"
793 printf '\033[10;1Hpty-doctor-glyphs' >> "$OUT.tp1.doc"
794 if converged_quiet "$OUT.tp1.doc" "$SOCK13"; then
795 echo "e2e FAIL: convergence control did not fire on a doctored pty capture"; exit 1
796 fi
797 rm_swept "$OUT.tp1.doc" "$OUT.tp1.doc.render" "$OUT.tp1.doc.dump" \
798 "$OUT.tp1.doc.render.n" "$OUT.tp1.doc.dump.n" "$OUT.tp1.doc.diff" \
799 "$OUT.tp1.doc.rvt" "$OUT.tp1.doc.dvt"
800 softkill "$D13PID" || true
801 D13PID=""
802 rm_swept "$OUT.tp1" "$OUT.tp1.err" "$OUT.tp1.log" "$OUT.tp1.d" "$TP1SH"
803 ok "reconnect while scrolled: view restored, prediction resumed"