a73x

5a69a787

feat(m9): prediction measured on a real path — the record

a73x   2026-08-08 17:11

Commit message
feat(m9): prediction measured on a real path — the record

netem 150ms egress on the LAN box, and the measured round trip agrees:
baseline med 150.4ms, so leg 1 is exercised rather than vacuous.

  predict      min 0.1  med 0.1  max 0.2   (n=20)
  predictauth  min 0.1  med 154.0 max 154.2 (n=20)
  predictburst min 233.6 med 233.9 max 237.1 (n=10)

Leg 1 asks for a predicted paint under 30ms. It is 0.1ms — and the number
that makes that meaningful is the second line, because it comes from the
SAME keystroke on the SAME connection: the authoritative echo of the very
character whose prediction was measured takes 154ms, tracking the path.
One keystroke, two clocks, so no argument about warm caches or differing
conditions can be made about the gap between them. Echo latency has stopped
being a function of the round trip where prediction applies, and the control
proves the harness was timing the round trip all along.

Leg 2: ten adversarial bursts, every prediction accounted for — made=55
confirmed=54 abandoned=1 pending=0 — and the last burst in the daemon's
grid. Contradicted=0 and expired=0 across the whole run, which answers the
risk banked at Task 3: judge-by-next-frame would have refuted every
character after the first of each burst, once per round trip. Reconcile v2
holds on a real path.

Convergence is asserted through the daemon's grid plus attribution rather
than by comparing the client's paint stream to `muxd dump` byte for byte —
the same narrowing already recorded for M7 at the top of this file, and for
the same reason: that comparison needs a second terminal emulator inside
the harness, whose own bugs would be indistinguishable from the ones it is
meant to catch.

Two things the first run of this phase got wrong, both fixed and both worth
recording because each made a leg pass while proving nothing. The burst
characters were sent back to back, so they reached the client as ONE read —
a multi-byte chunk, refused for being one — and the burst was measured with
no predictions in it at all (made=5 for 50 keystrokes). They are now 20ms
apart: five separate keystrokes inside 80ms, all outstanding when the first
answer arrives. And the convergence dump was taken after an EOF sent to end
`cat`, which over a 150ms path may not have been processed before the
client detached; the grid is now read while the client is still attached,
and no EOF is sent at all.

Box verified clean afterwards: no netem on the interface, no mux-tagged
firewall rule, no tagged process or file left behind.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
test/wan.sh
Old New
@@ -102,6 +102,10 @@ REPS_HOL="${MUX_WAN_REPS_HOL:-5}"
102 # M7: the kill criterion names 10 consecutive tears; the default is the 102 # M7: the kill criterion names 10 consecutive tears; the default is the
103 # criterion, not a convenience. 103 # criterion, not a convenience.
104 REPS_RECONNECT="${MUX_WAN_REPS_RECONNECT:-10}" 104 REPS_RECONNECT="${MUX_WAN_REPS_RECONNECT:-10}"
105 # M9: keystrokes timed for leg 1, and adversarial bursts for leg 2. Ten
106 # bursts is the plan's number and therefore the default.
107 REPS_PREDICT="${MUX_WAN_REPS_PREDICT:-20}"
108 REPS_BURST="${MUX_WAN_REPS_BURST:-10}"
105 109
106 # Unique remote names: the box may be shared, and a crashed run must never 110 # Unique remote names: the box may be shared, and a crashed run must never
107 # leave a socket another run mistakes for its own. 111 # leave a socket another run mistakes for its own.
@@ -861,7 +865,171 @@ def cmd_hol(argv):
861 report("hol", samples) 865 report("hol", samples)
862 866
863 867
868 def predict_stats(errlog):
869 """The client's MUX_PREDICT_STATS line, as a dict. The LAST one in the
870 file: the log is appended to across a run, and only this phase's client
871 is being asked about."""
872 try:
873 with open(errlog, "rb") as f:
874 lines = [l for l in f.read().split(b"\n") if l.startswith(b"predict ")]
875 except OSError:
876 return {}
877 if not lines:
878 return {}
879 out = {}
880 for tok in lines[-1].decode("ascii", "replace").split()[1:]:
881 k, _, v = tok.partition("=")
882 try:
883 out[k] = int(v)
884 except ValueError:
885 pass
886 return out
887
888
889 def enter_cat(c, what):
890 """Put the remote session into a genuinely canonical, echoing reader.
891
892 An interactive bash prompt is icanon=0/echo=0 — readline echoes for
893 itself — so it is the ADAPTIVE tier, where display has to be earned over
894 two confirmations. `cat` is the tier this leg is about: the line
895 discipline echoes, so a prediction is a deduction and paints from the
896 first keystroke. Returns once a prediction has actually been observed,
897 which is the only proof that the mode reached the client.
898 """
899 c.send(b"cat\n")
900 deadline = now() + 20.0
901 probe = b"~"
902 while now() < deadline:
903 start = len(c.buf)
904 c.send(probe)
905 if c.wait_for(lambda b, s=start: b"\x1b[4m" in b[s:], timeout=3.0) is not None:
906 # Let the authoritative echo of the probe land, so the run
907 # measured below starts from a settled screen.
908 c.drain(0.5)
909 return
910 fail("%s: no prediction ever painted; the session never reached the "
911 "always-predict tier (is `cat` running on the far side?)" % what)
912
913
914 def cmd_predict(argv):
915 """M9 leg 1: the predicted glyph, and the authoritative echo of the SAME
916 keystroke, timed from the same instant.
917
918 That pairing is the measurement. One number is supposed to be a function
919 of the local paint and nothing else; the other is supposed to be a
920 function of the path. Taking them from one keystroke on one connection
921 means no argument about warm caches or differing conditions can be made
922 about the gap between them — and the control is what proves the harness
923 is measuring the path at all rather than the hardware.
924 """
925 mux, via, reps, errlog = argv[0], argv[1], int(argv[2]), argv[3]
926 if reps > len(ECHO_RUN) - 1:
927 fail("predict: at most %d reps" % (len(ECHO_RUN) - 1))
928 c = Client(mux, via, errlog)
929 pred, auth = [], []
930 try:
931 settle(c, "predict")
932 enter_cat(c, "predict")
933
934 run = b""
935 for i in range(reps):
936 ch = ECHO_RUN[i + 1:i + 2] # skip the leading '#'
937 run += ch
938 start = len(c.buf)
939 t0 = now()
940 c.send(ch)
941 # The prediction: painted locally, underlined, at the cell the
942 # cursor is on. Searched from `start` so a repeat of the same
943 # character cannot match an earlier rep's paint.
944 if c.wait_for(lambda b, s=start, x=ch: b"\x1b[4m" + x in b[s:],
945 timeout=10.0) is None:
946 fail("predict: rep %d was never predicted" % i)
947 pred.append((now() - t0) * 1000.0)
948 # The authoritative echo of the same keystroke. The run only
949 # grows and predictions paint one cell at a time, so the run as
950 # a contiguous string appears only in a daemon row paint.
951 if c.wait_for(lambda b, w=run: w in b, timeout=20.0) is None:
952 fail("predict: rep %d never echoed authoritatively" % i)
953 auth.append((now() - t0) * 1000.0)
954 # Deliberately no EOF to end `cat`. Ending it means the next
955 # keystroke lands at a bash prompt, which is a different termios and
956 # therefore a different prediction tier — and over a 150ms path the
957 # EOF may not even have been processed before this client detaches,
958 # so the following phase would find a session in a state that
959 # depended on timing. Leaving the reader running is the state the
960 # next phase wants anyway; the daemon is torn down at cleanup.
961 c.drain(0.5)
962 finally:
963 c.detach()
964
965 st = predict_stats(errlog)
966 report("predict", pred, **{k: st.get(k, -1) for k in
967 ("made", "displayed", "confirmed",
968 "contradicted", "expired", "abandoned")})
969 report("predictauth", auth)
970
971
972 def cmd_predictburst(argv):
973 """M9 leg 2: adversarial bursts, then convergence by attribution.
974
975 Each burst types faster than the path can answer, which is the case that
976 used to refute itself once per round trip. Convergence is asserted the
977 way this harness asserts everything else about a client's grid — through
978 the daemon and the counters, not by parsing VT in the harness. Every
979 prediction must end accounted for (made = confirmed + abandoned, with
980 nothing left pending) and the daemon's grid must hold what was typed.
981 """
982 mux, via, bursts, errlog = argv[0], argv[1], int(argv[2]), argv[3]
983 ssh_cmd, rbin, rsock = argv[4], argv[5], argv[6]
984 c = Client(mux, via, errlog)
985 conv = []
986 try:
987 settle(c, "predictburst")
988 enter_cat(c, "predictburst")
989 for b_i in range(bursts):
990 word = bytes("bst%02d" % b_i, "ascii")
991 start = len(c.buf)
992 t0 = now()
993 # Paced, not blasted. Sent back to back they arrive at the
994 # client as ONE read, which is a multi-byte chunk and is refused
995 # for being one — so the burst would be measured with no
996 # predictions in it at all, and the leg would pass while proving
997 # nothing. 20ms apart is five separate keystrokes inside 80ms,
998 # still far inside the 150ms round trip, so every one of them is
999 # outstanding when the first answer arrives. That is the burst
1000 # this leg is about.
1001 for ch in word:
1002 c.send(bytes([ch]))
1003 time.sleep(0.02)
1004 if c.wait_for(lambda b, w=word, s=start: w in b[s:],
1005 timeout=25.0) is None:
1006 fail("predictburst: burst %d never converged" % b_i)
1007 conv.append((now() - t0) * 1000.0)
1008 c.send(b"\n")
1009 c.drain(0.3)
1010 # Read the daemon's grid while this client is still attached: the
1011 # detach below is what makes the counters printable, and anything
1012 # sent to get there could change the screen the check is about.
1013 grid = remote_dump(ssh_cmd, rbin, rsock)
1014 finally:
1015 c.detach()
1016
1017 st = predict_stats(errlog)
1018 made = st.get("made", -1)
1019 confirmed = st.get("confirmed", 0)
1020 abandoned = st.get("abandoned", 0)
1021 pending = made - confirmed - abandoned if made >= 0 else -1
1022 last = "bst%02d" % (bursts - 1)
1023 converged = 1 if last in grid else 0
1024 report("predictburst", conv, made=made, confirmed=confirmed,
1025 abandoned=abandoned, pending=pending,
1026 contradicted=st.get("contradicted", -1),
1027 expired=st.get("expired", -1), converged=converged)
1028
1029
864 COMMANDS = { 1030 COMMANDS = {
1031 "predict": cmd_predict,
1032 "predictburst": cmd_predictburst,
865 "baseline": cmd_baseline, 1033 "baseline": cmd_baseline,
866 "viafloor": cmd_viafloor, 1034 "viafloor": cmd_viafloor,
867 "attach": cmd_attach, 1035 "attach": cmd_attach,
@@ -1039,6 +1207,43 @@ if [ "${MUX_WAN_NETEM:-}" = "1" ]; then
1039 netem_off 1207 netem_off
1040 fi 1208 fi
1041 1209
1210 if [ "${MUX_WAN_PREDICT:-}" = "1" ]; then
1211 # M9 leg 1 is only meaningful at RTT >= 50ms and the criterion names
1212 # 150ms, so this phase brings its own delay rather than borrowing the
1213 # 75ms one above. Same interface question, same dual-home trap: ask the
1214 # box which device it would use to reach US.
1215 CLIENT_IP="$($MUX_WAN_SSH 'echo $SSH_CLIENT' | awk '{print $1}' | tr -d '\r\n')"
1216 [ -n "$CLIENT_IP" ] || { echo "wan FAIL: cannot tell which address the box sees us on" >&2; exit 1; }
1217 PIFACE="$($MUX_WAN_SSH "ip route get $CLIENT_IP" | sed -n 's/.* dev \([^ ]*\).*/\1/p' | head -1 | tr -d '\r\n')"
1218 [ -n "$PIFACE" ] || { echo "wan FAIL: no interface toward this client" >&2; exit 1; }
1219
1220 # A root qdisc shapes EGRESS only, so 150ms here is ~150ms of round
1221 # trip, not 300ms. The criterion asks for >=150ms RTT; the baseline
1222 # printed below is what settles whether it got it.
1223 say "M9: netem delay 150ms on $PIFACE (~150ms added round-trip)"
1224 $MUX_WAN_SSH "sudo -n tc qdisc add dev $PIFACE root netem delay 150ms"
1225 NETEM_IFACE="$PIFACE"
1226 DEADMAN_PGID="$($MUX_WAN_SSH "setsid sudo -n sh -c \
1227 'sleep $DEADMAN_SECS; tc qdisc del dev $PIFACE root' \
1228 >/dev/null 2>&1 </dev/null & echo \$!" | tr -cd '0-9')" || true
1229
1230 # The path's own number under this delay, so leg 1's control has
1231 # something measured to be compared against rather than the nominal
1232 # 150ms the qdisc was asked for.
1233 measure predict baseline "$MUX_WAN_SSH" "$REPS_BASE"
1234
1235 say "M9 leg 1: predicted paint vs the same keystroke's authoritative echo"
1236 MUX_PREDICT_STATS=1 measure predict predict \
1237 "$MUX" "$VIA" "$REPS_PREDICT" "$WORK/predict.err"
1238
1239 say "M9 leg 2: $REPS_BURST adversarial bursts, convergence by attribution"
1240 MUX_PREDICT_STATS=1 measure predict predictburst \
1241 "$MUX" "$VIA" "$REPS_BURST" "$WORK/burst.err" \
1242 "$MUX_WAN_SSH" "$RBIN" "$RSOCK"
1243
1244 netem_off
1245 fi
1246
1042 # ---- summary + the kill criterion ----------------------------------------- 1247 # ---- summary + the kill criterion -----------------------------------------
1043 FAILED=0 1248 FAILED=0
1044 1249
@@ -1207,5 +1412,63 @@ echo "kill criteria (M6 + M7): PASS"
1207 else 1412 else
1208 echo "kill criteria (M6 + M7): FAIL — report the numbers; do not tune the thresholds." 1413 echo "kill criteria (M6 + M7): FAIL — report the numbers; do not tune the thresholds."
1209 fi 1414 fi
1415
1416 # ---- M9 prediction ---------------------------------------------------------
1417 if [ -n "$(val predict predict med)" ]; then
1418 P_RTT="$(val predict baseline med)"
1419 P_MED="$(val predict predict med)"
1420 P_MAX="$(val predict predict max)"
1421 P_AUTH="$(val predict predictauth med)"
1422 echo
1423 echo "M9 prediction (netem 150ms egress):"
1424 printf ' %-34s %8s %8s %8s %5s\n' "" min med max n
1425 for name in baseline predict predictauth predictburst; do
1426 [ -n "$(val predict "$name" med)" ] || continue
1427 printf ' %-34s %8s %8s %8s %5s\n' "$name" \
1428 "$(val predict "$name" min)" "$(val predict "$name" med)" \
1429 "$(val predict "$name" max)" "$(val predict "$name" n)"
1430 done
1431 echo " (all figures milliseconds)"
1432
1433 # The floor rule, applied rather than described: leg 1 says nothing at a
1434 # round trip prediction could not plausibly beat, so below 50ms it is
1435 # reported as not exercised — never as passed.
1436 if awk -v r="$P_RTT" 'BEGIN{exit !(r < 50)}'; then
1437 echo " leg 1: NOT EXERCISED — round trip $P_RTT ms is below the 50ms floor"
1438 elif awk -v m="$P_MED" 'BEGIN{exit !(m <= 30)}'; then
1439 echo " leg 1: predicted paint med $P_MED <= 30ms -> PASS (round trip $P_RTT ms)"
1440 echo " control: the SAME keystroke's authoritative echo med $P_AUTH ms,"
1441 echo " which tracks the measured round trip — so this harness is timing"
1442 echo " the path, and the predicted number is independent of it."
1443 else
1444 echo " leg 1: predicted paint med $P_MED > 30ms -> FAIL (round trip $P_RTT ms)"
1445 FAILED=1
1446 fi
1447
1448 B_MADE="$(val predict predictburst made)"
1449 B_PEND="$(val predict predictburst pending)"
1450 B_CONF="$(val predict predictburst confirmed)"
1451 B_ABND="$(val predict predictburst abandoned)"
1452 B_CONV="$(val predict predictburst converged)"
1453 B_CONTRA="$(val predict predictburst contradicted)"
1454 B_EXP="$(val predict predictburst expired)"
1455 if [ -n "$B_MADE" ]; then
1456 echo " counters after $REPS_BURST bursts: made=$B_MADE confirmed=$B_CONF" \
1457 "contradicted=$B_CONTRA expired=$B_EXP abandoned=$B_ABND pending=$B_PEND"
1458 # Leg 2, in this harness's substituted form. Comparing the client's
1459 # paint stream to `muxd dump` byte for byte needs a second terminal
1460 # emulator in the harness — the same narrowing recorded for M7 at
1461 # the top of this file — so convergence is asserted through the
1462 # daemon's grid plus attribution: no prediction may outlive the run
1463 # unaccounted for.
1464 if [ "$B_PEND" = "0" ] && [ "$B_CONV" = "1" ]; then
1465 echo " leg 2: every prediction accounted for (pending=0) and the last"
1466 echo " burst is in the daemon's grid -> PASS"
1467 else
1468 echo " leg 2: pending=$B_PEND converged=$B_CONV -> FAIL"
1469 FAILED=1
1470 fi
1471 fi
1472 fi
1210 echo "==================================================================" 1473 echo "=================================================================="
1211 exit "$FAILED" 1474 exit "$FAILED"