a73x

2862f603

test: a green run that leaves residue now says so instead of passing

a73x   2026-08-22 06:29

Commit message
test: a green run that leaves residue now says so instead of passing

e2e stranded 102 files in $TMPDIR on every green run. They are the prefix,
wall and zoom captures from the M1-M6 legs plus the agent legs' three: added
to the scenarios that write them, never to the trap that removes them. The
count is per run and cumulative, so by 2026-08-19 there were thousands, and
`make soak` — which refuses to start when any exist, because it cannot tell
stranded evidence from a leak of its own — had not run since. Nothing said
anything; the suite printed "e2e OK" throughout.

Verified pre-existing rather than assumed: 43b9ed1's e2e.sh, run against the
same binaries, strands the same 102.

The names are added, which fixes it once. The guard is what keeps it fixed:
after the trap has run, a PASSING run finds anything left matching soak's own
patterns, prints it, removes it, and fails. Pattern-matched rather than
listed, because the list is precisely what failed — every capture has to be
remembered in two places. Pinned to the run's pid, so a concurrent suite is
never counted and never deleted. Failing runs are untouched: their captures
are the evidence, and soak collects them.

Observed firing before being trusted: a planted stray turns a green run red
and names the file.

Same reasoning applied to coverage.sh, which had the same shape of bug in the
other direction — it swallowed e2e's status and exited 0 on a suite that
failed at scenario 13, reporting coverage figures for a third of a run as
though they described the suite. The report is still written, then the
status is re-raised.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test/coverage.sh
Old New
@@ -146,10 +146,13 @@ pick() { if [ -x "$SHIM/$1" ]; then echo "$SHIM/$1"; else echo "$2"; fi; }
146 E2E_RC=$? 146 E2E_RC=$?
147 set -e 147 set -e
148 148
149 # Reported, never fatal. A failing suite still collected everything up to 149 # A failing suite still collected everything up to the failure, and throwing
150 # the failure, and throwing that away would make the one run you most want 150 # that away would make the run you most want to look at the one that produces
151 # to look at the one run that produces nothing. 151 # nothing — so the report below is still built. But the status is NOT
152 [ "$E2E_RC" -eq 0 ] || echo "coverage NOTE: e2e exited $E2E_RC; report covers the run up to that point" 152 # swallowed: this exited 0 on a suite that failed at scenario 13, which is a
153 # green `make coverage` reporting on a third of the suite. It is re-raised at
154 # the bottom, after the report exists.
155 [ "$E2E_RC" -eq 0 ] || echo "coverage NOTE: e2e exited $E2E_RC; the report below covers the run up to that point"
153 156
154 DIRS=$(find "$RAW" -mindepth 1 -maxdepth 1 -type d | wc -l) 157 DIRS=$(find "$RAW" -mindepth 1 -maxdepth 1 -type d | wc -l)
155 [ "$DIRS" -gt 0 ] || { echo "coverage FAIL: no kcov databases in $RAW"; exit 1; } 158 [ "$DIRS" -gt 0 ] || { echo "coverage FAIL: no kcov databases in $RAW"; exit 1; }
@@ -180,3 +183,13 @@ awk -F'[:,]' '
180 echo 183 echo
181 echo "html: file://$OUT/merged/kcov-merged/index.html" 184 echo "html: file://$OUT/merged/kcov-merged/index.html"
182 echo "per-process databases: $RAW (mtimes attribute to $OK_LOG)" 185 echo "per-process databases: $RAW (mtimes attribute to $OK_LOG)"
186
187 # The suite's verdict, re-raised now that the report is written. A coverage
188 # run whose suite failed has measured part of a suite, and saying so is the
189 # difference between a number and a misleading number.
190 if [ "$E2E_RC" -ne 0 ]; then
191 echo
192 echo "coverage FAIL: e2e exited $E2E_RC — the figures above describe an"
193 echo " incomplete run, not this suite's coverage."
194 exit "$E2E_RC"
195 fi
test/e2e.sh
Old New
@@ -1402,16 +1402,84 @@ cleanup() {
1402 "$OUT.flip.d" \ 1402 "$OUT.flip.d" \
1403 "$OUT.flipa.env" "$OUT.flipa" "$OUT.flipa.err" "$OUT.flipa.log" \ 1403 "$OUT.flipa.env" "$OUT.flipa" "$OUT.flipa.err" "$OUT.flipa.log" \
1404 "$OUT.flipb.env" "$OUT.flipb" "$OUT.flipb.err" "$OUT.flipb.log" 1404 "$OUT.flipb.env" "$OUT.flipb" "$OUT.flipb.err" "$OUT.flipb.log"
1405 # The M1-M6 prefix/wall/zoom captures and the agent legs' three. Absent
1406 # from this trap until 2026-08-22, which is why every green run since
1407 # those features landed left 102 files in $TMPDIR — enough, by 08-19, that
1408 # `make soak` refused to start and no soak had run since. The guard below
1409 # is what makes the next omission say so instead of accumulating.
1410 # agent forwarding: the nested-agent leg and the three no-offer captures.
1411 rm -f "$OUT.agtnest" "$OUT.agtnest.err" "$OUT.agtnest.log" "$OUT.anoag" \
1412 "$OUT.anoag2" "$OUT.anoag3"
1413 # the CLI wall: both halves (cw*, wc*) — captures, injections and stop files.
1414 rm -f "$OUT.cwa" "$OUT.cwa.err" "$OUT.cwall.d" "$OUT.cwb" "$OUT.cwb.err" \
1415 "$OUT.cwcap" "$OUT.cwcap.err" "$OUT.cwinj" "$OUT.cwpc" "$OUT.cwst" \
1416 "$OUT.cwstop" "$OUT.wc" "$OUT.wccap" "$OUT.wc.d" "$OUT.wc.err" \
1417 "$OUT.wcinj" "$OUT.wc.log" "$OUT.wcst" "$OUT.wcstop"
1418 # the new-session-from-the-wall leg.
1419 rm -f "$OUT.nsw" "$OUT.nswcap" "$OUT.nswcap3" "$OUT.nsw.d" \
1420 "$OUT.nsw.err" "$OUT.nsw.log" "$OUT.nswst" "$OUT.nswst4" \
1421 "$OUT.nswstop"
1422 # the Ctrl-\ prefix leg.
1423 rm -f "$OUT.pfx" "$OUT.pfx.d" "$OUT.pfx.err" "$OUT.pfx.log" "$OUT.pfxst" \
1424 "$OUT.pfxstop"
1425 # the bell leg's ring captures.
1426 rm -f "$OUT.ring" "$OUT.ringcap" "$OUT.ring.d" "$OUT.ring.err" \
1427 "$OUT.ring.log" "$OUT.ringstop"
1428 # self-attach refusal, and the wall file it reads.
1429 rm -f "$OUT.sacap1" "$OUT.sacap2" "$OUT.sacap3" "$OUT.sa.d" "$OUT.saenv" \
1430 "$OUT.saself" "$OUT.sast" "$OUT.sastop" "$OUT.sawall"
1431 # the wall zoom leg.
1432 rm -f "$OUT.wza" "$OUT.wza.err" "$OUT.wzb" "$OUT.wzb.err" "$OUT.wzcap" \
1433 "$OUT.wzcapa" "$OUT.wzcapb" "$OUT.wzcap.err" "$OUT.wz.d" "$OUT.wzpc" \
1434 "$OUT.wzsta" "$OUT.wzstb" "$OUT.wzstop" "$OUT.wzwatch"
1435 # zoom demote.
1436 rm -f "$OUT.zda" "$OUT.zda.err" "$OUT.zdcap" "$OUT.zdcap.err" \
1437 "$OUT.zd.d" "$OUT.zdpc" "$OUT.zdstop"
1438 # zoom between tiles.
1439 rm -f "$OUT.zma" "$OUT.zma.err" "$OUT.zmb" "$OUT.zmb.err" "$OUT.zmcap" \
1440 "$OUT.zmcap.err" "$OUT.zm.d" "$OUT.zmfa" "$OUT.zmfb" "$OUT.zmpc" \
1441 "$OUT.zmstb" "$OUT.zmstop"
1442 # zoom onto a session, and its watchers.
1443 rm -f "$OUT.zsa" "$OUT.zsa.err" "$OUT.zsb" "$OUT.zsb.err" "$OUT.zscap" \
1444 "$OUT.zscap.err" "$OUT.zs.d" "$OUT.zsfa" "$OUT.zsfb" "$OUT.zspc" \
1445 "$OUT.zssta" "$OUT.zsstb" "$OUT.zsstop" "$OUT.zswatch"
1405 # The convergence files a FAILING assert_converged leaves behind 1446 # The convergence files a FAILING assert_converged leaves behind
1406 # (.render/.dump/.rvt/.dvt/.diff for that capture) are deliberately not 1447 # (.render/.dump/.rvt/.dvt/.diff for that capture) are deliberately not
1407 # chased here: on a failing run they are the evidence. 1448 # chased here: on a failing run they are the evidence.
1408 rm -rf "$XDG_CONFIG_HOME" "$XDG_STATE_HOME" "$XDG_CACHE_HOME" "${NOKEY_CFG:-}" 1449 rm -rf "$XDG_CONFIG_HOME" "$XDG_STATE_HOME" "$XDG_CACHE_HOME" "${NOKEY_CFG:-}"
1409 rm -f "$LEAKBANK" 1450 rm -f "$LEAKBANK"
1451 # Residue, in the leak sweep's shape and for its reason: a run that PASSED
1452 # must leave nothing behind, and this trap is the last place that can
1453 # still say so. Matched by pattern rather than by the list above, because
1454 # the list is what failed — every capture has to be remembered in two
1455 # places, and 102 of them were not. Pinned to this run's pid so a suite
1456 # running concurrently is never counted, and never deleted.
1457 #
1458 # Green runs only. A failing run keeps everything: the convergence files
1459 # noted above are its evidence, and soak sweeps them into its failure dir.
1460 _stray=0
1461 if [ "$_rc" -eq 0 ]; then
1462 _left=$(find "${TMPDIR:-/tmp}" -maxdepth 1 \
1463 \( -name "mux*-$$" -o -name "mux*-$$.*" \) 2>/dev/null)
1464 if [ -n "$_left" ]; then
1465 _n=$(printf '%s\n' "$_left" | wc -l)
1466 if [ "$_n" -eq 1 ]; then _w=file; else _w=files; fi
1467 echo "e2e FAIL: the suite passed but left $_n $_w in ${TMPDIR:-/tmp}:"
1468 printf '%s\n' "$_left" | sed 's/^/ /'
1469 echo " A capture is removed by the scenario that made it or"
1470 echo " by this trap. One added to neither is invisible until"
1471 echo " soak refuses to start; that is what this catches."
1472 printf '%s\n' "$_left" | xargs -r rm -f
1473 _stray=1
1474 fi
1475 fi
1410 # This trap can change the suite's answer in exactly one direction: a run 1476 # This trap can change the suite's answer in exactly one direction: a run
1411 # that was green and swept up a leak. Every other path returns normally 1477 # that was green and swept up a leak. Every other path returns normally
1412 # and leaves the status alone — a suite that failed at scenario 6 must 1478 # and leaves the status alone — a suite that failed at scenario 6 must
1413 # exit with scenario 6's failure, not with the trap's opinion of it. 1479 # exit with scenario 6's failure, not with the trap's opinion of it.
1414 if [ "$_rc" -eq 0 ] && [ "$_leak" -eq 1 ]; then exit 1; fi 1480 if [ "$_rc" -eq 0 ] && { [ "$_leak" -eq 1 ] || [ "$_stray" -eq 1 ]; }; then
1481 exit 1
1482 fi
1415 } 1483 }
1416 trap cleanup EXIT INT TERM 1484 trap cleanup EXIT INT TERM
1417 1485