96febc7b

test: line coverage for the e2e suite, via kcov

a73x 2026-08-21 20:24

Commit message

test: line coverage for the e2e suite, via kcov

`make coverage` runs the suite with the daemons under kcov and merges the
per-process databases into one report. Nothing is built differently for it:
kcov is a ptrace tracer reading DWARF, so the report describes the same
binaries the suite already grades. Zig 0.15 exposes no instrumentation flag
of its own, which is what settles the choice.

Three ptrace properties shaped the whole design, each found by a hang:

- kcov cannot nest. PTRACE_O_TRACEFORK means every descendant of a traced
  process is already traced, so an inner kcov's TRACEME is refused;
  MUX_KCOV_ACTIVE makes nested shims exec bare.
- a traced process that leaves a daemon behind never exits, because kcov
  waits for its last traced descendant and that descendant is designed
  never to end. This is why `mux` is not traced by default: the auto-start
  and --via legs spawn exactly that, and SIGTERM will not free it.
- a SIGKILLed wrapper writes nothing at all, at any --output-interval. So
  hardkill signals the child, lets the wrapper commit, and only then kills
  it — and it tests what the process IS rather than whether it has
  children, because a daemon has children too and killing a session shell
  first ends the session cleanly, which the abort legs must never see.

E2E_TIME_SCALE multiplies the polling budgets and nothing else. A traced
binary runs about half again slower, enough to blow budgets that are
generous at native speed; scaling changes how long the suite will WAIT,
never what it demands. E2E_OK_LOG stamps scenario boundaries, which is what
lets a database be attributed to the scenario that wrote it — and is also
the first per-scenario timing this suite has been able to report.

The suite is unchanged when none of this is set: `make e2e` still runs 60
scenarios in 329s, the same as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Makefile

Old		New
@@ -2,7 +2,7 @@
2	# default zig is 0.17-dev. Override with ZIG=... if yours lives elsewhere.	2	# default zig is 0.17-dev. Override with ZIG=... if yours lives elsewhere.
3	ZIG ?= $(HOME)/Downloads/zig-x86_64-linux-0.15.2/zig	3	ZIG ?= $(HOME)/Downloads/zig-x86_64-linux-0.15.2/zig
4		4
5	.PHONY: build check ci test e2e soak bench agent throughput deps clean clean-deps xversion xversion-build install release	5	.PHONY: build check ci test e2e soak bench agent throughput coverage deps clean clean-deps xversion xversion-build install release
6		6
7	# The QUIC stack (deps/quic) is built on demand by build.zig, so no target	7	# The QUIC stack (deps/quic) is built on demand by build.zig, so no target
8	# here needs to depend on this one. It exists to make the one-time cost	8	# here needs to depend on this one. It exists to make the one-time cost
@@ -94,6 +94,30 @@ bench:
94	# reading, so a timing assertion there would measure ghostty's debug wiring	94	# reading, so a timing assertion there would measure ghostty's debug wiring
95	# and never this code. That is also why there is no build.zig step — a step	95	# and never this code. That is also why there is no build.zig step — a step
96	# is handed the default-optimize artifacts, which are the wrong ones.	96	# is handed the default-optimize artifacts, which are the wrong ones.
		97	# Line coverage for the e2e suite, via kcov (pacman -S kcov).
		98	#
		99	# ReleaseSafe in its own prefix, for throughput's reason: kcov traps on every
		100	# instrumented line, so its cost rides on top of whatever the binary already
		101	# pays per line, and a Debug ghostty pays the page-integrity check documented
		102	# above on top of that. Nothing here has measured the Debug pairing to be
		103	# unusable — the runs that looked slow were hanging, for the reason
		104	# test/coverage.sh gives — so this is the documented 600x avoided, not a
		105	# number this target established.
		106	#
		107	# The trade is line granularity: an optimised build inlines, so lines merge
		108	# into their callers and the denominator shrinks. Measured on muxd: 1639
		109	# instrumented lines here against 2651 in Debug.
		110	#
		111	# Outside `ci` deliberately — it needs an optional system package and is
		112	# still several times slower than the suite it wraps.
		113	COVDIR ?= dist/coverage
		114	coverage:
		115	$(ZIG) build -Doptimize=ReleaseSafe -p $(COVDIR)
		116	./test/coverage.sh $(COVDIR)/bin/muxd $(COVDIR)/bin/mux \
		117	$(COVDIR)/bin/rawmode $(COVDIR)/bin/delaypipe $(COVDIR)/bin/render \
		118	$(COVDIR)/bin/ptyclient $(COVDIR)/bin/wsclient $(COVDIR)/bin/muxweb \
		119	$(COVDIR)/bin/muxa
		120
97	THRUDIR ?= dist/throughput	121	THRUDIR ?= dist/throughput
98	throughput:	122	throughput:
99	$(ZIG) build -Doptimize=ReleaseSafe -p $(THRUDIR)	123	$(ZIG) build -Doptimize=ReleaseSafe -p $(THRUDIR)

test/coverage.sh

@@ @@ -0,0 +1,173 @@ @@
+#!/bin/sh
+# Line coverage for the e2e suite. kcov wraps the four user binaries, the
+# suite runs UNCHANGED, and the per-process databases are merged into one
+# report.
+#
+# kcov is a ptrace tracer, not an instrumentation pass, so nothing here
+# changes how anything is built: the report describes the same Debug
+# artifacts `zig build e2e` grades, not a special coverage build. That is
+# the whole reason kcov was chosen over -fprofile-instr-generate, which Zig
+# 0.15 does not expose at all.
+#
+# Two consequences of ptrace are load-bearing, and both are why this file
+# exists instead of a one-line kcov invocation:
+#
+#   - kcov cannot nest. It sets PTRACE_O_TRACEFORK, so every descendant of
+#     a traced process is ALREADY traced, and an inner kcov's TRACEME is
+#     refused with EPERM. `mux --via` spawns a muxd, which is exactly that
+#     shape. MUX_KCOV_ACTIVE makes a nested shim exec the real binary bare:
+#     those lines are attributed to whichever ancestor is traced, or lost,
+#     but the suite never dies.
+#   - the pid the suite holds becomes kcov's, not the daemon's. SIGKILL is
+#     the one signal a wrapper cannot forward, so a leg that kill -9s a
+#     daemon would leave the real one alive holding its socket. e2e.sh's
+#     hardkill answers that; see the comment there.
+#   - kcov waits for the LAST traced descendant, and `muxd start` daemonises:
+#     its launcher forks, the parent exits, and the daemon reparents to init.
+#     Traced, that daemon outlives the scenario, so kcov never exits and the
+#     suite's `wait` on it never returns — a hang at the auto-start leg that
+#     reads as a slow run, and one that ignores SIGTERM, so `timeout` will not
+#     free it either. The shims therefore run `start` BARE, exporting
+#     MUX_KCOV_ACTIVE so the `muxd run` it re-execs stays bare too. The cost
+#     is the auto-started daemons' lines; the alternative was a suite that
+#     could not finish.
+#
+#     Not --exit-first-process, which is kcov's own answer to daemons: it
+#     reparents the tracer, and then the real process is no longer a CHILD of
+#     the pid the suite holds — which is precisely what hardkill needs it to
+#     be. It trades this hang for a dead hardkill and a hang one leg later.
+#
+# A SIGKILLed wrapper writes nothing. kcov commits its database on clean exit
+# only — measured across --collect-only, --output-interval=1000 and full
+# report mode, all three yield no coverage.db when the wrapper is SIGKILLed.
+# That is why e2e.sh's hardkill signals the child and then WAITS for the
+# wrapper rather than killing both at once, and why the daemons this suite
+# tears down with SIGKILL would otherwise contribute nothing at all.
+set -eu
+MUXD="$1"; MUX="$2"; RAWMODE="$3"; DELAYPIPE="$4"; RENDER="$5"; PTYCLIENT="$6"
+WSCLIENT="$7"; MUXWEB="$8"
+# Same list as the e2e and soak steps, muxa included — see the note in
+# build.zig; all three must be handed the same binaries in the same order.
+MUXA="$9"
+command -v kcov > /dev/null 2>&1 || {
+    echo "coverage FAIL: this target needs kcov (pacman -S kcov)"
+    echo "               It is a ptrace tracer, so no rebuild is required —"
+    echo "               it reads DWARF from the binaries already built."
+    exit 1
+}
+E2E="$(dirname "$0")/e2e.sh"
+SRC="$(cd "$(dirname "$0")/../src" && pwd)"
+OUT="${COV_OUT:-$(cd "$(dirname "$0")/.." && pwd)/zig-out/coverage}"
+SHIM="$OUT/shim"
+RAW="$OUT/raw"
+rm -rf "$OUT"
+mkdir -p "$SHIM" "$RAW"
+# One shim per wrapped binary, each writing to its own $$-suffixed directory:
+# kcov cannot have two live processes share one output dir, and this suite
+# runs a dozen daemons and hundreds of clients. They are merged below.
+# COV_WRAP names which binaries to trace. `mux` is NOT in the default set,
+# and that is a limitation rather than a preference: a traced `mux` that
+# auto-starts a daemon can never exit. The daemon reparents to init but stays
+# TRACED — PTRACE_O_TRACEFORK follows it there — and kcov waits for its last
+# traced descendant, so it waits on a process designed never to end. The
+# auto-start and `--via` legs wedge on this, and a SIGTERM will not free it.
+#
+# Excluding the `start` verb (below) does not cover it: the daemon is spawned
+# by the client, not named on the shim's command line. --exit-first-process is
+# kcov's answer and does fix this one, but it reparents the tracer, and then
+# the real process is no longer a child of the pid the suite holds — which
+# breaks hardkill and moves the hang one leg later. Daemons carry server.zig,
+# so tracing them and not the client is the side of the trade worth taking.
+#
+# Set COV_WRAP="mux" to measure the client over a slice that avoids those
+# legs (E2E_STOP_AFTER pairs with it), and merge the two runs by hand.
+for spec in $(echo "${COV_WRAP:-muxd muxa muxweb}" | tr ' ' '\n' | while read -r w; do
+    case "$w" in
+        muxd)   echo "muxd|$MUXD" ;;
+        mux)    echo "mux|$MUX" ;;
+        muxa)   echo "muxa|$MUXA" ;;
+        muxweb) echo "muxweb|$MUXWEB" ;;
+        "")     ;;
+        *)      echo "coverage FAIL: COV_WRAP names unknown binary '$w'" >&2; exit 1 ;;
+    esac
+done); do
+    name=${spec%%|*}
+    real=${spec#*|}
+    real=$(cd "$(dirname "$real")" && pwd)/$(basename "$real")
+    cat > "$SHIM/$name" <<SHIM_EOF
+#!/bin/sh
+[ -n "\$MUX_KCOV_ACTIVE" ] && exec "$real" "\$@"
+MUX_KCOV_ACTIVE=1; export MUX_KCOV_ACTIVE
+case " \$* " in
+    *" start "*) exec "$real" "\$@" ;;
+esac
+exec kcov --collect-only --include-path=$SRC "$RAW/$name.\$\$" "$real" "\$@"
+SHIM_EOF
+    chmod +x "$SHIM/$name"
+done
+# e2e.sh prepends `dirname "$MUXD"` to PATH so an auto-started daemon is the
+# one under test. That directory is $SHIM here, which is what makes the
+# daemons the suite starts BY NAME rather than by path get wrapped too.
+# Scenario boundaries, stamped by e2e.sh's ok(). kcov writes a database when
+# a traced process EXITS, so a database's mtime and this file together say
+# which scenario was running when it was written — which is the only
+# per-scenario attribution available from a single run. It is exact for the
+# short-lived clients and coarse for the dozen daemons that outlive several
+# scenarios and land wherever they were killed.
+OK_LOG="$OUT/scenarios.tsv"
+export E2E_OK_LOG="$OK_LOG"
+# Tracing costs roughly half again in wall clock, and several legs assert on a
+# reconnect or a handoff healing inside a real-time budget. Those budgets are
+# generous at native speed and merely tight under a tracer, so the suite is
+# told to wait longer — not to demand less. See e2e.sh's TIME_SCALE note.
+export E2E_TIME_SCALE="${COV_TIME_SCALE:-4}"
+set +e
+# Each argument is the shim if that binary is wrapped, the real thing if not.
+pick() { if [ -x "$SHIM/$1" ]; then echo "$SHIM/$1"; else echo "$2"; fi; }
+"$E2E" "$(pick muxd "$MUXD")" "$(pick mux "$MUX")" "$RAWMODE" "$DELAYPIPE" \
+    "$RENDER" "$PTYCLIENT" "$WSCLIENT" "$(pick muxweb "$MUXWEB")" \
+    "$(pick muxa "$MUXA")"
+E2E_RC=$?
+set -e
+# Reported, never fatal. A failing suite still collected everything up to
+# the failure, and throwing that away would make the one run you most want
+# to look at the one run that produces nothing.
+[ "$E2E_RC" -eq 0 ] || echo "coverage NOTE: e2e exited $E2E_RC; report covers the run up to that point"
+DIRS=$(find "$RAW" -mindepth 1 -maxdepth 1 -type d | wc -l)
+[ "$DIRS" -gt 0 ] || { echo "coverage FAIL: no kcov databases in $RAW"; exit 1; }
+kcov --merge --include-path="$SRC" "$OUT/merged" "$RAW"/* > /dev/null 2>&1
+JSON="$OUT/merged/kcov-merged/coverage.json"
+[ -f "$JSON" ] || { echo "coverage FAIL: merge produced no $JSON"; exit 1; }
+# Sorted by UNCOVERED lines, not by percentage: the question this report
+# answers is "what is worth a test", and 0% of a 24-line file is not the
+# same size of gap as 54% of server.zig.
+echo
+echo "coverage: $DIRS processes traced, e2e exited $E2E_RC"
+echo
+printf '%-22s %8s %13s %10s\n' file cov covered/total uncovered
+awk -F'"' '/"file":/ {
+    n = split($4, p, "/")
+    printf "%-22s %7s%% %6d/%-6d %10d\n", p[n], $8, $12, $16, $16 - $12
+}' "$JSON" | sort -k4 -rn
+# The totals live at the top level of the same file, past the files array,
+# and are the only bare (unquoted) numbers in it.
+awk -F'[:,]' '
+    /^  "percent_covered"/ { gsub(/[ "]/, "", $2); pct = $2 }
+    /^  "covered_lines"/   { gsub(/[ "]/, "", $2); cov = $2 }
+    /^  "total_lines"/     { gsub(/[ "]/, "", $2); tot = $2 }
+    END { printf "\nTOTAL %s%%  %s/%s lines\n", pct, cov, tot }
+' "$JSON"
+echo
+echo "html: file://$OUT/merged/kcov-merged/index.html"
+echo "per-process databases: $RAW (mtimes attribute to $OK_LOG)"

test/e2e.sh

Old		New
@@ -508,8 +508,24 @@ want_stat_ge() {
508	# Wait until PATTERN shows up in FILE (default 15s). Timing that keys off the	508	# Wait until PATTERN shows up in FILE (default 15s). Timing that keys off the
509	# session's own output instead of a fixed sleep: the marker is proof the	509	# session's own output instead of a fixed sleep: the marker is proof the
510	# client is attached and idle, which is exactly the state the tear needs.	510	# client is attached and idle, which is exactly the state the tear needs.
		511	# E2E_TIME_SCALE multiplies the polling BUDGETS below, and nothing else. It
		512	# exists for test/coverage.sh: a ptrace-traced binary runs about half again
		513	# slower, which is enough to blow a 25s convergence budget that is generous
		514	# at native speed — and the failure reads as "the attach never converged"
		515	# rather than "the tracer is slow", which is a lie about the product.
		516	#
		517	# Budgets only. Not the sleeps between polls, not a `timeout` that a scenario
		518	# asserts on, and never a threshold: scaling this changes how long the suite
		519	# is WILLING TO WAIT, never what it demands to see. A run at scale 4 that
		520	# passes proves the same facts as a run at scale 1, just later.
		521	TIME_SCALE="${E2E_TIME_SCALE:-1}"
		522	case "$TIME_SCALE" in
		523	''\|[!0-9]) echo "e2e FAIL: E2E_TIME_SCALE must be a positive integer"; exit 1 ;;
		524	esac
		525	[ "$TIME_SCALE" -ge 1 ] \|\| { echo "e2e FAIL: E2E_TIME_SCALE must be >= 1"; exit 1; }
		526
511	wait_for() {	527	wait_for() {
512	_file="$1"; _pat="$2"; _ticks=$(( ${3:-15} * 10 )); _i=0	528	_file="$1"; _pat="$2"; _ticks=$(( ${3:-15} * 10 * TIME_SCALE )); _i=0
513	while [ "$_i" -lt "$_ticks" ]; do	529	while [ "$_i" -lt "$_ticks" ]; do
514	if [ -f "$_file" ] && grep -q "$_pat" "$_file" 2>/dev/null; then return 0; fi	530	if [ -f "$_file" ] && grep -q "$_pat" "$_file" 2>/dev/null; then return 0; fi
515	sleep 0.1; _i=$((_i+1))	531	sleep 0.1; _i=$((_i+1))
@@ -517,12 +533,46 @@ wait_for() {
517	return 1	533	return 1
518	}	534	}
519		535
		536	# hardkill PID — SIGKILL a pid and anything it fathered, children first.
		537	#
		538	# Plain `kill -9 "$pid"` is right only while the pid the suite holds IS the
		539	# process under test. Under the coverage harness (test/coverage.sh) it is a
		540	# kcov wrapper instead, and SIGKILL is the one signal a wrapper cannot
		541	# forward: killing the node alone would leave the real daemon alive holding
		542	# its socket, and every leg that waits for that daemon to die would hang
		543	# rather than fail. Children first so a traced child's death is observed by
		544	# its tracer, which is when kcov writes the coverage it has collected.
		545	#
		546	# With no wrapper in the picture there are no children and this is exactly
		547	# `kill -9`, which is why the abort legs keep the semantics they assert on:
		548	# the daemon still dies by SIGKILL, still without unlinking its socket.
		549	hardkill() {
		550	# The test is what the process IS, not whether it has children. A daemon
		551	# has children too — a session shell per attach — and killing those first
		552	# ends the session CLEANLY, which is the one thing the abort legs must not
		553	# see: they assert on a client whose daemon vanished under it, and a tidy
		554	# session exit takes a different path out of the client (measured: exit
		555	# 128 where the leg wants 0). Only a wrapper gets the two-step treatment.
		556	if [ "$(ps -o comm= -p "$1" 2>/dev/null)" = kcov ]; then
		557	# A wrapper writes its coverage when it sees its child die, so give it
		558	# that moment. Killing both at once loses the database entirely —
		559	# measured: SIGKILLing kcov yields no coverage.db at any
		560	# --output-interval, so the write only ever happens on a clean exit.
		561	# Bounded at 1s, and it costs nothing when the wrapper exits at once.
		562	_i=0
		563	while kill -0 "$1" 2>/dev/null && [ "$_i" -lt 20 ]; do
		564	sleep 0.05; _i=$((_i + 1))
		565	done
		566	fi
		567	kill -9 "$1" 2>/dev/null \|\| true
		568	}
		569
520	# Poll until nothing answers on a socket path (2s). Keyed off the daemon's	570	# Poll until nothing answers on a socket path (2s). Keyed off the daemon's
521	# own liveness rather than a fixed sleep, same reasoning as wait_for.	571	# own liveness rather than a fixed sleep, same reasoning as wait_for.
522	wait_gone() {	572	wait_gone() {
523	_i=0	573	_i=0
524	while "$MUXD" dump --sock "$1" > /dev/null 2>&1; do	574	while "$MUXD" dump --sock "$1" > /dev/null 2>&1; do
525	_i=$((_i + 1)); [ "$_i" -lt 40 ] \|\| { echo "e2e FAIL: daemon on $1 never died"; exit 1; }	575	_i=$((_i + 1)); [ "$_i" -lt $(( 40 * TIME_SCALE )) ] \|\| { echo "e2e FAIL: daemon on $1 never died"; exit 1; }
526	sleep 0.05	576	sleep 0.05
527	done	577	done
528	}	578	}
@@ -534,7 +584,7 @@ wait_gone() {
534	# to bind almost always said why; pass "" for the spawns that have none.	584	# to bind almost always said why; pass "" for the spawns that have none.
535	wait_sock() {	585	wait_sock() {
536	_i=0	586	_i=0
537	while [ ! -S "$1" ] && [ "$_i" -lt 50 ]; do sleep 0.1; _i=$((_i+1)); done	587	while [ ! -S "$1" ] && [ "$_i" -lt $(( 50 * TIME_SCALE )) ]; do sleep 0.1; _i=$((_i+1)); done
538	[ -S "$1" ] \|\| {	588	[ -S "$1" ] \|\| {
539	echo "e2e FAIL: $3"	589	echo "e2e FAIL: $3"
540	# An `if` rather than `[ -n "$2" ] && cat "$2"`, for converged_quiet's	590	# An `if` rather than `[ -n "$2" ] && cat "$2"`, for converged_quiet's
@@ -572,7 +622,7 @@ dump_session() {
572	# reach and fail 10s later blaming the wrong thing.	622	# reach and fail 10s later blaming the wrong thing.
573	wait_grid() {	623	wait_grid() {
574	_i=0	624	_i=0
575	while [ "$_i" -lt 100 ]; do	625	while [ "$_i" -lt $(( 100 * TIME_SCALE )) ]; do
576	dump_session "$1" "${4:-}" 2>/dev/null \| grep -q "$2" && return 0	626	dump_session "$1" "${4:-}" 2>/dev/null \| grep -q "$2" && return 0
577	sleep 0.1; _i=$((_i+1))	627	sleep 0.1; _i=$((_i+1))
578	done	628	done
@@ -589,7 +639,7 @@ wait_grid() {
589	# assertions further down.	639	# assertions further down.
590	wait_sessions() {	640	wait_sessions() {
591	_i=0	641	_i=0
592	while [ "$_i" -lt 100 ]; do	642	while [ "$_i" -lt $(( 100 * TIME_SCALE )) ]; do
593	"$MUXD" stats --sock "$1" 2>/dev/null \| grep -q "sessions=$2" && return 0	643	"$MUXD" stats --sock "$1" 2>/dev/null \| grep -q "sessions=$2" && return 0
594	sleep 0.1; _i=$((_i+1))	644	sleep 0.1; _i=$((_i+1))
595	done	645	done
@@ -851,6 +901,29 @@ OK_COUNT=0
851	ok() {	901	ok() {
852	OK_COUNT=$((OK_COUNT + 1))	902	OK_COUNT=$((OK_COUNT + 1))
853	echo "e2e OK: $1"	903	echo "e2e OK: $1"
		904	# Scenario boundaries, stamped for whoever needs to attribute something
		905	# to the scenario that produced it. Nothing in this suite reads the file:
		906	# test/coverage.sh maps each kcov database to the scenario that was
		907	# running when the traced process wrote it, and the timestamps are also
		908	# the only per-scenario timing this suite has ever been able to report.
		909	#
		910	# An `if` rather than `[ -n ... ] && printf`, for wait_sock's reason: a
		911	# false guard as the last command in a function becomes that function's
		912	# exit status, and under `set -e` every unstamped run would abort at its
		913	# first passing scenario.
		914	if [ -n "${E2E_OK_LOG:-}" ]; then
		915	printf '%s\t%s\t%s\n' "$OK_COUNT" "$(date +%s.%N)" "$1" >> "$E2E_OK_LOG"
		916	fi
		917	# Prefix slicing. This suite is linear and stateful — the M13 scenarios
		918	# run inside sessions the M10 scenarios created — so a prefix is the only
		919	# slice that means anything, and a "run just scenario 40" filter would be
		920	# a filter that lies. Exiting rather than skipping keeps that honest, and
		921	# exiting 0 goes out through the trap, so a sliced run still tears down
		922	# its daemons and still reports its leak verdict.
		923	if [ -n "${E2E_STOP_AFTER:-}" ] && [ "$OK_COUNT" -ge "$E2E_STOP_AFTER" ]; then
		924	echo "e2e STOP: sliced after $OK_COUNT scenarios (E2E_STOP_AFTER)"
		925	exit 0
		926	fi
854	}	927	}
855		928
856	# ---- the leak sweep's two halves (hygiene kit, 6a) ---------------------	929	# ---- the leak sweep's two halves (hygiene kit, 6a) ---------------------
@@ -1427,7 +1500,7 @@ ok "styled content survives the paint path"
1427	{ printf 'seq 1 60\n'; sleep 2; } \| "$MUX" --sock "$SOCK" > "$OUT.kill" &	1500	{ printf 'seq 1 60\n'; sleep 2; } \| "$MUX" --sock "$SOCK" > "$OUT.kill" &
1428	CPID=$!	1501	CPID=$!
1429	sleep 1	1502	sleep 1
1430	kill -9 "$CPID" 2>/dev/null \|\| true	1503	hardkill "$CPID"
1431	sleep 1	1504	sleep 1
1432	kill -0 "$DPID" \|\| { echo "e2e FAIL: daemon died after client kill -9"; exit 1; }	1505	kill -0 "$DPID" \|\| { echo "e2e FAIL: daemon died after client kill -9"; exit 1; }
1433		1506
@@ -1569,7 +1642,7 @@ set +e
1569	CLIPID=$!	1642	CLIPID=$!
1570	# Let the session establish, then take the daemon away mid-session.	1643	# Let the session establish, then take the daemon away mid-session.
1571	sleep 2	1644	sleep 2
1572	kill -9 "$D2PID" 2>/dev/null	1645	hardkill "$D2PID"
1573	D2PID=""	1646	D2PID=""
1574	wait "$CLIPID"	1647	wait "$CLIPID"
1575	RC=$?	1648	RC=$?
@@ -1658,7 +1731,7 @@ wait_for "$OUT.m7b" "m7b-pre-restart" 20 \|\| {
1658		1731
1659	# SIGKILL leaves the socket file behind; the new daemon's stale-socket	1732	# SIGKILL leaves the socket file behind; the new daemon's stale-socket
1660	# recovery (ECONNREFUSED + S_ISSOCK -> unlink) is what lets it rebind here.	1733	# recovery (ECONNREFUSED + S_ISSOCK -> unlink) is what lets it rebind here.
1661	kill -9 "$D3PID"	1734	hardkill "$D3PID"
1662	sleep 0.5	1735	sleep 0.5
1663	"$MUXD" run --sock "$SOCK3" --shell /bin/sh > "$OUT.d3b.d" 2>&1 &	1736	"$MUXD" run --sock "$SOCK3" --shell /bin/sh > "$OUT.d3b.d" 2>&1 &
1664	D3PID=$!	1737	D3PID=$!
@@ -2030,7 +2103,7 @@ wait_for "$OUT.qk" "quic-pre-restart" 20 \|\| {
2030	cat "$OUT.qk" "$OUT.qk.err" 2>/dev/null; exit 1;	2103	cat "$OUT.qk" "$OUT.qk.err" 2>/dev/null; exit 1;
2031	}	2104	}
2032		2105
2033	kill -9 "$D4PID"	2106	hardkill "$D4PID"
2034	D4PID=""	2107	D4PID=""
2035	sleep 0.5	2108	sleep 0.5
2036	"$MUXD" run --sock "$SOCK4" --shell /bin/sh \	2109	"$MUXD" run --sock "$SOCK4" --shell /bin/sh \