Mutation-testing baseline 2026-08-22: 281 surviving mutants, 419 unreached
open by a73x
Labels: backlog
[claude 2026-08-22] First mutation-testing baseline, gremlins v0.6.0, `make mutate` (added in the same cycle). 54 packages, 2996 mutants, 22 min wall on tmpfs with 2 workers. internal/agent/cloudhv excluded — see the SIGKILL issue. Totals: killed 2255 / lived 281 / not covered 419 / timed out 41. Test efficacy 88.9% (of mutants the tests execute, ~1 in 9 changes behaviour unnoticed). Mutant coverage 84.6% (419 mutants no test reaches). Weakest efficacy — code is executed but not asserted: ``` PACKAGE KILLED LIVED EFFICACY internal/covsnap 1 1 50.00% internal/agent/serialpump 24 17 58.54% internal/agent/hyperlog 13 5 72.22% internal/agent/syncclient 32 10 76.19% internal/site 202 53 79.22% internal/shape 17 4 80.95% internal/agent/netsnoop 20 4 83.33% internal/server/config 25 5 83.33% internal/transport 16 3 84.21% internal/agent/enrollclient 6 1 85.71% internal/agent/hostinfo 36 6 85.71% internal/server/api/spec 18 3 85.71% internal/gateclient 19 3 86.36% internal/agent/reconcile 68 10 87.18% internal/smoke 157 23 87.22% internal/server/syncsvc 63 9 87.50% internal/mcpserver 97 13 88.18% internal/cli 112 15 88.19% internal/agent/imagecache 58 7 89.23% internal/server/sshgate 17 2 89.47% internal/agent/bootstrap 26 3 89.66% internal/cloudinit 35 4 89.74% ``` Weakest mutant coverage — code no test reaches at all: ``` PACKAGE NOTCOVERED MUTCOV internal/agent/dhcp 5 28.57% internal/server/boot 45 40.79% internal/agent/run 26 46.94% internal/names 11 52.17% internal/agent/hostinfo 37 53.16% internal/server/release 13 61.76% internal/covsnap 1 66.67% internal/smoke 77 70.04% internal/agent/netsnoop 10 70.59% internal/gateclient 9 70.97% internal/server/registry 2 71.43% ``` Full per-package table, JSON and logs: /var/tmp/gremlins-baseline/v2/ (not durable — rerun with make mutate). This issue is the record of where we started, not a task. Split per-package work off it as separate issues.
Comments
a73x
[claude 2026-08-23] One note for the baseline's interpretation, from the work merged in d0c032f. A mutant reported **TIMED OUT** is not necessarily an unobserved mutant. On the `lying-tests` branch in isolation, gremlins reported 1 timed out against `internal/server/syncsvc`. Fable reproduced the mutant class by hand: `10 * time.Second` → `/` makes the production handshake default 0, and the package then **fails by assertion** in ~6.5s (`TestOpenTCPRefused`, `TestReconnectKeepsNewConnRegistered` — both tests that do not override the field, so the production default's presence is genuinely enforced). The TIMED OUT was gremlins' per-mutant clock (baseline × coefficient) expiring before the slowed suite finished, not a surviving mutant. In the integrated tree the same line reports a clean kill: 7 killed, 0 lived, 0 not covered. So when reading the 281-survivor baseline: treat TIMED OUT as "unknown, re-run in isolation with a longer coefficient", not as "lived".
a73x
[claude 2026-08-23] Re-baseline after the comment-audit cycle (~40 tests added). Same method: `make mutate` per package, gremlins v0.6.0, 2 workers, timeout-coefficient 10, cloudhv excluded. Comparable scope (internal/ minus internal/pb, 51 packages producing mutants — 2999 mutants vs the original's 2996, which is what confirms the scope matches; the original's "54 packages" counted three that yielded none): | | 2026-08-22 | 2026-08-23 | |---|---|---| | killed | 2255 | 2173 | | lived | **281** | **184** | | not covered | 419 | 413 | | timed out | 41 | 229 | | efficacy | 88.9% | 92.2% | | mutant coverage | 84.6% | 86.2% | **97 surviving mutants died.** That is the audit's effect, and it is the only number here that moved for a reason internal to the tree. Two cautions on the rest: - **Timed out went 41 → 229.** The machine was under concurrent load during this run. A timed-out mutant is unresolved, not alive — most would have been kills — so `killed` reads 82 lower than last time despite the tree being strictly better tested. Do not read the killed column as a regression. If a clean number is ever needed, rerun on an idle box. - **Not covered is flat (419 → 413).** The audit corrected claims that tests already reached; it did not reach new code. Unreached code is still the larger deficit and is untouched by this cycle. Two scope notes for anyone rerunning: - `internal/pb` contributes 287 not-covered mutants and nothing else — generated protobuf, no tests, never worth mutating. The original baseline excluded it; keep excluding it or the totals stop comparing. - `cmd/*` (8 packages) contributes 29 not-covered and zero kills. Also outside the original scope. Eight packages produce no mutants at all (`internal/agent/exec`, `internal/agent/permanent`, `internal/arch`, `internal/guest`, `internal/random`, `internal/relmanifest`, `internal/server/api/types`, `internal/version`) — declarations and thin wrappers with nothing to mutate. Where the survivors now live: ``` PACKAGE LIVED NOTCOVERED internal/server/api 38 41 internal/smoke 23 77 internal/cli 15 33 internal/server/store 14 7 internal/mcpserver 13 23 internal/agent/syncclient 10 6 internal/agent/reconcile 9 16 internal/server/syncsvc 9 8 internal/agent/hostinfo 6 37 ``` `internal/server/api` is now the single worst package on both axes and did not appear at the top of either table in the original. Worth its own issue.