Mutation-testing baseline 2026-08-22: 281 surviving mutants, 419 unreached

a73x commented

2026-08-22 15:50:00 +00:00

(Migrated from git.a73x.sh)

[claude 2026-08-22] First mutation-testing baseline, gremlins v0.6.0, make mutate (added in the same cycle). 54 packages, 2996 mutants, 22 min wall on tmpfs with 2 workers. internal/agent/cloudhv excluded — see the SIGKILL issue.

Totals: killed 2255 / lived 281 / not covered 419 / timed out 41.
Test efficacy 88.9% (of mutants the tests execute, ~1 in 9 changes behaviour unnoticed). Mutant coverage 84.6% (419 mutants no test reaches).

Weakest efficacy — code is executed but not asserted:

PACKAGE                      KILLED LIVED EFFICACY
internal/covsnap                  1     1   50.00%
internal/agent/serialpump        24    17   58.54%
internal/agent/hyperlog          13     5   72.22%
internal/agent/syncclient        32    10   76.19%
internal/site                   202    53   79.22%
internal/shape                   17     4   80.95%
internal/agent/netsnoop          20     4   83.33%
internal/server/config           25     5   83.33%
internal/transport               16     3   84.21%
internal/agent/enrollclient       6     1   85.71%
internal/agent/hostinfo          36     6   85.71%
internal/server/api/spec         18     3   85.71%
internal/gateclient              19     3   86.36%
internal/agent/reconcile         68    10   87.18%
internal/smoke                  157    23   87.22%
internal/server/syncsvc          63     9   87.50%
internal/mcpserver               97    13   88.18%
internal/cli                    112    15   88.19%
internal/agent/imagecache        58     7   89.23%
internal/server/sshgate          17     2   89.47%
internal/agent/bootstrap         26     3   89.66%
internal/cloudinit               35     4   89.74%

Weakest mutant coverage — code no test reaches at all:

PACKAGE                      NOTCOVERED   MUTCOV
internal/agent/dhcp                 5   28.57%
internal/server/boot               45   40.79%
internal/agent/run                 26   46.94%
internal/names                     11   52.17%
internal/agent/hostinfo            37   53.16%
internal/server/release            13   61.76%
internal/covsnap                    1   66.67%
internal/smoke                     77   70.04%
internal/agent/netsnoop            10   70.59%
internal/gateclient                 9   70.97%
internal/server/registry            2   71.43%

Full per-package table, JSON and logs: /var/tmp/gremlins-baseline/v2/ (not durable — rerun with make mutate).

This issue is the record of where we started, not a task. Split per-package work off it as separate issues.

Migrated from git-collab issue d78939caed3f63f2810e04ecf8484dd5aad38080.

Original author: a73x dev@a73x.sh.

[claude 2026-08-22] First mutation-testing baseline, gremlins v0.6.0, `make mutate` (added in the same cycle). 54 packages, 2996 mutants, 22 min wall on tmpfs with 2 workers. internal/agent/cloudhv excluded — see the SIGKILL issue. Totals: killed 2255 / lived 281 / not covered 419 / timed out 41. Test efficacy 88.9% (of mutants the tests execute, ~1 in 9 changes behaviour unnoticed). Mutant coverage 84.6% (419 mutants no test reaches). Weakest efficacy — code is executed but not asserted: ``` PACKAGE KILLED LIVED EFFICACY internal/covsnap 1 1 50.00% internal/agent/serialpump 24 17 58.54% internal/agent/hyperlog 13 5 72.22% internal/agent/syncclient 32 10 76.19% internal/site 202 53 79.22% internal/shape 17 4 80.95% internal/agent/netsnoop 20 4 83.33% internal/server/config 25 5 83.33% internal/transport 16 3 84.21% internal/agent/enrollclient 6 1 85.71% internal/agent/hostinfo 36 6 85.71% internal/server/api/spec 18 3 85.71% internal/gateclient 19 3 86.36% internal/agent/reconcile 68 10 87.18% internal/smoke 157 23 87.22% internal/server/syncsvc 63 9 87.50% internal/mcpserver 97 13 88.18% internal/cli 112 15 88.19% internal/agent/imagecache 58 7 89.23% internal/server/sshgate 17 2 89.47% internal/agent/bootstrap 26 3 89.66% internal/cloudinit 35 4 89.74% ``` Weakest mutant coverage — code no test reaches at all: ``` PACKAGE NOTCOVERED MUTCOV internal/agent/dhcp 5 28.57% internal/server/boot 45 40.79% internal/agent/run 26 46.94% internal/names 11 52.17% internal/agent/hostinfo 37 53.16% internal/server/release 13 61.76% internal/covsnap 1 66.67% internal/smoke 77 70.04% internal/agent/netsnoop 10 70.59% internal/gateclient 9 70.97% internal/server/registry 2 71.43% ``` Full per-package table, JSON and logs: /var/tmp/gremlins-baseline/v2/ (not durable — rerun with make mutate). This issue is the record of where we started, not a task. Split per-package work off it as separate issues. --- Migrated from git-collab issue `d78939caed3f63f2810e04ecf8484dd5aad38080`. Original author: a73x <dev@a73x.sh>.

a73x commented

2026-08-23 11:27:42 +00:00

(Migrated from git.a73x.sh)

[claude 2026-08-23] One note for the baseline's interpretation, from the work merged in d0c032f.

A mutant reported TIMED OUT is not necessarily an unobserved mutant. On the lying-tests branch in isolation, gremlins reported 1 timed out against internal/server/syncsvc. Fable reproduced the mutant class by hand: 10 * time.Second → / makes the production handshake default 0, and the package then fails by assertion in ~6.5s (TestOpenTCPRefused, TestReconnectKeepsNewConnRegistered — both tests that do not override the field, so the production default's presence is genuinely enforced).

The TIMED OUT was gremlins' per-mutant clock (baseline × coefficient) expiring before the slowed suite finished, not a surviving mutant. In the integrated tree the same line reports a clean kill: 7 killed, 0 lived, 0 not covered.

So when reading the 281-survivor baseline: treat TIMED OUT as "unknown, re-run in isolation with a longer coefficient", not as "lived".

[claude 2026-08-23] One note for the baseline's interpretation, from the work merged in d0c032f. A mutant reported **TIMED OUT** is not necessarily an unobserved mutant. On the `lying-tests` branch in isolation, gremlins reported 1 timed out against `internal/server/syncsvc`. Fable reproduced the mutant class by hand: `10 * time.Second` → `/` makes the production handshake default 0, and the package then **fails by assertion** in ~6.5s (`TestOpenTCPRefused`, `TestReconnectKeepsNewConnRegistered` — both tests that do not override the field, so the production default's presence is genuinely enforced). The TIMED OUT was gremlins' per-mutant clock (baseline × coefficient) expiring before the slowed suite finished, not a surviving mutant. In the integrated tree the same line reports a clean kill: 7 killed, 0 lived, 0 not covered. So when reading the 281-survivor baseline: treat TIMED OUT as "unknown, re-run in isolation with a longer coefficient", not as "lived".

a73x commented

2026-08-23 14:42:47 +00:00

(Migrated from git.a73x.sh)

[claude 2026-08-23] Re-baseline after the comment-audit cycle (~40 tests added). Same method: make mutate per package, gremlins v0.6.0, 2 workers, timeout-coefficient 10, cloudhv excluded.

Comparable scope (internal/ minus internal/pb, 51 packages producing mutants — 2999 mutants vs the original's 2996, which is what confirms the scope matches; the original's "54 packages" counted three that yielded none):

	2026-08-22	2026-08-23
killed	2255	2173
lived	281	184
not covered	419	413
timed out	41	229
efficacy	88.9%	92.2%
mutant coverage	84.6%	86.2%

97 surviving mutants died. That is the audit's effect, and it is the only number here that moved for a reason internal to the tree.

Two cautions on the rest:

Timed out went 41 → 229. The machine was under concurrent load during this run. A timed-out mutant is unresolved, not alive — most would have been kills — so killed reads 82 lower than last time despite the tree being strictly better tested. Do not read the killed column as a regression. If a clean number is ever needed, rerun on an idle box.
Not covered is flat (419 → 413). The audit corrected claims that tests already reached; it did not reach new code. Unreached code is still the larger deficit and is untouched by this cycle.

Two scope notes for anyone rerunning:

internal/pb contributes 287 not-covered mutants and nothing else — generated protobuf, no tests, never worth mutating. The original baseline excluded it; keep excluding it or the totals stop comparing.
cmd/* (8 packages) contributes 29 not-covered and zero kills. Also outside the original scope.

Eight packages produce no mutants at all (internal/agent/exec, internal/agent/permanent, internal/arch, internal/guest, internal/random, internal/relmanifest, internal/server/api/types, internal/version) — declarations and thin wrappers with nothing to mutate.

Where the survivors now live:

PACKAGE                    LIVED  NOTCOVERED
internal/server/api           38          41
internal/smoke                23          77
internal/cli                  15          33
internal/server/store         14           7
internal/mcpserver            13          23
internal/agent/syncclient     10           6
internal/agent/reconcile       9          16
internal/server/syncsvc        9           8
internal/agent/hostinfo        6          37

internal/server/api is now the single worst package on both axes and did not appear at the top of either table in the original. Worth its own issue.

[claude 2026-08-23] Re-baseline after the comment-audit cycle (~40 tests added). Same method: `make mutate` per package, gremlins v0.6.0, 2 workers, timeout-coefficient 10, cloudhv excluded. Comparable scope (internal/ minus internal/pb, 51 packages producing mutants — 2999 mutants vs the original's 2996, which is what confirms the scope matches; the original's "54 packages" counted three that yielded none): | | 2026-08-22 | 2026-08-23 | |---|---|---| | killed | 2255 | 2173 | | lived | **281** | **184** | | not covered | 419 | 413 | | timed out | 41 | 229 | | efficacy | 88.9% | 92.2% | | mutant coverage | 84.6% | 86.2% | **97 surviving mutants died.** That is the audit's effect, and it is the only number here that moved for a reason internal to the tree. Two cautions on the rest: - **Timed out went 41 → 229.** The machine was under concurrent load during this run. A timed-out mutant is unresolved, not alive — most would have been kills — so `killed` reads 82 lower than last time despite the tree being strictly better tested. Do not read the killed column as a regression. If a clean number is ever needed, rerun on an idle box. - **Not covered is flat (419 → 413).** The audit corrected claims that tests already reached; it did not reach new code. Unreached code is still the larger deficit and is untouched by this cycle. Two scope notes for anyone rerunning: - `internal/pb` contributes 287 not-covered mutants and nothing else — generated protobuf, no tests, never worth mutating. The original baseline excluded it; keep excluding it or the totals stop comparing. - `cmd/*` (8 packages) contributes 29 not-covered and zero kills. Also outside the original scope. Eight packages produce no mutants at all (`internal/agent/exec`, `internal/agent/permanent`, `internal/arch`, `internal/guest`, `internal/random`, `internal/relmanifest`, `internal/server/api/types`, `internal/version`) — declarations and thin wrappers with nothing to mutate. Where the survivors now live: ``` PACKAGE LIVED NOTCOVERED internal/server/api 38 41 internal/smoke 23 77 internal/cli 15 33 internal/server/store 14 7 internal/mcpserver 13 23 internal/agent/syncclient 10 6 internal/agent/reconcile 9 16 internal/server/syncsvc 9 8 internal/agent/hostinfo 6 37 ``` `internal/server/api` is now the single worst package on both axes and did not appear at the top of either table in the original. Worth its own issue.

Rows
Columns

Mutation-testing baseline 2026-08-22: 281 surviving mutants, 419 unreached #87