Revisit mutate-report: threshold, or drop it #92

New issue

Open

opened 2026-08-22 16:01:01 +00:00 by a73x · 0 comments

a73x commented

2026-08-22 16:01:01 +00:00

(Migrated from git.a73x.sh)

[claude 2026-08-22] make ci now runs mutate-report: gremlins over the diff against origin/main, reported and never failing (the warn tier lint-extra uses). It is deliberately toothless for now — this issue is the reminder to come back and decide whether it grows teeth.

What to check when revisiting:

What do real branches score? First observations, for calibration:
- the 8 unpushed commits on main (74 files, 10k lines): 166 killed / 55 lived / 6 timed out, 75.11% efficacy, 43s
- whole-tree baseline for comparison: 88.9% efficacy (see the baseline issue)
  A diff scores well below the tree, which is the point — recently changed lines are the least asserted.
Is 43s per push acceptable? It scales with diff size, not tree size, so it should stay flat. Watch for a branch where it does not.
Can a threshold be set? --threshold-efficacy is the flag. 100% would be wrong: the tree contains equivalent mutants that cannot be killed (internal/cloudinit's i+1 < len(root.Content) -> <= is the same loop for every even-length YAML mapping). A floor near the observed median, raised over time, is the ratchet shape the coverage gate already uses (scripts/coverage.sh).
Does the report get read? A warn tier nobody reads is worse than no tier — it costs time and buys nothing. If three cycles pass with survivors nobody looks at, either gate it or delete it.

Decide by looking at the numbers, not by preference.

Migrated from git-collab issue f283feff2201f2a6831032aec0b7863d730f9fce.

Original author: a73x dev@a73x.sh.

[claude 2026-08-22] `make ci` now runs `mutate-report`: gremlins over the diff against origin/main, reported and never failing (the warn tier lint-extra uses). It is deliberately toothless for now — this issue is the reminder to come back and decide whether it grows teeth. What to check when revisiting: 1. What do real branches score? First observations, for calibration: - the 8 unpushed commits on main (74 files, 10k lines): 166 killed / 55 lived / 6 timed out, 75.11% efficacy, 43s - whole-tree baseline for comparison: 88.9% efficacy (see the baseline issue) A diff scores well below the tree, which is the point — recently changed lines are the least asserted. 2. Is 43s per push acceptable? It scales with diff size, not tree size, so it should stay flat. Watch for a branch where it does not. 3. Can a threshold be set? --threshold-efficacy is the flag. 100% would be wrong: the tree contains equivalent mutants that cannot be killed (internal/cloudinit's `i+1 < len(root.Content)` -> `<=` is the same loop for every even-length YAML mapping). A floor near the observed median, raised over time, is the ratchet shape the coverage gate already uses (scripts/coverage.sh). 4. Does the report get read? A warn tier nobody reads is worse than no tier — it costs time and buys nothing. If three cycles pass with survivors nobody looks at, either gate it or delete it. Decide by looking at the numbers, not by preference. --- Migrated from git-collab issue `f283feff2201f2a6831032aec0b7863d730f9fce`. Original author: a73x <dev@a73x.sh>.