RETRO.md

Ref: main v0.0.5   Size: 7.1 KiB   History

# Retro

One line per push to `main`: what slowed the work down. Enforced by
`.githooks/pre-push` — a push that adds no line here is refused.

## 2026-08-22

- Mutation testing: two full sweeps were spent before noticing the workdir
  filesystem, not the tool, was producing the numbers — ext4 reported 207
  timed-out mutants and a flattering 100% efficacy where tmpfs reported 199
  killed and an honest 93%.
- The diff-scoped mutation check was wired without asking what an empty diff
  means. Pushing main answered it: origin/main became HEAD, gremlins read a
  diff with no entries as "everything changed", and the whole-tree run it
  started was killed by the one mutant that signals its own process group.
- Wire rename: `make ci` failed twice on nothing wrong. A fresh worktree has
  an empty web/dist, so the web package misses its coverage floor until
  `make web` runs; and proto-check diffs generated code against the COMMIT,
  so it always fails on an uncommitted .proto change. Neither gate says so.
- Wire-rename followups: proving "a v0.0.7 agent still talks to this server"
  meant rebuilding the old descriptor set from the tag by hand. The tree keeps
  every version of the .proto and no artifact of what a release actually put on
  the wire, so the compatibility claim had nowhere to be checked against.
- Comment audit: six agents worked in parallel worktrees forked from the same
  commit, and by the time they finished, main had moved a rename underneath
  them. The merge was clean and would have silently reverted it — nothing in
  git distinguishes "did not touch this" from "restored the old name". Six
  deletions were abandoned because their mutation evidence was gathered against
  a tree that no longer existed. Proving the rest was comments-only needed a
  purpose-built token-stream comparator; go/printer reflows, so a diff that
  round-trips through it reads as changed when nothing was.
- Volumes: the plan specified a schema FK (`volumes.host_id → hosts`) that
  broke host removal, and a 120s smoke cleanup window against a 5-minute reap
  grace; both were only caught by review against the live contracts, not by
  the task tests.

## 2026-08-23

- Loud misconfiguration: the survey of both call sites was done in a worktree
  still parked on the previous main, so the two tests the work was told not to
  duplicate looked like they did not exist and were nearly rewritten. A
  worktree's branch is not the branch you are about to fork from, and nothing
  says so. `make shape-check` also diffs the generated graph against the
  COMMIT, so adding one import fails the gate until the regenerated docs are
  committed — the same trap proto-check sets, in a second place.
- Constant sweep: the third time this month `make ci` failed on a package the
  branch never touched. internal/server/web's floor is baselined against a
  built dist and a fresh worktree has none, so the gate reports a coverage
  regression where the real fact is "the UI was not compiled". It is written
  down twice above and was still rediscovered from scratch; a RETRO entry is
  not a gate. Filed.
- Concurrency observers: the same stale-fork tax, one lane later. The worktree
  forked from a main five commits behind, so the first make ci failed on a
  coverage floor in a package this branch never touched, and the rebase landed
  on a wire rename mid-branch. Then a plain `git checkout <file>` to undo a
  mutation silently ate the uncommitted comment deletions sitting in the same
  file — mutation-test rounds and unstaged edits do not share a working tree.
  Stage before mutating, and restore with an explicit --source=HEAD.
- Wrong comments: verifying cost more than fixing. Every line number in the
  filed issue was stale — the audit branch had already moved the files, and two
  of the eleven were fixed in passing there — so each site had to be found by
  content and then re-proved against the tree rather than trusted. Worse, the
  callers grep to disprove "the snapshot path uses FreezeCAs" printed `no
  matches found` because zsh expanded --include=*.go, not because there were no
  callers. A negative that arrives as a shell error reads exactly like a
  negative that arrives as an answer. Quote the glob, and treat an empty result
  as a claim to check rather than a claim to make.
- Factory fixes: the merge_how proof ran the merger by hand and skipped the
  handler that strips merge_how from a part before the cross-source merge, so
  the fix that passed in the guest failed the first boot gate. Two more gate
  runs died at placement: a gate that dies does not reap its smoke VM, and the
  leftover plus a live test VM filled the devhost. Prove at the layer the code
  runs through, and check the host is empty before a run.
- Volumes release: the branch's own refactor broke the release. Ordering moved
  to internal/version so the agent could reach it, and the ship script's tag
  probe kept compiling against internal/server/release.Less — a symbol nothing
  in the build graph references, so vet, lint and the boot gate all passed and
  stage 1 of the stg ship found it instead. Scripts that compile Go inline are
  outside every check the tree runs on itself. The rebase cost the same way
  twice over: HardDeleteVM had gained a hostID argument on main, and a fake API
  the new cloud-init test replaced wholesale stopped registering its VM, so the
  first ci after a clean rebase failed in two places that had nothing to do
  with volumes.
- Site image: the ship stopped at stage 3 because nginx:alpine still carried
  seven HIGH util-linux advisories that Alpine had already fixed in its own
  package repo, and the image gate is right to refuse them. The base tag lags
  the distro, so a build is only as current as whenever upstream last rebuilt;
  taking the upgrade at build time is the difference between shipping and
  waiting on someone else's release cadence.
- Backup image: the same base-tag lag as the site image, found one stage later
  and only on prod. BACKUPS=1 is a prod-only setting, so the new BACKUP_IMAGE
  requirement had never been exercised — stg proved nine stages and still could
  not have caught this. A plane-specific config gate is only proven on the plane
  that sets it.
- Backup pull secret: baking sqlite in moved the job's image from Docker Hub to
  the plane's own registry, and the manifest kept the pull spec it had when the
  image was public. Nothing failed at apply time — a CronJob validates fine with
  no credential — so the break would have surfaced at 03:20 as an
  ImagePullBackOff nobody was watching, which is the exact failure the change
  set out to end. A workload that changes registries changes its auth, and only
  running the job proves it.
- Scanner blindness: the flood of "Version matching error" lines during the
  v0.0.8 ship read as cosmetic noise and was the finding. trivy 0.58 could not
  parse the toolchain's go1.27.0-X:nodwarf5 stamp, so it matched no Go module
  against any advisory and said so once per module rather than once per image —
  loud enough to be ignored, quiet enough to look like a warning. A scanner that
  cannot read a version passes; only reading the summary and noticing the
  gobinary targets were missing would have caught it.