docs/superpowers/specs/2026-09-01-e2e-dedupe-and-vm-gate-design.md

Ref: main worktree-web-client-spec   Size: 6.8 KiB   History

# e2e harness: dedupe, and a real-VM gate

2026-09-01. Two improvements to the test harness, independent enough to land
separately, ordered so the cheap one goes first.

## Why now

The retro record names the costs this design answers:

- Two askpass legs went green about a prompt that was never raised, because
  the ssh *shim* is an assumption wearing a test's clothes (2026-08-29,
  twice). "Assert behavior, don't assume it. Mocks are assumptions" — the
  shim is the one standing mock in the suite.
- Four group files each hand-build the same shim rig; ~123 sites spell the
  same four-line rc-capture idiom; ~57 ptyclient scripts end in the same
  detach tail. Every copy is a place a fix has to be found.
- `wan.sh` proves the real-remote harness pattern works but is hand-run
  against pet boxes, outside every gate.

## Part 1 — dedupe (no behavior change)

As designed: three lib helpers. As measured on contact (2026-09-01): one
survived. The scenario pins (105/38 in `test/e2e.sh`) do not move; that
they do not move is the delivery check that this part changed nothing.

**1. `ssh_shim_rig DIR` — DROPPED.** Reading all five shim bodies showed
the common factor is already extracted: `ssh_shim_head` IS it, done
deliberately. The bodies differ in exactly what each group pins — 04 logs
pid/comm/args, 09 remaps a runtime dir per fake box, 09's second shim
fakes unreachable-with-a-reason, 16 remaps a whole HOME — and 15 refuses
the shared head on purpose, with a comment saying why (its subject is the
options the head discards). A rig helper would have served one caller and
blurred the other four.

**2. `rc0 "message" [FILE...]`** (renamed from the spec's `must`, reshaped
to read the `RC` the enclosing `set +e … RC=$? … set -e` capture already
holds — a wrapper taking `cmd args...` cannot carry the heredocs most
sites feed ptyclient). Fails the suite printing the caller's own prose and
the named capture files. It appends no "(rc N)" suffix: the messages carry
`$RC` in bespoke positions, some with rc legends ("134 = panic"). 58 of
the 82 plain `-eq 0` sites converted; the ~24 that compare RC to specific
values, kill something, or dump state on the way out keep their bespoke
form, and that boundary is the point — the helper covers the shape, never
the judgment.

**3. A shared detach tail — DROPPED.** The heredocs are quote-protected so
`$$` and `$?` reach the REMOTE shell literally; interpolating a shared
tail means unquoting every script, which breaks every marker. Two clear
lines per site beat that trade.

Conversion gate: full `make e2e` after each group file converts (the suite is
linear and stateful; `E2E_ONLY` cannot vouch for a group's neighbors), and
`make ci` before delivery.

## Part 2 — `make vm`: user journeys against a real box

A new `test/vm.sh` and Makefile target. It is a delivery-ritual gate beside
`make ci`, never inside it: `make ci` stays hermetic and offline. The e2e ssh
shims stay — they are the fast local pins; `make vm` is where their
assumptions get audited against a real sshd, a real network, a real Ubuntu.

### Fixture

One long-lived VM, like a real user's server:

- **`mux-e2e`** on the eitri fleet, pinned to host **charizard** (this
  machine), x86_64, 2 vCPU / 2 GB / 10 GB, underlay address `10.78.5.3`.
- Reachable by plain non-interactive ssh as `ubuntu@10.78.5.3` — proven
  2026-09-01: cert auth, no gate hop, `mux d endpoint` on the mux-less box
  answers rc 127.
- Auth is pinned by a `~/.ssh/config` Host block for `10.78.5.3` naming the
  harness identity (`~/.config/mux-vm/vm_key` + its 5-year cert, CA
  registered with the tenant as `mux-vm-gate`), so the gate never depends on
  the freshness of anyone's interactive cert.
- The hosts-file spelling the journeys use is `ubuntu@10.78.5.3` —
  `[user@]host` is in `hosts.zig`'s grammar.

Hermeticity comes from a **scorched-earth reset inside the VM** at the top of
every run — kill every mux process, remove the installed binary and all four
XDG dirs — not from re-creating the VM. The client side runs on this box
under `ptyclient` with every XDG dir isolated to scratch, per the standing
rig rule.

`vm.sh` preflights `ssh ubuntu@10.78.5.3 true` and REFUSES with the recovery
recipe when it fails (VM gone: recreate on charizard and re-read the underlay
IP; cert stale: re-sign against `~/.config/mux-vm/vm_ca`). A missing VM is a
loud failure, never a skip — the same stance e2e.sh takes on nvim and curl:
a gate that can quietly not run is the one failure a green tree cannot show.

### Journeys, v1

Each claim about the VM is asserted by a separate ssh reading `/proc` or the
filesystem there — never by the daemon's own report. The build under test is
the static musl `mux` (same target `make release` ships).

1. **Install like a user.** scp the binary to `~/.local/bin/mux` on the
   mux-less box; `mux d endpoint` over ssh now answers instead of 127.
2. **Entry dial.** `ptyclient` runs `mux ubuntu@10.78.5.3`: real ssh, real
   `mux d endpoint --start` auto-start, session paints. Type a marker,
   assert it via ssh against the VM's own pty (grid dump run remotely),
   detach, reattach, marker still present. Asserts the daemon's pid file /
   `/proc` state remotely: exactly one daemon, exec'd from the installed
   image.
3. **No autostart.** `mux d stop` on the VM; hold a wall polling the host
   for >3 poll cycles; assert via ssh that NO mux process exists on the VM
   after — the poll must not have resurrected it (the live-box bug,
   2026-08-30 era, now pinned against a real remote).
4. **Remote upgrade.** `mux d upgrade ubuntu@10.78.5.3` pushing the same
   image (the shape `e2e_16_push` already pins locally): assert the daemon
   pid did
   NOT change across the exec (`/proc/PID/exe` re-read through ssh), the
   session and its OSC 133 marks mechanism survive, and `mux a` still
   answers with `mechanism: marks`.

### Stretch (recorded, not v1)

- QUIC journey over the underlay address.
- Version-skew journey: install the released v15 tarball on the VM, assert
  the documented `[unreachable]` reading and the stop-then-start migration.
- Reconnect-latency journey: kill the ssh under a live tile, measure
  tear-to-repaint against a wall clock (the "why does reconnect take so
  long" question, made a number).

### Counting

`vm.sh` carries its own `ok()` count and its own literal pin at the bottom,
e2e.sh-style: a journey added means editing the pin by hand, on purpose. It
does not touch e2e's pins.

## Non-goals

- Parallelizing the e2e groups, per-group pins, keep-going mode: real
  candidates, out of scope here.
- Multi-VM topologies (client VM + server VM): the laptop-to-server shape is
  this box to `mux-e2e`; a second VM buys nothing until a journey needs an
  observer from a third box.
- eitri MCP/API calls from `vm.sh`: the harness speaks ssh only. VM
  lifecycle (create/destroy/recreate) stays a documented human/Claude
  action, not harness code.