038bf188
docs: the e2e dedupe and vm-gate design
a73x 2026-09-01 18:31
Commit message
docs/superpowers/specs/2026-09-01-e2e-dedupe-and-vm-gate-design.md
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,134 @@ | |||
| 1 | # e2e harness: dedupe, and a real-VM gate | ||
| 2 | |||
| 3 | 2026-09-01. Two improvements to the test harness, independent enough to land | ||
| 4 | separately, ordered so the cheap one goes first. | ||
| 5 | |||
| 6 | ## Why now | ||
| 7 | |||
| 8 | The retro record names the costs this design answers: | ||
| 9 | |||
| 10 | - Two askpass legs went green about a prompt that was never raised, because | ||
| 11 | the ssh *shim* is an assumption wearing a test's clothes (2026-08-29, | ||
| 12 | twice). "Assert behavior, don't assume it. Mocks are assumptions" — the | ||
| 13 | shim is the one standing mock in the suite. | ||
| 14 | - Four group files each hand-build the same shim rig; ~123 sites spell the | ||
| 15 | same four-line rc-capture idiom; ~57 ptyclient scripts end in the same | ||
| 16 | detach tail. Every copy is a place a fix has to be found. | ||
| 17 | - `wan.sh` proves the real-remote harness pattern works but is hand-run | ||
| 18 | against pet boxes, outside every gate. | ||
| 19 | |||
| 20 | ## Part 1 — dedupe (no behavior change) | ||
| 21 | |||
| 22 | Three lib helpers replace three copied idioms. The scenario pins (105/38 in | ||
| 23 | `test/e2e.sh`) do not move; that they do not move is the delivery check that | ||
| 24 | this part changed nothing. | ||
| 25 | |||
| 26 | **1. `ssh_shim_rig DIR`** in `e2e_lib.sh`: creates DIR, the three logs | ||
| 27 | (`DIR/pids`, `DIR/comms`, `DIR/args`), writes the option-loop head (today's | ||
| 28 | `ssh_shim_head`) plus the pid/comm/arg logging lines every rig repeats, | ||
| 29 | registers `defer_rm`, and leaves the file open for the caller to append its | ||
| 30 | bespoke behavior before `chmod +x`. `04_handoff`, `09_hosts`, `15_askpass` | ||
| 31 | and `16_push` convert; their per-group behaviors (HRUN homes, fakebin/uname | ||
| 32 | fronting) stay in the group files as appended bodies. | ||
| 33 | |||
| 34 | **2. `must "message" cmd args...`**: runs the command with the rc captured | ||
| 35 | inside the helper, and on nonzero prints the message and the rc and exits 1. | ||
| 36 | Converts only the sites whose failure block is a simple message — the | ||
| 37 | multi-line diagnostic blocks keep their current shape, because the per-leg | ||
| 38 | failure prose is load-bearing and `must` must never genericize it. Expect | ||
| 39 | roughly half to two-thirds of the ~123 sites to convert; the true count is | ||
| 40 | measured during implementation, not promised here. | ||
| 41 | |||
| 42 | **3. A shared detach tail** for ptyclient scripts: one lib variable holding | ||
| 43 | `send \x1cd` + `waitexit`, interpolated into the ~57 heredocs that end a | ||
| 44 | scripted session the same way. | ||
| 45 | |||
| 46 | Conversion gate: full `make e2e` after each group file converts (the suite is | ||
| 47 | linear and stateful; `E2E_ONLY` cannot vouch for a group's neighbors), and | ||
| 48 | `make ci` before delivery. | ||
| 49 | |||
| 50 | ## Part 2 — `make vm`: user journeys against a real box | ||
| 51 | |||
| 52 | A new `test/vm.sh` and Makefile target. It is a delivery-ritual gate beside | ||
| 53 | `make ci`, never inside it: `make ci` stays hermetic and offline. The e2e ssh | ||
| 54 | shims stay — they are the fast local pins; `make vm` is where their | ||
| 55 | assumptions get audited against a real sshd, a real network, a real Ubuntu. | ||
| 56 | |||
| 57 | ### Fixture | ||
| 58 | |||
| 59 | One long-lived VM, like a real user's server: | ||
| 60 | |||
| 61 | - **`mux-e2e`** on the eitri fleet, pinned to host **charizard** (this | ||
| 62 | machine), x86_64, 2 vCPU / 2 GB / 10 GB, underlay address `10.78.5.3`. | ||
| 63 | - Reachable by plain non-interactive ssh as `ubuntu@10.78.5.3` — proven | ||
| 64 | 2026-09-01: cert auth, no gate hop, `mux d endpoint` on the mux-less box | ||
| 65 | answers rc 127. | ||
| 66 | - Auth is pinned by a `~/.ssh/config` Host block for `10.78.5.3` naming the | ||
| 67 | harness identity (`~/.config/mux-vm/vm_key` + its 5-year cert, CA | ||
| 68 | registered with the tenant as `mux-vm-gate`), so the gate never depends on | ||
| 69 | the freshness of anyone's interactive cert. | ||
| 70 | - The hosts-file spelling the journeys use is `ubuntu@10.78.5.3` — | ||
| 71 | `[user@]host` is in `hosts.zig`'s grammar. | ||
| 72 | |||
| 73 | Hermeticity comes from a **scorched-earth reset inside the VM** at the top of | ||
| 74 | every run — kill every mux process, remove the installed binary and all four | ||
| 75 | XDG dirs — not from re-creating the VM. The client side runs on this box | ||
| 76 | under `ptyclient` with every XDG dir isolated to scratch, per the standing | ||
| 77 | rig rule. | ||
| 78 | |||
| 79 | `vm.sh` preflights `ssh ubuntu@10.78.5.3 true` and REFUSES with the recovery | ||
| 80 | recipe when it fails (VM gone: recreate on charizard and re-read the underlay | ||
| 81 | IP; cert stale: re-sign against `~/.config/mux-vm/vm_ca`). A missing VM is a | ||
| 82 | loud failure, never a skip — the same stance e2e.sh takes on nvim and curl: | ||
| 83 | a gate that can quietly not run is the one failure a green tree cannot show. | ||
| 84 | |||
| 85 | ### Journeys, v1 | ||
| 86 | |||
| 87 | Each claim about the VM is asserted by a separate ssh reading `/proc` or the | ||
| 88 | filesystem there — never by the daemon's own report. The build under test is | ||
| 89 | the static musl `mux` (same target `make release` ships). | ||
| 90 | |||
| 91 | 1. **Install like a user.** scp the binary to `~/.local/bin/mux` on the | ||
| 92 | mux-less box; `mux d endpoint` over ssh now answers instead of 127. | ||
| 93 | 2. **Entry dial.** `ptyclient` runs `mux ubuntu@10.78.5.3`: real ssh, real | ||
| 94 | `mux d endpoint --start` auto-start, session paints. Type a marker, | ||
| 95 | assert it via ssh against the VM's own pty (grid dump run remotely), | ||
| 96 | detach, reattach, marker still present. Asserts the daemon's pid file / | ||
| 97 | `/proc` state remotely: exactly one daemon, exec'd from the installed | ||
| 98 | image. | ||
| 99 | 3. **No autostart.** `mux d stop` on the VM; hold a wall polling the host | ||
| 100 | for >3 poll cycles; assert via ssh that NO mux process exists on the VM | ||
| 101 | after — the poll must not have resurrected it (the live-box bug, | ||
| 102 | 2026-08-30 era, now pinned against a real remote). | ||
| 103 | 4. **Remote upgrade.** `mux d upgrade ubuntu@10.78.5.3` pushing the same | ||
| 104 | image (the shape `e2e_16_push` already pins locally): assert the daemon | ||
| 105 | pid did | ||
| 106 | NOT change across the exec (`/proc/PID/exe` re-read through ssh), the | ||
| 107 | session and its OSC 133 marks mechanism survive, and `mux a` still | ||
| 108 | answers with `mechanism: marks`. | ||
| 109 | |||
| 110 | ### Stretch (recorded, not v1) | ||
| 111 | |||
| 112 | - QUIC journey over the underlay address. | ||
| 113 | - Version-skew journey: install the released v15 tarball on the VM, assert | ||
| 114 | the documented `[unreachable]` reading and the stop-then-start migration. | ||
| 115 | - Reconnect-latency journey: kill the ssh under a live tile, measure | ||
| 116 | tear-to-repaint against a wall clock (the "why does reconnect take so | ||
| 117 | long" question, made a number). | ||
| 118 | |||
| 119 | ### Counting | ||
| 120 | |||
| 121 | `vm.sh` carries its own `ok()` count and its own literal pin at the bottom, | ||
| 122 | e2e.sh-style: a journey added means editing the pin by hand, on purpose. It | ||
| 123 | does not touch e2e's pins. | ||
| 124 | |||
| 125 | ## Non-goals | ||
| 126 | |||
| 127 | - Parallelizing the e2e groups, per-group pins, keep-going mode: real | ||
| 128 | candidates, out of scope here. | ||
| 129 | - Multi-VM topologies (client VM + server VM): the laptop-to-server shape is | ||
| 130 | this box to `mux-e2e`; a second VM buys nothing until a journey needs an | ||
| 131 | observer from a third box. | ||
| 132 | - eitri MCP/API calls from `vm.sh`: the harness speaks ssh only. VM | ||
| 133 | lifecycle (create/destroy/recreate) stays a documented human/Claude | ||
| 134 | action, not harness code. | ||