1baf9412

docs: why the upgrade is an exec and not a handover

a73x 2026-08-26 15:17

Commit message

docs: why the upgrade is an exec and not a handover

The 2026-08-19 design passed fds to a new process. Every hard problem in
it was caused by the process changing rather than the binary: waitpid's
ECHILD arm is `unreachable`, the shim and agent dirs are named after the
pid and held open by every live shell, and Zig 0.15.2 has no SCM_RIGHTS to
pass anything with. execve changes only the binary.

The entry records what deliberately does NOT cross and why each one is
safe to drop, the return-watermark bug (a verdict may cross, the
coordinate it was recorded at may not) and the leg shape it dictates, the
two env variables and why neither is a flag, the CONNECTION_CLOSE number
the QUIC leg measures (~60ms against 19948ms without it), and the
cross-version gap: a handover leg needs an old side that already carries
the feature, so the first release with it is the floor.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

CLAUDE.md

Old		New
@@ -114,6 +114,16 @@ real pty), `wsclient` (browser stand-in), `rawmode`, `delaypipe`, `render`.
114	- Detach is a goodbye, not a tear. A pump that has sent its detach frame	114	- Detach is a goodbye, not a tear. A pump that has sent its detach frame
115	never redials — redial checks `detach_ack` — so the daemon's close after	115	never redials — redial checks `detach_ack` — so the daemon's close after
116	a detach is final, not a network event to recover from.	116	a detach is final, not a network event to recover from.
		117	- An upgrade is an exec, not a restart. `muxd upgrade` execs the candidate
		118	over the RUNNING daemon: same pid, same children, same fds — so `waitpid`,
		119	the pid-named shim and agent dirs, and the socket path all survive untouched,
		120	and `Server.deinit` never runs on that path. The manifest memfd carries only
		121	what cannot be rebuilt (pty fd + child pid, the VT dump and title, the
		122	tracker, the QUIC arm's key bytes, the cumulative counters); scrollback,
		123	clients, delta trackers, agent channels and per-connection QUIC state are
		124	dropped on purpose. A return watermark is RE-STAMPED, never carried — it
		125	belongs to the seq space that minted it. Rollback is another exec, of the
		126	binary the manifest names, with `MUX_UPGRADE_ROLLBACK` as the loop guard.
117	- No socket stealing: `muxd run` refuses a path another daemon owns.	127	- No socket stealing: `muxd run` refuses a path another daemon owns.
118		128
119	## Working rules	129	## Working rules

docs/decisions.md

+precedence and both badge words, and each check was watched failing against the
+mutation that removes its rule. A real-browser pass stays the gate for what only
+a browser has: rendering, input, and the platform APIs the shell talks to.
+## 2026-08-26 — `muxd upgrade`: the pid stays, only the binary changes
+Rolling a new muxd was `stop` + `run`, which kills every session's shell. The
+-08-19 design handed the pty masters and listeners to a NEW process over
+`SCM_RIGHTS`. An inventory of the daemon as it stands found that every hard
+problem in that design is caused by the PROCESS changing, not by the binary
+changing:
+- A shell whose pty crossed is still the old daemon's child. The new daemon's
+  `checkExited` calls `waitpid`, which answers `ECHILD`, which std marks
+  `unreachable` — a panic on the first pump. (Not a guess: mutating the
+  adoption's `Pty.adopt(rec.pty_fd, rec.child_pid)` to pass `0` reproduces it
+  exactly, `posix.zig:4427 .CHILD => unreachable`.) Rebuilding exit detection
+  on master-EOF loses the shell's real exit code forever after.
+- The shell-integration shim dir (`ZDOTDIR`, `--init-file`) and the agent dir
+  (`SSH_AUTH_SOCK`) are named after the daemon's PID and are held open by every
+  live shell. `Server.deinit` `deleteTree`s both, unlinks the socket path and
+  every agent socket, and signals every shell.
+- Zig 0.15.2 has no `recvmsg` wrapper, no `cmsghdr`, no `SCM_RIGHTS`, no
+  `CMSG_*`. All of it would be hand-rolled for this one feature.
+`execve` keeps the pid, the children, the cwd, the environment and every fd not
+marked close-on-exec. It changes only the binary. So the daemon writes what only
+it knows into an anonymous memfd, clears `FD_CLOEXEC` on exactly the fds it
+keeps — unix listener, QUIC UDP socket, each pty master, each agent listener,
+the memfd — and execs the candidate as `muxd run --resume-fd N`. Nothing in
+`Server.deinit` runs on that path: no unlink, no `deleteTree`, no signal. The
+process that owns those names never exits.
+**What deliberately does not cross.** Scrollback: `Engine.dumpState` is
+viewport-only by construction, so a client observes `history_rows` going to 0
+and its absolute row space renumbering. Clients: their fds are close-on-exec
+and the exec is their EOF; they redial as they do for any tear. Delta trackers
+and the `*_sent` latches: rebuilt fresh over the replayed engine, safe only
+because the epoch is re-minted, and honest because no client survived to have
+been told anything. Agent channels and pending awaits: bound to clients that
+are gone. Per-connection QUIC/ngtcp2/TLS state: C state, dropped — which is why
+the exec sends CONNECTION_CLOSE first (below).
+**The watermark bug, and what it teaches.** The manifest carries `last_return`,
+whose `seq` is the RETURN watermark `muxa await --since` compares against. The
+adopted session mints a fresh delta tracker at seq 0, so a carried watermark is
+a watermark from the future that no post-upgrade return can exceed. Measured on
+a live daemon: the FIRST `muxa run` after every upgrade timed out at 30s while
+the shell had already answered, and only the second worked. The verdict crosses;
+its watermark is re-stamped into the new seq space. The lesson is the one
+`awaits-answer-from-snapshots` already paid for, in a new place: a number is
+only meaningful inside the space that minted it, and carrying a verdict is not
+the same as carrying the coordinate it was recorded at. It is also why the e2e
+leg asserts the FIRST await after the exec rather than "an await" — a leg that
+ran two would have gone green on the bug.
+**The refusals are safety, not authority.** `upgrade_req` execs a path of the
+requester's choosing as the daemon's user, which is the same power `stop_req`
+already grants anyone who can open the socket, and the socket is the user's own.
+The version rule, the absolute-and-executable path check, the candidate's own
+`--version`, and the candidate's `run --resume-fd N --check` exist against
+mistakes — wrong path, wrong arch, half-copied binary — not against an
+adversary who is already inside.
+**Rollback is another exec.** If adoption fails before the pump starts, the new
+image execs the OLD binary, whose path the manifest carries, with the same
+`--resume-fd`. The loop guard is `MUX_UPGRADE_ROLLBACK=1` in the env rather than
+a flag: two binaries that both refuse one manifest would exec each other
+forever, sessions alive and unreachable, and an unknown VARIABLE is ignored by
+any muxd where an unknown FLAG is fatal usage to the older one being rolled back
+to. The same reasoning picks `MUX_RESUME_FAIL_AT` as the e2e's handle on the
+abort: `execUpgrade` builds a fixed argv, so nothing a leg types can put
+`--resume-fail-at` in front of the candidate, but the daemon's environment
+crosses the exec untouched.
+A rollback is invisible to the operator: `muxd upgrade` prints its verdict the
+moment the daemon accepts, and if adoption then fails, the image that would
+correct it is gone. The e2e leg reads the daemon's own log for the abort line,
+because without it every other assertion in that leg passes just as well on an
+upgrade that simply worked. A version field in `stats_reply` for `confirmServing`
+to compare is the cheap fix if this ever matters to a human; it is not in v1.
+**The QUIC goodbye is worth a number.** Unix clients need nothing. A QUIC peer
+cannot tell an exec from a bad network, so every connection gets CONNECTION_CLOSE
+before the exec (`quic_server.closeAll`, which drains once — `closeConn` is
+deliberately quiet). Measured with the leg's own stopwatch: ~60ms back in
+service with the goodbye, 19948ms without it against `--quic-idle-ms 15000`.
+That gap is the whole reason `closeAll` exists.
+**The xversion gap.** A cross-version handover leg needs an old side that
+already carries this feature, so the first release with it is the floor. Until
+then the same-binary leg (`--allow-same-version`, a flag whose help text says
+it exists for the e2e) is the v1 gate, and the cross-version leg is a documented
+gap rather than a test that quietly does not run.