971a17e9

docs: M18 decisions, field verification, and what the version string cannot tell you

a73x 2026-08-15 12:22

Commit message

docs: M18 decisions, field verification, and what the version string cannot tell you

The locked decisions and, where execution disagreed with the plan, the
measured truth rather than the prediction. Two of those matter beyond
this milestone.

The cross-version limitation is TWO failures, not one, split by
transport: socket/ssh get a clean exit 1 because the pre-M18
serviceObserver drops the connection, while QUIC HANGS because its
handleFrame ignores the frame in silence — a QUIC connection takes a
client slot at quicOnOpen and never goes through the observer path.
Remote boxes are the QUIC ones. A hello frame cannot fix it: MsgType is
non-exhaustive and both arms end in `else => {}`, so an old daemon
answers an unknown frame with exactly the silence that IS the bug. The
banked mitigation probes a verb old daemons already answer.

Field-verified on both boxes, which had been on 0.0.1-3 with no daemon
running after a power cut — a clean upgrade window. Two QUIC dials each
gave `sessions=3 session 0 / a / b` with isolation clean. The note that
will save someone an hour: `muxd --version` still prints 0.0.1-5 after
the upgrade, because M18 is untagged. The version string cannot tell you
whether a box has multi-session; `muxd stats | grep sessions=` can.

Also recorded: why a bare attach joins the zeroth session and never
creates one. init spawns "0" before any client connects, and the M14
handoff plus every reconnect path re-attaches with an empty name — all of
them must land in the same shell. tmux-new semantics would make a handoff
a new login.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

docs/decisions.md

+  back is the one the up-line named, so every dial targets what was
+  bound. Two suites congruent mod 900, or a tuned ephemeral range, no
+  longer collide.
+## 2026-08-15 (M18 — the multi-session daemon)
+One daemon, N sessions, one socket path / QUIC port / key. The design was
+locked in conversation on 2026-08-13 and executed as eight tasks; what
+follows is the locked set, then the places execution disagreed with it.
+- **A connection IS a session.** The attach payload grew an optional
+  UTF-8 name tail after its fixed 20 bytes, and *nothing after attach
+  changed*: no frame tagging, no per-frame session id, no QUIC stream
+  surgery. Same-host tiles are separate connections to the same port.
+  This is the decision every other one falls out of, and it is why the
+  diff is a name tail plus a `Session` struct rather than a protocol.
+- **Empty name means the default session, and that is the compat
+  story.** `encodeAttachNamed("")` writes the same twenty bytes
+  `encodeAttach` always did, so an M17 client works against an M18
+  daemon and an M18 client with no `--session` works against an older
+  one. `status_req`, `await_req` and `debug_dump` took the same tail for
+  the same reason — one rule across every session-scoped verb.
+- **Known limitation, documented rather than fixed — and it is two
+  different failures, not one.** The plan predicted a single symptom
+  ("silently ignored by `decodeAttach catch return`, the client hangs");
+  the cross-version run measured both arms and they differ by transport,
+  because a pre-M18 daemon has two attach paths:
+  - **Unix socket, ssh and `--via`** reach `serviceObserver`, which
+    DROPS the connection on an unparseable attach. The client says `mux:
+    connection to muxd lost` and exits 1. A clean, bounded failure.
+  - **QUIC** reaches `handleFrame`, whose `decodeAttach catch return`
+    ignores the frame in silence — and there the client really does
+    hang, measured as an 8s timeout with no output. A plain `quic://`
+    attach from the same M18 client to the same old daemon works, so it
+    is the NAME that is fatal, not the transport.
+  This is the worse half landing on the deployment that most needs it
+  right: remote boxes are exactly the QUIC ones, so upgrading the client
+  or the wall ahead of the box turns `--session` into a hang with
+  nothing printed. Still not worth a capability dance for a flag that
+  did not exist when that daemon shipped, but a bounded first-snapshot
+  wait on a NAMED attach — "no answer; that daemon may predate
+  `--session`" — is a cheap mitigation and is banked as a candidate.
+- **Purpose bounds the surface.** The wall showing one host twice is the
+  whole point, so there is no list verb (the wall names its sessions;
+  attach-or-create makes discovery unnecessary), no kill verb (a session
+  ends when its shell exits), no rename, no per-session shell or
+  scrollback config. `max_sessions = 4`, wall-sized, in the spirit of
+  `max_clients = 8`.
+- **A bare attach joins the zeroth session and never creates one —
+  seamless handoff is the reason.** The default session is spawned in
+  `init` (`createSession(..., proto.default_session, ...)`), so `"0"`
+  exists before any client connects and an empty name always *finds* it.
+  The alternative — tmux's `new-session` semantics, where an unnamed
+  invocation makes a fresh one — would break the product's central
+  trick: the M14 handoff attaches over ssh, re-dials, and re-attaches
+  over QUIC, and every reconnect and resume path in `client.zig`
+  re-attaches the same way. All of those carry an empty name, so all of
+  them must land in the SAME shell. Under create-on-bare-attach a
+  handoff would be a new login and a network blip would spawn a shell.
+  The tmux analogy holds once mapped correctly: mux has no separate
+  "new" verb — `mux` IS the attach verb — so its bare form is `tmux
+  attach`, not `tmux new`. Naming a session that does not exist is the
+  explicit request, and that is the arm that creates.
+- **An attach is answered, always — there is no wait state.** Both arms
+  refuse the same way `resolveSession` returns null: `exit_status 1`,
+  queued on the client path and written-then-dropped on the observer
+  path. Resolution is synchronous — map `""`→`"0"`, validate, find, else
+  create, else no. This is worth stating because the pre-M18 hang
+  documented above looks like a wait and is not one: that daemon's
+  `decodeAttach catch return` fails to PARSE a named attach and returns
+  before any resolve-or-refuse logic runs, so it never decides anything.
+  Silence is not a state this protocol has; it is what a peer that
+  predates the rule does.
+- **Creation requires a real size.** Attach-or-create only creates when
+  the attach carries nonzero cols×rows. muxa attaches at 0×0 — it makes
+  no size claim, deliberately — so `muxa send` to a dead session is
+  refused with `exit_status 1` instead of silently spawning a 0×0 shell.
+- **A session name means one session, whichever transport carried the
+  attach.** The session table belongs to the daemon, not to the listener
+  that accepted the connection, so `--session a` over QUIC and
+  `--session a` over the unix socket join the *same* shell — and two
+  QUIC dials to one port come out as two shells. This is the operator's
+  half of decision 6 and it is now pinned end to end: the QUIC block
+  drives two dials into two sessions, then joins one of them from the
+  socket and asserts both clients' markers on one grid with the session
+  count unmoved. QUIC is where the claim is least obvious, since a QUIC
+  client is promoted to a slot when its *handshake* completes — before
+  any attach — so its slot starts session-less and the name is the only
+  thing that ever binds it.
+- **Command tracking is per-session**, forced by the agent-surface
+  merge. A tracker fed by two shells would interleave their command
+  lifecycles into nonsense, so `cmd` and `last_return` live on the
+  session; `AwaitState` stays on the client slot, because an await is
+  one client's question.
+**Amended in execution — reality disagreeing with the plan.**
+- **A `/bin/sh` session has no marks, so the planned `muxa status` phase
+  assertion could not hold.** The plan's e2e sketch asserted that
+  `status --session b` names a running command while `--session a`
+  reports `at_prompt`. `shellint` detects shells by basename and unknown
+  shells get nothing, so `cmd.phase` in a `/bin/sh` session never leaves
+  `at_prompt` and the assertion would have been asserting a constant
+  against itself. Switching the block to bash was rejected for the
+  reason `e2e.sh` already pins `SHELL=/bin/sh` at the top: `bash_init`
+  sources `$HOME/.bashrc`, which is arbitrary code on the session under
+  test. What the e2e asserts instead is what it can honestly observe —
+  that `status --session` returns a real StatusReply for a named session
+  and that `capture --session` returns one session's grid and not its
+  neighbour's. The per-session tracker itself stays pinned where marks
+  actually exist: `server.zig`'s unit tests, and `agent.sh`'s bash
+  daemons.
+- **`muxa` is now an e2e.sh argument (`$9`).** The plan assumed the
+  suite could reach it; it could not — muxa went only to `agent.sh`. It
+  is asked for here rather than there because the verbs under test
+  address a session *without attaching*, and the sessions they address
+  can only be CREATED by `mux`, which `agent.sh` has no copy of. The
+  soak step takes the same argument, since soak *is* the e2e suite run N
+  times — an argument added to one and not the other makes every soak
+  run abort on an unbound variable before its first scenario, which is
+  exactly how it was caught.
+- **The scenario detaches before asserting.** The plan's sketch held
+  both clients open across the assertions; the block as written has each
+  client plant its marker and leave. A session ends when its shell
+  exits, not when its last client leaves, so every routing assertion now
+  runs against sessions with nobody attached — which is also what makes
+  the answers impossible to explain by a live client holding something
+  open.
+**The one real defect, and the e2e found it rather than review** (see
+`fix(pty)`, f25f62c). `forkpty` returns the master without CLOEXEC, so
+every session spawned later inherited every earlier session's master —
+observed on a three-session daemon as fds 0,1,2,255 in the first shell,
+those plus fd 3 in the second, plus fd 3 AND fd 6 in the third. A master
+with a second holder never sees its last close, so `Pty.deinit`'s close
+stopped hanging up the far side; the shell is interactive and therefore
+ignores the SIGTERM that follows; and the blocking `waitpid` after that
+never returned. `muxd stop` unlinked the socket and printed `muxd:
+stopped` while the daemon sat in `do_wait` forever with every session's
+shell alive — and because deinit walks sessions in slot order, wedging on
+the first meant none of the others were torn down either.
+Two things worth keeping from it. **A fd leak is a shutdown bug, not a
+tidiness bug** — the daemon's own sockets were already CLOEXEC, and the
+pty master was the single oversight in an otherwise consistent
+convention. And **the bug was strictly unreachable at N=1**: nothing in
+the unit layer, the review rounds, or fifteen prior milestones could have
+surfaced it, because none of them ever spawned a second pty from a
+process that already held one. The first test that ran two sessions on
+one daemon found it on its first execution.

docs/roadmap.md

+And `muxa capture` is **the one verb with no automated coverage at all** —
+`test/agent.sh` drives run, await, send and status; capture is where
+`--diff-since` would land, so the test and the flag are owed together.
+*(Closed in M18: the multi-session e2e block drives `capture --session`
+and asserts it returns one session's grid and not its neighbour's.
+`--diff-since` is still owed.)*
+## M18 — multi-session daemon — complete
+Shipped 2026-08-15 (e7285d9..2b1c9bb + close-out). One daemon now holds
+up to **four** independent sessions — engine, pty, tracker, epoch,
+command tracker and name each — over **one socket path, one QUIC port,
+one key**. The wall can finally show the same host twice, which is what
+the milestone was for.
+**A connection is a session.** The attach payload grew an optional UTF-8
+name tail after its fixed 20 bytes and nothing after attach changed: no
+frame tagging, no QUIC stream surgery, the transport still one bidi
+stream per connection. Same-host tiles are separate connections to the
+same port. Attach-or-create is tmux-style; a session ends when its shell
+exits and the daemon exits with the last one. `status_req`, `await_req`
+and `debug_dump` grew the same tail, which is what lets `muxa` and `muxd
+dump` address one session of several without attaching to it.
+**Field-verified 2026-08-15 on both real boxes.** They had been on
+**0.0.1-3** — behind the agent surface as well as M18 — and a site power
+cut had taken them down, which made the upgrade window a clean one: no
+daemon running on either, so nothing to tear down and no session to lose.
+Static musl `muxd` (stripped, 23MB → 8.6MB) installed to
+`/usr/local/bin`, identical sha256 on both boxes and on the build host,
+then `muxd start --quic` on each. On the LAN box over real QUIC, and on
+the WAN box over QUIC through the LAN-side UDP forward (the gate carries
+TCP only): two dials, `--session a` and `--session b`, produced
+`sessions=3 session 0 / session a / session b` on **both** boxes, with
+each session holding its own marker and neither holding its neighbour's,
+and the default session holding neither. The milestone's central claim,
+on real hardware over a real link.
+Two deployment notes worth keeping. `muxd --version` still prints
+**0.0.1-5** after the upgrade — M18 is untagged, so the version string
+cannot tell you whether a box has it; `muxd stats | grep sessions=` is
+the check that can, which is the same discriminator the banked
+capability probe uses. And the trial PSK survived the power cycle
+identical on both boxes (`ba046148…`), so the M17 key-drift failure did
+not repeat — fingerprint before blaming the network stays the rule.
+**One name, one session, either door.** The session table is the
+daemon's, not the accepting listener's, so `--session a` reaches the same
+shell over QUIC as over the unix socket, and two QUIC dials to one port
+are two shells. Pinned end to end by the QUIC block: two dials into two
+sessions, then a socket client joins one of them and both clients'
+markers land on one grid with the session count unmoved. That also makes
+a mixed-transport wall of a single host work — `muxweb HOST#a
+quic://HOST:8787#b` is tile 0 over ssh and tile 1 over QUIC against one
+daemon. (`muxd run --sock PATH --quic ADDR` binds both; ssh needs no
+listener of its own, since `muxd proxy` pumps into the same socket.)
+**The compat story held**, measured against main @ 5d41e10 (the branch
+point) in both directions. An empty name is the default session and
+encodes to byte-identical pre-M18 payloads: an M18 client drove a
+pre-M18 daemon's default session; a pre-M18 client drove an M18
+daemon's and created nothing (`sessions=1`, `session 0`); and a pre-M18
+`muxa`'s empty `status_req` still answered. The known limitation is
+documented rather than fixed, and turns out to be **two** failures
+split by transport — socket/ssh get a clean exit 1 (the old
+`serviceObserver` drops the connection), **QUIC hangs** (the old
+`handleFrame` ignores the frame in silence). Remote boxes are the QUIC
+ones, so a client upgraded ahead of its box hangs with nothing printed;
+a bounded first-snapshot wait on a named attach is banked as the cheap
+mitigation. See decisions.md.
+**One real defect, found by the new e2e and not by review** (f25f62c).
+`forkpty` returns the master without CLOEXEC, so every session spawned
+later inherited every earlier session's master — one spare `/dev/ptmx`
+per predecessor, riding through exec into a shell that had no idea it
+held it. A master with a second holder never sees its last close, so
+`Pty.deinit`'s close stopped hanging up, the interactive shell ignored
+the SIGTERM that followed, and the blocking `waitpid` never returned:
+`muxd stop` unlinked the socket, printed `muxd: stopped`, and left the
+daemon in `do_wait` forever with every session's shell alive. Strictly
+unreachable at one session per daemon. The pin closes a master while a
+later session is live and polls for the hangup with a deadline — the bug
+is a hang, and a hung `zig build test` step prints nothing at all.
+**The cross-version check stopped being hand-rolled** (`test/xversion.sh`,
+`make xversion`). This tree's client against a previous version's daemon
+and the reverse, each daemon in a container of its own, both sides built
+static musl so one artifact runs in the container and on the host. Eight
+legs: both directions on the default session, both halves of the
+`--session` limitation, an old `muxa`'s empty `status_req`, the
+`stats_req` version discriminator the banked mitigation would need, and a
+teardown observed rather than claimed. It exists because M18's by-hand
+run found a *documentation* error days after the sentence was written; a
+gate finds that on the next run.
+Two things about it stated rather than left implied. **Containers share
+the host kernel**, so nothing there validates pty, poll or fd semantics
+against a different one — most of what muxd is. That needs VMs and is not
+claimed. And the rig is **falsifiable on purpose**: point `XVER_OLD_BIN`
+at the new binaries and legs must fail. Doing exactly that caught a false
+pass in the gate's own first draft — the QUIC arm keyed on the client's
+exit code, and a daemon that HONOURS the name also leaves the client
+attached until `timeout` kills it, so 124 meant both things. It now asks
+the daemon whether a session called `zz` appeared, which only a daemon
+that understood the name can produce.
+`muxa capture` is **no longer the one verb with no automated coverage**:
+the M18 block drives it by name and asserts it returns one session's
+grid and not its neighbour's. `--diff-since` is still owed.
+## M16 candidates — and still outranked by trial feedback
+plus the seven items M15 created. Nothing here is a field finding; the
+next one that arrives outranks all of it.
+- **Probe for named-session support before sending a named attach** —
+  M18's measured cross-version gap. `--session NAME` against a pre-M18
+  daemon is dropped cleanly on the socket path but ignored in **silence
+  over QUIC**, where the client hangs with nothing printed. The two
+  differ because a pre-M18 QUIC connection takes a client slot at
+  `quicOnOpen`, so its attach reaches `handleFrame`'s silent
+  `decodeAttach catch return`, while a socket connection is an observer
+  until it attaches and `serviceObserver` closes on a bad one.
+  A `hello`/version frame cannot fix this: `MsgType` is non-exhaustive
+  and both switch arms end in `else => {}`, so an old daemon ignores an
+  unknown frame in exactly the silence that is the bug. What works is a
+  probe using a verb old daemons ALREADY answer — `stats_req`, which
+  pre-M18 `handleFrame` serves without an attach, on QUIC included — and
+  reading the reply: M18's stats text carries `sessions=`, pre-M18's
+  does not. Zero protocol change, effective against shipped binaries,
+  and it yields a true message rather than a timeout's guess.
+  Gate it on the capability and refuse only the ATTACH, never the
+  connection: an unnamed attach is byte-identical to pre-M18 and works
+  today, so a version-equality check would break configurations that are
+  currently fine. Ranks above the rest of this list for anyone running a
+  wall against boxes that upgrade on their own schedule, which is every
+  wall. A `hello` frame is still worth adding for its own sake — free,
+  safe, and it makes the NEXT version pair a negotiation instead of
+  another archaeology exercise.
+- **A refused attach should say which refusal it was** — M18 review
+  finding. `resolveSession` returning null is answered with
+  `exit_status 1` for four different causes: a name that failed
+  validation, a full session table, an unknown name with no size to
+  create at, and a spawn that failed. That byte is also what a shell
+  exiting 1 sends, so five distinct outcomes arrive identical. The
+  full-table case was the model — "the same honest no a full client
+  table gives" — and it is honest there, where there is genuinely one
+  answer; it stopped being honest once four causes started sharing it.
+  The evidence that this costs something is that **the tests cannot read
+  it either**. `test/e2e.sh` has to disambiguate through a side channel
+  (`muxd stats | grep sessions=`), and `test/xversion.sh` had to invent
+  `assert_name_not_honoured` to ask "did a session appear?" because the
+  reply itself could not say. When a protocol's own suite has to go
+  around the answer to find out what happened, an operator staring at a
+  bare exit code has strictly less to work with.
+  Not worth a protocol change on its own — bank it with the probe above,
+  which is the other half of the same story and would land in the same
+  place. The cheap client-side half needs no wire change at all: on a
+  **named** attach that never produced a snapshot, `mux` can print "the
+  daemon refused the attach or the shell exited 1" instead of exiting
+  silently. That one line covers the case an operator actually hits,
+  which is a typo in `--session`.
+- ~~**`drain()` swallows `ECONNREFUSED`**~~ — paid by M15 (Task 4). Both
+  socket paths now route the refusal through one verdict, and an
+  unreachable port fails in **1ms** instead of a full deadline
@@ @@ -380,7 +537,14 @@ next one that arrives outranks all of it. @@
+. **`addCSourceFiles` for the QUIC deps**, replacing the build script.
+. **Autotools wolfSSL + ticket resumption** — the only route to 0-RTT.
+. **Certificates / TOFU · QUIC datagrams for input · multi-session per
+   *connection* · NAT traversal** — parked as a group. Multi-session
+   **shipped in M18, but per *daemon***: one socket path / port / key
+   holding N sessions, each its own connection. What stays banked here
+   is the narrower thing this line originally meant — several sessions
+   multiplexed inside ONE connection, which needs QUIC stream surgery
+   the M18 design deliberately refused (a connection IS a session).
+   Worth doing only if the connection count per host ever becomes the
+   cost, which on a four-tile wall it is not.
+. **Scrollback retention tuning** — policy decided (engine-native 10k
+   ring); making the limit configurable is what remains, and nobody has
+   hit the ceiling.