a73x

4d48bfad

feat: mux d stop, dump and stats take --quic for a daemon whose socket is gone

a73x   2026-09-05 19:04

Commit message
feat: mux d stop, dump and stats take --quic for a daemon whose socket is gone

The other half of 145807a2. A re-bind takes a deleted path back only
when nothing else holds it; when a successor already has it the first
daemon is still holding sessions that nothing could stop, dump or
count, because every admin verb took `--sock` alone. The daemon has
served these three verbs on a QUIC client slot since `mux a --quic`
existed; this is the client learning the spelling.

`--quic HOST[:PORT] [--key FILE]` on stop, dump and stats, with the key
resolved by the same three-way rule mux a and `mux d start --quic` use.
`--sock` with `--quic` on those verbs is a parse refusal (two daemons
named for one question); `start` still takes both. A QUIC ask is
bounded at 5 s, because UDP has no "nothing listening" errno. Stop's
verdict over QUIC is the daemon's own CONNECTION_CLOSE, and a port that
never answered is rc 1 rather than the socket's idempotent 0 — silence
may be a firewall or the wrong key. `upgrade` stays on the socket: its
request is served to observers only and its manifest is local.

Pinned by parse tests and an e2e leg on the session group's real
--quic daemon: socket file deleted, all three verbs over UDP, the stop
graded by the pid going away.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
README.md
Old New
@@ -740,6 +740,8 @@ has none.
740 mux d start [-d] # the daemon: in this shell, or -d to fork one and wait for it 740 mux d start [-d] # the daemon: in this shell, or -d to fork one and wait for it
741 mux d dump [--vt] # debug: print the authoritative grid (what the screen *should* be) 741 mux d dump [--vt] # debug: print the authoritative grid (what the screen *should* be)
742 mux d stats # live sessions by name + clients each; wire stats (deltas vs snapshot bytes) 742 mux d stats # live sessions by name + clients each; wire stats (deltas vs snapshot bytes)
743 mux d stop # ask the daemon to exit; dump, stats and stop also take
744 # --quic HOST[:PORT] [--key FILE], for a daemon whose socket file is gone
743 mux d upgrade # exec a newly installed binary in place; every session keeps running 745 mux d upgrade # exec a newly installed binary in place; every session keeps running
744 mux d upgrade HOST # push THIS binary onto HOST over ssh, then upgrade its daemon the same way 746 mux d upgrade HOST # push THIS binary onto HOST over ssh, then upgrade its daemon the same way
745 make bench # typing-workload bytes-per-keystroke measurement 747 make bench # typing-workload bytes-per-keystroke measurement
build.zig
Old New
@@ -322,7 +322,7 @@ const mod_table = [_]ModSpec{
322 // fourth copy of connect-write-poll-read here. `testtmp` is the keygen 322 // fourth copy of connect-write-poll-read here. `testtmp` is the keygen
323 // round-trip's: it needs a directory to generate into, which the daemon 323 // round-trip's: it needs a directory to generate into, which the daemon
324 // never touches. 324 // never touches.
325 .{ .name = "mux", .path = "src/cli/mux.zig", .link_libc = true, .imports = &.{ "daemon", "client", "wall", "agent", "webhub", "term", "proxy", "quic", "xdg", "spawn", "sockpath", "cliflags", "dial", "server_os", "client_os" }, .test_imports = &.{"testtmp"}, .quic_tests = true }, 325 .{ .name = "mux", .path = "src/cli/mux.zig", .link_libc = true, .imports = &.{ "daemon", "client", "wall", "agent", "webhub", "term", "proxy", "quic", "xdg", "spawn", "sockpath", "cliflags", "dial", "link", "server_os", "client_os" }, .test_imports = &.{"testtmp"}, .quic_tests = true },
326 // Opt-in rows: no default artifact or test names them, so machines 326 // Opt-in rows: no default artifact or test names them, so machines
327 // without the viewer's system libraries keep all existing gates. 327 // without the viewer's system libraries keep all existing gates.
328 .{ .name = "native_core", .path = "src/gui/native_core.zig", .link_libc = true, .imports = &.{ "client", "term" }, .opt_in = true }, 328 .{ .name = "native_core", .path = "src/gui/native_core.zig", .link_libc = true, .imports = &.{ "client", "term" }, .opt_in = true },
docs/superpowers/specs/2026-09-05-daemon-socket-lifecycle-design.md
Old New
@@ -58,7 +58,7 @@ would have dated the second daemon's birth against the first one's loss.
58 ### 2. Watching the path and taking it back (145807a2, option a) 58 ### 2. Watching the path and taking it back (145807a2, option a)
59 59
60 `Server.pumpOnce` gains a once-a-second stat of `sock_path` against 60 `Server.pumpOnce` gains a once-a-second stat of `sock_path` against
61 `bound.path_id` (`sock_watch_ms`, same shape as the agent relay's timers). 61 `bound.path_id` (`Server.sock_watch_interval_ms`, same shape as the agent relay's timers).
62 Cost: one `fstatat` per second per daemon. 62 Cost: one `fstatat` per second per daemon.
63 63
64 On loss: 64 On loss:
@@ -82,7 +82,7 @@ a re-bound fd is just the current `bound.fd`, so nothing there changes.
82 `upgrade_req` is refused while a re-bind is pending? No pending state 82 `upgrade_req` is refused while a re-bind is pending? No pending state
83 exists: the re-bind is synchronous inside one tick. 83 exists: the re-bind is synchronous inside one tick.
84 84
85 ### 3. Admin verbs by QUIC (145807a2, option b) — pending the user's call 85 ### 3. Admin verbs by QUIC (145807a2, option b) — taken, 2026-09-05
86 86
87 `mux d stop|dump|stats --quic HOST:PORT [--key PATH]` for a daemon whose 87 `mux d stop|dump|stats --quic HOST:PORT [--key PATH]` for a daemon whose
88 path is gone AND whose slot the successor now holds. Client-side only: 88 path is gone AND whose slot the successor now holds. Client-side only:
@@ -109,6 +109,9 @@ Not in scope: an admin socket of its own, a signal handler.
109 daemon, `sleep 2`, `mux d stats --sock PATH` answers, and the log carries 109 daemon, `sleep 2`, `mux d stats --sock PATH` answers, and the log carries
110 the loss line and the re-bound line in that order; `mux d stop` then 110 the loss line and the re-bound line in that order; `mux d stop` then
111 unlinks and logs `unlinked`. 111 unlinks and logs `unlinked`.
112 - Log-line unit test: the trail is asserted through `std.testing` capture 112 - Log lines: as built, `logSocket` writes stderr like the rest of the
113 of a writer, so the daemon's `socketLog` takes a writer like every other 113 daemon's diagnostics (fd 2, never fd 1, so the fd-1 wedge rule is not in
114 stdout-adjacent path in this repo (the fd-1 wedge rule). 114 play) and the trail is graded off the real log by the e2e leg in the boot
115 group; the unit test grades the watch's STATE (`sock_watch.lost`,
116 `sock_watch.refused`, the bound inode), not the wording. The wall's
117 auto-start note has a unit test of its own on `xdg.appendLogLineTo`.
src/cli/main.zig
Old New
@@ -20,15 +20,16 @@ const sockpath = @import("sockpath");
20 const upgrade = @import("daemon").upgrade; 20 const upgrade = @import("daemon").upgrade;
21 const cliflags = @import("cliflags"); 21 const cliflags = @import("cliflags");
22 const dial = @import("dial"); 22 const dial = @import("dial");
23 const link_mod = @import("link");
23 24
24 const usage = 25 const usage =
25 \\usage: 26 \\usage:
26 \\ mux d start [-d] [--sock PATH] [--shell PATH] [--cols N] [--rows N] 27 \\ mux d start [-d] [--sock PATH] [--shell PATH] [--cols N] [--rows N]
27 \\ [--quic HOST[:PORT] --key FILE] [--quic-idle-ms N] 28 \\ [--quic HOST[:PORT] --key FILE] [--quic-idle-ms N]
28 \\ (-d comes first; forks it off and waits, no-op if up) 29 \\ (-d comes first; forks it off and waits, no-op if up)
29 \\ mux d dump [--vt] [--session NAME] [--sock PATH] 30 \\ mux d dump [--vt] [--session NAME] [--sock PATH | --quic HOST[:PORT] [--key FILE]]
30 \\ mux d stats [--sock PATH] 31 \\ mux d stats [--sock PATH | --quic HOST[:PORT] [--key FILE]]
31 \\ mux d stop [--sock PATH] (ask the daemon on PATH to exit) 32 \\ mux d stop [--sock PATH | --quic HOST[:PORT] [--key FILE]] (ask the daemon to exit)
32 \\ mux d proxy [--sock PATH] (byte pump: stdio <-> session socket) 33 \\ mux d proxy [--sock PATH] (byte pump: stdio <-> session socket)
33 \\ mux d endpoint [--sock PATH] [--start] (announce QUIC port+key, then proxy) 34 \\ mux d endpoint [--sock PATH] [--start] (announce QUIC port+key, then proxy)
34 \\ mux d keygen (write a fresh key to ~/.config/mux/key) 35 \\ mux d keygen (write a fresh key to ~/.config/mux/key)
@@ -182,10 +183,22 @@ const UsageResponse = union(enum) {
182 /// locally while HOST is honored remotely would leave the user 183 /// locally while HOST is honored remotely would leave the user
183 /// believing both. 184 /// believing both.
184 sock_with_host, 185 sock_with_host,
186 /// `stop|dump|stats --sock PATH --quic HOST:PORT`: two daemons named
187 /// for one question. `start` takes both, because a daemon can listen
188 /// on both; the admin verbs ask one thing of one daemon.
189 sock_with_quic,
185 }; 190 };
186 191
187 const DaemonInvocation = union(enum) { command: DaemonArguments, usage: UsageResponse }; 192 const DaemonInvocation = union(enum) { command: DaemonArguments, usage: UsageResponse };
188 193
194 /// The verbs that may reach a daemon over QUIC instead of its socket path:
195 /// one frame in, one frame out, which `handleDaemonVerb` already serves on a
196 /// QUIC client slot. `upgrade` stays on the socket — its request is served
197 /// to observers only, and the manifest it carries is local to the box.
198 fn quicAdmin(cmd: DaemonCommand) bool {
199 return cmd == .stop or cmd == .dump or cmd == .stats;
200 }
201
189 fn parseArgs(args: []const [:0]const u8) DaemonInvocation { 202 fn parseArgs(args: []const [:0]const u8) DaemonInvocation {
190 if (args.len < 2) return .{ .usage = .no_command }; 203 if (args.len < 2) return .{ .usage = .no_command };
191 const spec = specForName(args[1]) orelse return .{ .usage = .{ .unknown_command = args[1] } }; 204 const spec = specForName(args[1]) orelse return .{ .usage = .{ .unknown_command = args[1] } };
@@ -230,6 +243,12 @@ fn parseArgs(args: []const [:0]const u8) DaemonInvocation {
230 if (o.key != null and o.quic == null) return .{ .usage = .key_without_quic }; 243 if (o.key != null and o.quic == null) return .{ .usage = .key_without_quic };
231 244
232 if (o._host != null and o.sock != null) return .{ .usage = .sock_with_host }; 245 if (o._host != null and o.sock != null) return .{ .usage = .sock_with_host };
246 if (quicAdmin(spec.cmd) and o.quic != null and o.sock != null) return .{ .usage = .sock_with_quic };
247 // `--quic` is a bind for `start` and a door for the admin verbs. Every
248 // other verb is served on the socket alone, and a flag it parsed and
249 // dropped would run `mux d upgrade --quic HOST` against the socket's
250 // daemon while the user believed HOST was being upgraded.
251 if (o.quic != null and spec.cmd != .start and !quicAdmin(spec.cmd)) return .{ .usage = .{ .unknown_arg = "--quic" } };
233 252
234 return .{ .command = o }; 253 return .{ .command = o };
235 } 254 }
@@ -255,6 +274,10 @@ fn usageExit(u: UsageResponse) u8 {
255 "mux d: upgrade HOST uses that box's own default socket; --sock only names a local one\n", 274 "mux d: upgrade HOST uses that box's own default socket; --sock only names a local one\n",
256 .{}, 275 .{},
257 ), 276 ),
277 .sock_with_quic => std.debug.print(
278 "mux d: --sock and --quic name two daemons; ask one of them\n",
279 .{},
280 ),
258 } 281 }
259 return usageCode(u); 282 return usageCode(u);
260 } 283 }
@@ -284,7 +307,11 @@ pub fn main(args: []const [:0]const u8) !u8 {
284 // A remote upgrade names a HOST, and the socket that matters is that 307 // A remote upgrade names a HOST, and the socket that matters is that
285 // box's own default; resolving a local one here would refuse the verb on 308 // box's own default; resolving a local one here would refuse the verb on
286 // a machine with no XDG_RUNTIME_DIR for no reason it could name. 309 // a machine with no XDG_RUNTIME_DIR for no reason it could name.
287 const uses_socket = specForCmd(o._cmd).uses_socket and o.resume_fd == null and o._host == null; 310 // An admin verb aimed at `--quic` resolves no socket path either: the
311 // daemon it is for may have none, and the box asking may have no
312 // runtime dir.
313 const uses_socket = specForCmd(o._cmd).uses_socket and o.resume_fd == null and o._host == null and
314 !(quicAdmin(o._cmd) and o.quic != null);
288 const sock_path = if (o.sock) |s| 315 const sock_path = if (o.sock) |s|
289 try alloc.dupe(u8, s) 316 try alloc.dupe(u8, s)
290 else if (!uses_socket) 317 else if (!uses_socket)
@@ -307,9 +334,9 @@ pub fn main(args: []const [:0]const u8) !u8 {
307 resumeRun(alloc, o, fd) 334 resumeRun(alloc, o, fd)
308 else 335 else
309 startCmd(alloc, o, sock_path, args[2..]), 336 startCmd(alloc, o, sock_path, args[2..]),
310 .dump => return dump(alloc, sock_path, o.vt, if (o.session) |n| n.name else ""), 337 .dump => return dump(alloc, adminTarget(o, sock_path), o.vt, if (o.session) |n| n.name else ""),
311 .stats => return stats(alloc, sock_path), 338 .stats => return stats(alloc, adminTarget(o, sock_path)),
312 .stop => return stopCmd(alloc, sock_path), 339 .stop => return stopCmd(alloc, adminTarget(o, sock_path)),
313 .upgrade => return if (o._host) |h| 340 .upgrade => return if (o._host) |h|
314 remoteUpgradeCmd(alloc, h, o.allow_same_version) 341 remoteUpgradeCmd(alloc, h, o.allow_same_version)
315 else 342 else
@@ -609,52 +636,206 @@ fn run(alloc: std.mem.Allocator, o: DaemonArguments, sock_path: []const u8) !u8
609 return try srv.run(); 636 return try srv.run();
610 } 637 }
611 638
612 /// Perform an unbounded one-shot query. A connected daemon that stops replying 639 /// Where an admin verb sends its one frame. `sock` is the path every verb
613 /// remains a visible hang rather than being reported like an absent socket. 640 /// has always taken. `quic` is the second door, for the daemon the first
641 /// cannot reach: one whose socket file was deleted and whose path a
642 /// successor now holds, so the re-bind in `Server.watchSockPath` is
643 /// refused and the daemon is still holding sessions nobody can stop,
644 /// dump or count (issue 145807a2). `mux a` has dialled daemons this way
645 /// since QUIC existed; the daemon serves these verbs on a client slot
646 /// exactly as on an observer, so this is the client learning the spelling.
647 const AdminTarget = union(enum) {
648 sock: []const u8,
649 quic: struct { host_port: []const u8, key: ?[]const u8 },
650 };
651
652 fn adminTarget(o: DaemonArguments, sock_path: []const u8) AdminTarget {
653 if (o.quic) |hp| return .{ .quic = .{ .host_port = hp, .key = o.key } };
654 return .{ .sock = sock_path };
655 }
656
657 /// The one bound QUIC asks live under. A unix socket says "nothing
658 /// listening" as an errno in a microsecond, so `oneShotQuery` on a path
659 /// may wait forever on a daemon that answered the connect; UDP has no
660 /// such errno — an unanswered port is silence — so every QUIC ask is
661 /// bounded, and the silence is reported as what it might be.
662 const admin_quic_deadline_ms: u32 = 5000;
663
664 /// One deadline for the whole ask, minted by the caller and spent by the
665 /// handshake and the reply together: two full budgets in a row made "5 s"
666 /// a ten-second wait on a port that took the handshake and nothing else.
667 fn adminDeadline() i64 {
668 return std.time.milliTimestamp() + admin_quic_deadline_ms;
669 }
670
671 fn msLeft(deadline: i64) u32 {
672 const left = deadline - std.time.milliTimestamp();
673 return if (left <= 0) 0 else @intCast(@min(left, admin_quic_deadline_ms));
674 }
675
676 /// Dial a daemon's QUIC arm for one admin verb: key by the same three-way
677 /// rule `mux a --quic` and `mux d start --quic` use (`--key`, then
678 /// `MUX_KEY_FILE`, then the default path), handshake within the deadline.
679 /// Every refusal is printed here in the verb's voice and answered with
680 /// null; the caller's only job is the frame.
681 fn openAdminQuic(alloc: std.mem.Allocator, verb: []const u8, q: anytype, deadline: i64) ?link_mod.Link {
682 const res = xdg.resolveKeyPath(alloc, xdg.pickKey(q.key, std.posix.getenv(xdg.key_env))) catch |e| {
683 std.debug.print("mux d {s}: quic: cannot resolve a key path: {s}\n", .{ verb, @errorName(e) });
684 return null;
685 };
686 defer res.deinit(alloc);
687 const key_path = switch (res) {
688 .given, .default => |p| p,
689 .missing => |p| {
690 std.debug.print(
691 "mux d {s}: no key: pass --key, set MUX_KEY_FILE, or run `mux d keygen` (default {s})\n",
692 .{ verb, p },
693 );
694 return null;
695 },
696 };
697 const key = quic.Key.load(key_path) catch |e| {
698 var buf: [quic.key_refusal_len]u8 = undefined;
699 std.debug.print("mux d {s}: {s}\n", .{ verb, quic.keyRefusalBody(&buf, e, key_path) });
700 return null;
701 };
702 const addr = quic.parseAddr(alloc, q.host_port) catch |e| {
703 std.debug.print("mux d {s}: --quic wants HOST[:PORT], got {s}: {s}\n", .{ verb, q.host_port, @errorName(e) });
704 return null;
705 };
706 const cl = quic.Client.connect(alloc, addr, key, quic.default_idle_ms) catch |e| {
707 std.debug.print("mux d {s}: cannot dial quic://{s}: {s}\n", .{ verb, q.host_port, @errorName(e) });
708 return null;
709 };
710 while (!cl.isReady()) {
711 cl.pump();
712 if (cl.dead or std.time.milliTimestamp() >= deadline) {
713 cl.deinit();
714 std.debug.print(
715 "mux d {s}: quic://{s} did not answer in {d}s (nothing there, or not this key)\n",
716 .{ verb, q.host_port, admin_quic_deadline_ms / 1000 },
717 );
718 return null;
719 }
720 var fds = [_]std.posix.pollfd{
721 .{ .fd = cl.pollFd(), .events = std.posix.POLL.IN, .revents = 0 },
722 };
723 _ = std.posix.poll(&fds, cl.timeoutMs(50)) catch {};
724 }
725 return .{ .quic = .{ .cl = cl, .alloc = alloc } };
726 }
727
728 /// Perform a one-shot query. On a socket it is unbounded: a connected
729 /// daemon that stops replying remains a visible hang rather than being
730 /// reported like an absent socket. Over QUIC it is bounded, for the reason
731 /// on `admin_quic_deadline_ms`.
614 fn oneShotQuery( 732 fn oneShotQuery(
615 alloc: std.mem.Allocator, 733 alloc: std.mem.Allocator,
616 sock_path: []const u8, 734 target: AdminTarget,
617 verb: []const u8, 735 verb: []const u8,
618 req: proto.MsgType, 736 req: proto.MsgType,
619 req_payload: []const u8, 737 req_payload: []const u8,
620 want: proto.MsgType, 738 want: proto.MsgType,
621 ) !u8 { 739 ) !u8 {
622 const frame = (dial.ask(alloc, sock_path, req, req_payload, want, null) catch |e| switch (e) { 740 const frame = switch (target) {
623 error.NoDaemon => { 741 .sock => |sock_path| (dial.ask(alloc, sock_path, req, req_payload, want, null) catch |e| switch (e) {
624 std.debug.print( 742 error.NoDaemon => {
625 "mux d {s}: nothing listening on {s} (`mux d start -d` starts one)\n", 743 std.debug.print(
626 .{ verb, sock_path }, 744 "mux d {s}: nothing listening on {s} (`mux d start -d` starts one)\n",
627 ); 745 .{ verb, sock_path },
628 return 1; 746 );
747 return 1;
748 },
749 // A reply this side could not read, or a daemon that hung up over the
750 // request: both stay errors, so a broken daemon never reads as an
751 // absent one.
752 else => return e,
753 }) orelse return 1,
754 .quic => |q| blk: {
755 const deadline = adminDeadline();
756 var l = openAdminQuic(alloc, verb, q, deadline) orelse return 1;
757 // The goodbye: a QUIC connection holds one of the daemon's client
758 // slots until CONNECTION_CLOSE or its idle timer, and close()
759 // writes the former.
760 defer l.close();
761 l.sendFrame(req, req_payload) catch |e| {
762 std.debug.print("mux d {s}: quic: could not send the request: {s}\n", .{ verb, @errorName(e) });
763 return 1;
764 };
765 const got = l.awaitFrame(alloc, want, msLeft(deadline), .{}) catch |e| switch (e) {
766 error.Closed => null,
767 else => return e,
768 };
769 break :blk got orelse {
770 std.debug.print("mux d {s}: quic://{s} took the request and did not answer\n", .{ verb, q.host_port });
771 return 1;
772 };
629 }, 773 },
630 // A reply this side could not read, or a daemon that hung up over the 774 };
631 // request: both stay errors, so a broken daemon never reads as an
632 // absent one.
633 else => return e,
634 }) orelse return 1;
635 defer frame.deinit(alloc); 775 defer frame.deinit(alloc);
636 try proto.writeAllFd(std.posix.STDOUT_FILENO, frame.payload); 776 try proto.writeAllFd(std.posix.STDOUT_FILENO, frame.payload);
637 try proto.writeAllFd(std.posix.STDOUT_FILENO, "\n"); 777 try proto.writeAllFd(std.posix.STDOUT_FILENO, "\n");
638 return 0; 778 return 0;
639 } 779 }
640 780
641 fn dump(alloc: std.mem.Allocator, sock_path: []const u8, vt_mode: bool, session: []const u8) !u8 { 781 fn dump(alloc: std.mem.Allocator, target: AdminTarget, vt_mode: bool, session: []const u8) !u8 {
642 // vt byte ++ session-name tail, built by the wire module — empty is the 782 // vt byte ++ session-name tail, built by the wire module — empty is the
643 // wire's own default spelling, so a bare `mux d dump` sends exactly the 783 // wire's own default spelling, so a bare `mux d dump` sends exactly the
644 // one-byte payload that predates session names. 784 // one-byte payload that predates session names.
645 var buf: [proto.debug_dump_max_len]u8 = undefined; 785 var buf: [proto.debug_dump_max_len]u8 = undefined;
646 const payload = proto.encodeDebugDumpNamed(&buf, vt_mode, session); 786 const payload = proto.encodeDebugDumpNamed(&buf, vt_mode, session);
647 return oneShotQuery(alloc, sock_path, "dump", .debug_dump, payload, .dump_reply); 787 return oneShotQuery(alloc, target, "dump", .debug_dump, payload, .dump_reply);
648 } 788 }
649 789
650 fn stats(alloc: std.mem.Allocator, sock_path: []const u8) !u8 { 790 fn stats(alloc: std.mem.Allocator, target: AdminTarget) !u8 {
651 return oneShotQuery(alloc, sock_path, "stats", .stats_req, "", .stats_reply); 791 return oneShotQuery(alloc, target, "stats", .stats_req, "", .stats_reply);
652 } 792 }
653 793
654 /// Request daemon shutdown, then wait for the peer process to exit rather than 794 /// Request daemon shutdown, then wait for the peer process to exit rather than
655 /// only for socket unlink. Both an already-absent daemon and a completed stop 795 /// only for socket unlink. Both an already-absent daemon and a completed stop
656 /// return zero, making the command idempotent for scripts. 796 /// return zero, making the command idempotent for scripts.
657 fn stopCmd(alloc: std.mem.Allocator, sock_path: []const u8) !u8 { 797 ///
798 /// Over QUIC there is no pid to wait on and no errno that says "absent", so
799 /// the verdict is the daemon's own goodbye: `Server.deinit` tears every
800 /// client slot down before it frees anything else, and a QUIC slot's
801 /// teardown is a CONNECTION_CLOSE to us. A port that never answered is a
802 /// refusal (rc 1), not a no-op — the silence may be a firewall or the wrong
803 /// key, and a script that read it as "already stopped" would be lied to.
804 /// The goodbye is read off `quic.Client.dead`, which a lost packet stream
805 /// or an idle expiry also sets, so the message says what was seen — the
806 /// connection ended after the request was delivered — and does not claim
807 /// the pid is gone; `mux d stats --quic` afterwards is the check.
808 fn stopCmd(alloc: std.mem.Allocator, target: AdminTarget) !u8 {
809 const sock_path = switch (target) {
810 .sock => |p| p,
811 .quic => |q| {
812 const deadline = adminDeadline();
813 var l = openAdminQuic(alloc, "stop", q, deadline) orelse return 1;
814 defer l.close();
815 l.sendFrame(.stop_req, "") catch |e| {
816 std.debug.print("mux d stop: quic: could not deliver the stop request: {s}\n", .{@errorName(e)});
817 return 1;
818 };
819 // Nothing is ever wanted: the loop ends on the daemon's close.
820 // `stop_req` as the wanted type is one the daemon never sends.
821 const got = l.awaitFrame(alloc, .stop_req, msLeft(deadline), .{}) catch |e| switch (e) {
822 error.Closed => {
823 std.debug.print(
824 "mux d: stopped (quic://{s} ended the connection after taking the request)\n",
825 .{q.host_port},
826 );
827 return 0;
828 },
829 else => return e,
830 };
831 if (got) |f| f.deinit(alloc);
832 std.debug.print(
833 "mux d stop: quic://{s} took the request and is still connected after {d}s\n",
834 .{ q.host_port, admin_quic_deadline_ms / 1000 },
835 );
836 return 1;
837 },
838 };
658 const stream = dial.dial(sock_path) catch { 839 const stream = dial.dial(sock_path) catch {
659 std.debug.print("mux d stop: nothing listening on {s}\n", .{sock_path}); 840 std.debug.print("mux d stop: nothing listening on {s}\n", .{sock_path});
660 return 0; 841 return 0;
@@ -666,30 +847,31 @@ fn stopCmd(alloc: std.mem.Allocator, sock_path: []const u8) !u8 {
666 const peer = peerPid(stream.handle); 847 const peer = peerPid(stream.handle);
667 stream.close(); 848 stream.close();
668 849
669 // Probe before checking the deadline so the final interval is observed. 850 // The pid is the verdict whenever the kernel names one. The path is
670 // Connection refusal indicates that the shutdown unlink has completed; 851 // not: a daemon that lost its path to a successor watches that path
671 // a wedged event loop can still accept through the socket backlog. 852 // and re-binds it within a second of the successor's unlink
853 // (`Server.watchSockPath`), so "the socket still answers" can mean the
854 // OTHER daemon answered — and a stop that graded the path reported a
855 // clean exit as a failure. Process exit is also what a supervisor
856 // wants: the unlink happens before shell reaping and directory
857 // cleanup, so a stop that returned on the unlink returned early.
858 if (peer) |pid| return waitPidGone(alloc, pid, sock_path, asked);
859
860 // No pid (a kernel that does not expose the peer): the socket's
861 // silence is all there is. Probe before checking the deadline so the
862 // final interval is observed; a wedged event loop can still accept
863 // through the backlog, so silence here means the unlink ran.
672 const stop_deadline_ms: i64 = 2000; 864 const stop_deadline_ms: i64 = 2000;
673 const t0 = std.time.milliTimestamp(); 865 const t0 = std.time.milliTimestamp();
674 while (true) { 866 while (true) {
675 if (!sockpath.answers(sock_path)) return waitPidGone(peer, sock_path); 867 if (!sockpath.answers(sock_path)) {
868 std.debug.print("mux d: stopped\n", .{});
869 return 0;
870 }
676 if (std.time.milliTimestamp() - t0 >= stop_deadline_ms) break; 871 if (std.time.milliTimestamp() - t0 >= stop_deadline_ms) break;
677 std.Thread.sleep(50 * std.time.ns_per_ms); 872 std.Thread.sleep(50 * std.time.ns_per_ms);
678 } 873 }
679 const secs = @divTrunc(stop_deadline_ms, 1000); 874 return stillRunning(alloc, sock_path, asked, @divTrunc(stop_deadline_ms, 1000));
680 if (!asked) {
681 std.debug.print(
682 "mux d stop: could not deliver the stop request to {s}, and it is still answering after {d}s\n",
683 .{ sock_path, secs },
684 );
685 return 1;
686 }
687 var hint: [log_hint_len]u8 = undefined;
688 std.debug.print(
689 "mux d stop: {s} still answering after {d}s{s}\n",
690 .{ sock_path, secs, logHint(alloc, &hint) },
691 );
692 return 1;
693 } 875 }
694 876
695 /// The peer's pid, or null when the kernel cannot expose it; callers then 877 /// The peer's pid, or null when the kernel cannot expose it; callers then
@@ -699,26 +881,15 @@ fn peerPid(fd: std.posix.socket_t) ?std.posix.pid_t {
699 return cred.pid; 881 return cred.pid;
700 } 882 }
701 883
702 /// Wait for the peer process after its socket disappears. Socket unlink occurs 884 /// Wait for the daemon's process to leave. Only a live process answers
703 /// before shell reaping and directory cleanup, so process exit is the reliable 885 /// signal 0 with success: ESRCH is the answer wanted, and EPERM means the
704 /// completion signal for supervisors. 886 /// pid was reused by someone else's process, so the daemon is just as gone.
705 fn waitPidGone(peer: ?std.posix.pid_t, sock_path: []const u8) u8 { 887 fn waitPidGone(alloc: std.mem.Allocator, pid: std.posix.pid_t, sock_path: []const u8, asked: bool) u8 {
706 const pid = peer orelse { 888 const gone_deadline_ms: i64 = 5000;
707 std.debug.print("mux d: stopped\n", .{});
708 return 0;
709 };
710 const gone_deadline_ms: i64 = 3000;
711 const t0 = std.time.milliTimestamp(); 889 const t0 = std.time.milliTimestamp();
712 // Only a live process answers signal 0 with success. ESRCH is the
713 // answer wanted; EPERM means the pid was reused by someone else's
714 // process, and the daemon is just as gone.
715 while (std.posix.kill(pid, 0)) |_| { 890 while (std.posix.kill(pid, 0)) |_| {
716 if (std.time.milliTimestamp() - t0 >= gone_deadline_ms) { 891 if (std.time.milliTimestamp() - t0 >= gone_deadline_ms) {
717 std.debug.print( 892 return stillRunning(alloc, sock_path, asked, @divTrunc(gone_deadline_ms, 1000));
718 "mux d stop: {s} is closed, but pid {d} is still running {d}s later\n",
719 .{ sock_path, pid, @divTrunc(gone_deadline_ms, 1000) },
720 );
721 return 1;
722 } 893 }
723 std.Thread.sleep(20 * std.time.ns_per_ms); 894 std.Thread.sleep(20 * std.time.ns_per_ms);
724 } else |_| {} 895 } else |_| {}
@@ -726,6 +897,25 @@ fn waitPidGone(peer: ?std.posix.pid_t, sock_path: []const u8) u8 {
726 return 0; 897 return 0;
727 } 898 }
728 899
900 /// The stop's failure line. Whether the request was delivered decides
901 /// whether the log is worth pointing at: a daemon that never read the
902 /// frame has nothing to say about it there.
903 fn stillRunning(alloc: std.mem.Allocator, sock_path: []const u8, asked: bool, secs: i64) u8 {
904 if (!asked) {
905 std.debug.print(
906 "mux d stop: could not deliver the stop request to {s}, and it is still running after {d}s\n",
907 .{ sock_path, secs },
908 );
909 return 1;
910 }
911 var hint: [log_hint_len]u8 = undefined;
912 std.debug.print(
913 "mux d stop: the daemon on {s} is still running after {d}s{s}\n",
914 .{ sock_path, secs, logHint(alloc, &hint) },
915 );
916 return 1;
917 }
918
729 /// What the remote-upgrade preflight learned. One ssh run answers everything: 919 /// What the remote-upgrade preflight learned. One ssh run answers everything:
730 /// `uname -m && command -v mux && mux d endpoint` — the bare endpoint verb, 920 /// `uname -m && command -v mux && mux d endpoint` — the bare endpoint verb,
731 /// which never starts a daemon — and the LINE COUNT is the diagnosis, because 921 /// which never starts a daemon — and the LINE COUNT is the diagnosis, because
@@ -1771,6 +1961,35 @@ test "parseArgs: stop is a command and takes --sock" {
1771 try std.testing.expectEqualStrings("/tmp/x.sock", s.command.sock.?); 1961 try std.testing.expectEqualStrings("/tmp/x.sock", s.command.sock.?);
1772 } 1962 }
1773 1963
1964 test "parseArgs: stop, dump and stats take --quic as the other door, and never both doors" {
1965 // The path-less daemon's door (145807a2): the same `--quic HOST:PORT
1966 // [--key]` mux a spells, on the three verbs the daemon serves to a
1967 // QUIC client slot.
1968 const q = parse(&.{ "d", "stop", "--quic", "127.0.0.1:4433", "--key", "/k" });
1969 try std.testing.expect(q == .command);
1970 try std.testing.expect(q.command._cmd == .stop);
1971 try std.testing.expectEqualStrings("127.0.0.1:4433", q.command.quic.?);
1972 try std.testing.expectEqualStrings("/k", q.command.key.?);
1973 try std.testing.expect(parse(&.{ "d", "stats", "--quic", "box" }) == .command);
1974 try std.testing.expect(parse(&.{ "d", "dump", "--session", "a", "--quic", "box:1" }) == .command);
1975
1976 // Two daemons named for one question is a refusal, not a preference.
1977 try std.testing.expect(parse(&.{ "d", "stop", "--sock", "/x", "--quic", "box" }).usage == .sock_with_quic);
1978 try std.testing.expect(parse(&.{ "d", "stats", "--quic", "box", "--sock", "/x" }).usage == .sock_with_quic);
1979 // `start` listens on both, so both stay legal there.
1980 try std.testing.expect(parse(&.{ "d", "start", "--sock", "/x", "--quic", "0.0.0.0:1" }) == .command);
1981 // And the key rule is unchanged: --key still wants --quic.
1982 try std.testing.expect(parse(&.{ "d", "stop", "--key", "/k" }).usage == .key_without_quic);
1983
1984 // A verb with no QUIC door refuses the flag rather than dropping it:
1985 // `upgrade --quic HOST` used to upgrade the socket's daemon in silence.
1986 inline for (.{ "upgrade", "proxy", "endpoint" }) |verb| {
1987 const r = parse(&.{ "d", verb, "--quic", "box" });
1988 try std.testing.expect(r == .usage);
1989 try std.testing.expectEqualStrings("--quic", r.usage.unknown_arg);
1990 }
1991 }
1992
1774 test "parseArgs: endpoint is a command and takes --sock" { 1993 test "parseArgs: endpoint is a command and takes --sock" {
1775 const r = parse(&.{ "d", "endpoint" }); 1994 const r = parse(&.{ "d", "endpoint" });
1776 try std.testing.expect(r == .command); 1995 try std.testing.expect(r == .command);
@@ -2032,11 +2251,11 @@ test "oneShotQuery: a socket nobody serves is exit 1" {
2032 // therefore return one for an absent socket. 2251 // therefore return one for an absent socket.
2033 try std.testing.expectEqual( 2252 try std.testing.expectEqual(
2034 @as(u8, 1), 2253 @as(u8, 1),
2035 try oneShotQuery(std.testing.allocator, sock, "dump", .debug_dump, "", .dump_reply), 2254 try oneShotQuery(std.testing.allocator, .{ .sock = sock }, "dump", .debug_dump, "", .dump_reply),
2036 ); 2255 );
2037 try std.testing.expectEqual( 2256 try std.testing.expectEqual(
2038 @as(u8, 1), 2257 @as(u8, 1),
2039 try oneShotQuery(std.testing.allocator, sock, "stats", .stats_req, "", .stats_reply), 2258 try oneShotQuery(std.testing.allocator, .{ .sock = sock }, "stats", .stats_req, "", .stats_reply),
2040 ); 2259 );
2041 } 2260 }
2042 2261
@@ -2046,7 +2265,7 @@ test "stopCmd: a socket path with nothing on it is exit 0, not a failure" {
2046 defer tmp.cleanup(); 2265 defer tmp.cleanup();
2047 var buf: [280]u8 = undefined; 2266 var buf: [280]u8 = undefined;
2048 const sock = try std.fmt.bufPrint(&buf, "{s}/absent.sock", .{tmp.path()}); 2267 const sock = try std.fmt.bufPrint(&buf, "{s}/absent.sock", .{tmp.path()});
2049 try std.testing.expectEqual(@as(u8, 0), try stopCmd(std.testing.allocator, sock)); 2268 try std.testing.expectEqual(@as(u8, 0), try stopCmd(std.testing.allocator, .{ .sock = sock }));
2050 } 2269 }
2051 2270
2052 test "peerPid: the kernel names the peer" { 2271 test "peerPid: the kernel names the peer" {
@@ -2079,7 +2298,7 @@ test "waitPidGone: returns only once the OS has no such process" {
2079 _ = try child.wait(); 2298 _ = try child.wait();
2080 const pid = try std.fmt.parseInt(std.posix.pid_t, std.mem.trim(u8, buf[0..n], "\n "), 10); 2299 const pid = try std.fmt.parseInt(std.posix.pid_t, std.mem.trim(u8, buf[0..n], "\n "), 10);
2081 try std.posix.kill(pid, 0); // alive when we start, or the wait proves nothing 2300 try std.posix.kill(pid, 0); // alive when we start, or the wait proves nothing
2082 try std.testing.expectEqual(@as(u8, 0), waitPidGone(pid, "(test)")); 2301 try std.testing.expectEqual(@as(u8, 0), waitPidGone(std.testing.allocator, pid, "(test)", true));
2083 try std.testing.expectError(error.ProcessNotFound, std.posix.kill(pid, 0)); 2302 try std.testing.expectError(error.ProcessNotFound, std.posix.kill(pid, 0));
2084 } 2303 }
2085 2304
test/e2e.sh
Old New
@@ -176,8 +176,8 @@ done
176 # one of those and adds a convergence point would be pinning a fact every 176 # one of those and adds a convergence point would be pinning a fact every
177 # leg above already establishes. 177 # leg above already establishes.
178 178
179 [ "$OK_COUNT" = "115" ] || { 179 [ "$OK_COUNT" = "117" ] || {
180 echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 115 —" 180 echo "e2e FAIL: $OK_COUNT scenario checkpoints ran, the pin says 117 —"
181 echo " a scenario was added (update the pin) or silently lost" 181 echo " a scenario was added (update the pin) or silently lost"
182 exit 1 182 exit 1
183 } 183 }
test/e2e_05_session.sh
Old New
@@ -402,6 +402,53 @@ if dump_session "$SOCK23" b | grep -q "x18a-pin"; then
402 dump_session "$SOCK23" b; exit 1 402 dump_session "$SOCK23" b; exit 1
403 fi 403 fi
404 404
405 assert_stopped "$SOCK23" "$D20PID" "M18 quic multi-session" "$OUT.m18qstop"
406 D20PID=""
407 ok "quic: two dials are two sessions, and one name is one session on either transport" 405 ok "quic: two dials are two sessions, and one name is one session on either transport"
406
407 # --- the admin verbs by QUIC: the path-less daemon's door (145807a2) ------
408 #
409 # The incident's end state: the daemon's socket file is gone and (in the
410 # real one) a successor holds the path, so the re-bind is refused and
411 # `--sock` reaches the wrong daemon or none. This daemon has a QUIC arm,
412 # and the verbs the daemon serves on a client slot — stats, dump, stop —
413 # now take `--quic HOST:PORT --key` as `mux a` always has. Graded with
414 # the file really deleted: the daemon takes it back within a second (the
415 # boot group pins that), so every ask below is made in the window where
416 # only QUIC can reach it, or after, and either way through UDP alone.
417 rm "$SOCK23"
418 "$MUX" d stats --quic "127.0.0.1:$QPORT5" --key "$M18KEY" > "$OUT.m18qstats" 2>&1 || {
419 echo "e2e FAIL: mux d stats --quic failed"; cat "$OUT.m18qstats"; exit 1; }
420 grep -q 'sessions=3' "$OUT.m18qstats" || {
421 echo "e2e FAIL: stats over quic did not count the three sessions"; cat "$OUT.m18qstats"; exit 1; }
422 "$MUX" d dump --quic "127.0.0.1:$QPORT5" --key "$M18KEY" --session a > "$OUT.m18qdump" 2>&1 || {
423 echo "e2e FAIL: mux d dump --quic failed"; cat "$OUT.m18qdump"; exit 1; }
424 grep -q 'q18a-pin' "$OUT.m18qdump" || {
425 echo "e2e FAIL: dump over quic did not carry session a's marker"; cat "$OUT.m18qdump"; exit 1; }
426 # Both doors at once is a refusal at parse, rc 2, and no frame is sent.
427 set +e
428 "$MUX" d stats --sock "$SOCK23" --quic "127.0.0.1:$QPORT5" > /dev/null 2> "$OUT.m18qboth"
429 _rc=$?
430 set -e
431 [ "$_rc" = "2" ] || { echo "e2e FAIL: --sock with --quic exited $_rc, want 2"; cat "$OUT.m18qboth"; exit 1; }
432 grep -q 'name two daemons' "$OUT.m18qboth" || {
433 echo "e2e FAIL: --sock with --quic did not say why"; cat "$OUT.m18qboth"; exit 1; }
434 # The stop, and the OS says it took: the pid goes, and whatever socket file
435 # the re-bind put back goes with it (deinit's unlink), so no leftover
436 # names a daemon that is not there.
437 set +e
438 "$MUX" d stop --quic "127.0.0.1:$QPORT5" --key "$M18KEY" 2> "$OUT.m18qstop"
439 _rc=$?
440 set -e
441 [ "$_rc" = "0" ] || { echo "e2e FAIL: mux d stop --quic exited $_rc, want 0"; cat "$OUT.m18qstop"; exit 1; }
442 grep -q '^mux d: stopped' "$OUT.m18qstop" || {
443 echo "e2e FAIL: stop over quic did not report stopped"; cat "$OUT.m18qstop"; exit 1; }
444 wait_pid_gone "$D20PID" "stop over quic reported stopped"
445 [ ! -S "$SOCK23" ] || { echo "e2e FAIL: stop over quic left $SOCK23"; ls -l "$SOCK23"; exit 1; }
446 # And a stopped port is a refusal, not a silent no-op: a script must not
447 # read a firewall or a wrong key as "already stopped".
448 set +e
449 "$MUX" d stop --quic "127.0.0.1:$QPORT5" --key "$M18KEY" 2> "$OUT.m18qstop2"
450 _rc=$?
451 set -e
452 [ "$_rc" = "1" ] || { echo "e2e FAIL: stop on a dead quic port exited $_rc, want 1"; cat "$OUT.m18qstop2"; exit 1; }
453 D20PID=""
454 ok "quic: stats, dump and stop reach a daemon whose socket file is gone (145807a2)"