a73x

3bf0fc41

fix: stop native reconnect on SSH authentication refusal

a73x   2026-09-05 14:01

Commit message
fix: stop native reconnect on SSH authentication refusal
RETRO.md
Old New
@@ -421,8 +421,83 @@ this milestone.
421 - [x] Record final Sprint 5 demo acceptance and any requested ergonomic adjustment; 421 - [x] Record final Sprint 5 demo acceptance and any requested ergonomic adjustment;
422 distinguish recorded approval from any hands-on trial. Accepted on 2026-09-05 422 distinguish recorded approval from any hands-on trial. Accepted on 2026-09-05
423 with no requested changes; recorded-demo approval only. 423 with no requested changes; recorded-demo approval only.
424 - [ ] Start with available Luna/Terra slots and short briefs containing the pinned 424 - [x] Start with available Luna/Terra slots and short briefs containing the pinned
425 compiler/cache command; record model selection, review yield and available cost 425 compiler/cache command; record model selection, review yield and available cost
426 evidence without estimating missing token totals. 426 evidence without estimating missing token totals.
427 - [ ] When shared handoff errors are changed, separate unreachable-host EOF from 427 Completed in the SSH follow-up below; local session metadata confirms both models.
428 - [x] When shared handoff errors are changed, separate unreachable-host EOF from
428 authentication refusal and verify bounded reconnect and join-only restore. 429 authentication refusal and verify bounded reconnect and join-only restore.
430 Implemented and validated in the SSH follow-up; recorded-demo acceptance pending.
431
432 ## SSH failure classification — follow-up sprint, 2026-09-05
433
434 The user authorized this bounded sprint after accepting the native tiling milestone.
435 Implementation, adversarial review, closing cleanup assessment, and validation are
436 complete. The [delivery record](docs/superpowers/plans/2026-09-05-ssh-error-classification.md)
437 contains commands and evidence. The new recorded trial is awaiting user acceptance.
438
439 ### Results and review
440
441 - Canonical OpenSSH authentication refusal now stops native automatic reconnect
442 during both restore and post-attachment redial. The pane retains its target and
443 diagnostic; correcting credentials requires explicit Retry. Unknown EOF and
444 unavailable-host recovery retain their existing behavior. End, creation, and
445 terminal layout policy were not changed.
446 - Opening review found the existing handoff/Reason/pump boundaries sufficient;
447 no separate structural cleanup was warranted. Closing review removed an unused
448 classification category and a bespoke ASCII scan, and extracted the retry-policy
449 predicate to verify existing non-auth behavior explicitly.
450 - Review tightened banner/identity parsing and authentication-method coverage,
451 required real execution counts for both retry paths, replaced a timing-dependent
452 reconnect test with an attached-then-release handshake, and fixed test-server
453 teardown so a stop failure cannot skip GUI/daemon cleanup. Root's final finding
454 preserved retries for post-attachment non-auth errors such as connection reset.
455 - The permanent real OpenSSH/native test failed on the accepted previous release:
456 a rejected key left the pane reconnecting and caused repeated SSH attempts.
457 It passes on the final release, including manual recovery, independent neighbour
458 input, unavailable-host return, missing-session preservation, and shutdown.
459 - `make ci` passed. After the final pump-policy correction, `make check`, native
460 unit/integration gates, real SSH acceptance, and the actual GUI recording were
461 rerun. Wayland 200%/100%/150%/200% transitions and raw-output stress passed.
462 Fixtures and the owned compositor were stopped.
463
464 ### Model economy and workflow lessons
465
466 Both agents started with short briefs and no conversation fork. Local session
467 `turn_context.model` records confirm Luna for implementation and Terra for review.
468 An earlier reviewer claim that it used Astra was an unsupported self-description;
469 the reviewer retracted it. Verify metadata at startup rather than after delivery.
470
471 The local cumulative counters below include cached input, repeated context, and
472 all focused follow-up turns; they are not unique context sizes or monetary bills.
473 The evidence snapshot is retained with the demo logs.
474
475 | Agent | Input tokens | Cached input (included) | Output tokens |
476 | --- | ---: | ---: | ---: |
477 | Luna implementer | 5,424,261 | 5,296,640 | 17,644 |
478 | Terra reviewer | 6,733,586 | 6,560,000 | 13,092 |
479
480 The first/last token events span about 17 minutes for implementation and 16 minutes
481 for review, including follow-up waits; active model time and monetary cost are not
482 available. Review produced concrete correctness and test-quality improvements,
483 but repeated socket-denied unit runs and large failure output expanded context.
484 Retain Luna/Terra defaults; reduce redundant validation output and route socket
485 checks to the authorized environment after the first denial. No measured cost
486 saving is claimed.
487
488 ### Retained limits and next actions
489
490 The classifier in `src/client/handoff.zig` recognizes complete OpenSSH diagnostic
491 forms, not every SSH implementation, localization, or truncated message. `Reason`
492 retains its existing 120-byte ASCII line limit. Unrecognized text stays unknown
493 and therefore recoverable rather than being guessed to mean auth refusal. The
494 next shared diagnostic-format change owns expanding classification and must retain
495 negative-banner, fragmented-line, reconnect, and shutdown checks.
496
497 - [ ] Record this sprint's demo acceptance separately from any hands-on trial.
498 - [ ] Obtain explicit approval before exposing this demo page/video at the proposed
499 private `/ssh-followup/` Tailscale route. Automatic approval review rejected that
500 publication because the specific payload/destination were not explicitly authorized.
501 The video is retained locally; existing private demo routes were left unchanged.
502 - [ ] In the next authorized sprint, verify agent model metadata immediately after
503 startup and avoid rerunning a socket-denied suite in the same sandbox.
docs/native-sprint-workflow.md
Old New
@@ -29,6 +29,13 @@ packages. Keep one sprint active, ending in a working demo and explicit acceptan
29 of inheriting the full conversation. Escalate a concrete unresolved problem 29 of inheriting the full conversation. Escalate a concrete unresolved problem
30 to a stronger model when needed. Root coordinates and integrates; assign each 30 to a stronger model when needed. Root coordinates and integrates; assign each
31 investigation one owner so agents do not repeat it independently by accident. 31 investigation one owner so agents do not repeat it independently by accident.
32
33 Check each agent's model immediately after startup. Prefer recorded session
34 `turn_context.model` metadata when available; distinguish it from the requested
35 spawn setting, and do not rely on an agent's self-description. Retain available
36 token counters with cached input identified separately; counters are not a
37 monetary bill. After a sandbox denial, use the authorized validation route
38 instead of repeating the same blocked suite or streaming its full failure log.
32 4. **Build a functional slice.** Establish the smallest model/interface contract, 39 4. **Build a functional slice.** Establish the smallest model/interface contract,
33 then connect actual input, rendering, and daemon behavior. Keep changes small 40 then connect actual input, rendering, and daemon behavior. Keep changes small
34 enough to review. Avoid expanding into later sprint features. 41 enough to review. Avoid expanding into later sprint features.
@@ -66,6 +73,13 @@ packages. Keep one sprint active, ending in a working demo and explicit acceptan
66 its original target. For persistence, retain original bytes across real 73 its original target. For persistence, retain original bytes across real
67 permission failures, reopen from a different working directory and scale, and 74 permission failures, reopen from a different working directory and scale, and
68 keep a sentinel in the terminal client's layout file. 75 keep a sentinel in the terminal client's layout file.
76
77 For SSH handoff or reconnect changes on Linux, also run the real OpenSSH gate
78 from the [SSH follow-up plan](superpowers/plans/2026-09-05-ssh-error-classification.md).
79 `test/native_ssh.py` accepts the release daemon and GUI paths and uses its own
80 loopback sshd, keys, state, and sessions. Verify authentication refusal on both
81 restoration and reconnect, explicit Retry after correcting credentials,
82 unavailable-host recovery, join-only restoration, and bounded shutdown.
69 7. **Refactor the integrated result.** Reserve this as work, even after the opening 83 7. **Refactor the integrated result.** Reserve this as work, even after the opening
70 cleanup. Remove duplication, superseded paths, unnecessary state, and temporary 84 cleanup. Remove duplication, superseded paths, unnecessary state, and temporary
71 scaffolding exposed by implementation. Review the result again. For retained 85 scaffolding exposed by implementation. Review the result again. For retained
docs/superpowers/plans/2026-09-05-native-tiling.md
Old New
@@ -6,6 +6,10 @@ validated, and accepted by the user on 2026-09-05.
6 Sprint 4 is implemented, reviewed, and validated; separate recorded-demo 6 Sprint 4 is implemented, reviewed, and validated; separate recorded-demo
7 acceptance was not given. Sprint 3 acceptance passed. 7 acceptance was not given. Sprint 3 acceptance passed.
8 8
9 The retained SSH error-classification debt is addressed in the separately
10 authorized [follow-up sprint](2026-09-05-ssh-error-classification.md), whose
11 implementation and validation are complete and recorded-demo acceptance is pending.
12
9 ## Working agreement 13 ## Working agreement
10 14
11 Follow the reusable [sprint workflow](../../native-sprint-workflow.md), including 15 Follow the reusable [sprint workflow](../../native-sprint-workflow.md), including
docs/superpowers/plans/2026-09-05-ssh-error-classification.md
Old New
@@ -0,0 +1,108 @@
1 # SSH failure classification — follow-up sprint
2
3 The user authorized this bounded follow-up after accepting native tiling Sprint 5.
4 The native tiling milestone remains complete. This sprint closes the shared
5 handoff debt recorded in [RETRO.md](../../../RETRO.md), using the
6 [native sprint workflow](../../native-sprint-workflow.md).
7
8 ## Scope and acceptance
9
10 Recognize SSH authentication refusal separately from an unavailable host when
11 the endpoint announcement fails. A native pane must stop automatic reconnect on
12 a recognized authentication refusal, retain its identity and diagnostic, and
13 offer explicit Retry after credentials are corrected. Apply this both during
14 initial workspace restoration and after a previously attached session disconnects.
15
16 An unavailable host or an unclassified announcement EOF remains eligible for
17 bounded-backoff reconnect. Unknown diagnostics must not be guessed to mean
18 authentication refusal. Preserve cancellation and bounded shutdown, join-only
19 restoration, and the prohibition on automatically retrying End or creation.
20 Terminal layout policy, tabs, pane rearrangement, and credential UI are outside
21 this sprint.
22
23 Acceptance uses actual SSH authentication and real daemon sessions, with an
24 independent attempt counter, session catalogue, rendered output, and saved pane
25 identity. Cover failed credentials on restore and reconnect, corrected credentials
26 with explicit Retry, unavailable-host recovery, missing-session preservation,
27 an independently usable neighbouring pane, and shutdown during retries. Focused
28 tests cover diagnostic fragmentation and ambiguous/non-authentication failures.
29
30 ## Delivery
31
32 - Implementation: GPT-5.6 Luna (`handoff_impl`). Adversarial review: GPT-5.6 Terra
33 (`handoff_review`), with independent integration and acceptance by root.
34 - Briefs specify `deps/zig/zig` and `ZIG_GLOBAL_CACHE_DIR=/tmp/muxg-zig-cache`.
35 - Opening and closing cleanup assessments, final frozen-source gates, real GUI
36 demonstration, and the retrospective are required before handoff. Recorded-demo
37 acceptance remains pending until the user gives it.
38 - Model selection and concrete review yield will be recorded. Token/cost figures
39 will be reported only if available.
40
41 The optional Linux real-SSH gate requires OpenSSH client, server, and keygen:
42
43 ```sh
44 export ZIG_GLOBAL_CACHE_DIR=/tmp/muxg-zig-cache
45 deps/zig/zig build install native -Doptimize=ReleaseSafe -p dist/native-ssh-release
46 python3 test/native_ssh.py dist/native-ssh-release/bin/mux dist/native-ssh-release/bin/muxg
47 ```
48
49 This complements the standard gates in the sprint workflow. It starts an owned
50 unprivileged loopback SSH server, with temporary keys and isolated state; local
51 socket/PTY permissions are required. `MUXG_VIDEODRIVER=wayland` selects a real
52 compositor for the same checks.
53
54 ## Delivery evidence
55
56 Status: implemented, reviewed, and validated. Recorded-demo acceptance is pending.
57 The native tiling milestone remains accepted; this follow-up has its own demo gate.
58
59 Opening review found no justified standalone cleanup. Closing review kept the
60 classifier at the shared handoff boundary, removed an unused result category and
61 manual ASCII scan, and extracted the pump retry predicate. Review covered banner
62 false positives, authentication-method lists, initial and post-attachment retries,
63 deterministic test ordering, and cleanup of owned fixtures. A final correction
64 preserves all previous non-auth post-attachment retry behavior; the initial-open
65 whitelist was not extended to established connections.
66
67 Validation logs are under `/tmp/muxg-ssh-followup/`, with retained copies under
68 `/home/xanderle/.codex/visualizations/2026/09/05/01a071c2-5767-7790-a335-1a84536c9ca5/ssh-evidence/`:
69
70 - `before.log`: real-SSH negative control against the accepted prior release.
71 Rejected keys repeatedly spawned SSH while the pane remained reconnecting.
72 - `ci.log`: complete `make ci` passed. Following the final pump-only correction,
73 `make check` passed again (`check-final.log`); all affected native checks below
74 were rerun on the rebuilt final release.
75 - `build-final.log`: paired ReleaseSafe daemon/GUI and native unit tests passed.
76 `native-e2e-final.log`: the complete standard native integration gate passed.
77 - `real-ssh-final.log`: all five grouped real OpenSSH acceptance checks passed
78 offscreen. `demo-final.log`: the same checks passed on real Sway at 200%.
79 These use a real loopback SSH server and real daemon sessions, not an SSH stub.
80 Existing repository gates also passed; no new separate-VM validation is claimed.
81 - `scale.log`: real Wayland 200% → 100% → 150% → 200%, drag cancellation,
82 independently queried PTY dimensions, and framebuffer checks passed.
83 - `stress.log`, `stress-result.json`: `make native-stress` passed alone on the
84 isolated GPU compositor (NVIDIA GeForce RTX 3080, GLES2). Both binaries came
85 from the ReleaseSafe build graph. About 103.7 MB ran over 30.2 seconds;
86 sampled input-to-painted maxima were 55.4 ms on the shared daemon and 65.2 ms
87 on the separate daemon. Frame p99 was 18.1 ms. Input observations include
88 5 ms polling and are upper bounds; the recording is not a latency measurement.
89
90 The 30.4-second actual GUI demo is retained at
91 `/home/xanderle/.codex/visualizations/2026/09/05/01a071c2-5767-7790-a335-1a84536c9ca5/native-ssh-trial.mp4`.
92 It is 1100×700 H.264 with no audio, joining three continuous clips across window
93 restarts. It shows auth refusal with explicit Retry, correction and recovery,
94 post-attachment refusal, unavailable-host recovery, and missing-session refusal.
95 Use Ctrl+\\ then `p` to open recovery actions. No new ergonomic bindings were added.
96
97 Only a loopback demo server remains, serving the prepared page and video on 18771.
98 The proposed private route `https://charizard.folk-amberjack.ts.net/ssh-followup/`
99 was rejected by automatic approval review because this payload/destination lacks
100 explicit authorization. It has not been published. Existing routes were preserved.
101 The server PID is recorded in `/tmp/muxg-ssh-followup/server.pid`; stop that owned
102 server when the demo is no longer needed. Scripts, clip metadata, and server source
103 remain in `/tmp/muxg-ssh-followup/`. Test daemons, SSH fixtures, and compositor were
104 stopped; final process checks found no remaining final-test fixture processes.
105
106 Canonical diagnostic parsing deliberately leaves unknown/localized/truncated text
107 recoverable. The shared `Reason` line limit remains 120 ASCII bytes. Model evidence,
108 available token counters, lessons, and the follow-up trigger are in RETRO.md.
docs/superpowers/specs/2026-09-05-native-tiling-design.md
Old New
@@ -222,6 +222,13 @@ or exited sessions remain visible; restoration must not silently replace a missi
222 session with a newly created shell. Check the existing attach/create semantics 222 session with a newly created shell. Check the existing attach/create semantics
223 before implementing this guarantee and add a narrow protocol capability if needed. 223 before implementing this guarantee and add a narrow protocol capability if needed.
224 224
225 The authorized [SSH follow-up](../plans/2026-09-05-ssh-error-classification.md)
226 adds a stop condition to automatic reconnect: a recognized SSH authentication
227 refusal retains the pane and its diagnostic but requires explicit Retry after
228 credentials are corrected. This applies during restoration and after connection
229 loss. An unavailable host or unclassified announcement EOF remains recoverable
230 through bounded-backoff reconnect; EOF alone never proves authentication refusal.
231
225 Confirmed: GUI and terminal clients can deliberately select the same daemon 232 Confirmed: GUI and terminal clients can deliberately select the same daemon
226 session from a shared catalogue, while their layouts and automatic pane choices 233 session from a shared catalogue, while their layouts and automatic pane choices
227 remain independent. Sharing a session retains the daemon's existing resize and 234 remain independent. Sharing a session retains the daemon's existing resize and
src/client/client.zig
Old New
@@ -1046,7 +1046,11 @@ pub fn openFailure(buf: []u8, target: Target, err: anyerror, reason: []const u8)
1046 // ssh, or the remote, said why in a whole sentence. It beats 1046 // ssh, or the remote, said why in a whole sentence. It beats
1047 // both of the lines below: `UnterminatedLine` names what mux 1047 // both of the lines below: `UnterminatedLine` names what mux
1048 // observed, while `No route to host` names what happened. 1048 // observed, while `No route to host` names what happened.
1049 else => if (reason.len > 0) failedMsg( 1049 else => if (handoff.classifyReason(reason) == .authentication_refused) failedMsg(
1050 buf,
1051 "mux: authentication refused by {s} over ssh: {s}\n",
1052 .{ h.host, reason },
1053 ) else if (reason.len > 0) failedMsg(
1050 buf, 1054 buf,
1051 "mux: {s} over ssh: {s}\n", 1055 "mux: {s} over ssh: {s}\n",
1052 .{ h.host, reason }, 1056 .{ h.host, reason },
@@ -2749,6 +2753,15 @@ test "openFailure: a reason replaces the error name for a hand target" {
2749 ); 2753 );
2750 } 2754 }
2751 2755
2756 test "openFailure: an OpenSSH authentication refusal is visible as terminal" {
2757 var buf: [open_err_len]u8 = undefined;
2758 const h: Target = .{ .hand = .{ .host = "box", .ssh_argv = &.{"ssh"}, .cache_path = null } };
2759 try std.testing.expectEqualStrings(
2760 "mux: authentication refused by box over ssh: Permission denied (publickey).\n",
2761 openFailure(&buf, h, error.UnterminatedLine, "Permission denied (publickey).").msg,
2762 );
2763 }
2764
2752 test "openFailure: --via and --sock say what they know and nothing more" { 2765 test "openFailure: --via and --sock say what they know and nothing more" {
2753 var buf: [open_err_len]u8 = undefined; 2766 var buf: [open_err_len]u8 = undefined;
2754 2767
src/client/handoff.zig
Old New
@@ -65,6 +65,51 @@ pub const ReadLineError = error{
65 /// host`) and narrow enough to sit on a picker row beside a spelling. 65 /// host`) and narrow enough to sit on a picker row beside a spelling.
66 pub const reason_max = 120; 66 pub const reason_max = 120;
67 67
68 /// The small set of outcomes an SSH announce can have for an automatic
69 /// reconnect. A missing diagnostic is deliberately `unknown`: an exit code
70 /// or EOF does not prove that authentication was refused.
71 pub const FailureClass = enum {
72 authentication_refused,
73 unknown,
74 };
75
76 /// Classify only diagnostics whose wording is characteristic of OpenSSH
77 /// refusing authentication. Keep this conservative: in particular, exit
78 /// status 255 and a bare announce EOF are not authentication evidence.
79 pub fn classifyReason(reason: []const u8) FailureClass {
80 const line = std.mem.trim(u8, reason, " \t");
81 const denial = "permission denied (";
82 if (startsWithIgnoreCase(line, denial) and completeDenial(line[denial.len..])) return .authentication_refused;
83 if (std.ascii.indexOfIgnoreCase(line, ": " ++ denial)) |colon| {
84 const prefix = std.mem.trim(u8, line[0..colon], " \t");
85 if (validUserHostPrefix(prefix) and completeDenial(line[colon + 2 + denial.len ..])) return .authentication_refused;
86 }
87 // These server-side messages are only trusted with OpenSSH's framing;
88 // the phrase by itself could have come from a remote banner or command.
89 if (startsWithIgnoreCase(line, "received disconnect") and
90 (std.ascii.indexOfIgnoreCase(line, "too many authentication failures") != null or
91 std.ascii.indexOfIgnoreCase(line, "no more authentication methods to try") != null)) return .authentication_refused;
92 return .unknown;
93 }
94
95 fn validUserHostPrefix(prefix: []const u8) bool {
96 const at = std.mem.indexOfScalar(u8, prefix, '@') orelse return false;
97 if (at == 0 or at + 1 == prefix.len) return false;
98 const user = prefix[0..at];
99 const host = prefix[at + 1 ..];
100 for (user) |c| if (c == ':' or c == ' ' or c == '\t') return false;
101 for (host) |c| if (c == ' ' or c == '\t') return false;
102 return true;
103 }
104
105 fn startsWithIgnoreCase(haystack: []const u8, needle: []const u8) bool {
106 return haystack.len >= needle.len and std.ascii.indexOfIgnoreCase(haystack[0..needle.len], needle) != null;
107 }
108
109 fn completeDenial(methods: []const u8) bool {
110 return methods.len >= 2 and methods[methods.len - 2] == ')' and methods[methods.len - 1] == '.';
111 }
112
68 /// The last COMPLETE line a stderr pipe carried, kept across the pipe's 113 /// The last COMPLETE line a stderr pipe carried, kept across the pipe's
69 /// fragmentation. A struct rather than a buffer the reader appends to: ssh says 114 /// fragmentation. A struct rather than a buffer the reader appends to: ssh says
70 /// several things and dies on the last, and a caller wants one sentence. 115 /// several things and dies on the last, and a caller wants one sentence.
@@ -655,6 +700,29 @@ test "Reason: clear forgets the last line and the partial one behind it" {
655 try std.testing.expectEqualStrings(" a line", r.slice()); 700 try std.testing.expectEqualStrings(" a line", r.slice());
656 } 701 }
657 702
703 test "classifyReason: only conservative OpenSSH authentication diagnostics stop retry" {
704 try std.testing.expectEqual(FailureClass.authentication_refused, classifyReason("Permission denied (publickey,password)."));
705 try std.testing.expectEqual(FailureClass.authentication_refused, classifyReason("Permission denied (keyboard-interactive)."));
706 try std.testing.expectEqual(FailureClass.authentication_refused, classifyReason("xanderle@127.0.0.1: Permission denied (gssapi-with-mic,hostbased)."));
707 try std.testing.expectEqual(FailureClass.authentication_refused, classifyReason("Received disconnect: Too Many Authentication Failures"));
708 try std.testing.expectEqual(FailureClass.unknown, classifyReason("No supported authentication methods available"));
709 try std.testing.expectEqual(FailureClass.unknown, classifyReason("ssh: connect to host box port 22: No route to host"));
710 try std.testing.expectEqual(FailureClass.unknown, classifyReason("banner: authentication failed maintenance notice"));
711 try std.testing.expectEqual(FailureClass.unknown, classifyReason("banner: Too many authentication failures"));
712 try std.testing.expectEqual(FailureClass.unknown, classifyReason("banner: Permission denied (publickey)."));
713 try std.testing.expectEqual(FailureClass.unknown, classifyReason("banner: user@host: Permission denied (publickey)."));
714 try std.testing.expectEqual(FailureClass.unknown, classifyReason(""));
715 // OpenSSH's generic exit status is not enough to identify auth refusal.
716 try std.testing.expectEqual(FailureClass.unknown, classifyReason("255"));
717 }
718
719 test "classifyReason: fragmented stderr keeps a canonical denial terminal" {
720 var reason: Reason = .{};
721 reason.feed("xanderle@127.0.0.1: Permission denied (public");
722 reason.feed("key,password).\n");
723 try std.testing.expectEqual(FailureClass.authentication_refused, classifyReason(reason.slice()));
724 }
725
658 test "announce: format → parse round-trip, with and without the newline" { 726 test "announce: format → parse round-trip, with and without the newline" {
659 var buf: [announce_max_len]u8 = undefined; 727 var buf: [announce_max_len]u8 = undefined;
660 728
src/client/session_pump.zig
Old New
@@ -175,7 +175,7 @@ pub const Pump = struct {
175 if (self.closing.load(.acquire)) return; 175 if (self.closing.load(.acquire)) return;
176 var buf: [1024]u8 = undefined; 176 var buf: [1024]u8 = undefined;
177 const failure = client.openFailure(&buf, self.opts.target, err, dial.reason.slice()); 177 const failure = client.openFailure(&buf, self.opts.target, err, dial.reason.slice());
178 if (first and !(self.opts.retry_initial and retryableOpen(err))) { 178 if (terminalOpenFailure(first, self.opts.retry_initial, err, dial.reason.slice())) {
179 self.publish(.dial_failed, 2, failure.msg); 179 self.publish(.dial_failed, 2, failure.msg);
180 return; 180 return;
181 } 181 }
@@ -386,13 +386,36 @@ pub const Pump = struct {
386 } 386 }
387 }; 387 };
388 388
389 fn retryableOpen(err: anyerror) bool { 389 fn retryableOpen(err: anyerror, reason: []const u8) bool {
390 if (client.handoff.classifyReason(reason) == .authentication_refused) return false;
390 return switch (err) { 391 return switch (err) {
391 error.FileNotFound, error.ConnectionRefused, error.ConnectionTimedOut, error.Timeout, error.NetworkUnreachable, error.HostUnreachable, error.QuicHandshakeFailed, error.UnknownHostName, error.UnterminatedLine => true, 392 error.FileNotFound, error.ConnectionRefused, error.ConnectionTimedOut, error.Timeout, error.NetworkUnreachable, error.HostUnreachable, error.QuicHandshakeFailed, error.UnknownHostName, error.UnterminatedLine => true,
392 else => false, 393 else => false,
393 }; 394 };
394 } 395 }
395 396
397 fn terminalOpenFailure(first: bool, retry_initial: bool, err: anyerror, reason: []const u8) bool {
398 return (first and !(retry_initial and retryableOpen(err, reason))) or
399 (!first and client.handoff.classifyReason(reason) == .authentication_refused);
400 }
401
402 test "retryableOpen: authentication diagnostics are terminal while unknown announce EOF retries" {
403 try std.testing.expect(!retryableOpen(error.UnterminatedLine, "Permission denied (publickey)."));
404 try std.testing.expect(!retryableOpen(error.UnterminatedLine, "Received disconnect from box: 2: Too many authentication failures"));
405 try std.testing.expect(retryableOpen(error.UnterminatedLine, ""));
406 try std.testing.expect(retryableOpen(error.UnterminatedLine, "ssh: connect to host box port 22: No route to host"));
407 // The status code alone does not establish an authentication refusal.
408 try std.testing.expect(retryableOpen(error.UnterminatedLine, "255"));
409 }
410
411 test "reconnect policy: non-auth transport failures retain retry behavior" {
412 try std.testing.expect(!terminalOpenFailure(false, false, error.ConnectionResetByPeer, "connection reset"));
413 try std.testing.expect(!terminalOpenFailure(false, false, error.SystemResources, "resource unavailable"));
414 try std.testing.expect(terminalOpenFailure(false, false, error.UnterminatedLine, "Permission denied (publickey)."));
415 try std.testing.expect(terminalOpenFailure(true, false, error.UnterminatedLine, ""));
416 try std.testing.expect(!terminalOpenFailure(true, true, error.UnterminatedLine, ""));
417 }
418
396 // Only a lost connection earns a redial. Resource exhaustion, poll errors, 419 // Only a lost connection earns a redial. Resource exhaustion, poll errors,
397 // and other local failures must reach entry's failure publication. 420 // and other local failures must reach entry's failure publication.
398 fn connectionFailure(err: anyerror) anyerror!bool { 421 fn connectionFailure(err: anyerror) anyerror!bool {
@@ -917,3 +940,51 @@ test "restored SSH retries an initial announce EOF and joins when the host answe
917 try std.testing.expect(pump.snapshot_ready); 940 try std.testing.expect(pump.snapshot_ready);
918 try std.testing.expectEqual(@as(u64, 37), pump.replica.last_seq); 941 try std.testing.expectEqual(@as(u64, 37), pump.replica.last_seq);
919 } 942 }
943
944 test "restored SSH authentication refusal is terminal and does not retry" {
945 const a = std.testing.allocator;
946 var tmp = std.testing.tmpDir(.{});
947 defer tmp.cleanup();
948 const path = try tmp.dir.realpathAlloc(a, ".");
949 defer a.free(path);
950 const marker = try std.fmt.allocPrint(a, "{s}/runs", .{path});
951 defer a.free(marker);
952 const script = try std.fmt.allocPrint(a, "printf x >> {s}; printf 'Permission denied (publickey).\\n' >&2; exit 255", .{marker});
953 defer a.free(script);
954 const pump = try Pump.start(a, .{ .target = .{ .hand = .{ .host = "fixture", .ssh_argv = &.{ "/bin/sh", "-c", script }, .cache_path = null, .deadline_ms = 200 } }, .session = "native-test", .cols = 11, .rows = 3, .existing_only = true, .retry_initial = true, .open_timeout_ms = 1000 });
955 defer pump.stop();
956 const state = try testPhase(pump, .dial_failed);
957 try std.testing.expectEqualStrings("mux: authentication refused by fixture over ssh: Permission denied (publickey).\n", state.reasonText());
958 const runs = try tmp.dir.readFileAlloc(a, "runs", 16);
959 defer a.free(runs);
960 try std.testing.expectEqual(@as(usize, 1), runs.len);
961 }
962
963 test "reconnecting SSH authentication refusal is terminal after an established pipe" {
964 const a = std.testing.allocator;
965 var tmp = std.testing.tmpDir(.{});
966 defer tmp.cleanup();
967 const path = try tmp.dir.realpathAlloc(a, ".");
968 defer a.free(path);
969 var bytes: std.ArrayList(u8) = .empty;
970 defer bytes.deinit(a);
971 try proto.appendFrame(&bytes, a, .snapshot, &testSnapshot());
972 try tmp.dir.writeFile(.{ .sub_path = "snapshot", .data = bytes.items });
973 const marker = try std.fmt.allocPrint(a, "{s}/runs", .{path});
974 defer a.free(marker);
975 const once = try std.fmt.allocPrint(a, "{s}/once", .{path});
976 defer a.free(once);
977 const release = try std.fmt.allocPrint(a, "{s}/release", .{path});
978 defer a.free(release);
979 const script = try std.fmt.allocPrint(a, "printf x >> {s}; if test ! -e {s}; then touch {s}; printf 'endpoint none\\n'; cat {s}/snapshot; i=0; while test ! -e {s} && test $i -lt 200; do i=$((i+1)); sleep 0.01; done; else printf 'Permission denied (publickey).\\n' >&2; exit 255; fi", .{ marker, once, once, path, release });
980 defer a.free(script);
981 const pump = try Pump.start(a, .{ .target = .{ .hand = .{ .host = "fixture", .ssh_argv = &.{ "/bin/sh", "-c", script }, .cache_path = null, .deadline_ms = 200 } }, .session = "native-test", .cols = 11, .rows = 3, .existing_only = true, .retry_initial = true, .open_timeout_ms = 1000 });
982 defer pump.stop();
983 _ = try testPhase(pump, .attached);
984 try tmp.dir.writeFile(.{ .sub_path = "release", .data = "done" });
985 const state = try testPhase(pump, .dial_failed);
986 try std.testing.expectEqualStrings("mux: authentication refused by fixture over ssh: Permission denied (publickey).\n", state.reasonText());
987 const runs = try tmp.dir.readFileAlloc(a, "runs", 16);
988 defer a.free(runs);
989 try std.testing.expectEqual(@as(usize, 2), runs.len);
990 }
test/native_ssh.py
Old New
@@ -0,0 +1,251 @@
1 #!/usr/bin/env python3
2 """Optional Linux native acceptance with an owned loopback OpenSSH server.
3
4 Run with release mux and muxg paths. Requires sshd, ssh, and ssh-keygen; all keys,
5 configuration, daemon state, sockets, and processes belong to this fixture.
6 The SSH adapter supplies fixture options and records attempts, never outcomes.
7 """
8 import os
9 from pathlib import Path
10 import pwd
11 import shlex
12 import shutil
13 import signal
14 import socket
15 import subprocess
16 import sys
17 import time
18
19 sys.dont_write_bytecode = True
20 from native_lifecycle import LifecycleRig, identity
21 from native_picker import create, end, sessions
22 from native_tiling import eventually, require
23
24
25 class SSHFixture:
26 target = 'mux-native-ssh-fixture'
27
28 def __init__(self, rig):
29 self.rig = rig
30 self.root = rig.root / 'openssh'
31 self.root.mkdir(mode=0o700)
32 self.server = None
33 self.ssh = shutil.which('ssh')
34 self.sshd = shutil.which('sshd')
35 require(self.ssh and self.sshd and shutil.which('ssh-keygen'),
36 'real OpenSSH client, server, and keygen are required')
37 self.attempt_log = self.root / 'attempts'
38 self.attempt_log.touch()
39 self.sock, _ = rig.daemon('ssh-daemon')
40 for key in ('host', 'good', 'bad'):
41 subprocess.run(['ssh-keygen', '-q', '-t', 'ed25519', '-N', '',
42 '-f', str(self.root / key)], check=True)
43 (self.root / 'authorized_keys').write_bytes((self.root / 'good.pub').read_bytes())
44 self.credentials(True)
45 with socket.socket() as reserve:
46 reserve.bind(('127.0.0.1', 0))
47 self.port = reserve.getsockname()[1]
48 user = pwd.getpwuid(os.getuid()).pw_name
49 config = self.root / 'sshd_config'
50 config.write_text(f'''ListenAddress 127.0.0.1
51 Port {self.port}
52 HostKey {self.root}/host
53 PidFile {self.root}/sshd.pid
54 AuthorizedKeysFile {self.root}/authorized_keys
55 StrictModes no
56 PasswordAuthentication no
57 KbdInteractiveAuthentication no
58 UsePAM no
59 AllowUsers {user}
60 LogLevel VERBOSE
61 ''')
62 subprocess.run([self.sshd, '-t', '-f', str(config)], check=True)
63 host_key = (self.root / 'host.pub').read_text().split()
64 (self.root / 'known_hosts').write_text(
65 f'[127.0.0.1]:{self.port} {host_key[0]} {host_key[1]}\n')
66 options = ['-F', '/dev/null', '-p', str(self.port), '-i', str(self.root / 'identity'),
67 '-o', 'IdentitiesOnly=yes', '-o', 'IdentityAgent=none',
68 '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=2',
69 '-o', 'StrictHostKeyChecking=yes',
70 '-o', 'UserKnownHostsFile=' + str(self.root / 'known_hosts'),
71 '-o', 'GlobalKnownHostsFile=/dev/null']
72 self.remote_bin = self.root / 'remote-bin'
73 self.remote_bin.mkdir()
74 env = {key: str(rig.root / 'ssh-daemon-state' / key) for key in
75 ('XDG_STATE_HOME', 'XDG_RUNTIME_DIR', 'XDG_CONFIG_HOME', 'XDG_CACHE_HOME')}
76 env.update(SHELL='/bin/sh', MUX_KEY_FILE=str(self.root / 'absent-quic-key'),
77 MUX_SOCK='', MUX_SESSION='')
78 mux_wrapper = self.remote_bin / 'mux'
79 mux_wrapper.write_text('#!/bin/sh\nexec env ' +
80 shlex.join([key + '=' + value for key, value in env.items()]) +
81 ' ' + shlex.quote(rig.mux) + ' "$@" --sock ' +
82 shlex.quote(self.sock) + '\n')
83 mux_wrapper.chmod(0o700)
84 adapter_bin = self.root / 'client-bin'
85 adapter_bin.mkdir()
86 adapter = adapter_bin / 'ssh'
87 prefix = 'export PATH=' + shlex.quote(str(self.remote_bin)) + ':"$PATH"; '
88 adapter.write_text('#!' + sys.executable + '\nimport os,sys,time\n' +
89 'args=sys.argv[1:]\n' +
90 'i=args.index(' + repr(self.target) + ')\n' +
91 'with open(' + repr(str(self.attempt_log)) + ',"a") as f:\n' +
92 ' f.write(str(os.getpid())+" "+str(time.monotonic())+"\\n")\n' +
93 'command=' + repr(prefix) + '+" ".join(args[i+1:])\n' +
94 'os.execv(' + repr(self.ssh) + ',' + repr([self.ssh, *options]) +
95 '+args[:i]+[' + repr(user + '@127.0.0.1') + ',command])\n')
96 adapter.chmod(0o700)
97 rig.env['PATH'] = str(adapter_bin) + os.pathsep + rig.env.get('PATH', '')
98 self.start()
99
100 def credentials(self, valid):
101 shutil.copyfile(self.root / ('good' if valid else 'bad'), self.root / 'identity')
102 (self.root / 'identity').chmod(0o600)
103
104 def attempts(self):
105 return [int(line.split()[0]) for line in self.attempt_log.read_text().splitlines()]
106
107 def start(self):
108 self.server = self.rig.spawn([self.sshd, '-D', '-e', '-f',
109 str(self.root / 'sshd_config')],
110 'sshd-' + str(time.time_ns()))
111 def ready():
112 require(self.server.poll() is None, 'fixture sshd exited; inspect sshd log')
113 with socket.socket() as probe:
114 return probe.connect_ex(('127.0.0.1', self.port)) == 0
115 eventually(ready, 'fixture sshd did not listen')
116
117 def stop(self):
118 if self.server is not None and self.server.poll() is None:
119 self.server.terminate()
120 try:
121 self.server.wait(timeout=3)
122 except subprocess.TimeoutExpired:
123 self.server.kill()
124 self.server.wait(timeout=3)
125
126
127 def run(r, ssh, show=lambda label: None):
128 local, _ = r.daemon('neighbour')
129 create(local, 'neighbour')
130 create(ssh.sock, 'remote')
131 r.catalogue(['--sock ' + local, ssh.target])
132 r.launch_gui([], 'initial', attached=False)
133 r.choose('--sock ' + local)
134 r.picker('sessions')
135 r.choose('neighbour')
136 r.wait_state(lambda s: len(s['panes']) == 1 and s['panes'][0]['phase'] == 'attached')
137 r.host(ssh.target, 'v')
138 r.choose('remote')
139 state = r.wait_state(lambda s: len(s['panes']) == 2 and
140 all(p['phase'] == 'attached' for p in s['panes']))
141 local_id, remote_id = [p['id'] for p in state['panes']]
142 r.mark(remote_id, ssh.sock, 'remote', 'REAL-SSH-SESSION')
143 before = identity(r.state())
144 r.quit()
145 ssh.credentials(False)
146 count = len(ssh.attempts())
147 r.launch_gui([], 'restore-auth-refused', attached=False)
148 state = r.wait_state(lambda s: any(p['id'] == remote_id and p['phase'] == 'dial_failed'
149 for p in s['panes']))
150 require(identity(state) == before, 'authentication refusal changed saved pane identity')
151 r.chord('p')
152 state = r.wait_state(lambda s: s['recovery'] is not None)
153 require('Permission denied' in state['recovery']['notice'], 'SSH refusal reason missing from recovery')
154 r.pixels()
155 show('Authentication refused: pane retained; explicit Retry available')
156 time.sleep(1.4)
157 require(len(ssh.attempts()) == count + 1, 'authentication refusal automatically retried on restore')
158 r.key('escape')
159 r.mark(local_id, local, 'neighbour', 'NEIGHBOUR-STILL-USABLE')
160 require('remote' in sessions(ssh.sock), 'authentication failure ended the remote session')
161 r.ok('real SSH auth refusal stops restore retries, retains identity/reason, and isolates neighbour')
162
163 ssh.credentials(True)
164 count = len(ssh.attempts())
165 time.sleep(.5)
166 require(len(ssh.attempts()) == count, 'correcting credentials resumed without explicit Retry')
167 r.focus(remote_id)
168 r.recover('Retry')
169 r.wait_state(lambda s: all(p['phase'] == 'attached' for p in s['panes']))
170 r.mark(remote_id, ssh.sock, 'remote', 'EXPLICIT-RETRY-RECOVERED')
171 show('Corrected key + Retry: existing SSH session resumes')
172 r.ok('explicit Retry with corrected credentials rejoins the existing real SSH session')
173
174 ssh.credentials(False)
175 count = len(ssh.attempts())
176 attached_ssh = ssh.attempts()[-1]
177 require(Path(f'/proc/{attached_ssh}/exe').resolve() == Path(ssh.ssh).resolve(),
178 'recorded attachment PID is not the actual OpenSSH client')
179 os.kill(attached_ssh, signal.SIGTERM)
180 r.wait_state(lambda s: any(p['id'] == remote_id and p['phase'] == 'dial_failed'
181 for p in s['panes']))
182 time.sleep(1.4)
183 require(len(ssh.attempts()) == count + 1, 'authentication refusal automatically retried after disconnect')
184 r.chord('p')
185 require('Permission denied' in r.wait_state(lambda s: s['recovery'] is not None)['recovery']['notice'],
186 'post-attachment auth refusal lost reason')
187 show('Connection lost, key refused: automatic reconnect stops')
188 r.key('escape')
189 ssh.credentials(True)
190 r.recover('Retry')
191 r.wait_state(lambda s: all(p['phase'] == 'attached' for p in s['panes']))
192 r.ok('real SSH authentication refusal also stops retries after a previously attached connection')
193
194 r.quit()
195 ssh.stop()
196 count = len(ssh.attempts())
197 r.launch_gui([], 'restore-unavailable', attached=False)
198 r.wait_state(lambda s: any(p['id'] == remote_id and p['phase'] == 'reconnecting'
199 for p in s['panes']))
200 eventually(lambda: len(ssh.attempts()) >= count + 2, 'unavailable SSH host did not retry')
201 require(len(ssh.attempts()) <= count + 5, 'unavailable host retries are unbounded')
202 times = [float(line.split()[1]) for line in ssh.attempt_log.read_text().splitlines()[count:]]
203 require(times[1] - times[0] >= .15, 'unavailable host skipped reconnect backoff')
204 r.mark(local_id, local, 'neighbour', 'OFFLINE-NEIGHBOUR-USABLE')
205 show('SSH host unavailable: bounded reconnect; neighbour remains usable')
206 ssh.start()
207 r.wait_state(lambda s: all(p['phase'] == 'attached' for p in s['panes']), seconds=10)
208 r.mark(remote_id, ssh.sock, 'remote', 'HOST-RETURNED-AUTOMATICALLY')
209 show('SSH host returns: session reconnects automatically')
210 r.ok('unavailable real SSH host retries and recovers automatically')
211
212 r.focus(remote_id)
213 r.quit()
214 end(ssh.sock, 'remote')
215 r.launch_gui([], 'missing-session', attached=False)
216 r.wait_state(lambda s: any(p['id'] == remote_id and p['phase'] == 'refused'
217 for p in s['panes']))
218 require('remote' not in sessions(ssh.sock), 'restore recreated a missing session')
219 show('Missing remote session stays missing: restore never creates it')
220 r.quit()
221 ssh.stop()
222 r.launch_gui([], 'shutdown-offline', attached=False)
223 r.wait_state(lambda s: any(p['phase'] == 'reconnecting' for p in s['panes']))
224 start = time.monotonic()
225 r.quit()
226 require(time.monotonic() - start < 2, 'shutdown during SSH retries exceeded two seconds')
227 r.assert_cli_untouched()
228 r.ok('SSH restoration stays join-only; shutdown during reconnect is bounded; CLI layout untouched')
229
230
231 def main():
232 require(len(sys.argv) == 3, 'usage: native_ssh.py RELEASE_MUX RELEASE_MUXG')
233 r = LifecycleRig(*sys.argv[1:])
234 ssh = None
235 try:
236 ssh = SSHFixture(r)
237 run(r, ssh)
238 except BaseException:
239 r.failure_artifacts()
240 raise
241 finally:
242 try:
243 if ssh:
244 ssh.stop()
245 finally:
246 r.close()
247 print('Real SSH native checks passed; owned fixtures stopped. Evidence:', r.root, flush=True)
248
249
250 if __name__ == '__main__':
251 main()