quic: a multi-homed daemon replies from the wrong source address, so every dial silently falls back to ssh #78

New issue

Open

opened 2026-09-04 12:38:27 +00:00 by a73x · 0 comments

a73x commented

2026-09-04 12:38:27 +00:00

(Migrated from git.a73x.sh)

[claude 2026-09-04] Found while investigating eight long-lived ssh ... mux d endpoint
processes on a wall (they turned out to be correct use_pipe transports; the real
defect was two hops upstream).

Symptom

Every QUIC dial from 192.168.0.190 to the daemon on 192.168.0.207 times out. The wall
therefore runs all four of that host's panes over the ssh pipe, permanently, with no
user-visible error. handoff.zig:547 (afterAnnounce) is working as designed here: the
warm dial failed, ssh re-announced the SAME endpoint, so it refuses to re-dial
coordinates it just proved dead and pins the pane to use_pipe for that pane's life.
There is no re-promotion, so a transient-looking failure is permanent.

Root cause

The daemon's box is multi-homed on ONE subnet:

wlp0s20f3  192.168.0.207/24   default metric 600   (wifi)
enp63s0    192.168.0.208/24   default metric 100   (ethernet)

The client dials the wifi address. The daemon's QUIC listener binds the wildcard
0.0.0.0:45464, so when it replies, the kernel picks the route to the client
(ip route get 192.168.0.190 -> dev enp63s0 src 192.168.0.208) and stamps the
ethernet source address on the reply. tcpdump on the daemon's box, during a dial:

wlp0s20f3 In  IP 192.168.0.190.43000 > 192.168.0.207.45464: UDP, length 1200
enp63s0   Out IP 192.168.0.208.45464 > 192.168.0.190.43000: UDP, length 101

The client dialled .207 and the answer arrives from .208, so it discards it. Silence is
indistinguishable from a blackholed port, which is exactly the case handoff.zig's
attach-budget comment describes.

src/server/quic_server.zig carries local_storage/local_len per Connection, but
nothing in it (or src/quic.zig) uses recvmsg/IP_PKTINFO/cmsg — it is plain
recvfrom/sendto, so the listener cannot know which of its addresses a datagram was
sent to and cannot pin the reply's source to it.

Reproduction

On a box with two interfaces on the same subnet, where the dialled address is NOT the
one the return route prefers:

mux a status --quic <wifi-addr>:<port> --key K --session 0   # Timeout
mux a status --quic <eth-addr>:<port>  --key K --session 0   # works, same key

Evidence that ruled everything else out

Endpoint cache correct and fresh; announced key matches the daemon's key byte-for-byte.
UDP reaches the daemon: 197 of 200 x 1200B arrived (/proc/net/snmp InDatagrams).
Path MTU clean to 1472B.
clients=8 of max_clients=32 — not slot exhaustion.
The daemon answers the identical dial from ITSELF (loopback and its own LAN address).
The same mux client dials another host over tailscale successfully, so the QUIC client
is healthy.
A plain nc -u -l on the client box DOES receive datagrams from that daemon — an
unconnected socket accepts any source, which is why a naive reachability test passes
while QUIC fails. Worth knowing for whoever writes the test.

Suggested fix

Set IP_PKTINFO / IPV6_RECVPKTINFO on the listener socket, read the destination
address from the cmsg in recvmsg, and pass it back as the source in sendmsg (the
standard multi-homed QUIC server fix). local_storage/local_len is already the right
place to keep it.

Worth considering separately: the failure mode is a silent, permanent downgrade to a
slower transport. Even with the fix, a pane pinned to use_pipe might deserve a visible
notice, since today nothing on screen says the wall gave up on QUIC.

Migrated from git-collab issue df3935c1e00381688ce26de4f6ac3b724a38a882.

Original author: a73x dev@a73x.sh.

[claude 2026-09-04] Found while investigating eight long-lived `ssh ... mux d endpoint` processes on a wall (they turned out to be correct `use_pipe` transports; the real defect was two hops upstream). ## Symptom Every QUIC dial from 192.168.0.190 to the daemon on 192.168.0.207 times out. The wall therefore runs all four of that host's panes over the ssh pipe, permanently, with no user-visible error. `handoff.zig:547` (`afterAnnounce`) is working as designed here: the warm dial failed, ssh re-announced the SAME endpoint, so it refuses to re-dial coordinates it just proved dead and pins the pane to `use_pipe` for that pane's life. There is no re-promotion, so a transient-looking failure is permanent. ## Root cause The daemon's box is multi-homed on ONE subnet: wlp0s20f3 192.168.0.207/24 default metric 600 (wifi) enp63s0 192.168.0.208/24 default metric 100 (ethernet) The client dials the wifi address. The daemon's QUIC listener binds the wildcard `0.0.0.0:45464`, so when it replies, the kernel picks the route to the client (`ip route get 192.168.0.190` -> `dev enp63s0 src 192.168.0.208`) and stamps the ethernet source address on the reply. tcpdump on the daemon's box, during a dial: wlp0s20f3 In IP 192.168.0.190.43000 > 192.168.0.207.45464: UDP, length 1200 enp63s0 Out IP 192.168.0.208.45464 > 192.168.0.190.43000: UDP, length 101 The client dialled .207 and the answer arrives from .208, so it discards it. Silence is indistinguishable from a blackholed port, which is exactly the case `handoff.zig`'s attach-budget comment describes. `src/server/quic_server.zig` carries `local_storage`/`local_len` per Connection, but nothing in it (or `src/quic.zig`) uses `recvmsg`/`IP_PKTINFO`/`cmsg` — it is plain `recvfrom`/`sendto`, so the listener cannot know which of its addresses a datagram was sent to and cannot pin the reply's source to it. ## Reproduction On a box with two interfaces on the same subnet, where the dialled address is NOT the one the return route prefers: mux a status --quic <wifi-addr>:<port> --key K --session 0 # Timeout mux a status --quic <eth-addr>:<port> --key K --session 0 # works, same key ## Evidence that ruled everything else out - Endpoint cache correct and fresh; announced key matches the daemon's key byte-for-byte. - UDP reaches the daemon: 197 of 200 x 1200B arrived (`/proc/net/snmp` InDatagrams). - Path MTU clean to 1472B. - `clients=8` of `max_clients=32` — not slot exhaustion. - The daemon answers the identical dial from ITSELF (loopback and its own LAN address). - The same mux client dials another host over tailscale successfully, so the QUIC client is healthy. - A plain `nc -u -l` on the client box DOES receive datagrams from that daemon — an unconnected socket accepts any source, which is why a naive reachability test passes while QUIC fails. Worth knowing for whoever writes the test. ## Suggested fix Set `IP_PKTINFO` / `IPV6_RECVPKTINFO` on the listener socket, read the destination address from the cmsg in `recvmsg`, and pass it back as the source in `sendmsg` (the standard multi-homed QUIC server fix). `local_storage`/`local_len` is already the right place to keep it. Worth considering separately: the failure mode is a silent, permanent downgrade to a slower transport. Even with the fix, a pane pinned to `use_pipe` might deserve a visible notice, since today nothing on screen says the wall gave up on QUIC. --- Migrated from git-collab issue `df3935c1e00381688ce26de4f6ac3b724a38a882`. Original author: a73x <dev@a73x.sh>.