quic: a multi-homed daemon replies from the wrong source address, so every dial silently falls back to ssh

open   by a73x

Labels: backlog

[claude 2026-09-04] Found while investigating eight long-lived `ssh ... mux d endpoint`
processes on a wall (they turned out to be correct `use_pipe` transports; the real
defect was two hops upstream).

## Symptom

Every QUIC dial from 192.168.0.190 to the daemon on 192.168.0.207 times out. The wall
therefore runs all four of that host's panes over the ssh pipe, permanently, with no
user-visible error. `handoff.zig:547` (`afterAnnounce`) is working as designed here: the
warm dial failed, ssh re-announced the SAME endpoint, so it refuses to re-dial
coordinates it just proved dead and pins the pane to `use_pipe` for that pane's life.
There is no re-promotion, so a transient-looking failure is permanent.

## Root cause

The daemon's box is multi-homed on ONE subnet:

    wlp0s20f3  192.168.0.207/24   default metric 600   (wifi)
    enp63s0    192.168.0.208/24   default metric 100   (ethernet)

The client dials the wifi address. The daemon's QUIC listener binds the wildcard
`0.0.0.0:45464`, so when it replies, the kernel picks the route to the client
(`ip route get 192.168.0.190` -> `dev enp63s0 src 192.168.0.208`) and stamps the
ethernet source address on the reply. tcpdump on the daemon's box, during a dial:

    wlp0s20f3 In  IP 192.168.0.190.43000 > 192.168.0.207.45464: UDP, length 1200
    enp63s0   Out IP 192.168.0.208.45464 > 192.168.0.190.43000: UDP, length 101

The client dialled .207 and the answer arrives from .208, so it discards it. Silence is
indistinguishable from a blackholed port, which is exactly the case `handoff.zig`'s
attach-budget comment describes.

`src/server/quic_server.zig` carries `local_storage`/`local_len` per Connection, but
nothing in it (or `src/quic.zig`) uses `recvmsg`/`IP_PKTINFO`/`cmsg` — it is plain
`recvfrom`/`sendto`, so the listener cannot know which of its addresses a datagram was
sent to and cannot pin the reply's source to it.

## Reproduction

On a box with two interfaces on the same subnet, where the dialled address is NOT the
one the return route prefers:

    mux a status --quic <wifi-addr>:<port> --key K --session 0   # Timeout
    mux a status --quic <eth-addr>:<port>  --key K --session 0   # works, same key

## Evidence that ruled everything else out

- Endpoint cache correct and fresh; announced key matches the daemon's key byte-for-byte.
- UDP reaches the daemon: 197 of 200 x 1200B arrived (`/proc/net/snmp` InDatagrams).
- Path MTU clean to 1472B.
- `clients=8` of `max_clients=32` — not slot exhaustion.
- The daemon answers the identical dial from ITSELF (loopback and its own LAN address).
- The same mux client dials another host over tailscale successfully, so the QUIC client
  is healthy.
- A plain `nc -u -l` on the client box DOES receive datagrams from that daemon — an
  unconnected socket accepts any source, which is why a naive reachability test passes
  while QUIC fails. Worth knowing for whoever writes the test.

## Suggested fix

Set `IP_PKTINFO` / `IPV6_RECVPKTINFO` on the listener socket, read the destination
address from the cmsg in `recvmsg`, and pass it back as the source in `sendmsg` (the
standard multi-homed QUIC server fix). `local_storage`/`local_len` is already the right
place to keep it.

Worth considering separately: the failure mode is a silent, permanent downgrade to a
slower transport. Even with the fix, a pane pinned to `use_pipe` might deserve a visible
notice, since today nothing on screen says the wall gave up on QUIC.