a73x

5cb25cf4

spike: QUIC stack chosen — ngtcp2 + wolfSSL builds static-musl under the pinned toolchain

a73x   2026-08-08 14:08

Commit message
spike: QUIC stack chosen — ngtcp2 + wolfSSL builds static-musl under the pinned toolchain
.gitignore
Old New
@@ -1,2 +1,5 @@
1 zig-out/ 1 zig-out/
2 .zig-cache/ 2 .zig-cache/
3
4 # TLS keylog dropped by wolfSSL debug builds — never commit session secrets.
5 sslkeylog.log
docs/decisions.md
Old New
@@ -575,12 +575,29 @@ path, and it is transport work, which is the verdict restated.
575 defence in depth against those two guards ever being collapsed into one, 575 defence in depth against those two guards ever being collapsed into one,
576 and it is recorded here so the next reviewer does not repeat the 576 and it is recorded here so the next reviewer does not repeat the
577 "can't mutate it, is it dead?" cycle. 577 "can't mutate it, is it dead?" cycle.
578 - **`pkill -f` is banned in this suite.** The client's own argv contains 578 - **Never select processes by command-line pattern — anywhere in this
579 the entire `--via` command string, so any pattern matching the proxy also 579 project, scripts or ad-hoc operations alike.** Not a test-suite
580 matches the client under test — which killed its own subject once here 580 convention: a standing rule, promoted after a fourth sighting. The reason
581 before the rule existed. Select by pid: `comm == muxd` plus argv, or the 581 is structural rather than incidental — this project's command lines
582 parent-pid of the client. The remote-side equivalent is the existing 582 *contain each other*. `mux --via "ssh host muxd proxy"` carries the proxy's
583 bracketed-pattern trick in `wan.sh`. 583 command line inside its own argv; an ssh invocation carries the entire
584 remote script, including whatever pattern that script is about to match.
585 So a pattern that describes the intended victim also describes the process
586 doing the killing, or the one under test. The four sightings:
587 1. e2e Scenario A — `pkill -f "muxd proxy"` would have matched the client
588 under test; caught while writing it, and the reason `proxy_pid` selects
589 by `comm` + argv.
590 2. M7 manual smoke test — `pkill -f "muxd proxy --sock …"` SIGTERM'd the
591 very client being tested (exit 143), which briefly looked like a
592 product bug.
593 3. M8 spike cleanup on the box — `pkill -f "quicecho-$TAG server"` matched
594 the ssh command line carrying that same text and killed the cleanup
595 script before it ran `tc qdisc del`, leaving netem installed on a
596 shared machine until it was noticed.
597 4. Independently hit in review.
598 The replacement is always the same: select by `comm` plus a pid, or by
599 parent pid. The remote-side equivalent is `wan.sh`'s bracketed-pattern
600 trick, which works for the same reason — it breaks the self-match.
584 - **The never-established gate has e2e-only coverage.** The property is 601 - **The never-established gate has e2e-only coverage.** The property is
585 process-level (a client that must exit rather than loop, with no tty to 602 process-level (a client that must exit rather than loop, with no tty to
586 rescue it), so it cannot be unit-tested honestly. `make e2e` is therefore 603 rescue it), so it cannot be unit-tested honestly. `make e2e` is therefore
spike/quic/.gitignore
Old New
@@ -0,0 +1,3 @@
1 # Fetched sources and build trees: pinned by build.sh, never committed.
2 work/
3 work-*/
spike/quic/README.md
Old New
@@ -0,0 +1,216 @@
1 # M8 Task 1 spike — can our toolchain ship a QUIC stack?
2
3 **Throwaway.** Nothing here is wired into `make test`, `make build`, `make e2e`
4 or `make bench`; the main gates are untouched by construction. This directory
5 exists to answer one question and to record the answer.
6
7 ## Verdict
8
9 **Candidate (a), ngtcp2 1.25.0 + wolfSSL 5.9.2-stable, clears the bar.**
10 Candidate (b) (picoquic + picotls) was not reached and not needed.
11
12 Every requirement the plan set for the spike, and what happened:
13
14 | Bar | Result |
15 |---|---|
16 | Built by our **pinned Zig 0.15.2** (as `zig cc`/`ar`/`ranlib`) | yes — no gcc, no system clang |
17 | **Native** build | yes |
18 | **x86_64-linux-musl static** build | yes — `ELF 64-bit LSB executable, statically linked` |
19 | Reproducible from **pinned sources** | yes — `build.sh`, URLs + sha256, fetch-not-vendor (see below) |
20 | **TLS 1.3 external PSK** | yes — no certificates, no CA, anywhere in the handshake |
21 | **fd-drivable** (no imposed event loop) | yes — the echo pair is one `poll()` over one UDP fd |
22 | Working **echo pair** | yes — `quicecho.c`, both roles, verified locally and on the box |
23
24 `build.sh` reruns the whole thing from nothing in a few minutes.
25
26 ## What was NOT proven, and why — read this before quoting the numbers
27
28 **The plan's "measure over the real WAN" step could not be run, for an
29 environmental reason that matters to the whole milestone: the box has no
30 inbound UDP path.** It sits behind NAT on `10.78.3.3/24`, reachable only by
31 ssh through a jump host. Verified rather than assumed: a UDP listener on the
32 box, a packet sent to its egress address `86.153.165.146`, and nothing
33 arrived.
34
35 So the "client here, server there" topology QUIC needs is impossible against
36 this box today, and **that is a finding about the milestone, not about the
37 spike**. The plan's out-of-scope list says "NAT traversal / hole punching
38 (VMs have reachable addresses; banked)" — for the one VM we have, that
39 assumption is false. A direct QUIC transport reaches a VM only if something
40 can send UDP *to* it. This needs a controller decision before Task 2:
41 reachable VMs, a relay, or NAT traversal stops being out of scope.
42
43 What was measured instead, and labelled as such below: the static binary
44 running **on the box**, with the real link's round trip (16.5ms, measured
45 that same day) emulated on the box's loopback via `netem`. Handshake cost in
46 *round trips* is exactly what that measures well, and round trips are what
47 the ssh comparison is about. Loss, queueing, and PMTU on the real path are
48 **not** exercised, and no number here should be quoted as if they were.
49
50 ## Numbers
51
52 Emulated 16.55ms RTT (`netem delay 8250us` on the box's `lo`; `ping` confirmed
53 16.547/16.552/16.561ms), static musl binary, 6 reps:
54
55 ```
56 #REP 0 handshake_ms=18.50 echo_ms=35.51 resumed=0
57 #REP 1 handshake_ms=18.65 echo_ms=35.68 resumed=1
58 #REP 5 handshake_ms=18.48 echo_ms=35.53 resumed=1
59 ```
60
61 - **Cold handshake: 18.5ms ≈ 1.1 × RTT.** A TLS 1.3 external-PSK handshake is
62 one round trip, and that is what the wire shows.
63 - **Handshake → first echoed byte: 35.5ms ≈ 2.1 × RTT.**
64 - **Resumption bought nothing (18.5 cold vs 18.5 resumed), and that is
65 expected, not a defect.** With an external PSK the *cold* handshake is
66 already 1-RTT; there is no certificate exchange to skip. The only thing
67 left to win is 0-RTT early data, which this spike did not wire up. Anyone
68 planning Task 3 around "resumption makes reconnects cheap" should read this
69 line twice: the win is already in the cold path, and the next win requires
70 0-RTT specifically.
71
72 **Against ssh, from the same day's `wan.sh` run on the same link** (baseline
73 RTT 16.4-16.8ms): ssh `viafloor` — channel open + remote exec, before one
74 protocol byte moves — measured **36.4-38.1ms, or 2.2-2.3 × RTT**. QUIC's
75 entire cold handshake is **1.1 × RTT**, and QUIC handshake-to-echoed-byte
76 (2.1 × RTT) is about what ssh spends on channel setup *alone*. That is the
77 milestone's thesis surviving its first contact with numbers — with the
78 caveat above about emulated versus real path.
79
80 Local loopback (~0 RTT), for the CPU-cost floor: cold 1.3-1.6ms, resumed
81 0.8-1.0ms.
82
83 ## Integration assessment (plan Step 4)
84
85 **How it wants to be driven.** ngtcp2 imposes no event loop and owns no
86 socket — it is a state machine over buffers. The daemon integration is the
87 shape muxd already has:
88
89 - one UDP fd added to the existing `poll()` set; readable → `recvfrom` →
90 `ngtcp2_conn_read_pkt(conn, path, pi, buf, len, now)`;
91 - after *any* event, drain egress: `ngtcp2_conn_writev_stream(...)` in a loop
92 until it returns 0, `sendto` each packet;
93 - one timer: `ngtcp2_conn_get_expiry(conn)` gives an absolute deadline, which
94 becomes the `poll()` timeout (`pumpOnce` already takes one); on expiry,
95 `ngtcp2_conn_handle_expiry(conn, now)` then drain egress again.
96
97 No timerfd is required — the existing poll timeout is enough, which is
98 convenient because `pumpOnce(timeout_ms)` is already the daemon's shape. Each
99 connection is one `ngtcp2_conn*` plus one wolfSSL object; a client slot binds
100 to one bidi stream id.
101
102 **Task 2 requirements this spike does NOT demonstrate.** Named here because
103 the echo pair's success is misleading about each of them — it exchanges a
104 few dozen bytes per connection and then exits, which is precisely the
105 workload under which all four stay invisible:
106
107 1. **Flow control is mandatory and its absence looks like a hang.** ngtcp2
108 does not self-extend windows: as data is consumed the receiver must call
109 `ngtcp2_conn_extend_max_stream_offset` (stream level — the header says so
110 explicitly) *and* `ngtcp2_conn_extend_max_offset` (connection level).
111 Omit them and the peer stalls forever once it has sent the initial window
112 — on the order of 64KB, which a muxd session of snapshots plus deltas
113 reaches in seconds. It presents as a silent mid-session freeze with no
114 error anywhere, which is about the worst failure shape available.
115 `quicecho.c` never calls either, and never sends enough to notice.
116 2. **The daemon is fd-centric and a QUIC connection is not an fd.**
117 `clients[i].fd` feeds the poll array directly, and `queueFrame` /
118 `flushClient` / `drainPending` all reach `std.posix.send` on that fd. A
119 QUIC client shares ONE UDP fd with every other QUIC client and is
120 identified by connection ID, not by descriptor. Task 2 therefore needs a
121 send/recv indirection over "client i" — a small tagged union or vtable at
122 the three or four call sites that touch `.fd` — before a QUIC connection
123 can present as an ordinary client slot. Sized work, not a line change,
124 and worth doing as its own step so the socket path stays provably intact.
125 3. **`get_expiry` becomes a minimum across connections.** One connection's
126 deadline is the poll timeout; N connections means folding
127 `ngtcp2_conn_get_expiry` over all of them and taking the earliest, then
128 servicing every connection whose deadline has passed. The spike has
129 exactly one connection and so never had to.
130 4. **Retry / address validation.** On a public UDP port a server that
131 answers unvalidated Initials is an amplification reflector.
132 `quicecho.c` deliberately has no Retry or token path; the daemon needs
133 one. This moves from "not in the spike" to a Task 2 requirement.
134
135 **PSK API.** wolfSSL TLS 1.3 external PSK is two callbacks, one per side:
136 `wolfSSL_CTX_set_psk_client_tls13_callback` (fills identity, key, and picks
137 the ciphersuite) and `wolfSSL_CTX_set_psk_server_tls13_callback` (maps
138 identity → key). Both are ordinary C function pointers with no global state
139 beyond what we give them. Note the server callback's last parameter is
140 `const char **ciphersuite` (out), not `const char *` — the compiler catches
141 it, but the wolfSSL headers carry both shapes for different callback
142 families and it is easy to pick the wrong one. `wolfSSL_CTX_set_cipher_list`
143 pins the suite. No certificate calls are needed on either side: possession of
144 the key *is* the authentication, which is exactly the deploy model.
145
146 **Stream API.** One bidi stream carries everything:
147 `ngtcp2_conn_open_bidi_stream` (client, after
148 `extend_max_local_streams_bidi`), then data goes out as `ngtcp2_vec`s through
149 `writev_stream` and arrives in the `recv_stream_data` callback as a plain
150 `(data, datalen)`. That is byte-stream semantics — the existing frame
151 protocol needs no change to ride it, which is the "zero protocol changes"
152 requirement met in the API rather than by hope.
153
154 **Binary size.** Static musl `quicecho`: 8.7MB unstripped, **1.68MB
155 stripped** — that is the whole QUIC + TLS stack plus a small program. Today's
156 `muxd` static musl is 15.5MB unstripped. So QUIC costs on the order of
157 **+1.7MB stripped**, against an artifact that is already 15MB because of
158 ghostty-vt.
159
160 ## Gotchas Task 2 will hit (both cost a link failure here)
161
162 1. **`zig cc` instruments C with UBSan by default.** A static musl link then
163 fails on undefined `__ubsan_handle_*`. Fix: `-fno-sanitize=undefined` (or
164 accept the runtime). This will bite again the moment these sources are
165 compiled via `addCSourceFiles` in `build.zig`.
166 2. **wolfSSL omits AES-ECB by default**, and ngtcp2's wolfSSL backend needs
167 it for QUIC header protection (undefined `wolfSSL_EVP_aes_128_ecb`). Fix:
168 `-DWOLFSSL_AESECB=yes`.
169 3. **Build ngtcp2's library only** (`ENABLE_LIB_ONLY`), not its examples.
170 The examples are what want brotli, and cmake would otherwise resolve it
171 to the host's shared libraries and break the static musl link. An earlier
172 version of this file credited the fix to
173 `CMAKE_DISABLE_FIND_PACKAGE_Libbrotli*`; those flags were inert —
174 ngtcp2's `find_package(Libbrotli*)` calls sit inside
175 `if(NOT ENABLE_LIB_ONLY)`, so with the library-only build they never run,
176 and cmake duly reported the flags as unused-cli. The lesson that carries
177 into Task 2 is the general one, and it survives the move away from cmake:
178 **compile the library sources, not the example sources.**
179 4. **`WOLFSSL_KEYLOG_EXPORT` writes TLS secrets to disk with nothing asking
180 it to — it is now OFF in `build.sh`, and this is the corrected record.**
181 Built with that flag, these binaries drop an `sslkeylog.log` of
182 `CLIENT_HANDSHAKE_TRAFFIC_SECRET` lines into the current directory on
183 every run: no env var, no flag, no message. Re-verified in an empty temp
184 directory after the fact — one run, one secrets file.
185
186 The first version of this README said the file "is deleted". That was
187 wrong. Three copies existed and one was deleted: the repo root. **A
188 second sat in the local scratch tree, and a third — 12,640 bytes,
189 world-readable, 20 handshakes — sat in `/home/ubuntu` on the shared box**,
190 left there by the spike runs. Review found both; the box copy was removed
191 in review, the scratch copy when this correction was written. All keys
192 involved were throwaway loopback/emulated-RTT test keys, and no real
193 session was ever carried over this stack — but "I cleaned it up" was a
194 claim I had not checked, on exactly the class of artifact where checking
195 is the whole job.
196
197 The flag is now `no` in the committed script, so anyone who runs it gets
198 no keylog by default. That is structure rather than advice, which is the
199 only reason to trust it. Task 2 keeps it off.
200
201 ## On "vendored" vs fetched
202
203 The plan said vendor pinned snapshots; `build.sh` fetches them against
204 sha256 pins instead. These are ~120MB of third-party C, this directory is
205 explicitly throwaway, and for the question the spike asks — *can our
206 toolchain build it* — a pinned fetch is the same evidence as a checked-in
207 copy. Vendoring is the right call at Task 2, when the answer is "yes, and we
208 are keeping it"; doing it now would put 120MB in the history for an
209 experiment we might have deleted.
210
211 ## Files
212
213 - `quicecho.c` — the echo pair, both roles, ~600 lines. Deliberately not a
214 design sketch: no retry/token, no migration, one connection at a time.
215 - `build.sh` — pinned fetch + both builds, pinned Zig only. `./build.sh` and
216 it produces `work/quicecho-native` and `work/quicecho-musl`.
spike/quic/build.sh
Old New
@@ -0,0 +1,128 @@
1 #!/bin/sh
2 # M8 Task 1 spike: build a QUIC echo pair from pinned sources using ONLY the
3 # repo's pinned Zig 0.15.2 as the C toolchain (zig cc / zig ar / zig ranlib).
4 #
5 # The point is not the echo pair. The point is that every line below runs
6 # under the toolchain muxd already ships with, for native AND
7 # x86_64-linux-musl static — which is the question Task 1 exists to answer.
8 #
9 # ./build.sh [workdir] (default: ./work, ~700MB, throwaway)
10 #
11 # Sources are fetched, not vendored into git: they are ~120MB of C, this is a
12 # throwaway spike, and a URL + sha256 pin answers "can our toolchain build
13 # it" exactly as well as a checked-in copy would. Vendoring is a Task 2
14 # decision, when the answer is "yes, and we are keeping it".
15 set -eu
16
17 NGTCP2_VER=1.25.0
18 WOLFSSL_VER=5.9.2-stable
19 NGTCP2_SHA=1c0843076528a87b65e9a9d455100941f4cb65d44f96c5da6ae56df146043955
20 WOLFSSL_SHA=2f4ef3d4fd387a9b3191d36a6316d69116c46ff69bb9583b6c82b36d7b8ca114
21
22 ZIG="${ZIG:-$HOME/Downloads/zig-x86_64-linux-0.15.2/zig}"
23 SELF="$(cd "$(dirname "$0")" && pwd)"
24 # An absolute workdir must be taken as given. Prepending the script's
25 # directory to one silently built a 274MB tree inside the repo during
26 # review, where .gitignore's `work/` then hid it.
27 case "${1:-work}" in
28 /*) W="$1" ;;
29 *) W="$SELF/${1:-work}" ;;
30 esac
31
32 [ -x "$ZIG" ] || { echo "no zig at $ZIG (set ZIG=...)" >&2; exit 1; }
33 mkdir -p "$W/src" "$W/bin"
34
35 # --- zig as the C toolchain -------------------------------------------------
36 cat > "$W/bin/zigcc" <<EOF
37 #!/bin/sh
38 exec $ZIG cc "\$@"
39 EOF
40 cat > "$W/bin/zigcc-musl" <<EOF
41 #!/bin/sh
42 exec $ZIG cc -target x86_64-linux-musl "\$@"
43 EOF
44 cat > "$W/bin/zigar" <<EOF
45 #!/bin/sh
46 exec $ZIG ar "\$@"
47 EOF
48 cat > "$W/bin/zigranlib" <<EOF
49 #!/bin/sh
50 exec $ZIG ranlib "\$@"
51 EOF
52 chmod +x "$W"/bin/*
53
54 # --- pinned sources ---------------------------------------------------------
55 fetch() { # url sha file
56 [ -f "$W/src/$3" ] || curl -sSL -o "$W/src/$3" "$1"
57 echo "$2 $W/src/$3" | sha256sum -c - >/dev/null || {
58 echo "checksum mismatch for $3" >&2; exit 1; }
59 }
60 fetch "https://github.com/ngtcp2/ngtcp2/releases/download/v$NGTCP2_VER/ngtcp2-$NGTCP2_VER.tar.gz" \
61 "$NGTCP2_SHA" "ngtcp2-$NGTCP2_VER.tar.gz"
62 fetch "https://github.com/wolfSSL/wolfssl/archive/refs/tags/v$WOLFSSL_VER.tar.gz" \
63 "$WOLFSSL_SHA" "wolfssl-$WOLFSSL_VER.tar.gz"
64 [ -d "$W/src/ngtcp2-$NGTCP2_VER" ] || tar -C "$W/src" -xzf "$W/src/ngtcp2-$NGTCP2_VER.tar.gz"
65 [ -d "$W/src/wolfssl-$WOLFSSL_VER" ] || tar -C "$W/src" -xzf "$W/src/wolfssl-$WOLFSSL_VER.tar.gz"
66
67 for T in native musl; do
68 CC="$W/bin/zigcc"; XTRA=""
69 if [ "$T" = musl ]; then
70 CC="$W/bin/zigcc-musl"
71 XTRA="-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=x86_64"
72 fi
73
74 # wolfSSL. Two flags here are load-bearing and were both found the hard
75 # way, by link failure:
76 # -fno-sanitize=undefined zig cc instruments C by default, and the
77 # ubsan runtime is not in a static musl link
78 # (undefined __ubsan_handle_*).
79 # WOLFSSL_AESECB ngtcp2's wolfSSL backend needs AES-ECB for
80 # QUIC header protection; wolfSSL omits it by
81 # default (undefined wolfSSL_EVP_aes_128_ecb).
82 #
83 # WOLFSSL_KEYLOG_EXPORT is deliberately OFF. Built with it on, these
84 # binaries write every handshake's secrets to ./sslkeylog.log with
85 # nothing asking them to — no env var, no flag, no message. Flip it to
86 # `yes` locally when you need to decrypt your own capture, and do not
87 # commit that. It is off here so that running this script cannot leak
88 # keys by default, which is a stronger guarantee than a warning.
89 cmake -S "$W/src/wolfssl-$WOLFSSL_VER" -B "$W/build/wolfssl-$T" $XTRA \
90 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="-fno-sanitize=undefined -O2" \
91 -DCMAKE_C_COMPILER="$CC" -DCMAKE_AR="$W/bin/zigar" -DCMAKE_RANLIB="$W/bin/zigranlib" \
92 -DBUILD_SHARED_LIBS=OFF -DWOLFSSL_QUIC=yes -DWOLFSSL_PSK=yes \
93 -DWOLFSSL_AESECB=yes -DWOLFSSL_AESCTR=yes -DWOLFSSL_KEYLOG_EXPORT=no \
94 -DWOLFSSL_SESSION_TICKET=yes -DWOLFSSL_EXAMPLES=no -DWOLFSSL_CRYPT_TESTS=no \
95 -DWOLFSSL_INSTALL=yes -DCMAKE_INSTALL_PREFIX="$W/out/$T" >/dev/null
96 cmake --build "$W/build/wolfssl-$T" -j"$(nproc)" >/dev/null
97 cmake --install "$W/build/wolfssl-$T" >/dev/null
98
99 # ngtcp2 + its wolfSSL crypto backend. ENABLE_LIB_ONLY is what keeps the
100 # static musl link clean: ngtcp2's find_package(Libbrotli*) calls live
101 # inside `if(NOT ENABLE_LIB_ONLY)`, so building the library alone never
102 # goes looking for the host's shared brotli in the first place. (An
103 # earlier version of this script also passed
104 # CMAKE_DISABLE_FIND_PACKAGE_Libbrotli* and credited them with the fix;
105 # cmake reported them unused-cli, because they were.)
106 PKG_CONFIG_PATH="$W/out/$T/lib/pkgconfig" \
107 cmake -S "$W/src/ngtcp2-$NGTCP2_VER" -B "$W/build/ngtcp2-$T" $XTRA \
108 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS="-fno-sanitize=undefined -O2" \
109 -DCMAKE_C_COMPILER="$CC" -DCMAKE_AR="$W/bin/zigar" -DCMAKE_RANLIB="$W/bin/zigranlib" \
110 -DENABLE_STATIC_LIB=ON -DENABLE_SHARED_LIB=OFF -DENABLE_WOLFSSL=ON \
111 -DENABLE_OPENSSL=OFF -DENABLE_GNUTLS=OFF -DENABLE_BORINGSSL=OFF \
112 -DENABLE_LIB_ONLY=ON \
113 -DCMAKE_INSTALL_PREFIX="$W/out/$T" >/dev/null
114 cmake --build "$W/build/ngtcp2-$T" -j"$(nproc)" >/dev/null
115 cmake --install "$W/build/ngtcp2-$T" >/dev/null
116 done
117
118 L() { echo "$W/out/$1/lib/libngtcp2_crypto_wolfssl.a $W/out/$1/lib/libngtcp2.a $W/out/$1/lib/libwolfssl.a"; }
119 # shellcheck disable=SC2046
120 "$W/bin/zigcc" -O2 -o "$W/quicecho-native" "$SELF/quicecho.c" \
121 -I"$W/out/native/include" $(L native) -lm -pthread
122 # shellcheck disable=SC2046
123 "$W/bin/zigcc-musl" -O2 -static -o "$W/quicecho-musl" "$SELF/quicecho.c" \
124 -I"$W/out/musl/include" $(L musl) -lm
125
126 echo "built:"
127 ls -la "$W/quicecho-native" "$W/quicecho-musl"
128 file "$W/quicecho-musl" | cut -d, -f1-3
spike/quic/quicecho.c
Old New
@@ -0,0 +1,598 @@
1 /* quicecho — M8 Task 1 spike.
2 *
3 * A QUIC echo pair (one binary, two roles) over ngtcp2 + wolfSSL, existing
4 * only to answer one question: can our pinned toolchain ship this? It is
5 * NOT a design sketch for the daemon — no retry/token, no migration, one
6 * connection at a time, minimal error recovery.
7 *
8 * quicecho server <port> <psk-hex>
9 * quicecho client <host> <port> <psk-hex> [reps]
10 *
11 * Authentication is TLS 1.3 external PSK: both sides hold the same 32-byte
12 * key, nobody holds a certificate, and there is no CA anywhere. That is the
13 * property the milestone's deploy model needs ("scp a binary, scp a key").
14 *
15 * The client reports, per rep: handshake completion and first echoed byte,
16 * both measured from the first packet sent. Rep 0 is cold; later reps reuse
17 * the wolfSSL session, which is what the resumed number measures.
18 */
19 #include <arpa/inet.h>
20 #include <assert.h>
21 #include <errno.h>
22 #include <netdb.h>
23 #include <poll.h>
24 #include <stdio.h>
25 #include <stdlib.h>
26 #include <string.h>
27 #include <sys/socket.h>
28 #include <sys/types.h>
29 #include <time.h>
30 #include <unistd.h>
31
32 #include <ngtcp2/ngtcp2.h>
33 #include <ngtcp2/ngtcp2_crypto.h>
34 #include <ngtcp2/ngtcp2_crypto_wolfssl.h>
35
36 #include <wolfssl/options.h>
37 #include <wolfssl/ssl.h>
38
39 #define ALPN "\x08""quicecho"
40 #define PSK_IDENTITY "mux"
41 #define PSK_CIPHERSUITE "TLS13-AES128-GCM-SHA256"
42 #define MAX_PSK 64
43 #define SERVER_SCIDLEN 8
44
45 static uint8_t g_psk[MAX_PSK];
46 static size_t g_psklen;
47
48 static uint64_t timestamp_ns(void) {
49 struct timespec tp;
50 clock_gettime(CLOCK_MONOTONIC, &tp);
51 return (uint64_t)tp.tv_sec * NGTCP2_SECONDS + (uint64_t)tp.tv_nsec;
52 }
53
54 static int hex2bin(const char *hex, uint8_t *out, size_t outcap, size_t *outlen) {
55 size_t n = strlen(hex), i;
56 if (n % 2 || n / 2 > outcap) return -1;
57 for (i = 0; i < n; i += 2) {
58 unsigned v;
59 if (sscanf(hex + i, "%2x", &v) != 1) return -1;
60 out[i / 2] = (uint8_t)v;
61 }
62 *outlen = n / 2;
63 return 0;
64 }
65
66 /* ---- PSK callbacks -------------------------------------------------- */
67
68 static unsigned int psk_client_cb(WOLFSSL *ssl, const char *hint, char *identity,
69 unsigned int id_max, unsigned char *key,
70 unsigned int key_max, const char **ciphersuite) {
71 (void)ssl;
72 (void)hint;
73 if (id_max < sizeof(PSK_IDENTITY) || key_max < g_psklen) return 0;
74 strcpy(identity, PSK_IDENTITY);
75 memcpy(key, g_psk, g_psklen);
76 *ciphersuite = PSK_CIPHERSUITE;
77 return (unsigned int)g_psklen;
78 }
79
80 static unsigned int psk_server_cb(WOLFSSL *ssl, const char *identity,
81 unsigned char *key, unsigned int key_max,
82 const char **ciphersuite) {
83 (void)ssl;
84 if (ciphersuite) *ciphersuite = PSK_CIPHERSUITE;
85 /* Possession of the key IS the authentication; the identity only selects
86 * which key, and this spike has exactly one. */
87 if (!identity || strcmp(identity, PSK_IDENTITY) != 0) return 0;
88 if (key_max < g_psklen) return 0;
89 memcpy(key, g_psk, g_psklen);
90 return (unsigned int)g_psklen;
91 }
92
93 /* ---- shared connection state ---------------------------------------- */
94
95 struct peer {
96 ngtcp2_crypto_conn_ref conn_ref;
97 WOLFSSL_CTX *ssl_ctx;
98 WOLFSSL *ssl;
99 ngtcp2_conn *conn;
100 int fd;
101 struct sockaddr_storage local_addr;
102 socklen_t local_addrlen;
103 struct sockaddr_storage remote_addr;
104 socklen_t remote_addrlen;
105 int64_t stream_id;
106 int handshake_done;
107 int got_echo;
108 int closed;
109 uint64_t t0;
110 uint64_t t_handshake;
111 uint64_t t_echo;
112 const char *payload;
113 size_t payload_len;
114 size_t payload_sent;
115 int is_server;
116 };
117
118 static ngtcp2_conn *get_conn(ngtcp2_crypto_conn_ref *ref) {
119 return ((struct peer *)ref->user_data)->conn;
120 }
121
122 static void rand_cb(uint8_t *dest, size_t destlen, const ngtcp2_rand_ctx *ctx) {
123 (void)ctx;
124 if (wolfSSL_RAND_bytes(dest, (int)destlen) != 1) abort();
125 }
126
127 static int get_new_connection_id_cb(ngtcp2_conn *conn, ngtcp2_cid *cid,
128 ngtcp2_stateless_reset_token *token,
129 size_t cidlen, void *user_data) {
130 (void)conn;
131 (void)user_data;
132 if (wolfSSL_RAND_bytes(cid->data, (int)cidlen) != 1) return NGTCP2_ERR_CALLBACK_FAILURE;
133 cid->datalen = cidlen;
134 if (wolfSSL_RAND_bytes(token->data, NGTCP2_STATELESS_RESET_TOKENLEN) != 1)
135 return NGTCP2_ERR_CALLBACK_FAILURE;
136 return 0;
137 }
138
139 static int handshake_completed_cb(ngtcp2_conn *conn, void *user_data) {
140 struct peer *p = user_data;
141 (void)conn;
142 p->handshake_done = 1;
143 p->t_handshake = timestamp_ns();
144 return 0;
145 }
146
147 static int recv_stream_data_cb(ngtcp2_conn *conn, uint32_t flags,
148 int64_t stream_id, uint64_t offset,
149 const uint8_t *data, size_t datalen,
150 void *user_data, void *stream_user_data) {
151 struct peer *p = user_data;
152 (void)flags;
153 (void)offset;
154 (void)stream_user_data;
155
156 if (p->is_server) {
157 /* Echo: the whole point. Bounce it back on the same stream. */
158 ngtcp2_vec vec = {.base = (uint8_t *)data, .len = datalen};
159 ngtcp2_ssize wcont;
160 uint8_t buf[1500];
161 ngtcp2_path_storage ps;
162 ngtcp2_pkt_info pi;
163 ngtcp2_path_storage_zero(&ps);
164 for (;;) {
165 ngtcp2_ssize n = ngtcp2_conn_writev_stream(
166 conn, &ps.path, &pi, buf, sizeof(buf), &wcont, NGTCP2_WRITE_STREAM_FLAG_NONE,
167 stream_id, &vec, 1, timestamp_ns());
168 if (n <= 0) break;
169 sendto(p->fd, buf, (size_t)n, 0, (struct sockaddr *)&p->remote_addr,
170 p->remote_addrlen);
171 if (wcont > 0) break;
172 }
173 } else {
174 p->got_echo = 1;
175 p->t_echo = timestamp_ns();
176 }
177 return 0;
178 }
179
180 static int extend_max_local_streams_bidi_cb(ngtcp2_conn *conn, uint64_t max_streams,
181 void *user_data) {
182 struct peer *p = user_data;
183 (void)max_streams;
184 if (!p->is_server && p->stream_id == -1) {
185 if (ngtcp2_conn_open_bidi_stream(conn, &p->stream_id, NULL) != 0) p->stream_id = -1;
186 }
187 return 0;
188 }
189
190 /* ---- TLS setup ------------------------------------------------------ */
191
192 static int tls_init(struct peer *p, int is_server) {
193 p->ssl_ctx = wolfSSL_CTX_new(is_server ? wolfTLSv1_3_server_method()
194 : wolfTLSv1_3_client_method());
195 if (!p->ssl_ctx) return -1;
196
197 if (is_server) {
198 if (ngtcp2_crypto_wolfssl_configure_server_context(p->ssl_ctx) != 0) return -1;
199 wolfSSL_CTX_set_psk_server_tls13_callback(p->ssl_ctx, psk_server_cb);
200 wolfSSL_CTX_use_psk_identity_hint(p->ssl_ctx, "");
201 } else {
202 if (ngtcp2_crypto_wolfssl_configure_client_context(p->ssl_ctx) != 0) return -1;
203 wolfSSL_CTX_set_psk_client_tls13_callback(p->ssl_ctx, psk_client_cb);
204 }
205 /* No certificates anywhere: possession of the PSK is the authentication. */
206 wolfSSL_CTX_set_cipher_list(p->ssl_ctx, PSK_CIPHERSUITE);
207
208 p->ssl = wolfSSL_new(p->ssl_ctx);
209 if (!p->ssl) return -1;
210 wolfSSL_set_app_data(p->ssl, &p->conn_ref);
211 wolfSSL_set_quic_use_legacy_codepoint(p->ssl, 0);
212 if (!is_server) {
213 wolfSSL_UseALPN(p->ssl, (char *)ALPN + 1, sizeof(ALPN) - 2, WOLFSSL_ALPN_FAILED_ON_MISMATCH);
214 wolfSSL_UseSNI(p->ssl, WOLFSSL_SNI_HOST_NAME, "quicecho", 8);
215 }
216 return 0;
217 }
218
219 /* ---- write/flush ---------------------------------------------------- */
220
221 static int flush_egress(struct peer *p) {
222 uint8_t buf[1500];
223 ngtcp2_path_storage ps;
224 ngtcp2_pkt_info pi;
225 ngtcp2_ssize wcont;
226 uint64_t ts = timestamp_ns();
227
228 ngtcp2_path_storage_zero(&ps);
229 for (;;) {
230 ngtcp2_vec vec;
231 ngtcp2_vec *pvec = NULL;
232 size_t vcnt = 0;
233 int64_t sid = -1;
234 uint32_t flags = NGTCP2_WRITE_STREAM_FLAG_NONE;
235
236 if (!p->is_server && p->stream_id != -1 && p->payload_sent < p->payload_len) {
237 vec.base = (uint8_t *)p->payload + p->payload_sent;
238 vec.len = p->payload_len - p->payload_sent;
239 pvec = &vec;
240 vcnt = 1;
241 sid = p->stream_id;
242 }
243
244 ngtcp2_ssize n = ngtcp2_conn_writev_stream(p->conn, &ps.path, &pi, buf,
245 sizeof(buf), &wcont, flags, sid,
246 pvec, vcnt, ts);
247 if (n < 0) {
248 if (n == NGTCP2_ERR_WRITE_MORE) {
249 if (wcont > 0) p->payload_sent += (size_t)wcont;
250 continue;
251 }
252 fprintf(stderr, "writev_stream: %s\n", ngtcp2_strerror((int)n));
253 return -1;
254 }
255 if (n == 0) return 0;
256 if (wcont > 0) p->payload_sent += (size_t)wcont;
257
258 if (sendto(p->fd, buf, (size_t)n, 0, (struct sockaddr *)&p->remote_addr,
259 p->remote_addrlen) < 0) {
260 fprintf(stderr, "sendto: %s\n", strerror(errno));
261 return -1;
262 }
263 }
264 }
265
266 static int feed_packet(struct peer *p, const uint8_t *data, size_t datalen,
267 struct sockaddr *from, socklen_t fromlen) {
268 ngtcp2_path path = {
269 .local = {.addr = (struct sockaddr *)&p->local_addr, .addrlen = p->local_addrlen},
270 .remote = {.addr = from, .addrlen = fromlen},
271 };
272 ngtcp2_pkt_info pi = {0};
273 int rv = ngtcp2_conn_read_pkt(p->conn, &path, &pi, data, datalen, timestamp_ns());
274 if (rv != 0) {
275 if (rv != NGTCP2_ERR_DRAINING && rv != NGTCP2_ERR_CLOSING)
276 fprintf(stderr, "read_pkt: %s\n", ngtcp2_strerror(rv));
277 return -1;
278 }
279 return 0;
280 }
281
282 /* Returns milliseconds until ngtcp2 wants attention, capped. */
283 static int poll_timeout_ms(struct peer *p) {
284 ngtcp2_tstamp expiry = ngtcp2_conn_get_expiry(p->conn);
285 uint64_t now = timestamp_ns();
286 if (expiry == UINT64_MAX) return 200;
287 if (expiry <= now) return 0;
288 uint64_t d = (expiry - now) / NGTCP2_MILLISECONDS;
289 return d > 200 ? 200 : (int)d;
290 }
291
292 /* ---- client --------------------------------------------------------- */
293
294 static int client_once(const char *host, const char *port, int rep,
295 WOLFSSL_SESSION *resume, WOLFSSL_SESSION **out_session) {
296 struct addrinfo hints = {.ai_family = AF_UNSPEC, .ai_socktype = SOCK_DGRAM};
297 struct addrinfo *res, *ai;
298 struct peer p;
299 char msg[64];
300 int rv = -1;
301
302 memset(&p, 0, sizeof(p));
303 p.stream_id = -1;
304 p.is_server = 0;
305 p.conn_ref.get_conn = get_conn;
306 p.conn_ref.user_data = &p;
307
308 if (getaddrinfo(host, port, &hints, &res) != 0) return -1;
309 for (ai = res; ai; ai = ai->ai_next) {
310 p.fd = socket(ai->ai_family, ai->ai_socktype, 0);
311 if (p.fd < 0) continue;
312 if (connect(p.fd, ai->ai_addr, ai->ai_addrlen) == 0) break;
313 close(p.fd);
314 p.fd = -1;
315 }
316 if (!ai) { freeaddrinfo(res); return -1; }
317 memcpy(&p.remote_addr, ai->ai_addr, ai->ai_addrlen);
318 p.remote_addrlen = ai->ai_addrlen;
319 freeaddrinfo(res);
320
321 p.local_addrlen = sizeof(p.local_addr);
322 getsockname(p.fd, (struct sockaddr *)&p.local_addr, &p.local_addrlen);
323
324 if (tls_init(&p, 0) != 0) goto out;
325 if (resume) wolfSSL_set_session(p.ssl, resume);
326
327 {
328 ngtcp2_path path = {
329 .local = {.addr = (struct sockaddr *)&p.local_addr, .addrlen = p.local_addrlen},
330 .remote = {.addr = (struct sockaddr *)&p.remote_addr, .addrlen = p.remote_addrlen},
331 };
332 ngtcp2_callbacks cb = {
333 .client_initial = ngtcp2_crypto_client_initial_cb,
334 .recv_crypto_data = ngtcp2_crypto_recv_crypto_data_cb,
335 .encrypt = ngtcp2_crypto_encrypt_cb,
336 .decrypt = ngtcp2_crypto_decrypt_cb,
337 .hp_mask = ngtcp2_crypto_hp_mask_cb,
338 .recv_retry = ngtcp2_crypto_recv_retry_cb,
339 .update_key = ngtcp2_crypto_update_key_cb,
340 .delete_crypto_aead_ctx = ngtcp2_crypto_delete_crypto_aead_ctx_cb,
341 .delete_crypto_cipher_ctx = ngtcp2_crypto_delete_crypto_cipher_ctx_cb,
342 .version_negotiation = ngtcp2_crypto_version_negotiation_cb,
343 .get_path_challenge_data2 = ngtcp2_crypto_get_path_challenge_data2_cb,
344 .rand = rand_cb,
345 .get_new_connection_id2 = get_new_connection_id_cb,
346 .handshake_completed = handshake_completed_cb,
347 .recv_stream_data = recv_stream_data_cb,
348 .extend_max_local_streams_bidi = extend_max_local_streams_bidi_cb,
349 };
350 ngtcp2_cid dcid, scid;
351 ngtcp2_settings settings;
352 ngtcp2_transport_params params;
353
354 dcid.datalen = NGTCP2_MIN_INITIAL_DCIDLEN;
355 scid.datalen = 8;
356 if (wolfSSL_RAND_bytes(dcid.data, (int)dcid.datalen) != 1) goto out;
357 if (wolfSSL_RAND_bytes(scid.data, (int)scid.datalen) != 1) goto out;
358
359 ngtcp2_settings_default(&settings);
360 settings.initial_ts = timestamp_ns();
361 ngtcp2_transport_params_default(&params);
362 params.initial_max_streams_bidi = 8;
363 params.initial_max_stream_data_bidi_local = 256 * 1024;
364 params.initial_max_stream_data_bidi_remote = 256 * 1024;
365 params.initial_max_data = 1024 * 1024;
366
367 if (ngtcp2_conn_client_new(&p.conn, &dcid, &scid, &path, NGTCP2_PROTO_VER_V1,
368 &cb, &settings, &params, NULL, &p) != 0)
369 goto out;
370 ngtcp2_conn_set_tls_native_handle(p.conn, p.ssl);
371 }
372
373 snprintf(msg, sizeof(msg), "quicecho-rep-%d\n", rep);
374 p.payload = msg;
375 p.payload_len = strlen(msg);
376 p.t0 = timestamp_ns();
377
378 if (flush_egress(&p) != 0) goto out;
379
380 while (!p.got_echo) {
381 struct pollfd pfd = {.fd = p.fd, .events = POLLIN};
382 int ready = poll(&pfd, 1, poll_timeout_ms(&p));
383 if (ready < 0) goto out;
384 if (ready > 0) {
385 uint8_t buf[65536];
386 struct sockaddr_storage from;
387 socklen_t fromlen = sizeof(from);
388 ssize_t n = recvfrom(p.fd, buf, sizeof(buf), 0, (struct sockaddr *)&from, &fromlen);
389 if (n > 0 && feed_packet(&p, buf, (size_t)n, (struct sockaddr *)&from, fromlen) != 0)
390 goto out;
391 } else if (ngtcp2_conn_handle_expiry(p.conn, timestamp_ns()) != 0) {
392 goto out;
393 }
394 if (flush_egress(&p) != 0) goto out;
395 if (timestamp_ns() - p.t0 > 10 * NGTCP2_SECONDS) {
396 fprintf(stderr, "rep %d: timed out (handshake_done=%d)\n", rep, p.handshake_done);
397 goto out;
398 }
399 }
400
401 /* Tell the server we are done. Without this it sits in its one-connection
402 * loop and the next rep's Initial never reaches a fresh conn — which is
403 * exactly how the resumed rep failed the first time this ran. */
404 {
405 uint8_t cbuf[1500];
406 ngtcp2_path_storage cps;
407 ngtcp2_pkt_info cpi;
408 ngtcp2_ccerr ccerr;
409 ngtcp2_ssize cn;
410 ngtcp2_path_storage_zero(&cps);
411 ngtcp2_ccerr_default(&ccerr);
412 cn = ngtcp2_conn_write_connection_close(p.conn, &cps.path, &cpi, cbuf,
413 sizeof(cbuf), &ccerr, timestamp_ns());
414 if (cn > 0)
415 sendto(p.fd, cbuf, (size_t)cn, 0, (struct sockaddr *)&p.remote_addr,
416 p.remote_addrlen);
417 }
418
419 printf("#REP %d handshake_ms=%.2f echo_ms=%.2f resumed=%d\n", rep,
420 (double)(p.t_handshake - p.t0) / 1e6, (double)(p.t_echo - p.t0) / 1e6,
421 resume ? 1 : 0);
422 fflush(stdout);
423
424 if (out_session) {
425 WOLFSSL_SESSION *s = wolfSSL_get1_session(p.ssl);
426 if (s) *out_session = s;
427 }
428 rv = 0;
429
430 out:
431 if (p.conn) ngtcp2_conn_del(p.conn);
432 if (p.ssl) wolfSSL_free(p.ssl);
433 if (p.ssl_ctx) wolfSSL_CTX_free(p.ssl_ctx);
434 if (p.fd >= 0) close(p.fd);
435 return rv;
436 }
437
438 /* ---- server --------------------------------------------------------- */
439
440 static int server_run(const char *port) {
441 struct addrinfo hints = {.ai_family = AF_INET, .ai_socktype = SOCK_DGRAM,
442 .ai_flags = AI_PASSIVE};
443 struct addrinfo *res;
444 int fd;
445
446 if (getaddrinfo(NULL, port, &hints, &res) != 0) return -1;
447 fd = socket(res->ai_family, res->ai_socktype, 0);
448 if (fd < 0) { freeaddrinfo(res); return -1; }
449 {
450 int on = 1;
451 setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
452 }
453 if (bind(fd, res->ai_addr, res->ai_addrlen) != 0) {
454 fprintf(stderr, "bind: %s\n", strerror(errno));
455 freeaddrinfo(res);
456 close(fd);
457 return -1;
458 }
459 freeaddrinfo(res);
460 fprintf(stderr, "quicecho server listening on udp/%s\n", port);
461
462 for (;;) {
463 /* One connection at a time: spike, not a server. */
464 struct peer p;
465 uint8_t buf[65536];
466 struct sockaddr_storage from;
467 socklen_t fromlen = sizeof(from);
468 ssize_t n;
469 ngtcp2_pkt_hd hd;
470 int alive = 1;
471
472 n = recvfrom(fd, buf, sizeof(buf), 0, (struct sockaddr *)&from, &fromlen);
473 if (n <= 0) continue;
474 if (ngtcp2_accept(&hd, buf, (size_t)n) != 0) continue;
475
476 memset(&p, 0, sizeof(p));
477 p.fd = fd;
478 p.is_server = 1;
479 p.stream_id = -1;
480 p.conn_ref.get_conn = get_conn;
481 p.conn_ref.user_data = &p;
482 memcpy(&p.remote_addr, &from, fromlen);
483 p.remote_addrlen = fromlen;
484 p.local_addrlen = sizeof(p.local_addr);
485 getsockname(fd, (struct sockaddr *)&p.local_addr, &p.local_addrlen);
486
487 if (tls_init(&p, 1) != 0) continue;
488
489 {
490 ngtcp2_path path = {
491 .local = {.addr = (struct sockaddr *)&p.local_addr, .addrlen = p.local_addrlen},
492 .remote = {.addr = (struct sockaddr *)&p.remote_addr, .addrlen = p.remote_addrlen},
493 };
494 ngtcp2_callbacks cb = {
495 .recv_client_initial = ngtcp2_crypto_recv_client_initial_cb,
496 .recv_crypto_data = ngtcp2_crypto_recv_crypto_data_cb,
497 .encrypt = ngtcp2_crypto_encrypt_cb,
498 .decrypt = ngtcp2_crypto_decrypt_cb,
499 .hp_mask = ngtcp2_crypto_hp_mask_cb,
500 .update_key = ngtcp2_crypto_update_key_cb,
501 .delete_crypto_aead_ctx = ngtcp2_crypto_delete_crypto_aead_ctx_cb,
502 .delete_crypto_cipher_ctx = ngtcp2_crypto_delete_crypto_cipher_ctx_cb,
503 .version_negotiation = ngtcp2_crypto_version_negotiation_cb,
504 .get_path_challenge_data2 = ngtcp2_crypto_get_path_challenge_data2_cb,
505 .rand = rand_cb,
506 .get_new_connection_id2 = get_new_connection_id_cb,
507 .handshake_completed = handshake_completed_cb,
508 .recv_stream_data = recv_stream_data_cb,
509 };
510 ngtcp2_cid scid;
511 ngtcp2_settings settings;
512 ngtcp2_transport_params params;
513
514 scid.datalen = SERVER_SCIDLEN;
515 if (wolfSSL_RAND_bytes(scid.data, SERVER_SCIDLEN) != 1) goto next;
516
517 ngtcp2_settings_default(&settings);
518 settings.initial_ts = timestamp_ns();
519 ngtcp2_transport_params_default(&params);
520 params.initial_max_streams_bidi = 8;
521 params.initial_max_stream_data_bidi_local = 256 * 1024;
522 params.initial_max_stream_data_bidi_remote = 256 * 1024;
523 params.initial_max_data = 1024 * 1024;
524 params.original_dcid = hd.dcid;
525 params.original_dcid_present = 1;
526
527 if (ngtcp2_conn_server_new(&p.conn, &hd.scid, &scid, &path, hd.version, &cb,
528 &settings, &params, NULL, &p) != 0)
529 goto next;
530 ngtcp2_conn_set_tls_native_handle(p.conn, p.ssl);
531 }
532
533 if (feed_packet(&p, buf, (size_t)n, (struct sockaddr *)&from, fromlen) != 0) goto next;
534 if (flush_egress(&p) != 0) goto next;
535
536 while (alive) {
537 struct pollfd pfd = {.fd = fd, .events = POLLIN};
538 int ready = poll(&pfd, 1, poll_timeout_ms(&p));
539 if (ready < 0) break;
540 if (ready > 0) {
541 fromlen = sizeof(from);
542 n = recvfrom(fd, buf, sizeof(buf), 0, (struct sockaddr *)&from, &fromlen);
543 if (n > 0 && feed_packet(&p, buf, (size_t)n, (struct sockaddr *)&from, fromlen) != 0)
544 alive = 0;
545 } else if (ngtcp2_conn_handle_expiry(p.conn, timestamp_ns()) != 0) {
546 alive = 0;
547 }
548 if (alive && flush_egress(&p) != 0) alive = 0;
549 }
550
551 next:
552 if (p.conn) ngtcp2_conn_del(p.conn);
553 if (p.ssl) wolfSSL_free(p.ssl);
554 if (p.ssl_ctx) wolfSSL_CTX_free(p.ssl_ctx);
555 fprintf(stderr, "connection over; waiting for the next\n");
556 }
557 }
558
559 int main(int argc, char **argv) {
560 wolfSSL_Init();
561
562 if (argc >= 4 && strcmp(argv[1], "server") == 0) {
563 if (hex2bin(argv[3], g_psk, sizeof(g_psk), &g_psklen) != 0) {
564 fprintf(stderr, "bad psk hex\n");
565 return 2;
566 }
567 return server_run(argv[2]) == 0 ? 0 : 1;
568 }
569
570 if (argc >= 5 && strcmp(argv[1], "client") == 0) {
571 int reps = argc >= 6 ? atoi(argv[5]) : 2;
572 WOLFSSL_SESSION *sess = NULL;
573 int i, failed = 0;
574 if (hex2bin(argv[4], g_psk, sizeof(g_psk), &g_psklen) != 0) {
575 fprintf(stderr, "bad psk hex\n");
576 return 2;
577 }
578 for (i = 0; i < reps; i++) {
579 WOLFSSL_SESSION *next = NULL;
580 if (client_once(argv[2], argv[3], i, sess, &next) != 0) {
581 failed = 1;
582 if (next) wolfSSL_SESSION_free(next);
583 break;
584 }
585 if (next) {
586 if (sess) wolfSSL_SESSION_free(sess);
587 sess = next;
588 }
589 }
590 if (sess) wolfSSL_SESSION_free(sess);
591 return failed;
592 }
593
594 fprintf(stderr,
595 "usage: quicecho server <port> <psk-hex>\n"
596 " quicecho client <host> <port> <psk-hex> [reps]\n");
597 return 2;
598 }
src/client.zig
Old New
@@ -781,7 +781,9 @@ test "drainStdinForQuit: the quit byte is seen, other input is dropped" {
781 test "drainStdinForQuit: a zero wait returns at once and consumes nothing" { 781 test "drainStdinForQuit: a zero wait returns at once and consumes nothing" {
782 const pipe = try std.posix.pipe(); 782 const pipe = try std.posix.pipe();
783 defer std.posix.close(pipe[0]); 783 defer std.posix.close(pipe[0]);
784 defer std.posix.close(pipe[1]); 784 // No `defer close(pipe[1])`: the write end is closed explicitly below,
785 // and closing it twice is EBADF — which std.posix maps to `unreachable`,
786 // i.e. the same panic Transport.close's idempotence exists to prevent.
785 // A quit byte is already queued, which makes this the interesting case: 787 // A quit byte is already queued, which makes this the interesting case:
786 // reconnect's iteration zero passes a zero-length wait, and it must 788 // reconnect's iteration zero passes a zero-length wait, and it must
787 // neither block nor eat the byte. Leaving it unread is what keeps 789 // neither block nor eat the byte. Leaving it unread is what keeps
@@ -793,6 +795,11 @@ test "drainStdinForQuit: a zero wait returns at once and consumes nothing" {
793 try std.testing.expect(!drainStdinForQuit(pipe[0], 0)); 795 try std.testing.expect(!drainStdinForQuit(pipe[0], 0));
794 try std.testing.expect(timer.read() / std.time.ns_per_ms < 50); 796 try std.testing.expect(timer.read() / std.time.ns_per_ms < 50);
795 797
798 // Closed before the read so that a regression which CONSUMED the byte
799 // but still returned false fails on EOF instead of blocking forever. A
800 // wedged test is worth less than a red one — the same lesson
801 // expectInitRefused exists for.
802 std.posix.close(pipe[1]);
796 var buf: [4]u8 = undefined; 803 var buf: [4]u8 = undefined;
797 const n = try std.posix.read(pipe[0], &buf); 804 const n = try std.posix.read(pipe[0], &buf);
798 try std.testing.expectEqualSlices(u8, "\x1c", buf[0..n]); 805 try std.testing.expectEqualSlices(u8, "\x1c", buf[0..n]);