a73x

d5d2c29f

feat(deploy): a release is one pipeline run twice, at stg then prod

a73x   2026-08-08 14:02

Commit message
feat(deploy): a release is one pipeline run twice, at stg then prod

eitri.sh gains a staging twin. `scripts/ship.sh --target stg --tag v0.0.4-pre.1`
builds a tagged tree, publishes its artifacts, rolls the plane and proves it
with the same smoke the branch gate runs; the release is that script again with
`--target prod`. Nothing reaches prod that a machine has not already done to
stg.

The two planes share a cluster, a node, and one set of manifests. What differs
is values — namespace, hostnames, the host-port triple the node cannot let them
share, storage size, which plane gets backups and a CDN purge — so stg and prod
cannot drift in shape, only in configuration. Rendering is how you read a
manifest now: `--render-only` prints what an apply would send and touches
nothing. envsubst is called with an explicit variable list, and every name on it
is checked non-empty first, because envsubst's own answer to an unset variable
is to substitute nothing at all. The site that serves /dl comes into the tree
with them, under the names the live objects already carry, so the manifests
adopt that site rather than standing a second one beside it. It rolls before the
server, always, because a server started ahead of its site pins the previous
release's boot manifest for twenty-four hours.

Stage 4 is why the script exists. It reads the config keys the tagged tree
declares straight out of the config schema, compares them against a committed
contract that classifies every key as required, optional or retired, and refuses
to deploy a plane whose Secret and whose binary disagree — naming the key and
the patch that fixes it. It then asserts the Secret's listeners, gate domain and
advertisements match the plane's own manifests, because each of those
disagreeing presents as something else entirely: a wrong gate domain reads as
"pubkey denied", a wrong sync advertisement as a host that enrolls and never
syncs. Both classes are reported in one pass. The Secret is read, never written:
a script that can write config secrets is a script that can overwrite prod's.

Versions understand the cycle a release actually walks. A pre-release orders
below the release it leads to and above the one before it —
`v0.0.4-pre.1 < v0.0.4-pre.1-3-gabc1234 < v0.0.4-pre.2 < v0.0.4 < v0.0.4-2-gabc1234`
— so an agent on a staging plane takes each pre-release in turn and then the
release itself, through the same advertisement path a fielded agent uses. That
is what lets stg rehearse the agent upgrade, which is the failure that defined
the v0.0.3 release day. Everything unparsable stays unordered and is never
offered an upgrade, exactly as before. The pipeline verifies its tag by running
that code rather than restating its rule in shell.

The hosted smoke exercises two MCP origins. `SMOKE_MCP_URL` takes a
space-separated list: the first entry gets the full cycle — register, create,
exec, expose, banner, destroy — and every later one gets an unauthenticated-401
and a toolset check. One long call is enough to observe a proxy's silent-origin
timeout; a second would only double the VM churn. Unset, the list is the console
origin alone and the branch gate is unchanged. The credential-chain proof
becomes opt-in for the same reason it has to be: it posts a password to a login
form, and a plane fronted by a real identity provider has none to post — so a
hosted run starts at the VM lifecycle, carried by the operator's token, and says
which leg it gave up.

Real hardware moves to stg and the branch gate re-homes to nested KVM.
`scripts/devhost.sh` defines that host rather than hand-building it, so a wedged
one is a three-minute recycle; `make deploy` itself does not change, only which
host it points at. What nested cannot prove — bridged networking, hardware
quirks — becomes coverage stg owns, per pre-release tag instead of per branch.
.gitignore
Old New
@@ -31,3 +31,8 @@ deploy.env
31 31
32 /dist/ 32 /dist/
33 site/dist/ 33 site/dist/
34
35 # Transient probe dir scripts/ship.sh builds to ask internal/server/release
36 # whether a tag is parsable (removed on exit; ignored so a crash can't leave
37 # the tree dirty for the next run's clean-tree check).
38 /.shipcheck.*/
Makefile
Old New
@@ -13,7 +13,7 @@ LINT_WARN := errcheck,revive,gocyclo,funlen,gocritic,misspell,unconvert,nakedret
13 .PHONY: build build-go build-darwin web test vet proto clean \ 13 .PHONY: build build-go build-darwin web test vet proto clean \
14 lint lint-extra arch cover fmt fmt-check tidy-check proto-check shape shape-check api api-check \ 14 lint lint-extra arch cover fmt fmt-check tidy-check proto-check shape shape-check api api-check \
15 site site-check ci deadcode \ 15 site site-check ci deadcode \
16 deploy release hooks site-image server-image 16 deploy release ship hooks site-image server-image
17 17
18 # Enable the repo's client-side merge gate: point git at .githooks, whose 18 # Enable the repo's client-side merge gate: point git at .githooks, whose
19 # pre-push hook runs `make ci` before any push that updates main. Run once per 19 # pre-push hook runs `make ci` before any push that updates main. Run once per
@@ -68,6 +68,14 @@ deploy:
68 release: web 68 release: web
69 ./scripts/release.sh 69 ./scripts/release.sh
70 70
71 # The release pipeline for one plane: build the tagged tree, publish it, roll the
72 # plane and prove it (see scripts/ship.sh). Run it twice — TARGET=stg with a
73 # pre-release tag, then TARGET=prod. FROM=<n> resumes at a stage after a partial
74 # failure; the script's other flags are for direct use.
75 ship:
76 @$(if $(and $(TARGET),$(TAG)),,$(error ship: TARGET and TAG are required, e.g. make ship TARGET=stg TAG=v0.0.4-pre.1))
77 ./scripts/ship.sh --target $(TARGET) --tag $(TAG)$(if $(FROM), --from $(FROM))
78
71 # --- quality gates ----------------------------------------------------------- 79 # --- quality gates -----------------------------------------------------------
72 80
73 # Architecture fitness functions (R1–R6). -count=1 is mandatory: these tests 81 # Architecture fitness functions (R1–R6). -count=1 is mandatory: these tests
deploy/server/Dockerfile
Old New
@@ -1,7 +1,13 @@
1 # eitri-server production image: one static Go binary (pure-Go sqlite, no 1 # eitri-server production image: static Go binaries (pure-Go sqlite, no CGO)
2 # CGO) on distroless/static — CA roots included for OIDC discovery, runs as 2 # on distroless/static — CA roots included for OIDC discovery, runs as nonroot
3 # nonroot (uid 65532). The console SPA is embedded in the binary; state 3 # (uid 65532). The console SPA is embedded in the binary; state lives on the
4 # lives on the mounted PVC and config in the mounted server.json Secret. 4 # mounted PVC and config in the mounted server.json Secret.
5 #
6 # eitri-oidc rides along as a second entrypoint. A plane with no external
7 # identity provider runs it as its own Deployment off this same image, so the
8 # issuer can never be a different version than the server it authenticates for
9 # — and a plane fronted by a real IdP simply never starts it.
5 FROM gcr.io/distroless/static-debian12:nonroot 10 FROM gcr.io/distroless/static-debian12:nonroot
6 COPY eitri-server /eitri-server 11 COPY eitri-server /eitri-server
12 COPY eitri-oidc /eitri-oidc
7 ENTRYPOINT ["/eitri-server"] 13 ENTRYPOINT ["/eitri-server"]
deploy/server/README.md
Old New
@@ -1,82 +1,454 @@
1 # eitri-server on the eitri.sh cluster 1 # The hosted planes and the release pipeline
2 2
3 One container runs the whole control plane (embedded console, API, OIDC 3 Two planes run on one cluster from one set of manifests: **stg**
4 auth, QUIC sync, SSH gate), pinned to vnic-1 — the cluster's public-IP 4 (`stg.eitri.sh`) and **prod** (`console.eitri.sh`). A release is
5 node — with state on a local-path PVC and config from a Secret. 5 `scripts/ship.sh` run twice — first at stg with a pre-release tag, then at prod
6 with the release tag, same script, same stages, artifacts rebuilt from the same
7 tree. Nothing reaches prod that a machine has not already done to stg.
6 8
7 ## One-time bring-up 9 Each plane is one container carrying the whole control plane (console, API,
10 OIDC relying party, QUIC sync, SSH gate) pinned to the cluster's public-IP
11 node, with state on a local-path PVC and config in a Secret, plus an nginx
12 deployment serving the static site and its `/dl` artifacts.
8 13
9 1. **Google OAuth client** (GCP console → Credentials): Web application, 14 ## The two planes
10 authorized redirect `https://console.eitri.sh/auth/callback`. Note the 15
11 client id + secret. 16 | | prod | stg |
12 2. **server.json** (never committed; lives only in the Secret): 17 |---|---|---|
18 | namespace | `eitri` | `eitri-stg` |
19 | console | `console.eitri.sh` | `stg.eitri.sh` |
20 | MCP host | `api.eitri.sh` | `api.stg.eitri.sh` |
21 | gate | `gate.eitri.sh:2222` | `gate.stg.eitri.sh:2223` |
22 | sync | `sync.eitri.sh:8443` | `sync.stg.eitri.sh:8444` |
23 | site / `/dl` | `eitri.sh` | `dl.stg.eitri.sh` |
24 | http listener | `:8081` | `:8082` |
25 | guest CIDR pool | `10.78.0.0/16` | `10.79.0.0/16` |
26 | sign-in | Google | bundled `eitri-oidc` at `oidc.stg.eitri.sh` |
27 | backups | nightly | none — the database is disposable |
28 | CDN purge | yes | no — served cache-bypassed |
29
30 The pod runs `hostNetwork`, so those ports are the **node's**: the two planes
31 share a node and cannot share a port triple. Deployment, Service, PVC and
32 IngressRoute keep the same names in both namespaces — they are namespace-scoped,
33 so per-plane naming would be noise, and identical names mean `kubectl -n eitri`
34 and `kubectl -n eitri-stg` take the same commands. Only what is node-scoped or
35 cluster-visible (host ports, hostnames, TLS secrets) differs.
36
37 The guest CIDR pools differ so a host flicked between planes never carries a
38 colliding guest subnet. The local dev plane is `10.77.0.0/16`.
39
40 ### Reading a manifest
41
42 The manifests are templates. Rendering is how you read them:
43
44 scripts/ship.sh --target stg --tag v0.0.4-pre.1 --render-only
45
46 That touches nothing — no cluster, no build — and prints exactly what an apply
47 would send. Values come from `deploy/server/plane.<target>.env` (committed) and
48 `~/eitri-deploy/<target>/ship.env` (not). `envsubst` is called with an explicit
49 variable list and every name on it is checked non-empty first, because
50 `envsubst`'s own answer to an unset variable is to substitute nothing at all.
51
52 One template set, two planes, so the planes cannot differ in *shape* — only in
53 *values*. That is the whole premise: the shape under test at stg is the real one.
54
55 Two components are *present or absent* rather than shaped differently, each
56 gated on a plane value: the backup CronJob (`BACKUPS`) and the bundled issuer
57 (`LOCAL_OIDC`). Their values join the template variable list only on a plane
58 that has them, so a plane without one can never render a half-filled manifest
59 for it.
60
61 ### Sign-in, and the one place the planes genuinely differ
62
63 prod signs in against Google. stg runs `eitri-oidc`, the issuer that already
64 serves the local plane, so stg is reproducible from nothing but this repo and a
65 Secret, its identities are ours to create and delete, and the smoke can drive a
66 real sign-in headlessly — which is what lets the credential chain be *proven* at
67 stg rather than skipped.
68
69 It runs as its own Deployment from the **server's image at the server's tag**,
70 so the two can never be different builds, and it is an ordinary pod behind a
71 Service — nothing about it needs a socket on the node, so it costs the shared
72 node no port. It gets **its own hostname**, `oidc.stg.eitri.sh`, rather than a
73 path on the console: the login form posts back to the absolute path
74 `/authorize`, so mounted under a prefix the browser's POST would land on the
75 console host's catch-all rule instead of the issuer. An issuer only works at the
76 root of its own origin.
77
78 **Its state is not disposable.** `users.json` holds each identity's subject and
79 the server derives a tenant from that subject, so losing the file signs the
80 operator into a brand-new tenant while the old one still owns the plane's hosts
81 — with no error anywhere to say so. Hence its own PVC, pinned to the same node
82 as the server's.
83
84 What this costs: **stg does not exercise prod's relying-party wiring against a
85 third-party identity provider.** Everything after sign-in is identical; the
86 Google-specific half of prod's login is proven only at prod.
87
88 ### Two facts about routing that will surprise someone
89
90 **The console host carries every path, so `console.eitri.sh/mcp` is a working
91 MCP endpoint** — and it is the one a proxy fronts. `api.eitri.sh` is not the
92 only way in; it is the way in that does not also route the console (routing the
93 whole API host would make it a second console origin, and OIDC sign-in works at
94 exactly `oidc.public_url`). The smoke exercises both.
95
96 **stg's console is proxied and prod's is not.** stg is *more* fronted than prod,
97 which inverts the usual staging relationship and has one practical consequence:
98 the console's SSE streams traverse Cloudflare at stg and go direct at prod. If a
99 stg run shows a dead live-updating console or a stalled event stream, the first
100 hypothesis is proxy buffering, not a code defect — and the confirming test is
101 the same page against `api.stg.eitri.sh`, which is grey-cloud.
102
103 ## The pipeline
104
105 scripts/ship.sh --target <stg|prod> --tag <vX.Y.Z[-pre.N]>
106 [--from <stage>] [--skip-smoke] [--render-only]
107
108 Every stage announces itself and is idempotent; re-running from the top is
109 always safe and is the documented default. `--from <n>` resumes after a partial
110 failure.
111
112 1. **Verify the tag** — clean tree, HEAD is exactly this tag, and the version is
113 parsable by `internal/server/release`, asked by running that code rather than
114 re-deriving its rule. An unparsable version silently disables the upgrade
115 button fleet-wide. Releases are `vX.Y.Z`, pre-releases `vX.Y.Z-pre.N`, and
116 the fleet orders the whole chain:
117 `v0.0.4-pre.1 < v0.0.4-pre.2 < v0.0.4 < v0.0.4-2-g<hex>`.
118 2. **Build the artifacts** into `dist/<tag>`, with the manifest base pointed at
119 the target's own `/dl`. This is the one build input the planes legitimately
120 differ on, and it is why the stg run *proves* the download and upgrade paths
121 rather than rehearsing them.
122 3. **Build and push the images**, server and site, tagged with the version.
123 4. **Check the config** (see below).
124 5. **Apply the plane's shape** — namespace, middleware, storage, service,
125 certificates, routes. Nothing here restarts anything.
126 6. **Roll the site, then purge.** Order is load-bearing: the server fetches the
127 release manifest at boot and pins the answer for 24 hours, so a server rolled
128 ahead of its site serves the previous release to the whole fleet for a day.
129 7. **Roll the control plane.** Recreate strategy — a brief gap; agents redial.
130 8. **Hosted smoke**: the same `eitri-smoke` the branch gate runs, against this
131 plane's public names.
132 9. **Report** what is actually running.
133
134 ### Stage 4, the one that justifies the script
135
136 Every incident on v0.0.3 release day was a hosted-shape failure the local gate
137 cannot see. Stage 4 is where that class dies. It reads the plane's Secret, never
138 writes it, and never prints its contents.
139
140 *Schema drift, both directions.* The keys the **tagged tree** declares are
141 extracted from `internal/server/config/config.go` and compared against
142 `deploy/server/config.required`, which classifies each as `required`, `optional`
143 or `retired`. A key the tree grew that nobody classified is a hard failure
144 naming the key; a required key missing from the Secret is a hard failure naming
145 the key and the patch that fixes it; a retired key still set is a warning. Had
146 this existed, `default_images` would have failed a pre-deploy check the moment
147 the struct grew the field, instead of becoming a live secret patch.
148
149 *Plane agreement.* The Secret's `http_listen`, `quic_listen`, `ssh_listen`,
150 `ssh_gate_domain`, `advertise_http`, `advertise_quic` and `oidc.public_url` must
151 match the plane's rendered manifests. Each of these disagreeing is a failure
152 that presents as something else entirely: a wrong gate domain reads as "pubkey
153 denied", a wrong `advertise_quic` as a host that enrolls and then never syncs.
154
155 ### What stays operator-manual
156
157 Ordered by when they bite.
158
159 1. **The Google OAuth client — prod only.** No API, and a mistake is invisible
160 until sign-in fails. stg runs the bundled issuer and needs nothing from
161 Google.
162 2. **First-time Secret creation.** The pipeline validates config secrets and
163 never writes them: a script that can write one is a script that can overwrite
164 prod's. That is two secrets on a plane with a bundled issuer, one on prod.
165 3. **`regcred` copied into the namespace.**
166 4. **Registering the plane's identity — stg only**, one `kubectl exec` at
167 bring-up. The pipeline stays plane-agnostic: it forwards whatever credential
168 `ship.env` names and never creates one.
169 5. **Firewall openings** for the plane's sync and gate ports, in the cloud
170 security list *and* the node's host firewall.
171 6. **The Cloudflare purge token**, once, into
172 `~/eitri-deploy/pipeline/cf-purge.env` (`CF_ZONE_ID`, `CF_API_TOKEN`). Until
173 it exists a prod run refuses to finish without `ALLOW_MANUAL_PURGE=1`, which
174 is how you say out loud that you will purge by hand.
175 7. **DNS.** Verify rather than create: the `*.eitri.sh` wildcard covers the HTTP
176 names. `gate` and `sync` are the exception and need explicit A records — see
177 "Known gaps".
178 8. **Minting the operator PAT — prod only.** Where the plane's issuer is one of
179 ours, the smoke signs in and mints its own, so there is no token to paste and
180 none to rotate.
181 9. **Tagging.** The pipeline verifies tags; it never creates or pushes them.
182
183 ## One-time bring-up of a plane
184
185 Shown for stg. prod is the same minus steps 5 and 7, plus a Google OAuth client
186 (Web application, authorized redirect `https://console.eitri.sh/auth/callback`)
187 whose id and secret go into `oidc` below, and a hand-minted operator PAT at the
188 end.
189
190 1. **`eitri-oidc.json`** — 0600, in `~/eitri-deploy/stg/`, the bundled issuer's
191 own config. `issuer` must be the URL browsers and the server reach it at, and
192 `redirect_url` must equal the server's callback exactly; stage 4 asserts both
193 against the plane rather than letting them drift into a 400 at the end of an
194 otherwise working login.
195
196 ```json
197 {
198 "listen": ":9111",
199 "issuer": "https://oidc.stg.eitri.sh",
200 "users_file": "/var/lib/eitri-oidc/users.json",
201 "signing_key": "/var/lib/eitri-oidc/signing.key",
202 "clients": [
203 {"id": "eitri-console", "redirect_url": "https://stg.eitri.sh/auth/callback"}
204 ]
205 }
206 ```
207
208 2. **`server.json`** — 0600, in `~/eitri-deploy/<target>/`, never committed and
209 never written by a script:
13 210
14 ```json 211 ```json
15 { 212 {
16 "db_path": "/var/lib/eitri/eitri.db", 213 "db_path": "/var/lib/eitri/eitri.db",
17 "http_listen": ":8081", 214 "http_listen": ":8082",
18 "quic_listen": ":8443", 215 "quic_listen": ":8444",
19 "default_image_url": "https://cloud-images.ubuntu.com/resolute/20260720/resolute-server-cloudimg-amd64.img", 216 "ssh_listen": ":2223",
20 "default_image_sha256": "117816726abbdefc5ef3e38902e81a76f1c76c3610e709999d0885f9d5d9b477", 217 "ssh_gate_domain": "gate.stg.eitri.sh",
21 "advertise_http": "https://console.eitri.sh",
22 "advertise_quic": "sync.eitri.sh:8443",
23 "cidr_pool": "10.78.0.0/16",
24 "ssh_listen": ":2222",
25 "ssh_gate_domain": "gate.eitri.sh",
26 "ssh_ca_key": "/var/lib/eitri/ssh_ca", 218 "ssh_ca_key": "/var/lib/eitri/ssh_ca",
27 "ssh_host_key": "/var/lib/eitri/ssh_host_key", 219 "ssh_host_key": "/var/lib/eitri/ssh_host_key",
220 "advertise_http": "https://stg.eitri.sh",
221 "advertise_quic": "sync.stg.eitri.sh:8444",
222 "cidr_pool": "10.79.0.0/16",
28 "host_secret": "<openssl rand -hex 32>", 223 "host_secret": "<openssl rand -hex 32>",
224 "release_manifest_url": "https://dl.stg.eitri.sh/dl/latest/manifest.json",
225 "default_images": {
226 "amd64": {"url": "...", "sha256": "..."},
227 "arm64": {"url": "...", "sha256": "..."}
228 },
29 "oidc": { 229 "oidc": {
30 "issuer": "https://accounts.google.com", 230 "issuer": "https://oidc.stg.eitri.sh",
31 "client_id": "<google client id>", 231 "client_id": "eitri-console",
32 "client_secret": "<google client secret>", 232 "public_url": "https://stg.eitri.sh",
33 "public_url": "https://console.eitri.sh" 233 "allowed_identities": ["ship@eitri.local"]
34 } 234 }
35 } 235 }
36 ``` 236 ```
37 237
38 ``` 238 No `client_secret`: the bundled issuer registers public PKCE clients, and the
39 kubectl -n eitri create secret generic eitri-server-config \ 239 server pins the token request's client-auth style accordingly. prod's Google
40 --from-file=server.json=./server.json 240 client is confidential and does set one.
41 ``` 241
242 `allowed_identities` closes the signup gate. Both architectures get a
243 `default_images` entry: a host handed an image it cannot execute fails at
244 boot, which is the mistake that map exists to prevent — and stage 4 enforces
245 it for every entry, not just the first.
246
247 3. **Namespace and pull secret**:
248
249 kubectl create ns eitri-stg
250 kubectl -n eitri get secret regcred -o yaml \
251 | sed 's/namespace: eitri/namespace: eitri-stg/' | kubectl apply -f -
252
253 4. **The config Secrets** — the server's, and the issuer's where there is one:
254
255 kubectl -n eitri-stg create secret generic eitri-stg-server-config \
256 --from-file=server.json=$HOME/eitri-deploy/stg/server.json
257 kubectl -n eitri-stg create secret generic eitri-stg-oidc-config \
258 --from-file=eitri-oidc.json=$HOME/eitri-deploy/stg/eitri-oidc.json
259
260 5. **`ship.env`**: `cp scripts/ship.env.example ~/eitri-deploy/stg/ship.env`
261 and fill it in. For stg that means `CI_USER` and `CI_PASSWORD_FILE`; generate
262 the password once and lock it down:
263
264 umask 077; openssl rand -hex 16 > ~/eitri-deploy/stg/ship-password
265
266 6. **First run**: `scripts/ship.sh --target stg --tag <pre-tag> --skip-smoke`
267 applies everything else and starts both deployments. The server creates its
268 own SSH CA and gate host key on the PVC at first boot, so the gate comes up
269 on its own. Skip the smoke on this one run: the identity it signs in as does
270 not exist yet.
271
272 7. **Register the plane's identity**, once, in the running issuer. It writes to
273 the issuer's PVC, so it survives every later roll:
274
275 kubectl -n eitri-stg exec deploy/eitri-oidc -i -- \
276 /eitri-oidc user add --config /etc/eitri/eitri-oidc.json \
277 --password-file /dev/stdin ship@eitri.local \
278 < ~/eitri-deploy/stg/ship-password
279
280 The password arrives on stdin rather than in an argument, which would put it
281 in the pod's process list. Re-running this changes the password and **keeps
282 the identity's subject**, so the tenant it owns survives a rotation.
283
284 8. **Sign in at exactly `https://stg.eitri.sh`** — anything else gives "invalid
285 oauth state" — which JIT-provisions the operator tenant that will own the
286 plane's hosts. Then `scripts/ship.sh --target stg --tag <pre-tag> --from 8`
287 proves the whole thing.
288
289 There is no PAT to mint by hand here. The smoke signs in as that same
290 identity and mints its own short-lived token for the run, deriving the tenant
291 from it rather than being told. prod, which cannot sign in headlessly against
292 Google, still needs a non-expiring PAT named `ship` saved to
293 `~/eitri-deploy/prod/deploy-pat` (0600) and named in its `ship.env`.
294
295 ### Adopting prod's site objects
296
297 The `site-*.yaml` files describe the objects that have served `eitri.sh` since
298 before they were in the tree, and they carry those objects' own names —
299 Deployment and Service `web`, IngressRoutes `web-http` and `web-https`,
300 Certificate `web-tls`. They adopt the live site; they do not stand a second one
301 beside it. Confirm that before the first prod run:
42 302
43 The pool is deliberately 10.78/16 — the dev fleet is 10.77/16, so a 303 scripts/ship.sh --target prod --tag <tag> --render-only > /tmp/prod.yaml
44 host flicked between fleets never carries colliding guest subnets. 304 kubectl diff -f /tmp/prod.yaml
45 3. **Image**: `make server-image` (needs SERVER_IMAGE in deploy.env), then
46 `kubectl -n eitri set image deployment/eitri-server eitri-server=<SERVER_IMAGE>:<version>`.
47 4. **Apply**: `kubectl apply -f pvc.yaml -f deployment.yaml -f service.yaml -f certificate.yaml -f ingressroute.yaml -f backup-cronjob.yaml`. The server creates its own SSH CA and gate host key on the PVC on first boot, so the gate comes up on its own — no key step is required.
48 5. **Network** (operator):
49 - Oracle security list AND vnic-1 host firewall: open 8443/udp, 2222/tcp.
50 - Cloudflare DNS: `console.eitri.sh` proxied to the eitri.sh origin (repoint off the old tailnet record — confirm nothing dev-side still resolves it); `sync.eitri.sh` and `gate.eitri.sh` grey-cloud A records to vnic-1's public IP.
51 - `api.eitri.sh` (the MCP endpoint): A record to the same origin as console. Proxied works only when the MCP client sends progress tokens — a `vm_create` that stays silent past Cloudflare's ~100s origin timeout gets a 524; if the client under test doesn't, use a grey-cloud record.
52 6. **Backups**: nightly CronJob writes dated sqlite backups on the PVC; run `backup-pull.sh` from cron on an off-cluster machine — local-path storage does not survive the node, the off-node copy is the DR story.
53 305
54 ## Rollout 306 Every object should read as an update, never a create. One intentional
307 difference is expected: `web-http` gains the `redirect-https` middleware the
308 console's HTTP route already uses, so plain-HTTP `eitri.sh` starts redirecting.
309 The pipeline never deletes, so anything the live namespace carries that these
310 manifests do not describe — an unused `web-data` PVC, say — is left alone.
55 311
56 make server-image 312 ## Runbook: the nested dev fleet
57 kubectl -n eitri set image deployment/eitri-server eitri-server=<SERVER_IMAGE>:<version>
58 313
59 Single replica + Recreate: a rollout is a short outage; agents reconnect 314 The branch gate's fleet lives on the workstation, defined by
60 and re-sync on their own (the sync registry is rebuilt from Hellos). 315 `scripts/devhost.sh` rather than hand-built. Bring it up **before** mewtwo
316 moves to stg — until it is green, the project has no branch gate.
61 317
62 The pod runs hostNetwork: the QUIC sync socket (8443/udp) and SSH gate 318 scripts/devhost.sh create # from nothing
63 (2222/tcp) bind vnic-1's interfaces directly — both flows carry pinned 319 scripts/devhost.sh recycle # destroy + create; a wedged host is 3 minutes
64 end-to-end cryptography and tolerate no middlebox, including the CNI's 320 scripts/devhost.sh destroy
65 hostPort NAT, which conntrack-drops long-lived single-tuple UDP flows.
66 The console listens on :8081 (svclb claims host 8080); the Service maps
67 8080 → 8081 so the IngressRoute is unaffected.
68 321
69 ## Re-homing a host from another fleet 322 `create` prints the `AGENT_HOSTS` and `AGENT_EXTRA_FLAGS` lines for
323 `~/eitri-deploy/deploy.env`. **`make deploy` itself does not change** — only
324 which host it points at.
325
326 All four of these must pass before mewtwo is allowed to move:
327
328 1. `devhost.sh create` from nothing, then `make deploy` → boot gate **PASS**
329 including `gate SSH: ok`, `exposed port: ok`, `remote MCP: ok`. A first pass
330 on a cold host also proves the agent bootstrap — cloud-hypervisor and
331 `CLOUDHV.fd` downloaded and sha-verified — works nested.
332 2. A second `make deploy` on the warm host → PASS. Proves the agent-swap path
333 and VM re-adoption.
334 3. `devhost.sh recycle`, then `make deploy` → PASS. Proves the recycle story is
335 real and not a one-time hand-built machine.
336 4. `coverage/integration` is written, so the gate's by-product survives the move.
337
338 If a nested guest cannot boot at all, everything downstream stops. That is why
339 this goes first.
340
341 ## Runbook: moving mewtwo from the local plane to stg
342
343 Preconditions: the dev fleet's four proofs are green, and stg is up — console
344 reachable, sign-in works, `/mcp` refuses an unauthenticated caller, certificate
345 valid. The local plane stays running throughout, so rollback is always a join
346 away.
347
348 1. **Drain.** List mewtwo's VMs through the local plane and delete every one.
349 Guests survive an agent swap, not a re-enrollment. Confirm nothing on mewtwo
350 is precious before starting.
351 2. **Remove mewtwo from `AGENT_HOSTS` in `~/eitri-deploy/deploy.env`, in the
352 same sitting.** This is the step most likely to be forgotten and the most
353 damaging to forget: the next `make deploy` would scp a coverage-instrumented
354 branch binary over mewtwo's release binary, restart it, and quietly drag it
355 back toward the dev fleet. Do it before the join, not after.
356 3. **Decommission from the local plane**: `DELETE /api/v1/hosts/{id}` with the
357 local operator PAT.
358 4. **Wipe the plane identity.** On mewtwo: `sudo systemctl stop eitri-agent`,
359 then remove `/var/lib/eitri-agent/{identity.json,epoch,vms}`. Keep `images/`
360 — the image cache is plane-agnostic and saves a large download. Removing
361 `epoch` is what avoids the epoch fence violation described below.
362 5. **Install the release binary, not a build.** Fetch
363 `eitri-server_<tag>_linux_amd64.tar.gz` from `https://dl.stg.eitri.sh/dl/<tag>/`,
364 verify it against `SHA256SUMS`, install. From here mewtwo runs release
365 artifacts — which is the coverage stg exists to provide.
366 6. **Join stg.** Mint a join token in the stg console as the stg operator, run
367 `eitri-agent join <blob>` on mewtwo, `systemctl start eitri-agent`. Confirm
368 the host appears with the right architecture, uplink and guest subnet.
369 7. **Prove it**: `scripts/ship.sh --target stg --tag <tag> --from 8`. A full
370 PASS including gate SSH, exposed port and remote MCP is the acceptance
371 criterion.
372
373 **Rollback**, at any point: re-mint a join token on the local plane, repeat step
374 4's wipe, re-join, restore `AGENT_HOSTS`. The cost is one more wipe and whatever
375 guests exist at the time.
376
377 ## Runbook: the first pre-release run
378
379 1. Land the pipeline on the version branch and get `make deploy` green on the
380 nested dev host — that is the gate for the pipeline code itself.
381 2. Tag `v0.0.4-pre.1` **on the version branch** and push the tag. Pre-tags are
382 branch tags; main stays untouched until release.
383 3. `scripts/ship.sh --target stg --tag v0.0.4-pre.1`. Expect stage 4 to fail the
384 first time — that is the mechanism working. The fix is a Secret patch plus a
385 `config.required` classification, both recorded.
386 4. Migrate mewtwo (above) and re-run stage 8.
387 5. **The proof this run exists to produce**: drive `vm_create` over
388 `https://stg.eitri.sh/mcp` — the proxied origin — and observe whether progress
389 tokens keep Cloudflare from 524-ing a ten-minute call. Record the answer in
390 `docs/assumptions.md` either way. A negative result is exactly as valuable
391 and changes what the release can claim.
392 6. Iterate with `-pre.2`, `-pre.3` as findings land. **The second pre-tag
393 rehearses the agent upgrade**, which is the failure that defined the v0.0.3
394 release day and the reason the version ordering understands pre-releases:
395 after `--target stg --tag v0.0.4-pre.2` has rolled, mewtwo is still on
396 `-pre.1` and the console must offer it the upgrade. Take it from the console,
397 confirm the agent comes back reporting `v0.0.4-pre.2` with its guests intact,
398 and that the artifact it fetched came from `dl.stg.eitri.sh`. An upgrade that
399 is not offered means the manifest, not the ordering — check
400 `release_manifest_url` and that the site rolled before the server.
401 7. When stg is clean, tag the release on main and run
402 `scripts/ship.sh --target prod --tag v0.0.4` — same script, same stages,
403 nothing new attempted. The last pre-release upgrades to it like any other
404 version, because it orders below it.
405
406 ## Known gaps
407
408 **The CDN purge's first real execution is at promote time.** stg is served
409 cache-bypassed, so a purge there would invalidate nothing and make the prod step
410 look practiced when it is not. The mitigation is to keep the step small enough
411 that an unrehearsed run is safe: one `curl` with an explicit file list built
412 from `dist/<tag>`, no cache tags, no zone-wide purge. A zone-wide purge is the
413 tempting simplification and the wrong one — it would evict the whole site's
414 cache on every release.
415
416 **`gate` and `sync` do not inherit the wildcard usefully.** Every `*.eitri.sh`
417 name resolves to the home origin, where Traefik runs, but the control-plane pod
418 is `hostNetwork` on the cloud node — so its SSH gate and QUIC sync bind *that*
419 node's interfaces. The HTTP path works through the Service; the raw TCP and UDP
420 listeners do not. Both planes want explicit A records for `gate.*` and `sync.*`
421 pointing at the pod's node, and the matching ports opened there. Verify with
422 `nc -vz gate.eitri.sh 2222` from outside the LAN before trusting a gate leg.
423
424 **Nested cannot prove bridged networking or real-hardware quirks.** That
425 coverage moved from per-branch to per-pre-tag when mewtwo moved to stg —
426 deliberate, and the reason stg gates releases on real metal.
427
428 **stg's sign-in is not prod's.** stg runs the bundled issuer, so what a stg run
429 proves is the credential chain end to end against an issuer of ours — and what
430 it does not touch is prod's relying-party wiring against Google. Everything
431 after sign-in is identical. See "Sign-in, and the one place the planes genuinely
432 differ".
433
434 ## Re-homing a host between planes
70 435
71 A host that previously belonged to a different control plane carries that 436 A host that previously belonged to a different control plane carries that
72 fleet's snapshot epoch in its state dir and will refuse the new server's 437 plane's snapshot epoch in its state dir and will refuse the new server's
73 lower-numbered snapshots ("epoch fence violation" in the server log). 438 lower-numbered snapshots ("epoch fence violation" in the server log). After
74 After `eitri-agent join` against the new fleet: 439 `eitri-agent join` against the new plane:
75 440
76 sudo systemctl stop eitri-agent 441 sudo systemctl stop eitri-agent
77 sudo rm /var/lib/eitri-agent/epoch 442 sudo rm /var/lib/eitri-agent/epoch
78 sudo systemctl start eitri-agent 443 sudo systemctl start eitri-agent
79 444
80 VMs from the old fleet are absent from the new fleet's desired state and 445 VMs from the old plane are absent from the new plane's desired state and are
81 are reaped through the normal quarantine grace — re-home a host only when 446 reaped through the normal quarantine grace — re-home a host only when its
82 its existing guests are disposable. 447 existing guests are disposable.
448
449 ## Backups
450
451 The nightly CronJob writes dated sqlite backups onto prod's PVC; run
452 `backup-pull.sh` from cron on an off-cluster machine. local-path storage does
453 not survive the node, so the off-node copy is the DR story. stg has no CronJob:
454 its database is disposable by design.
deploy/server/backup-cronjob.yaml
Old New
@@ -1,8 +1,11 @@
1 # Nightly dated sqlite backups onto the PVC. Applied only where BACKUPS=1:
2 # local-path storage does not survive the node, so the off-node copy pulled by
3 # backup-pull.sh is the real DR story and this is what it pulls.
1 apiVersion: batch/v1 4 apiVersion: batch/v1
2 kind: CronJob 5 kind: CronJob
3 metadata: 6 metadata:
4 name: eitri-server-backup 7 name: eitri-server-backup
5 namespace: eitri 8 namespace: ${NAMESPACE}
6 spec: 9 spec:
7 schedule: "20 3 * * *" 10 schedule: "20 3 * * *"
8 concurrencyPolicy: Forbid 11 concurrencyPolicy: Forbid
@@ -12,7 +15,7 @@ spec:
12 spec: 15 spec:
13 restartPolicy: Never 16 restartPolicy: Never
14 nodeSelector: 17 nodeSelector:
15 kubernetes.io/hostname: vnic-1 18 kubernetes.io/hostname: ${NODE_NAME}
16 containers: 19 containers:
17 - name: backup 20 - name: backup
18 image: alpine:3.20 21 image: alpine:3.20
@@ -28,4 +31,4 @@ spec:
28 - {name: data, mountPath: /data} 31 - {name: data, mountPath: /data}
29 volumes: 32 volumes:
30 - name: data 33 - name: data
31 persistentVolumeClaim: {claimName: eitri-server-data} 34 persistentVolumeClaim: {claimName: ${PVC_NAME}}
deploy/server/certificate.yaml
Old New
@@ -1,11 +1,18 @@
1 # The console and the MCP host share one certificate. The gate and sync names
2 # are deliberately absent: SSH and QUIC carry their own pinned credentials and
3 # never present a web certificate.
4 #
5 # The issuer solves DNS-01 through Cloudflare, so this issues before any route
6 # answers and regardless of whether the origin is reachable — which is why the
7 # certificate applies in the same pass as everything else rather than after it.
1 apiVersion: cert-manager.io/v1 8 apiVersion: cert-manager.io/v1
2 kind: Certificate 9 kind: Certificate
3 metadata: 10 metadata:
4 name: console-tls 11 name: ${TLS_SECRET}
5 namespace: eitri 12 namespace: ${NAMESPACE}
6 spec: 13 spec:
7 secretName: console-tls 14 secretName: ${TLS_SECRET}
8 dnsNames: [console.eitri.sh, api.eitri.sh] 15 dnsNames: [${CONSOLE_HOST}, ${API_HOST}]
9 issuerRef: 16 issuerRef:
10 name: letsencrypt-prod 17 name: letsencrypt-prod
11 kind: ClusterIssuer 18 kind: ClusterIssuer
deploy/server/config.required
Old New
@@ -0,0 +1,70 @@
1 # The server config contract: every key the tagged tree's config schema declares,
2 # classified. scripts/ship.sh reads the schema out of
3 # internal/server/config/config.go on the tree being shipped, compares it against
4 # this file, and refuses to deploy a plane whose Secret and whose binary disagree.
5 #
6 # required the plane's server.json must set it to a non-empty value
7 # optional the server has a working default, or the key is a deliberate opt-in
8 # retired the field survives only so the server can spot it in an old config
9 # and say what to write instead; a plane still setting it gets a warning
10 #
11 # A key the tree declares and this file does not classify is a hard failure, on
12 # purpose: that is exactly the shape of the v0.0.3 release-day incident, where the
13 # struct grew default_images and every plane's Secret silently lacked it. The
14 # schema and its contract now move together or not at all.
15 #
16 # Nested structs reach this file through their JSON path, declared here so a new
17 # one cannot slip in unclassified — the extractor fails on a struct with no
18 # prefix. A "*" segment stands for a map key: the rule holds for every entry.
19 #
20 # struct-prefix: Config=
21 # struct-prefix: OIDC=oidc.
22 # struct-prefix: DefaultImage=default_images.*.
23
24 db_path required
25 http_listen required
26 quic_listen required
27 host_secret required
28 # Seals every piece of key material the plane holds — the host CA and gate host
29 # key on the PVC, and each tenant's managed SSH CA in the database — so the
30 # volume, the database and its backups hold ciphertext. 64 hex chars, minted with
31 # `openssl rand -hex 32`, and never rotated in place: a plane that loses it loses
32 # its host CA, and with it the identity every client pins and every VM's host
33 # certificate names. This Secret is that key's only home; keep the operator's
34 # copy of server.json.
35 key_encryption_key required
36 cidr_pool required
37 advertise_http required
38 advertise_quic required
39 ssh_listen required
40 ssh_gate_domain required
41 ssh_ca_key required
42 ssh_host_key required
43
44 # One image per host architecture. A plane serving both arm64 and amd64 hosts
45 # needs both entries: a host handed an image it cannot execute fails at boot,
46 # which is the mistake this map exists to prevent.
47 default_images required
48 default_images.*.url required
49 default_images.*.sha256 required
50
51 oidc required
52 oidc.issuer required
53 oidc.client_id required
54 oidc.public_url required
55 # Confidential clients only — the bundled loopback issuer is public.
56 oidc.client_secret optional
57 # The signup gate. Empty means anyone the issuer authenticates gets a tenant, so
58 # a hosted plane that is not open for signups sets one of these.
59 oidc.allowed_domains optional
60 oidc.allowed_identities optional
61
62 # Absent means the built-in default (eitri.sh); an explicit empty string
63 # disables release discovery and every upgrade surface with it.
64 release_manifest_url optional
65 credential_max_age optional
66 audit_retention optional
67
68 admin_token retired
69 default_image_url retired
70 default_image_sha256 retired
deploy/server/deployment.yaml
Old New
@@ -2,7 +2,7 @@ apiVersion: apps/v1
2 kind: Deployment 2 kind: Deployment
3 metadata: 3 metadata:
4 name: eitri-server 4 name: eitri-server
5 namespace: eitri 5 namespace: ${NAMESPACE}
6 spec: 6 spec:
7 replicas: 1 7 replicas: 1
8 strategy: 8 strategy:
@@ -13,17 +13,17 @@ spec:
13 metadata: 13 metadata:
14 labels: {app: eitri-server} 14 labels: {app: eitri-server}
15 spec: 15 spec:
16 # vnic-1 is load-bearing twice over: the only node with a public IP 16 # The node is load-bearing twice over: the only one with a public IP
17 # (sync/gate listeners below), and local-path binds the PVC to the node 17 # (sync/gate listeners below), and local-path binds the PVC to the node
18 # the pod first lands on. 18 # the pod first lands on.
19 nodeSelector: 19 nodeSelector:
20 kubernetes.io/hostname: vnic-1 20 kubernetes.io/hostname: ${NODE_NAME}
21 # hostNetwork: the QUIC sync socket (8443/udp) and SSH gate (2222/tcp) 21 # hostNetwork: the QUIC sync socket and SSH gate bind the node's
22 # bind the node's interfaces directly. Both flows carry their own pinned 22 # interfaces directly. Both flows carry their own pinned cryptography
23 # cryptography end-to-end and tolerate no middlebox — that includes the 23 # end-to-end and tolerate no middlebox — that includes the CNI's own
24 # CNI's own hostPort NAT, which conntrack-drops long-lived single-tuple 24 # hostPort NAT, which conntrack-drops long-lived single-tuple UDP flows.
25 # UDP flows. The console listens on :8081 because svclb already claims 25 # It also means these ports are the NODE's: two planes on one node need
26 # host 8080. 26 # two port triples, which is what the plane env files carry.
27 hostNetwork: true 27 hostNetwork: true
28 dnsPolicy: ClusterFirstWithHostNet 28 dnsPolicy: ClusterFirstWithHostNet
29 imagePullSecrets: 29 imagePullSecrets:
@@ -32,28 +32,29 @@ spec:
32 fsGroup: 65532 # distroless nonroot; PVC files must be writable 32 fsGroup: 65532 # distroless nonroot; PVC files must be writable
33 containers: 33 containers:
34 - name: eitri-server 34 - name: eitri-server
35 image: REGISTRY/eitri-server:VERSION # set at rollout: kubectl -n eitri set image ... 35 # By TAG, never by digest: a podman-pushed digest differs from the
36 # Always: a re-cut release reuses its version tag, and podman-pushed 36 # registry's, and rolling by one has taken the fleet down overnight.
37 # digests differ from the registry's — the tag plus a forced pull is 37 image: ${SERVER_IMAGE}:${TAG}
38 # the one rollout shape that never serves a stale image. 38 # Always: a re-cut release reuses its version tag, and the tag plus a
39 # forced pull is the one rollout shape that never serves a stale image.
39 imagePullPolicy: Always 40 imagePullPolicy: Always
40 ports: 41 ports:
41 - {name: http, containerPort: 8081} 42 - {name: http, containerPort: ${HTTP_PORT}}
42 - {name: sync, containerPort: 8443, protocol: UDP} 43 - {name: sync, containerPort: ${SYNC_PORT}, protocol: UDP}
43 - {name: gate, containerPort: 2222, protocol: TCP} 44 - {name: gate, containerPort: ${GATE_PORT}, protocol: TCP}
44 volumeMounts: 45 volumeMounts:
45 - {name: data, mountPath: /var/lib/eitri} 46 - {name: data, mountPath: /var/lib/eitri}
46 - {name: config, mountPath: /etc/eitri, readOnly: true} 47 - {name: config, mountPath: /etc/eitri, readOnly: true}
47 livenessProbe: 48 livenessProbe:
48 httpGet: {path: /livez, port: 8081} 49 httpGet: {path: /livez, port: ${HTTP_PORT}}
49 periodSeconds: 10 50 periodSeconds: 10
50 readinessProbe: 51 readinessProbe:
51 httpGet: {path: /readyz, port: 8081} 52 httpGet: {path: /readyz, port: ${HTTP_PORT}}
52 periodSeconds: 10 53 periodSeconds: 10
53 volumes: 54 volumes:
54 - name: data 55 - name: data
55 persistentVolumeClaim: {claimName: eitri-server-data} 56 persistentVolumeClaim: {claimName: ${PVC_NAME}}
56 - name: config 57 - name: config
57 secret: 58 secret:
58 secretName: eitri-server-config 59 secretName: ${CONFIG_SECRET}
59 defaultMode: 0400 60 defaultMode: 0400
deploy/server/ingressroute.yaml
Old New
@@ -2,38 +2,41 @@ apiVersion: traefik.io/v1alpha1
2 kind: IngressRoute 2 kind: IngressRoute
3 metadata: 3 metadata:
4 name: console-https 4 name: console-https
5 namespace: eitri 5 namespace: ${NAMESPACE}
6 spec: 6 spec:
7 entryPoints: [websecure] 7 entryPoints: [websecure]
8 routes: 8 routes:
9 - match: Host(`console.eitri.sh`) 9 # The console host rule carries every path, so /mcp answers here too — and
10 # this is the origin a proxy fronts. The API host below is not the only way
11 # in; it is the way in that doesn't also route the console.
12 - match: Host(`${CONSOLE_HOST}`)
10 kind: Rule 13 kind: Rule
11 services: 14 services:
12 - {name: eitri-server, port: 8080} 15 - {name: eitri-server, port: 8080}
13 # api.eitri.sh carries ONLY /mcp: routing the whole host would make it a 16 # The API host carries ONLY /mcp: routing the whole host would make it a
14 # second console origin, and OIDC sign-in works at exactly oidc.public_url. 17 # second console origin, and OIDC sign-in works at exactly oidc.public_url.
15 - match: Host(`api.eitri.sh`) && PathPrefix(`/mcp`) 18 - match: Host(`${API_HOST}`) && PathPrefix(`/mcp`)
16 kind: Rule 19 kind: Rule
17 services: 20 services:
18 - {name: eitri-server, port: 8080} 21 - {name: eitri-server, port: 8080}
19 tls: 22 tls:
20 secretName: console-tls 23 secretName: ${TLS_SECRET}
21 --- 24 ---
22 apiVersion: traefik.io/v1alpha1 25 apiVersion: traefik.io/v1alpha1
23 kind: IngressRoute 26 kind: IngressRoute
24 metadata: 27 metadata:
25 name: console-http 28 name: console-http
26 namespace: eitri 29 namespace: ${NAMESPACE}
27 spec: 30 spec:
28 entryPoints: [web] 31 entryPoints: [web]
29 routes: 32 routes:
30 - match: Host(`console.eitri.sh`) 33 - match: Host(`${CONSOLE_HOST}`)
31 kind: Rule 34 kind: Rule
32 middlewares: 35 middlewares:
33 - name: redirect-https 36 - name: redirect-https
34 services: 37 services:
35 - {name: eitri-server, port: 8080} 38 - {name: eitri-server, port: 8080}
36 - match: Host(`api.eitri.sh`) && PathPrefix(`/mcp`) 39 - match: Host(`${API_HOST}`) && PathPrefix(`/mcp`)
37 kind: Rule 40 kind: Rule
38 middlewares: 41 middlewares:
39 - name: redirect-https 42 - name: redirect-https
deploy/server/middleware.yaml
Old New
@@ -0,0 +1,12 @@
1 # Every plain-HTTP route sends the caller back over TLS. Namespace-scoped, so
2 # each plane carries its own copy under the same name and the IngressRoutes
3 # reference it without qualification.
4 apiVersion: traefik.io/v1alpha1
5 kind: Middleware
6 metadata:
7 name: redirect-https
8 namespace: ${NAMESPACE}
9 spec:
10 redirectScheme:
11 scheme: https
12 permanent: true
deploy/server/namespace.yaml
Old New
@@ -0,0 +1,4 @@
1 apiVersion: v1
2 kind: Namespace
3 metadata:
4 name: ${NAMESPACE}
deploy/server/oidc-deployment.yaml
Old New
@@ -0,0 +1,50 @@
1 # The bundled issuer runs from the SERVER's image, at the server's tag, as its
2 # own Deployment — so it cannot be a different version than the server it
3 # authenticates for, and it rolls in the same stage.
4 #
5 # Not hostNetwork, unlike the control plane: nothing here needs a raw socket on
6 # the node, so it reaches the world through the Service and leaves the node's
7 # port space to the three listeners that genuinely cannot use it.
8 apiVersion: apps/v1
9 kind: Deployment
10 metadata:
11 name: eitri-oidc
12 namespace: ${NAMESPACE}
13 spec:
14 replicas: 1
15 strategy:
16 type: Recreate # single writer for the flat user file
17 selector:
18 matchLabels: {app: eitri-oidc}
19 template:
20 metadata:
21 labels: {app: eitri-oidc}
22 spec:
23 # local-path binds the PVC to the node the pod first lands on, and the
24 # issuer's state must land on the same node as the rest of the plane's.
25 nodeSelector:
26 kubernetes.io/hostname: ${NODE_NAME}
27 imagePullSecrets:
28 - name: regcred
29 securityContext:
30 fsGroup: 65532 # distroless nonroot; the user file must be writable
31 containers:
32 - name: eitri-oidc
33 image: ${SERVER_IMAGE}:${TAG}
34 imagePullPolicy: Always
35 command: ["/eitri-oidc", "-config", "/etc/eitri/eitri-oidc.json"]
36 ports:
37 - {name: http, containerPort: ${OIDC_PORT}}
38 volumeMounts:
39 - {name: data, mountPath: /var/lib/eitri-oidc}
40 - {name: config, mountPath: /etc/eitri, readOnly: true}
41 readinessProbe:
42 httpGet: {path: /.well-known/openid-configuration, port: ${OIDC_PORT}}
43 periodSeconds: 10
44 volumes:
45 - name: data
46 persistentVolumeClaim: {claimName: ${OIDC_PVC_NAME}}
47 - name: config
48 secret:
49 secretName: ${OIDC_CONFIG_SECRET}
50 defaultMode: 0400
deploy/server/oidc.yaml
Old New
@@ -0,0 +1,75 @@
1 # The bundled issuer, for a plane with no external identity provider. Applied
2 # only where LOCAL_OIDC=1 — prod signs in against Google and never starts it.
3 #
4 # It gets its own hostname rather than a path on the console's, because the
5 # login form posts back to the absolute path /authorize: mounted under a prefix,
6 # the browser's POST would land on the console host's catch-all rule instead of
7 # the issuer. An issuer only works at the root of its own origin.
8 #
9 # Its state is NOT disposable. users.json holds each identity's subject, and the
10 # server derives a tenant from that subject — lose the file and the operator
11 # signs in to a brand-new tenant while the old one still owns the plane's hosts,
12 # with no error anywhere to say so. Hence a PVC, pinned like the server's.
13 apiVersion: v1
14 kind: PersistentVolumeClaim
15 metadata:
16 name: ${OIDC_PVC_NAME}
17 namespace: ${NAMESPACE}
18 spec:
19 accessModes: [ReadWriteOnce]
20 storageClassName: local-path
21 resources:
22 requests:
23 storage: ${OIDC_PVC_SIZE}
24 ---
25 apiVersion: v1
26 kind: Service
27 metadata:
28 name: eitri-oidc
29 namespace: ${NAMESPACE}
30 spec:
31 selector: {app: eitri-oidc}
32 ports:
33 - {name: http, port: ${OIDC_PORT}, targetPort: ${OIDC_PORT}}
34 ---
35 apiVersion: cert-manager.io/v1
36 kind: Certificate
37 metadata:
38 name: ${OIDC_TLS_SECRET}
39 namespace: ${NAMESPACE}
40 spec:
41 secretName: ${OIDC_TLS_SECRET}
42 dnsNames: [${OIDC_HOST}]
43 issuerRef:
44 name: letsencrypt-prod
45 kind: ClusterIssuer
46 ---
47 apiVersion: traefik.io/v1alpha1
48 kind: IngressRoute
49 metadata:
50 name: oidc-https
51 namespace: ${NAMESPACE}
52 spec:
53 entryPoints: [websecure]
54 routes:
55 - match: Host(`${OIDC_HOST}`)
56 kind: Rule
57 services:
58 - {name: eitri-oidc, port: ${OIDC_PORT}}
59 tls:
60 secretName: ${OIDC_TLS_SECRET}
61 ---
62 apiVersion: traefik.io/v1alpha1
63 kind: IngressRoute
64 metadata:
65 name: oidc-http
66 namespace: ${NAMESPACE}
67 spec:
68 entryPoints: [web]
69 routes:
70 - match: Host(`${OIDC_HOST}`)
71 kind: Rule
72 middlewares:
73 - name: redirect-https
74 services:
75 - {name: eitri-oidc, port: ${OIDC_PORT}}
deploy/server/plane.prod.env
Old New
@@ -0,0 +1,54 @@
1 # prod plane values — eitri.sh.
2 #
3 # One template set renders both planes (see README.md), so stg and prod cannot
4 # have different SHAPES, only different values. Every key here is required:
5 # scripts/ship.sh refuses to render with any of them empty rather than letting
6 # envsubst quietly substitute nothing.
7 #
8 # Nothing here is a secret. Config values that are (client secrets, the host
9 # secret, the OIDC credentials) live only in the plane's Secret, which the
10 # pipeline reads and validates but never writes.
11
12 NAMESPACE=eitri
13
14 # The node the control plane is pinned to. Load-bearing twice over: the only
15 # node with a public IP (the sync and gate listeners bind its interfaces
16 # directly), and local-path storage binds the PVC to whichever node the pod
17 # first lands on.
18 NODE_NAME=vnic-1
19 # The architecture both images are built for (scripts/server-image.sh is arm64
20 # by construction; the site image follows SITE_PLATFORM).
21 IMAGE_ARCH=arm64
22
23 CONSOLE_HOST=console.eitri.sh
24 API_HOST=api.eitri.sh
25 GATE_HOST=gate.eitri.sh
26 SYNC_HOST=sync.eitri.sh
27 SITE_HOST=eitri.sh
28
29 # Host ports. hostNetwork means these are the NODE's ports, so the two planes
30 # cannot share them; every value here must equal the matching listener in the
31 # plane's server.json, which ship.sh asserts rather than trusts.
32 HTTP_PORT=8081
33 SYNC_PORT=8443
34 GATE_PORT=2222
35
36 # prod signs in against Google, so the bundled issuer is not deployed and none
37 # of its values apply. The manifests that describe it are skipped wholesale.
38 LOCAL_OIDC=0
39
40 CONFIG_SECRET=eitri-server-config
41 PVC_NAME=eitri-server-data
42 PVC_SIZE=5Gi
43 TLS_SECRET=console-tls
44 # The certificate already serving eitri.sh. These manifests adopt the live site
45 # objects (Deployment/Service `web`, IngressRoutes `web-http`/`web-https`,
46 # Certificate `web-tls`), so naming it anything else would mint a second
47 # certificate and swap the one in use.
48 SITE_TLS_SECRET=web-tls
49
50 # Nightly sqlite backup CronJob: prod's database is the one nobody can rebuild.
51 BACKUPS=1
52 # The CDN purge runs at prod and only at prod — stg is served cache-bypassed,
53 # so there is nothing there to invalidate.
54 CDN_PURGE=1
deploy/server/plane.stg.env
Old New
@@ -0,0 +1,55 @@
1 # stg plane values — stg.eitri.sh.
2 #
3 # The same keys as plane.prod.env, because the same templates render both. Where
4 # a value differs, it differs because the two planes share a cluster and a node
5 # — not because stg is a smaller product.
6
7 NAMESPACE=eitri-stg
8
9 NODE_NAME=vnic-1
10 IMAGE_ARCH=arm64
11
12 CONSOLE_HOST=stg.eitri.sh
13 API_HOST=api.stg.eitri.sh
14 GATE_HOST=gate.stg.eitri.sh
15 SYNC_HOST=sync.stg.eitri.sh
16 # stg publishes its own /dl rather than borrowing prod's: scripts/site-image.sh
17 # stages exactly one version into site/dist/dl, so pushing a pre-release to
18 # eitri.sh would 404 every released download URL in the wild. Its own origin
19 # also means the release artifacts, the manifest shape and the cache headers
20 # are rehearsed by the same script that will later do it to prod.
21 SITE_HOST=dl.stg.eitri.sh
22
23 # stg shares vnic-1 with prod, so it cannot share prod's host ports: a second
24 # hostNetwork pod claiming 8081/8443/2222 would fail to schedule or silently
25 # lose its listeners.
26 HTTP_PORT=8082
27 SYNC_PORT=8444
28 GATE_PORT=2223
29
30 # stg signs in against the bundled eitri-oidc rather than Google: the plane is
31 # then reproducible from nothing but this repo and a Secret, its identities are
32 # ours to create and delete, and the smoke can drive a real sign-in headlessly.
33 # What it gives up is stated in the README — prod's relying-party wiring against
34 # a third-party IdP is the one part of sign-in stg does not exercise.
35 LOCAL_OIDC=1
36 OIDC_HOST=oidc.stg.eitri.sh
37 # Not a host port: the issuer is an ordinary pod behind a Service.
38 OIDC_PORT=9111
39 OIDC_TLS_SECRET=stg-oidc-tls
40 OIDC_CONFIG_SECRET=eitri-stg-oidc-config
41 OIDC_PVC_NAME=eitri-stg-oidc-data
42 OIDC_PVC_SIZE=1Gi
43
44 CONFIG_SECRET=eitri-stg-server-config
45 PVC_NAME=eitri-stg-server-data
46 PVC_SIZE=2Gi
47 TLS_SECRET=stg-console-tls
48 SITE_TLS_SECRET=stg-web-tls
49
50 # No backups: stg's database is disposable by design, and a second nightly
51 # sqlite job against local-path storage on the same node buys nothing.
52 BACKUPS=0
53 # No purge: stg is served cache-bypassed, so a purge here would be theatre that
54 # made the prod step look rehearsed when it is not. See README, "Known gaps".
55 CDN_PURGE=0
deploy/server/pvc.yaml
Old New
@@ -1,11 +1,11 @@
1 apiVersion: v1 1 apiVersion: v1
2 kind: PersistentVolumeClaim 2 kind: PersistentVolumeClaim
3 metadata: 3 metadata:
4 name: eitri-server-data 4 name: ${PVC_NAME}
5 namespace: eitri 5 namespace: ${NAMESPACE}
6 spec: 6 spec:
7 accessModes: [ReadWriteOnce] 7 accessModes: [ReadWriteOnce]
8 storageClassName: local-path 8 storageClassName: local-path
9 resources: 9 resources:
10 requests: 10 requests:
11 storage: 5Gi 11 storage: ${PVC_SIZE}
deploy/server/service.yaml
Old New
@@ -1,9 +1,11 @@
1 # Cluster-internal, so the published port is the same on both planes and only
2 # the target follows the plane's HTTP listener.
1 apiVersion: v1 3 apiVersion: v1
2 kind: Service 4 kind: Service
3 metadata: 5 metadata:
4 name: eitri-server 6 name: eitri-server
5 namespace: eitri 7 namespace: ${NAMESPACE}
6 spec: 8 spec:
7 selector: {app: eitri-server} 9 selector: {app: eitri-server}
8 ports: 10 ports:
9 - {name: http, port: 8080, targetPort: 8081} 11 - {name: http, port: 8080, targetPort: ${HTTP_PORT}}
deploy/server/site-certificate.yaml
Old New
@@ -0,0 +1,11 @@
1 apiVersion: cert-manager.io/v1
2 kind: Certificate
3 metadata:
4 name: ${SITE_TLS_SECRET}
5 namespace: ${NAMESPACE}
6 spec:
7 secretName: ${SITE_TLS_SECRET}
8 dnsNames: [${SITE_HOST}]
9 issuerRef:
10 name: letsencrypt-prod
11 kind: ClusterIssuer
deploy/server/site-deployment.yaml
Old New
@@ -0,0 +1,36 @@
1 # The static site: nginx over the built webroot, with the release artifacts
2 # baked in under /dl (scripts/site-image.sh stages dist/<version> into the image
3 # — there is no volume and nothing to upload afterwards).
4 #
5 # The site rolls BEFORE the server on every run. A server started ahead of its
6 # site fetches the boot manifest, finds the previous release, and pins that
7 # answer for 24 hours.
8 apiVersion: apps/v1
9 kind: Deployment
10 metadata:
11 name: web
12 namespace: ${NAMESPACE}
13 spec:
14 replicas: 1
15 selector:
16 matchLabels: {app: web}
17 template:
18 metadata:
19 labels: {app: web}
20 spec:
21 # The image is single-arch, so say which arch rather than discovering it
22 # as a scheduling failure on a mixed cluster.
23 nodeSelector:
24 kubernetes.io/arch: ${IMAGE_ARCH}
25 imagePullSecrets:
26 - name: regcred
27 containers:
28 - name: web
29 # By tag, like the server: see the note in deployment.yaml.
30 image: ${SITE_IMAGE}:${TAG}
31 imagePullPolicy: Always
32 ports:
33 - {name: http, containerPort: 8080}
34 readinessProbe:
35 httpGet: {path: /, port: 8080}
36 periodSeconds: 10
deploy/server/site-ingressroute.yaml
Old New
@@ -0,0 +1,32 @@
1 # Named for the deployment they route to, matching the objects already serving
2 # eitri.sh — these manifests adopt the live site rather than standing a second
3 # one beside it.
4 apiVersion: traefik.io/v1alpha1
5 kind: IngressRoute
6 metadata:
7 name: web-https
8 namespace: ${NAMESPACE}
9 spec:
10 entryPoints: [websecure]
11 routes:
12 - match: Host(`${SITE_HOST}`)
13 kind: Rule
14 services:
15 - {name: web, port: 8080}
16 tls:
17 secretName: ${SITE_TLS_SECRET}
18 ---
19 apiVersion: traefik.io/v1alpha1
20 kind: IngressRoute
21 metadata:
22 name: web-http
23 namespace: ${NAMESPACE}
24 spec:
25 entryPoints: [web]
26 routes:
27 - match: Host(`${SITE_HOST}`)
28 kind: Rule
29 middlewares:
30 - name: redirect-https
31 services:
32 - {name: web, port: 8080}
deploy/server/site-service.yaml
Old New
@@ -0,0 +1,9 @@
1 apiVersion: v1
2 kind: Service
3 metadata:
4 name: web
5 namespace: ${NAMESPACE}
6 spec:
7 selector: {app: web}
8 ports:
9 - {name: http, port: 8080, targetPort: 8080}
docs/assumptions.md
Old New
@@ -419,6 +419,36 @@ Secret is the key's only home.
419 that will not open stops the server rather than being regenerated—a fresh host 419 that will not open stops the server rather than being regenerated—a fresh host
420 CA would invalidate every client's pin and every VM's host certificate at once. 420 CA would invalidate every client's pin and every VM's host certificate at once.
421 421
422 ### A staging plane on the same cluster catches hosted-shape failures
423
424 stg runs from the same manifests as prod, differing only in values, so a defect
425 in the deployed shape—a config key the Secret lacks, a listener that disagrees
426 with its advertisement, a route that does not carry /mcp—fails at stg first.
427 Underpins promoting to prod by re-running one script rather than by inspection.
428 **Unproven**: no pre-release has been shipped through it yet. One difference is
429 deliberate and known: stg signs in against the bundled issuer, so prod's
430 relying-party wiring against a third-party provider is proven only at prod.
431
432 ### A pre-release orders below the release it leads to
433
434 v0.0.4-pre.1 < v0.0.4-pre.2 < v0.0.4, and builds derived from either sit
435 between them, so an agent walks a release cycle's tags the way it reads.
436 Underpins a staging plane rehearsing the agent-upgrade path on the very tags
437 that cycle produces—the failure that defined the v0.0.3 release day.
438 **Proven** in code, by the ordering the fleet itself uses: the full chain is
439 pinned in a test, and everything else—dev, a dirty tree, rc.1, a malformed
440 tag—stays unordered and is never offered an upgrade.
441
442 ### Nested KVM is enough to gate a branch
443
444 A guest of a workstation VM boots under cloud-hypervisor and exercises the same
445 agent, bootstrap, gate and exposure paths bare metal does. Underpins moving the
446 branch gate off real hardware so a hosted outage cannot fail it.
447 **Unproven**: the four proofs in deploy/server/README.md—cold deploy, warm
448 deploy, deploy after a recycle, coverage merged—have not been run. Bridged
449 networking and real-hardware quirks are known to be out of its reach, which is
450 why real metal gates pre-release tags instead.
451
422 ### A proxied MCP origin carries long calls 452 ### A proxied MCP origin carries long calls
423 453
424 The public-MCP story rests on Cloudflare not cutting a `vm_create` that runs 454 The public-MCP story rests on Cloudflare not cutting a `vm_create` that runs
docs/quickstart.md
Old New
@@ -21,10 +21,10 @@ host bundle:
21 21
22 ```sh 22 ```sh
23 V=v0.0.1 23 V=v0.0.1
24 curl -fsSLO "https://eitri.sh/dl/$V/eitri_${V}_linux_amd64.tar.gz" 24 curl -fsSLO "https://eitri.sh/dl/$V/eitri-server_${V}_linux_amd64.tar.gz"
25 curl -fsSLO "https://eitri.sh/dl/$V/SHA256SUMS" 25 curl -fsSLO "https://eitri.sh/dl/$V/SHA256SUMS"
26 sha256sum -c SHA256SUMS --ignore-missing 26 sha256sum -c SHA256SUMS --ignore-missing
27 tar xzf "eitri_${V}_linux_amd64.tar.gz" && cd "eitri_${V}_linux_amd64" 27 tar xzf "eitri-server_${V}_linux_amd64.tar.gz" && cd "eitri-server_${V}_linux_amd64"
28 ``` 28 ```
29 29
30 Then click **+ Add host** in the console and run the command it prints 30 Then click **+ Add host** in the console and run the command it prints
@@ -175,7 +175,7 @@ manage them by hand instead, disable it in `/etc/default/eitri-agent`:
175 `EITRI_AGENT_FLAGS="--bootstrap-url="`. 175 `EITRI_AGENT_FLAGS="--bootstrap-url="`.
176 176
177 Tarballs live at <https://eitri.sh/dl/latest/>. The host bundle 177 Tarballs live at <https://eitri.sh/dl/latest/>. The host bundle
178 (`eitri_<version>_linux_amd64.tar.gz`) has `eitri-server`, `eitri-agent`, and 178 (`eitri-server_<version>_linux_amd64.tar.gz`) has `eitri-server`, `eitri-agent`, and
179 their systemd units. The issuer bundle 179 their systemd units. The issuer bundle
180 (`eitri-oidc_<version>_linux_amd64.tar.gz`) has `eitri-oidc`—the bundled 180 (`eitri-oidc_<version>_linux_amd64.tar.gz`) has `eitri-oidc`—the bundled
181 sign-in provider—and its unit. The client bundle 181 sign-in provider—and its unit. The client bundle
@@ -187,7 +187,7 @@ Download and verify—set `V` to the current release (shown at
187 187
188 ```sh 188 ```sh
189 V=v0.0.1 189 V=v0.0.1
190 curl -fsSLO "https://eitri.sh/dl/$V/eitri_${V}_linux_amd64.tar.gz" 190 curl -fsSLO "https://eitri.sh/dl/$V/eitri-server_${V}_linux_amd64.tar.gz"
191 curl -fsSLO "https://eitri.sh/dl/$V/eitri-oidc_${V}_linux_amd64.tar.gz" 191 curl -fsSLO "https://eitri.sh/dl/$V/eitri-oidc_${V}_linux_amd64.tar.gz"
192 curl -fsSLO "https://eitri.sh/dl/$V/eitri-cli_${V}_$(uname -s | tr A-Z a-z)_amd64.tar.gz" 192 curl -fsSLO "https://eitri.sh/dl/$V/eitri-cli_${V}_$(uname -s | tr A-Z a-z)_amd64.tar.gz"
193 curl -fsSLO "https://eitri.sh/dl/$V/SHA256SUMS" 193 curl -fsSLO "https://eitri.sh/dl/$V/SHA256SUMS"
@@ -200,7 +200,7 @@ for every tarball before you unpack anything.
200 ### The server 200 ### The server
201 201
202 ```sh 202 ```sh
203 tar xzf eitri_*_linux_amd64.tar.gz && cd eitri_*_linux_amd64 203 tar xzf eitri-server_*_linux_amd64.tar.gz && cd eitri-server_*_linux_amd64
204 sudo install -m 0755 eitri-server /usr/local/bin/eitri-server 204 sudo install -m 0755 eitri-server /usr/local/bin/eitri-server
205 sudo install -m 0644 eitri-server.service /etc/systemd/system/eitri-server.service 205 sudo install -m 0644 eitri-server.service /etc/systemd/system/eitri-server.service
206 sudo useradd --system --home-dir /var/lib/eitri --shell /usr/sbin/nologin eitri 206 sudo useradd --system --home-dir /var/lib/eitri --shell /usr/sbin/nologin eitri
docs/upgrade.md
Old New
@@ -23,7 +23,8 @@ hand, stop the agent, move `eitri-agent.prev` back over the binary, and start
23 it again. 23 it again.
24 24
25 **Requirements.** The button lights up only when the running agent reports a 25 **Requirements.** The button lights up only when the running agent reports a
26 release version (`vX.Y.Z`)—agents built from an untagged or dirty tree 26 release version (`vX.Y.Z`) or a pre-release of one (`vX.Y.Z-pre.N`, which orders
27 below the release it leads to)—agents built from an untagged or dirty tree
27 report a git hash instead and are never offered upgrades. The server needs 28 report a git hash instead and are never offered upgrades. The server needs
28 `release_manifest_url` reachable; set it to `""` in the server config to 29 `release_manifest_url` reachable; set it to `""` in the server config to
29 disable upgrade checks entirely. 30 disable upgrade checks entirely.
internal/agent/selfupdate/selfupdate_test.go
Old New
@@ -59,9 +59,9 @@ func tarball(t *testing.T, members ...[2]string) []byte {
59 func hostBundle(t *testing.T, agent string) []byte { 59 func hostBundle(t *testing.T, agent string) []byte {
60 t.Helper() 60 t.Helper()
61 return tarball(t, 61 return tarball(t,
62 [2]string{"eitri_v9_linux_amd64/eitri-server", "server-binary"}, 62 [2]string{"eitri-server_v9_linux_amd64/eitri-server", "server-binary"},
63 [2]string{"eitri_v9_linux_amd64/eitri-agent", agent}, 63 [2]string{"eitri-server_v9_linux_amd64/eitri-agent", agent},
64 [2]string{"eitri_v9_linux_amd64/eitri-agent.service", "[Unit]"}, 64 [2]string{"eitri-server_v9_linux_amd64/eitri-agent.service", "[Unit]"},
65 ) 65 )
66 } 66 }
67 67
@@ -133,8 +133,8 @@ func TestApplyTarballWithoutAgentMember(t *testing.T) {
133 exe := filepath.Join(dir, "eitri-agent") 133 exe := filepath.Join(dir, "eitri-agent")
134 os.WriteFile(exe, []byte("old"), 0o755) 134 os.WriteFile(exe, []byte("old"), 0o755)
135 body := tarball(t, 135 body := tarball(t,
136 [2]string{"eitri_v9_linux_amd64/eitri-server", "server-binary"}, 136 [2]string{"eitri-server_v9_linux_amd64/eitri-server", "server-binary"},
137 [2]string{"eitri_v9_linux_amd64/eitri-agent.service", "[Unit]"}, 137 [2]string{"eitri-server_v9_linux_amd64/eitri-agent.service", "[Unit]"},
138 ) 138 )
139 url, sha := serveBinary(t, body) 139 url, sha := serveBinary(t, body)
140 140
internal/server/release/release.go
Old New
@@ -117,14 +117,21 @@ func (c *Client) Poll(ctx context.Context, every time.Duration, onErr func(error
117 } 117 }
118 118
119 // Less reports whether version a orders strictly before b. Versions are 119 // Less reports whether version a orders strictly before b. Versions are
120 // eitri's own tags ("vX.Y.Z") and the git-describe builds derived from them 120 // eitri's own tags — releases ("v0.0.3"), pre-releases of them ("v0.0.4-pre.1")
121 // ("vX.Y.Z-N-g<hex>", N commits past the tag), ordered by (X, Y, Z, N) with a 121 // — and the git-describe builds derived from either ("v0.0.3-5-gabc1234",
122 // plain tag at N=0. So a hand-deployed "v0.0.2-2-g68f804d" still orders before 122 // "v0.0.4-pre.1-3-gabc1234", N commits past that tag). The whole chain orders
123 // the "v0.0.3" release and takes the upgrade, while a build ahead of the 123 // the way it reads:
124 // latest release ("v0.0.3-5-gabc1234") never orders before it and is never 124 //
125 // offered a downgrade. Anything unparsable — "dev", a "-dirty" working tree, 125 // v0.0.4-pre.1 < v0.0.4-pre.1-3-gabc1234 < v0.0.4-pre.2 < v0.0.4 < v0.0.4-2-gabc1234
126 // a malformed tag — never orders before anything, so an unstamped build never 126 //
127 // sees an upgrade. 127 // So a hand-deployed "v0.0.2-2-g68f804d" orders before the "v0.0.3" release and
128 // takes the upgrade; a build ahead of the latest release never orders before it
129 // and is never offered a downgrade; and an agent on a pre-release takes the next
130 // pre-release and then the release itself, which is what lets a staging plane
131 // rehearse the upgrade path on the same tags a release cycle produces.
132 //
133 // Anything unparsable — "dev", a "-dirty" working tree, a malformed tag — never
134 // orders before anything, so an unstamped build never sees an upgrade.
128 func Less(a, b string) bool { 135 func Less(a, b string) bool {
129 pa, oka := parse(a) 136 pa, oka := parse(a)
130 pb, okb := parse(b) 137 pb, okb := parse(b)
@@ -139,35 +146,56 @@ func Less(a, b string) bool {
139 return false 146 return false
140 } 147 }
141 148
142 // parse reads "vX.Y.Z" or "vX.Y.Z-N-g<hex>" into the ordering tuple 149 // parse reads a version into its ordering tuple. Four shapes are accepted:
143 // (X, Y, Z, N); a plain tag carries N=0. The describe suffix must be exactly a 150 //
144 // non-negative decimal commit count and a "g"-prefixed non-empty hex abbrev — 151 // vX.Y.Z a release
145 // a "-dirty" marker or any other trailing text makes the version unparsable. 152 // vX.Y.Z-pre.N the Nth pre-release leading up to it
146 func parse(v string) ([4]int, bool) { 153 // vX.Y.Z-C-g<hex> C commits past the release (git describe)
147 var out [4]int 154 // vX.Y.Z-pre.N-C-g<hex> C commits past the pre-release
148 core := strings.TrimPrefix(v, "v") 155 //
149 if rel, desc, found := strings.Cut(core, "-"); found { 156 // The tuple is (X, Y, Z, final, pre, count). `final` is 1 for a release and 0
150 core = rel 157 // for a pre-release of it, which is the whole trick: it sinks every
151 count, abbrev, ok := strings.Cut(desc, "-") 158 // vX.Y.Z-pre.N below vX.Y.Z without disturbing how anything else sorts, and
152 if !ok || !isDecimal(count) || !isGitAbbrev(abbrev) { 159 // leaves builds derived from either in their own place. Both counters must be
160 // exactly a non-negative decimal, and a describe suffix must carry a
161 // "g"-prefixed non-empty hex abbrev — a "-dirty" marker or any other trailing
162 // text makes the version unparsable.
163 func parse(v string) ([6]int, bool) {
164 var out [6]int
165 core, suffix, hasSuffix := strings.Cut(strings.TrimPrefix(v, "v"), "-")
166
167 out[3] = 1 // a release outranks every pre-release of itself
168 if hasSuffix {
169 if pre, isPre := strings.CutPrefix(suffix, "pre."); isPre {
170 out[3] = 0
171 num, rest, hasRest := strings.Cut(pre, "-")
172 n, ok := decimal(num)
173 if !ok {
174 return out, false
175 }
176 out[4] = n
177 suffix, hasSuffix = rest, hasRest
178 }
179 }
180 if hasSuffix {
181 count, abbrev, found := strings.Cut(suffix, "-")
182 if !found || !isGitAbbrev(abbrev) {
153 return out, false 183 return out, false
154 } 184 }
155 n, err := strconv.Atoi(count) 185 n, ok := decimal(count)
156 if err != nil { 186 if !ok {
157 return out, false 187 return out, false
158 } 188 }
159 out[3] = n 189 out[5] = n
160 } 190 }
191
161 parts := strings.SplitN(core, ".", 3) 192 parts := strings.SplitN(core, ".", 3)
162 if len(parts) != 3 { 193 if len(parts) != 3 {
163 return out, false 194 return out, false
164 } 195 }
165 for i, p := range parts { 196 for i, p := range parts {
166 if !isDecimal(p) { 197 n, ok := decimal(p)
167 return out, false 198 if !ok {
168 }
169 n, err := strconv.Atoi(p)
170 if err != nil {
171 return out, false 199 return out, false
172 } 200 }
173 out[i] = n 201 out[i] = n
@@ -175,6 +203,19 @@ func parse(v string) ([4]int, bool) {
175 return out, true 203 return out, true
176 } 204 }
177 205
206 // decimal parses one non-negative decimal component, rejecting the sign Atoi
207 // would otherwise accept and any value too large to hold.
208 func decimal(s string) (int, bool) {
209 if !isDecimal(s) {
210 return 0, false
211 }
212 n, err := strconv.Atoi(s)
213 if err != nil {
214 return 0, false
215 }
216 return n, true
217 }
218
178 // isDecimal reports whether s is a non-empty run of decimal digits — the sign 219 // isDecimal reports whether s is a non-empty run of decimal digits — the sign
179 // Atoi would otherwise accept is not part of a version component. 220 // Atoi would otherwise accept is not part of a version component.
180 func isDecimal(s string) bool { 221 func isDecimal(s string) bool {
internal/server/release/release_test.go
Old New
@@ -66,6 +66,33 @@ func TestLessOrdersGitDescribeBuilds(t *testing.T) {
66 } 66 }
67 } 67 }
68 68
69 // TestLessOrdersPreReleases pins the chain a release cycle walks: each
70 // pre-release, the builds derived from it, the next pre-release, the release
71 // itself, and builds past that. An agent on a pre-release takes the next one
72 // and eventually the release, which is what lets a staging plane rehearse the
73 // upgrade path on the same tags the cycle produces.
74 func TestLessOrdersPreReleases(t *testing.T) {
75 // Every entry orders strictly before every entry after it.
76 chain := []string{
77 "v0.0.3",
78 "v0.0.4-pre.1",
79 "v0.0.4-pre.1-3-gabc1234",
80 "v0.0.4-pre.2",
81 "v0.0.4-pre.10", // numeric, not lexicographic
82 "v0.0.4",
83 "v0.0.4-2-gabc1234",
84 "v0.0.5-pre.1",
85 }
86 for i, a := range chain {
87 for j, b := range chain {
88 want := i < j
89 if got := Less(a, b); got != want {
90 t.Errorf("Less(%q,%q) = %v, want %v", a, b, got, want)
91 }
92 }
93 }
94 }
95
69 // TestLessLeavesUnstampedBuildsUnordered pins the shapes that stay 96 // TestLessLeavesUnstampedBuildsUnordered pins the shapes that stay
70 // unparsable: a dirty working tree or a malformed version never orders before 97 // unparsable: a dirty working tree or a malformed version never orders before
71 // anything, in either direction, so it is never offered an upgrade. 98 // anything, in either direction, so it is never offered an upgrade.
@@ -75,6 +102,17 @@ func TestLessLeavesUnstampedBuildsUnordered(t *testing.T) {
75 "dev", 102 "dev",
76 "v0.0.3-dirty", 103 "v0.0.3-dirty",
77 "v0.0.2-2-g68f804d-dirty", 104 "v0.0.2-2-g68f804d-dirty",
105 "v0.0.4-pre", // no pre number
106 "v0.0.4-pre.", // empty pre number
107 "v0.0.4-pre.x", // pre number is not a number
108 "v0.0.4-pre.1.2", // pre number is not one component
109 "v0.0.4-pre.-1", // negative pre number
110 "v0.0.4-pre.1-dirty", // dirty past a pre-release
111 "v0.0.4-pre.1-3", // no g-abbrev past a pre-release
112 "v0.0.4-pre.1-g abc", // no commit count past a pre-release
113 "v0.0.4-prerelease.1", // not the pre. prefix
114 "v0.0.4-rc.1", // only "pre" is spelled this way
115 "v0.0.4-pre.99999999999999999999", // pre number out of range
78 "v0.0.2-x-g123", // commit count is not a number 116 "v0.0.2-x-g123", // commit count is not a number
79 "v0.0.2--2-g68f804d", // negative commit count 117 "v0.0.2--2-g68f804d", // negative commit count
80 "v0.0.2-2", // no g-abbrev 118 "v0.0.2-2", // no g-abbrev
internal/site/dl_test.go
Old New
@@ -14,9 +14,9 @@ func fixtureDist(t *testing.T) string {
14 t.Fatal(err) 14 t.Fatal(err)
15 } 15 }
16 files := map[string]string{ 16 files := map[string]string{
17 "eitri_v0.0.1_linux_amd64.tar.gz": strings.Repeat("x", 2048), 17 "eitri-server_v0.0.1_linux_amd64.tar.gz": strings.Repeat("x", 2048),
18 "eitri-agent_linux_amd64": "binary", 18 "eitri-agent_linux_amd64": "binary",
19 "SHA256SUMS": "abc123 eitri_v0.0.1_linux_amd64.tar.gz\ndef456 eitri-agent_linux_amd64\n", 19 "SHA256SUMS": "abc123 eitri-server_v0.0.1_linux_amd64.tar.gz\ndef456 eitri-agent_linux_amd64\n",
20 } 20 }
21 for name, body := range files { 21 for name, body := range files {
22 if err := os.WriteFile(filepath.Join(dist, name), []byte(body), 0o644); err != nil { 22 if err := os.WriteFile(filepath.Join(dist, name), []byte(body), 0o644); err != nil {
@@ -32,7 +32,7 @@ func TestDownloadsPageListsArtifacts(t *testing.T) {
32 t.Fatal(err) 32 t.Fatal(err)
33 } 33 }
34 for _, want := range []string{ 34 for _, want := range []string{
35 "[eitri_v0.0.1_linux_amd64.tar.gz](/dl/v0.0.1/eitri_v0.0.1_linux_amd64.tar.gz)", 35 "[eitri-server_v0.0.1_linux_amd64.tar.gz](/dl/v0.0.1/eitri-server_v0.0.1_linux_amd64.tar.gz)",
36 "abc123", 36 "abc123",
37 "sha256sum -c SHA256SUMS", 37 "sha256sum -c SHA256SUMS",
38 "[SHA256SUMS](/dl/v0.0.1/SHA256SUMS)", 38 "[SHA256SUMS](/dl/v0.0.1/SHA256SUMS)",
internal/site/manifest_test.go
Old New
@@ -20,7 +20,7 @@ func TestBuildManifestFromBareBinaries(t *testing.T) {
20 } 20 }
21 } 21 }
22 // Non-bare files must be ignored. 22 // Non-bare files must be ignored.
23 if err := os.WriteFile(filepath.Join(dist, "eitri_v0.0.1_linux_amd64.tar.gz"), body, 0o644); err != nil { 23 if err := os.WriteFile(filepath.Join(dist, "eitri-server_v0.0.1_linux_amd64.tar.gz"), body, 0o644); err != nil {
24 t.Fatal(err) 24 t.Fatal(err)
25 } 25 }
26 26
internal/smoke/config.go
Old New
@@ -9,13 +9,30 @@ import (
9 // Config holds the environment-sourced settings for one smoke run. It mirrors 9 // Config holds the environment-sourced settings for one smoke run. It mirrors
10 // the variables the deploy boot-gate reads from $EITRI_DEPLOY_ENV. 10 // the variables the deploy boot-gate reads from $EITRI_DEPLOY_ENV.
11 type Config struct { 11 type Config struct {
12 ServerURL string 12 ServerURL string
13 // CIUser and CIPasswordFile drive the credential-chain proof: a headless
14 // sign-in that posts a password to the issuer's login form. Both empty ⇒
15 // that proof is skipped, which is what a plane fronted by a real identity
16 // provider needs — a Google-backed console has no password to post, and the
17 // operator PAT below carries the run instead.
13 CIUser string 18 CIUser string
14 CIPasswordFile string 19 CIPasswordFile string
15 CIPATFile string 20 // CIPATFile is an operator-minted token for the VM lifecycle. Empty ⇒ the
16 AgentUserHost string 21 // PAT minted by the sign-in above is used instead, which is right whenever
17 AgentPort int 22 // the identity that signs in is itself the operator.
18 AgentStateDir string 23 CIPATFile string
24 AgentUserHost string
25 AgentPort int
26 AgentStateDir string
27
28 // MCPURLs are the origins the remote-MCP leg exercises (SMOKE_MCP_URL, a
29 // space-separated list; a single ServerURL when unset). The FIRST entry gets
30 // the full leg — register, create, exec, expose, banner, destroy — and every
31 // later entry gets the cheap ones: an unauthenticated POST must be refused,
32 // and the advertised toolset must be complete. One long call is enough to
33 // observe a proxy's silent-origin timeout; a second would only double the VM
34 // churn for coverage already held.
35 MCPURLs []string
19 36
20 // Carried for a later coverage-collection task; optional here. 37 // Carried for a later coverage-collection task; optional here.
21 ServerGocoverdir string 38 ServerGocoverdir string
@@ -44,14 +61,20 @@ func loadConfig(getenv func(string) string) (Config, error) {
44 if serverURL == "" { 61 if serverURL == "" {
45 missing = append(missing, "SERVER_URL") 62 missing = append(missing, "SERVER_URL")
46 } 63 }
47 if ciUser == "" { 64 // CI_USER/CI_PASSWORD_FILE are deliberately absent from this list: a plane
48 missing = append(missing, "CI_USER") 65 // whose issuer is a real IdP cannot offer a headless password sign-in, so
49 } 66 // the credential-chain proof is opt-in. Setting exactly one of the pair is
50 if ciPasswordFile == "" { 67 // a typo rather than a choice, and is rejected below.
51 missing = append(missing, "CI_PASSWORD_FILE") 68 if (ciUser == "") != (ciPasswordFile == "") {
69 return Config{}, fmt.Errorf("CI_USER and CI_PASSWORD_FILE must be set together (set neither to skip the credential-chain proof)")
52 } 70 }
53 if ciPATFile == "" { 71 // CI_PAT_FILE carries the VM lifecycle as an operator whose tenant owns the
54 missing = append(missing, "CI_PAT_FILE") 72 // fleet's hosts. It is required only when nothing else can produce such a
73 // token: where a password issuer exists, the sign-in above mints one, and
74 // on a plane whose signed-in identity IS the operator that is the whole
75 // credential story — no PAT to paste, nothing to rotate by hand.
76 if ciPATFile == "" && ciUser == "" {
77 missing = append(missing, "CI_PAT_FILE (or CI_USER, to mint one by signing in)")
55 } 78 }
56 if agentHosts == "" { 79 if agentHosts == "" {
57 missing = append(missing, "AGENT_HOSTS") 80 missing = append(missing, "AGENT_HOSTS")
@@ -70,8 +93,14 @@ func loadConfig(getenv func(string) string) (Config, error) {
70 smokeVMUser = "ubuntu" 93 smokeVMUser = "ubuntu"
71 } 94 }
72 95
96 mcpURLs := strings.Fields(getenv("SMOKE_MCP_URL"))
97 if len(mcpURLs) == 0 {
98 mcpURLs = []string{serverURL}
99 }
100
73 return Config{ 101 return Config{
74 ServerURL: serverURL, 102 ServerURL: serverURL,
103 MCPURLs: mcpURLs,
75 CIUser: ciUser, 104 CIUser: ciUser,
76 CIPasswordFile: ciPasswordFile, 105 CIPasswordFile: ciPasswordFile,
77 CIPATFile: ciPATFile, 106 CIPATFile: ciPATFile,
internal/smoke/config_test.go
Old New
@@ -69,9 +69,6 @@ func TestLoadConfigMissingRequiredVars(t *testing.T) {
69 wantErr string 69 wantErr string
70 }{ 70 }{
71 {"missing server url", "SERVER_URL", "SERVER_URL"}, 71 {"missing server url", "SERVER_URL", "SERVER_URL"},
72 {"missing ci user", "CI_USER", "CI_USER"},
73 {"missing ci password file", "CI_PASSWORD_FILE", "CI_PASSWORD_FILE"},
74 {"missing ci pat file", "CI_PAT_FILE", "CI_PAT_FILE"},
75 {"missing agent hosts", "AGENT_HOSTS", "AGENT_HOSTS"}, 72 {"missing agent hosts", "AGENT_HOSTS", "AGENT_HOSTS"},
76 {"missing agent state dir", "AGENT_STATE_DIR", "AGENT_STATE_DIR"}, 73 {"missing agent state dir", "AGENT_STATE_DIR", "AGENT_STATE_DIR"},
77 } 74 }
@@ -95,13 +92,115 @@ func TestLoadConfigMissingAllRequiredVars(t *testing.T) {
95 if err == nil { 92 if err == nil {
96 t.Fatal("loadConfig: want error, got nil") 93 t.Fatal("loadConfig: want error, got nil")
97 } 94 }
98 for _, want := range []string{"SERVER_URL", "CI_USER", "CI_PASSWORD_FILE", "CI_PAT_FILE", "AGENT_HOSTS", "AGENT_STATE_DIR"} { 95 for _, want := range []string{"SERVER_URL", "CI_PAT_FILE", "AGENT_HOSTS", "AGENT_STATE_DIR"} {
99 if !strings.Contains(err.Error(), want) { 96 if !strings.Contains(err.Error(), want) {
100 t.Errorf("error = %q, missing %q", err.Error(), want) 97 t.Errorf("error = %q, missing %q", err.Error(), want)
101 } 98 }
102 } 99 }
103 } 100 }
104 101
102 // TestLoadConfigCredentialChainIsOptional pins which credentials a plane may
103 // leave out. A plane fronted by a real identity provider has no password to
104 // post and hands over an operator PAT instead; a plane with a password issuer
105 // signs in and mints its own. Only a plane offering neither is misconfigured —
106 // and setting exactly one half of the sign-in pair is a typo, not a choice.
107 func TestLoadConfigCredentialChainIsOptional(t *testing.T) {
108 cases := []struct {
109 name string
110 unset []string
111 wantErr string
112 }{
113 {"both unset skips the proof", []string{"CI_USER", "CI_PASSWORD_FILE"}, ""},
114 {"user without password file", []string{"CI_PASSWORD_FILE"}, "must be set together"},
115 {"password file without user", []string{"CI_USER"}, "must be set together"},
116 // A sign-in mints its own token, so a plane with a password issuer needs
117 // no operator PAT pasted anywhere.
118 {"sign-in without an operator PAT", []string{"CI_PAT_FILE"}, ""},
119 // With neither, nothing can authenticate the VM lifecycle.
120 {
121 "neither a sign-in nor a PAT",
122 []string{"CI_USER", "CI_PASSWORD_FILE", "CI_PAT_FILE"},
123 "CI_PAT_FILE",
124 },
125 }
126 for _, tc := range cases {
127 t.Run(tc.name, func(t *testing.T) {
128 vals := requiredVals()
129 for _, k := range tc.unset {
130 delete(vals, k)
131 }
132 cfg, err := loadConfig(fakeGetenv(vals))
133 if tc.wantErr == "" {
134 if err != nil {
135 t.Fatalf("loadConfig: %v", err)
136 }
137 got := map[string]string{
138 "CI_USER": cfg.CIUser,
139 "CI_PASSWORD_FILE": cfg.CIPasswordFile,
140 "CI_PAT_FILE": cfg.CIPATFile,
141 }
142 for _, k := range tc.unset {
143 if got[k] != "" {
144 t.Errorf("%s = %q, want empty", k, got[k])
145 }
146 }
147 return
148 }
149 if err == nil {
150 t.Fatal("loadConfig: want error, got nil")
151 }
152 if !strings.Contains(err.Error(), tc.wantErr) {
153 t.Errorf("error = %q, want to mention %q", err.Error(), tc.wantErr)
154 }
155 })
156 }
157 }
158
159 // TestLoadConfigMCPURLs pins the one documented rule of the list: unset means
160 // the console origin alone, so the branch gate is unchanged, and the order of
161 // an explicit list is preserved — the first entry is the one that gets the full
162 // leg.
163 func TestLoadConfigMCPURLs(t *testing.T) {
164 cases := []struct {
165 name string
166 env string
167 want []string
168 }{
169 {"unset defaults to the server url", "", []string{"https://server.example:8443"}},
170 {"one origin", "https://api.stg.eitri.sh", []string{"https://api.stg.eitri.sh"}},
171 {
172 "two origins keep their order",
173 "https://stg.eitri.sh https://api.stg.eitri.sh",
174 []string{"https://stg.eitri.sh", "https://api.stg.eitri.sh"},
175 },
176 {
177 "surrounding and repeated whitespace is not an origin",
178 " https://stg.eitri.sh \t https://api.stg.eitri.sh ",
179 []string{"https://stg.eitri.sh", "https://api.stg.eitri.sh"},
180 },
181 }
182 for _, tc := range cases {
183 t.Run(tc.name, func(t *testing.T) {
184 vals := requiredVals()
185 if tc.env != "" {
186 vals["SMOKE_MCP_URL"] = tc.env
187 }
188 cfg, err := loadConfig(fakeGetenv(vals))
189 if err != nil {
190 t.Fatalf("loadConfig: %v", err)
191 }
192 if len(cfg.MCPURLs) != len(tc.want) {
193 t.Fatalf("MCPURLs = %v, want %v", cfg.MCPURLs, tc.want)
194 }
195 for i := range tc.want {
196 if cfg.MCPURLs[i] != tc.want[i] {
197 t.Errorf("MCPURLs[%d] = %q, want %q", i, cfg.MCPURLs[i], tc.want[i])
198 }
199 }
200 })
201 }
202 }
203
105 func TestLoadConfigSmokeGateDefaults(t *testing.T) { 204 func TestLoadConfigSmokeGateDefaults(t *testing.T) {
106 cfg, err := loadConfig(fakeGetenv(requiredVals())) 205 cfg, err := loadConfig(fakeGetenv(requiredVals()))
107 if err != nil { 206 if err != nil {
internal/smoke/mcp.go
Old New
@@ -129,16 +129,11 @@ type delegator struct {
129 now func() time.Time 129 now func() time.Time
130 } 130 }
131 131
132 // proveMCP drives one full cycle through the remote MCP endpoint and nothing 132 // proveToolList proves an origin advertises the whole remote toolset. It is the
133 // else: delegate access to eitri, create a VM, run a command in it, publish a 133 // cheap half of the MCP leg, and all a second origin gets: a plane fronted two
134 // port, read the guest's banner back through that port, and destroy it. 134 // ways routes and authenticates /mcp on both, but only one of them needs to
135 // 135 // carry the long call that a proxy's silent-origin timeout would cut.
136 // The CA is registered BEFORE the VM is created, because a guest bakes its CA 136 func proveToolList(ctx context.Context, c mcpTools) error {
137 // set at create and a CA registered afterwards is one it will never trust. The
138 // DELEGATION itself may happen on either side of the create — that is the whole
139 // improvement over holding a signing key, and it is worth stating plainly here
140 // so nobody reintroduces an ordering constraint that no longer exists.
141 func proveMCP(ctx context.Context, c mcpTools, vmName string, d delegator, now func() time.Time, sleep func(time.Duration), dial bannerFunc) error {
142 names, err := c.List(ctx) 137 names, err := c.List(ctx)
143 if err != nil { 138 if err != nil {
144 return fmt.Errorf("list remote MCP tools: %w", err) 139 return fmt.Errorf("list remote MCP tools: %w", err)
@@ -151,6 +146,36 @@ func proveMCP(ctx context.Context, c mcpTools, vmName string, d delegator, now f
151 return fmt.Errorf("FAIL: remote MCP does not advertise %s (has: %s)", want, strings.Join(names, ", ")) 146 return fmt.Errorf("FAIL: remote MCP does not advertise %s (has: %s)", want, strings.Join(names, ", "))
152 } 147 }
153 } 148 }
149 return nil
150 }
151
152 // proveRemoteMCPToolset dials one origin with a bearer PAT, proves its toolset,
153 // and hangs up.
154 func proveRemoteMCPToolset(ctx context.Context, serverURL, pat string) error {
155 tools, closeMCP, err := dialMCP(ctx, serverURL, pat)
156 if err != nil {
157 return err
158 }
159 defer closeMCP()
160 if err := proveToolList(ctx, tools); err != nil {
161 return fmt.Errorf("%s: %w", serverURL, err)
162 }
163 return nil
164 }
165
166 // proveMCP drives one full cycle through the remote MCP endpoint and nothing
167 // else: delegate access to eitri, create a VM, run a command in it, publish a
168 // port, read the guest's banner back through that port, and destroy it.
169 //
170 // The CA is registered BEFORE the VM is created, because a guest bakes its CA
171 // set at create and a CA registered afterwards is one it will never trust. The
172 // DELEGATION itself may happen on either side of the create — that is the whole
173 // improvement over holding a signing key, and it is worth stating plainly here
174 // so nobody reintroduces an ordering constraint that no longer exists.
175 func proveMCP(ctx context.Context, c mcpTools, vmName string, d delegator, now func() time.Time, sleep func(time.Duration), dial bannerFunc) error {
176 if err := proveToolList(ctx, c); err != nil {
177 return err
178 }
154 179
155 // Through the tool, not the endpoint behind it: registering a CA is the 180 // Through the tool, not the endpoint behind it: registering a CA is the
156 // first step a bare token has to take, so the gate drives the same call an 181 // first step a bare token has to take, so the gate drives the same call an
internal/smoke/mcp_test.go
Old New
@@ -248,8 +248,46 @@ func TestProveMCPDestroysItsVMOnFailure(t *testing.T) {
248 assert.Contains(t, f.calls, "vm_destroy", "the leg must reap its own VM even when it fails") 248 assert.Contains(t, f.calls, "vm_destroy", "the leg must reap its own VM even when it fails")
249 } 249 }
250 250
251 // TestProveMCPRejectsAShortToolset: a transport that advertises fewer tools than 251 // TestProveToolList covers the cheap leg every extra origin gets on its own: a
252 // the stdio binary means the two have drifted apart. 252 // transport advertising fewer tools than the stdio binary means the two have
253 // drifted apart, and a missing tool is named rather than counted.
254 func TestProveToolList(t *testing.T) {
255 cases := []struct {
256 name string
257 tools func(*fakeMCP)
258 wantErr string
259 }{
260 {"the whole toolset passes", func(*fakeMCP) {}, ""},
261 {"a short toolset fails", func(f *fakeMCP) { f.tools = f.tools[:5] }, "advertises 5 tools"},
262 {
263 "a renamed tool is named",
264 func(f *fakeMCP) {
265 for i, name := range f.tools {
266 if name == "delegate_begin" {
267 f.tools[i] = "vm_something_else"
268 }
269 }
270 },
271 "does not advertise delegate_begin",
272 },
273 }
274 for _, tc := range cases {
275 t.Run(tc.name, func(t *testing.T) {
276 f := newFakeMCP(t, newCA(t))
277 tc.tools(f)
278 err := proveToolList(t.Context(), f)
279 if tc.wantErr == "" {
280 require.NoError(t, err)
281 return
282 }
283 require.Error(t, err)
284 assert.Contains(t, err.Error(), tc.wantErr)
285 })
286 }
287 }
288
289 // TestProveMCPRejectsAShortToolset: the full leg refuses to spend a boot on an
290 // endpoint whose toolset is already wrong.
253 func TestProveMCPRejectsAShortToolset(t *testing.T) { 291 func TestProveMCPRejectsAShortToolset(t *testing.T) {
254 ca, stranger := newCA(t), newCA(t) 292 ca, stranger := newCA(t), newCA(t)
255 f := newFakeMCP(t, ca) 293 f := newFakeMCP(t, ca)
@@ -259,6 +297,7 @@ func TestProveMCPRejectsAShortToolset(t *testing.T) {
259 err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) 297 err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial)
260 require.Error(t, err) 298 require.Error(t, err)
261 assert.Contains(t, err.Error(), "advertises 5 tools") 299 assert.Contains(t, err.Error(), "advertises 5 tools")
300 assert.Empty(t, f.calls, "a wrong toolset must fail before any tool is called")
262 } 301 }
263 302
264 // TestProveMCPRejectsAMissingDelegateTool guards the tools a remote caller 303 // TestProveMCPRejectsAMissingDelegateTool guards the tools a remote caller
internal/smoke/run.go
Old New
@@ -38,39 +38,62 @@ func Run() error {
38 // touches VMs — the machine identity's JIT tenant owns no hosts (the fleet's 38 // touches VMs — the machine identity's JIT tenant owns no hosts (the fleet's
39 // `default` tenant is human-owned), so the lifecycle half runs as the operator 39 // `default` tenant is human-owned), so the lifecycle half runs as the operator
40 // below. 40 // below.
41 pwBytes, err := os.ReadFile(cfg.CIPasswordFile) 41 //
42 if err != nil { 42 // A plane whose issuer is a real identity provider has no password to post,
43 return fmt.Errorf("read CI password file %q: %w", cfg.CIPasswordFile, err) 43 // so the proof is opt-in: no CI_USER means the run starts at phase 2 and the
44 } 44 // operator PAT carries it. What that plane gives up is stated out loud
45 password := strings.TrimRight(string(pwBytes), "\r\n") 45 // rather than silently, because it is the one leg a hosted run cannot make.
46 var signedInPAT string
47 if cfg.CIUser == "" {
48 fmt.Println("credential chain: skipped (no CI_USER — this plane signs in against a real identity provider)")
49 } else {
50 pwBytes, err := os.ReadFile(cfg.CIPasswordFile)
51 if err != nil {
52 return fmt.Errorf("read CI password file %q: %w", cfg.CIPasswordFile, err)
53 }
54 password := strings.TrimRight(string(pwBytes), "\r\n")
46 55
47 token, err := loginPAT(cfg.ServerURL, cfg.CIUser, password) 56 signedInPAT, err = loginPAT(cfg.ServerURL, cfg.CIUser, password)
48 if err != nil { 57 if err != nil {
49 return err 58 return err
50 } 59 }
51 ciTenant, err := proveCredentialChain(cfg.ServerURL, token) 60 ciTenant, err := proveCredentialChain(cfg.ServerURL, signedInPAT)
52 if err != nil { 61 if err != nil {
53 return err 62 return err
63 }
64 fmt.Printf("credential chain OK (tenant %s)\n", ciTenant)
54 } 65 }
55 fmt.Printf("credential chain OK (tenant %s)\n", ciTenant)
56 66
57 // The remote MCP endpoint's cheap half: it must exist and must refuse an 67 // The remote MCP endpoint's cheap half: it must exist and must refuse an
58 // unauthenticated caller. It needs no guest, so a misrouted or unprotected 68 // unauthenticated caller. It needs no guest, so a misrouted or unprotected
59 // /mcp fails here in a second rather than after a VM has booted. 69 // /mcp fails here in a second rather than after a VM has booted. Every
60 if err := proveRemoteMCPNeedsACredential(cfg.ServerURL); err != nil { 70 // origin in the list answers for itself — a plane fronted both through a
61 return err 71 // proxy and directly can route one and not the other.
72 for _, origin := range cfg.MCPURLs {
73 if err := proveRemoteMCPNeedsACredential(origin); err != nil {
74 return err
75 }
62 } 76 }
63 fmt.Println("remote MCP endpoint refuses an unauthenticated caller") 77 fmt.Printf("remote MCP endpoint refuses an unauthenticated caller (%s)\n", strings.Join(cfg.MCPURLs, ", "))
64 78
65 // Phase 2 — VM lifecycle. Authenticate with an operator-minted PAT read from 79 // Phase 2 — VM lifecycle, on a normal tenant-scoped console PAT (spec §3's
66 // CI_PAT_FILE (a normal, tenant-scoped console PAT — spec §3's automation 80 // automation story). It comes from CI_PAT_FILE where an operator minted one
67 // story). The scenario's tenant is DERIVED via Me(), never assumed, and 81 // by hand, because the fleet's hosts belong to that operator's tenant and
82 // the machine identity's own JIT tenant owns none. Where the plane's issuer
83 // is one of ours, the identity that just signed in IS the operator, so the
84 // PAT it minted carries this phase and there is no token to paste anywhere.
85 // The scenario's tenant is DERIVED via Me() either way, never assumed, and
68 // threaded into the user-CA registration and gate connect name below. 86 // threaded into the user-CA registration and gate connect name below.
69 patBytes, err := os.ReadFile(cfg.CIPATFile) 87 pat := signedInPAT
70 if err != nil { 88 if cfg.CIPATFile != "" {
71 return fmt.Errorf("read CI PAT file %q: %w", cfg.CIPATFile, err) 89 patBytes, err := os.ReadFile(cfg.CIPATFile)
90 if err != nil {
91 return fmt.Errorf("read CI PAT file %q: %w", cfg.CIPATFile, err)
92 }
93 pat = strings.TrimRight(string(patBytes), "\r\n")
94 } else {
95 fmt.Println("VM lifecycle runs on the PAT minted by the sign-in above")
72 } 96 }
73 pat := strings.TrimRight(string(patBytes), "\r\n")
74 97
75 api := &client.Client{ 98 api := &client.Client{
76 BaseURL: cfg.ServerURL, 99 BaseURL: cfg.ServerURL,
@@ -83,7 +106,11 @@ func Run() error {
83 return fmt.Errorf("resolve operator PAT tenant: %w", err) 106 return fmt.Errorf("resolve operator PAT tenant: %w", err)
84 } 107 }
85 if me.Tenant == "" { 108 if me.Tenant == "" {
86 return fmt.Errorf("operator PAT (%s) resolved to an empty tenant", cfg.CIPATFile) 109 source := cfg.CIPATFile
110 if source == "" {
111 source = "minted by " + cfg.CIUser + "'s sign-in"
112 }
113 return fmt.Errorf("operator PAT (%s) resolved to an empty tenant", source)
87 } 114 }
88 tenant := me.Tenant 115 tenant := me.Tenant
89 fmt.Printf("operator PAT tenant: %s\n", tenant) 116 fmt.Printf("operator PAT tenant: %s\n", tenant)
@@ -117,9 +144,20 @@ func Run() error {
117 144
118 ctx := context.Background() 145 ctx := context.Background()
119 146
147 // Every origin past the first is proven routable and authenticated and then
148 // let go: it advertises the whole toolset over a bearer PAT. The full cycle
149 // below runs once, on the first origin — the entry a plane points at the
150 // path it most needs watched.
151 for _, origin := range cfg.MCPURLs[1:] {
152 if err := proveRemoteMCPToolset(ctx, origin, pat); err != nil {
153 return err
154 }
155 fmt.Printf("remote MCP toolset OK at %s\n", origin)
156 }
157
120 // Remote MCP, over the same PAT: no local install, no uploaded CA, nothing 158 // Remote MCP, over the same PAT: no local install, no uploaded CA, nothing
121 // but a token and an HTTP endpoint. The session lives for the whole run. 159 // but a token and an HTTP endpoint. The session lives for the whole run.
122 tools, closeMCP, err := dialMCP(ctx, cfg.ServerURL, pat) 160 tools, closeMCP, err := dialMCP(ctx, cfg.MCPURLs[0], pat)
123 if err != nil { 161 if err != nil {
124 return err 162 return err
125 } 163 }
scripts/coverage.sh
Old New
@@ -31,6 +31,8 @@ declare -A FLOOR=(
31 [internal/agent/exposeproxy]=80 31 [internal/agent/exposeproxy]=80
32 [internal/server/api]=76 32 [internal/server/api]=76
33 [internal/server/delegation]=90 33 [internal/server/delegation]=90
34 [internal/server/seal]=85
35 [internal/server/sshca]=65
34 [internal/server/mcphttp]=88 36 [internal/server/mcphttp]=88
35 [internal/server/vmssh]=90 37 [internal/server/vmssh]=90
36 [internal/server/boot]=20 38 [internal/server/boot]=20
@@ -49,7 +51,7 @@ declare -A FLOOR=(
49 [internal/transport]=77 51 [internal/transport]=77
50 [internal/shape]=88 52 [internal/shape]=88
51 [internal/site]=84 53 [internal/site]=84
52 [internal/smoke]=46 54 [internal/smoke]=50
53 [internal/cli]=64 55 [internal/cli]=64
54 [internal/mcpserver]=57 56 [internal/mcpserver]=57
55 [internal/oidcprovider]=79 57 [internal/oidcprovider]=79
scripts/deploy.env.example
Old New
@@ -26,7 +26,11 @@ SERVER_URL="http://127.0.0.1:8080" # http_listen, for health che
26 26
27 # ── Hosts (remote eitri-agent) ──────────────────────────────────────────────── 27 # ── Hosts (remote eitri-agent) ────────────────────────────────────────────────
28 # Space-separated list of ssh targets, each "user@host[:port]" (port defaults 22). 28 # Space-separated list of ssh targets, each "user@host[:port]" (port defaults 22).
29 AGENT_HOSTS="ubuntu@192.168.0.193:2222" 29 # The branch gate's fleet is nested VMs on this workstation: run
30 # `scripts/devhost.sh create` and paste the AGENT_HOSTS line it prints. Real
31 # hardware gates pre-release tags on stg instead (deploy/server/README.md), so a
32 # hosted outage can never fail the fastest, most-run check in the project.
33 AGENT_HOSTS="ubuntu@192.168.122.50"
30 AGENT_BIN="/usr/local/bin/eitri-agent" # install destination on each host 34 AGENT_BIN="/usr/local/bin/eitri-agent" # install destination on each host
31 AGENT_STATE_DIR="/var/lib/eitri-agent" 35 AGENT_STATE_DIR="/var/lib/eitri-agent"
32 # Agent logs live in journald once the systemd unit is adopted: 36 # Agent logs live in journald once the systemd unit is adopted:
scripts/devhost.cloud-init.yaml
Old New
@@ -0,0 +1,23 @@
1 #cloud-config
2 # Seed for a nested dev host (scripts/devhost.sh). It carries the bare minimum
3 # to let `make deploy` in: a user with the workstation's key and passwordless
4 # sudo. Nothing eitri needs is installed here — cloud-hypervisor, the guest
5 # firmware and the agent itself all arrive through the deploy and the agent's
6 # own bootstrap, which is precisely the code path a branch gate should be
7 # exercising rather than skipping.
8 #
9 # ${SSH_AUTHORIZED_KEY} is substituted by devhost.sh from the local key.
10 users:
11 - name: ubuntu
12 sudo: "ALL=(ALL) NOPASSWD:ALL"
13 shell: /bin/bash
14 lock_passwd: true
15 ssh_authorized_keys:
16 - ${SSH_AUTHORIZED_KEY}
17
18 package_update: true
19 packages:
20 - qemu-guest-agent
21
22 runcmd:
23 - [systemctl, enable, --now, qemu-guest-agent]
scripts/devhost.sh
Old New
@@ -0,0 +1,185 @@
1 #!/usr/bin/env bash
2 #
3 # The branch gate's fleet: a nested KVM host on this workstation, defined here
4 # rather than hand-built, so a wedged dev host is a three-minute rebuild instead
5 # of a debugging session.
6 #
7 # scripts/devhost.sh create [name] build it from nothing
8 # scripts/devhost.sh recycle [name] destroy, then create — the whole point
9 # scripts/devhost.sh destroy [name]
10 # scripts/devhost.sh address [name] print its AGENT_HOSTS line
11 #
12 # Workstation-local on purpose. The branch gate is the most-run check in the
13 # project, and a guest on a remote box would make it fail whenever something
14 # hosted broke — inverting what the gate is for. This host has no network
15 # coupling to any fleet and dies the moment `make deploy` is not running.
16 #
17 # Nested guests get hardware virtualization through --cpu host-passthrough, so
18 # the host CPU must expose nested KVM (kvm_amd nested=1 / kvm_intel nested=1).
19 # What nested CANNOT prove is bridged networking and real-hardware quirks —
20 # that coverage moved to stg, which gates pre-release tags on real metal.
21 #
22 # The seed installs nothing eitri needs: cloud-hypervisor, the guest firmware
23 # and the agent all arrive through `make deploy` and the agent's own bootstrap,
24 # so a cold first deploy exercises that path rather than skipping it.
25 #
26 # Disks live in the libvirt system pool, which is root-owned — the qemu-img and
27 # rm calls below use sudo for that and nothing else.
28 set -euo pipefail
29
30 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
31
32 # Pinned guest image. Bump deliberately: take the sha from the SHA256SUMS file
33 # beside the image in the same dated directory.
34 IMAGE_URL="${DEVHOST_IMAGE_URL:-https://cloud-images.ubuntu.com/noble/20260801/noble-server-cloudimg-amd64.img}"
35 IMAGE_SHA256="${DEVHOST_IMAGE_SHA256:-0533b0655c32e68b31d792ecd6ccfca95abdbc536c4446874fe0513bd4140ffe}"
36 OS_VARIANT="${DEVHOST_OS_VARIANT:-ubuntu24.04}"
37
38 # Sized so the agent's admission logic runs against a cap BELOW the machine's
39 # real size, the same shape a bare-metal host runs — which keeps the
40 # admission path honestly covered. The caps themselves go in deploy.env's
41 # AGENT_EXTRA_FLAGS; see the line this script prints.
42 MEM_MB="${DEVHOST_MEM_MB:-16384}"
43 VCPUS="${DEVHOST_VCPUS:-8}"
44 DISK_GB="${DEVHOST_DISK_GB:-120}"
45
46 NETWORK="${DEVHOST_NETWORK:-default}"
47 POOL_DIR="${DEVHOST_POOL_DIR:-/var/lib/libvirt/images}"
48 CACHE_DIR="${DEVHOST_CACHE_DIR:-$HOME/.cache/eitri/devhost}"
49 SSH_KEY="${DEVHOST_SSH_KEY:-$HOME/.ssh/id_ed25519.pub}"
50 SSH_PORT="${DEVHOST_SSH_PORT:-22}"
51
52 bold() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; }
53 die() { printf 'devhost: %s\n' "$*" >&2; exit 1; }
54
55 VERB="${1:-}"
56 NAME="${2:-eitri-dev1}"
57
58 case "$VERB" in
59 create | recycle | destroy | address) ;;
60 *)
61 sed -n '2,20p' "$0" | sed 's/^#\{1,2\} \{0,1\}//' >&2
62 exit 1
63 ;;
64 esac
65
66 # Address and MAC are derived from the name, so a recycled host comes back at
67 # the same address and AGENT_HOSTS never has to be edited. The index is the
68 # trailing digits of the name ("eitri-dev2" -> 2 -> .51).
69 index="$(printf '%s' "$NAME" | grep -oE '[0-9]+$' || echo 1)"
70 IP="${DEVHOST_IP:-192.168.122.$((49 + index))}"
71 # Locally-administered QEMU prefix plus three bytes of the name's digest: stable
72 # per name, and distinct enough not to collide with anything else on the bridge.
73 mac_tail="$(printf '%s' "$NAME" | sha256sum | cut -c1-6 | sed 's/../&:/g; s/:$//')"
74 MAC="${DEVHOST_MAC:-52:54:00:$mac_tail}"
75 DISK="$POOL_DIR/$NAME.qcow2"
76
77 need() { command -v "$1" >/dev/null 2>&1 || die "$1 is not installed"; }
78
79 destroy_host() {
80 bold "Destroying $NAME"
81 virsh destroy "$NAME" >/dev/null 2>&1 || true
82 # --remove-all-storage takes the root disk and the cloud-init seed ISO with
83 # the domain; the explicit rm is the belt for a domain that never defined.
84 virsh undefine "$NAME" --nvram --remove-all-storage >/dev/null 2>&1 || true
85 sudo rm -f "$DISK"
86 # The DHCP reservation deliberately survives: it is what makes the address
87 # stable across a recycle, and create re-adds it either way.
88 echo "$NAME is gone (its reservation at $IP is kept)"
89 }
90
91 create_host() {
92 need virsh
93 need virt-install
94 need qemu-img
95 [[ -e /dev/kvm ]] || die "/dev/kvm is missing — nested guests need hardware virtualization"
96 [[ -f "$SSH_KEY" ]] || die "no public key at $SSH_KEY (set DEVHOST_SSH_KEY)"
97 virsh net-info "$NETWORK" >/dev/null 2>&1 || die "libvirt network '$NETWORK' does not exist"
98 virsh net-info "$NETWORK" | grep -q 'Active:.*yes' || die "libvirt network '$NETWORK' is not running (virsh net-start $NETWORK)"
99 if virsh dominfo "$NAME" >/dev/null 2>&1; then
100 die "$NAME already exists — 'recycle' rebuilds it, 'destroy' removes it"
101 fi
102
103 bold "Fetching the guest image"
104 mkdir -p "$CACHE_DIR"
105 cached="$CACHE_DIR/$(basename "$IMAGE_URL")"
106 if [[ ! -f "$cached" ]]; then
107 curl -fsSL -o "$cached.part" "$IMAGE_URL"
108 mv "$cached.part" "$cached"
109 fi
110 echo "$IMAGE_SHA256 $cached" | sha256sum -c - >/dev/null || {
111 rm -f "$cached"
112 die "image sha mismatch — cache purged, re-run"
113 }
114 echo "image ok: $cached"
115
116 bold "Pinning $NAME to $IP on the '$NETWORK' network"
117 virsh net-update "$NETWORK" delete ip-dhcp-host "<host mac='$MAC'/>" \
118 --live --config >/dev/null 2>&1 || true
119 virsh net-update "$NETWORK" add ip-dhcp-host \
120 "<host mac='$MAC' name='$NAME' ip='$IP'/>" --live --config >/dev/null
121
122 bold "Creating $NAME ($VCPUS vCPU, $((MEM_MB / 1024)) GB, $DISK_GB GB)"
123 sudo qemu-img convert -O qcow2 "$cached" "$DISK"
124 sudo qemu-img resize "$DISK" "${DISK_GB}G" >/dev/null
125
126 seed="$(mktemp)"
127 trap 'rm -f "$seed"' EXIT
128 SSH_AUTHORIZED_KEY="$(cat "$SSH_KEY")" \
129 envsubst '${SSH_AUTHORIZED_KEY}' <"$REPO_ROOT/scripts/devhost.cloud-init.yaml" >"$seed"
130
131 virt-install \
132 --name "$NAME" \
133 --memory "$MEM_MB" \
134 --vcpus "$VCPUS" \
135 --cpu host-passthrough \
136 --disk "path=$DISK,format=qcow2,bus=virtio" \
137 --network "network=$NETWORK,mac=$MAC,model=virtio" \
138 --os-variant "$OS_VARIANT" \
139 --cloud-init "user-data=$seed" \
140 --graphics none \
141 --noautoconsole \
142 --import
143 rm -f "$seed"
144 trap - EXIT
145
146 # A rebuilt host presents a new host key at an address ssh already knows,
147 # which otherwise fails the first deploy with a warning that looks like an
148 # attack rather than a recycle.
149 ssh-keygen -R "$IP" >/dev/null 2>&1 || true
150
151 bold "Waiting for ssh on $IP"
152 for _ in $(seq 1 60); do
153 if ssh -p "$SSH_PORT" -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
154 -o ConnectTimeout=5 "ubuntu@$IP" true 2>/dev/null; then
155 break
156 fi
157 sleep 5
158 done
159 ssh -p "$SSH_PORT" -o BatchMode=yes -o ConnectTimeout=5 "ubuntu@$IP" true 2>/dev/null ||
160 die "$NAME did not answer ssh at $IP within 5 minutes (virsh console $NAME)"
161
162 print_address
163 }
164
165 print_address() {
166 cat <<-EOF
167
168 $NAME is up. Point the branch gate at it in ~/eitri-deploy/deploy.env:
169
170 AGENT_HOSTS="ubuntu@$IP"
171 AGENT_EXTRA_FLAGS="--max-vcpus $((VCPUS - 2)) --max-mem-mb $((MEM_MB - 4096)) --max-disk-gb $((DISK_GB - 30))"
172
173 then run: make deploy
174 EOF
175 }
176
177 case "$VERB" in
178 create) create_host ;;
179 destroy) destroy_host ;;
180 recycle)
181 destroy_host
182 create_host
183 ;;
184 address) print_address ;;
185 esac
scripts/release.sh
Old New
@@ -1,7 +1,8 @@
1 #!/usr/bin/env bash 1 #!/usr/bin/env bash
2 # Cross-compiled release artifacts for eitri.sh, into dist/<version>/: 2 # Cross-compiled release artifacts for eitri.sh, into dist/<version>/:
3 # eitri_<v>_linux_{amd64,arm64}.tar.gz host bundle: server+agent+systemd unit 3 # eitri-server_<v>_linux_{amd64,arm64}.tar.gz Linux host bundle: server+agent+unit
4 # eitri_<v>_darwin_arm64.tar.gz Mac host bundle: agent + LaunchAgent installer 4 # eitri-agent_<v>_darwin_arm64.tar.gz Mac host bundle: agent + LaunchAgent installer
5 # (a Mac runs guests; it carries no server)
5 # eitri-cli_<v>_<os>_<arch>.tar.gz client CLI (eitri) for linux+darwin 6 # eitri-cli_<v>_<os>_<arch>.tar.gz client CLI (eitri) for linux+darwin
6 # eitri-oidc_<v>_linux_{amd64,arm64}.tar.gz bundled OIDC issuer + its unit 7 # eitri-oidc_<v>_linux_{amd64,arm64}.tar.gz bundled OIDC issuer + its unit
7 # (optional sidecar; not in manifest.json) 8 # (optional sidecar; not in manifest.json)
@@ -64,14 +65,15 @@ trap 'rm -rf "$STAGE_ROOT"' EXIT
64 # bare copies exist only to carry the ones already in the field across. 65 # bare copies exist only to carry the ones already in the field across.
65 # 66 #
66 # NEXT RELEASE, once no agent older than this one is still out there: point 67 # NEXT RELEASE, once no agent older than this one is still out there: point
67 # manifest.json at eitri_<v>_<os>_<arch>.tar.gz (widen the matcher in 68 # manifest.json at eitri-server_<v>_linux_<arch>.tar.gz and
69 # eitri-agent_<v>_darwin_arm64.tar.gz (widen the matcher in
68 # internal/site.BuildManifest) and delete the two `cp` lines below. /dl then 70 # internal/site.BuildManifest) and delete the two `cp` lines below. /dl then
69 # serves tarballs and nothing is shipped twice. cloud-hypervisor and CLOUDHV.fd 71 # serves tarballs and nothing is shipped twice. cloud-hypervisor and CLOUDHV.fd
70 # stay bare in either case — agents bootstrap those directly and they duplicate 72 # stay bare in either case — agents bootstrap those directly and they duplicate
71 # nothing. 73 # nothing.
72 # ------------------------------------------------------------------------------ 74 # ------------------------------------------------------------------------------
73 for arch in amd64 arm64; do 75 for arch in amd64 arm64; do
74 bundle="eitri_${VERSION}_linux_${arch}" 76 bundle="eitri-server_${VERSION}_linux_${arch}"
75 stage="$STAGE_ROOT/$arch" 77 stage="$STAGE_ROOT/$arch"
76 mkdir -p "$stage/$bundle" 78 mkdir -p "$stage/$bundle"
77 echo "==> building linux/$arch" 79 echo "==> building linux/$arch"
@@ -98,7 +100,7 @@ done
98 # entitlement in its code signature and so can only come from a signed 100 # entitlement in its code signature and so can only come from a signed
99 # distribution (brew install vfkit), never from a mirror of ours. That is why 101 # distribution (brew install vfkit), never from a mirror of ours. That is why
100 # the darwin agent skips the bootstrap the Linux one runs at startup. 102 # the darwin agent skips the bootstrap the Linux one runs at startup.
101 mac_bundle="eitri_${VERSION}_darwin_arm64" 103 mac_bundle="eitri-agent_${VERSION}_darwin_arm64"
102 mac_stage="$STAGE_ROOT/darwin-arm64" 104 mac_stage="$STAGE_ROOT/darwin-arm64"
103 mkdir -p "$mac_stage/$mac_bundle" 105 mkdir -p "$mac_stage/$mac_bundle"
104 echo "==> building darwin/arm64" 106 echo "==> building darwin/arm64"
scripts/server-image.sh
Old New
@@ -3,6 +3,10 @@
3 # pinned to the cluster's arm64 public-IP node; amd64 installs are served by 3 # pinned to the cluster's arm64 public-IP node; amd64 installs are served by
4 # the release tarballs, not this image). 4 # the release tarballs, not this image).
5 # 5 #
6 # The image carries eitri-oidc as well as eitri-server. A plane with no
7 # external identity provider runs the bundled issuer from this same image, so
8 # the two can never be at different versions.
9 #
6 # Reads deploy.env (same file scripts/deploy.sh uses): 10 # Reads deploy.env (same file scripts/deploy.sh uses):
7 # SERVER_IMAGE registry/repo to push, e.g. registry.example/eitri-server (required) 11 # SERVER_IMAGE registry/repo to push, e.g. registry.example/eitri-server (required)
8 # 12 #
@@ -24,6 +28,8 @@ BUILD_DIR="$(mktemp -d)"
24 trap 'rm -rf "$BUILD_DIR"' EXIT 28 trap 'rm -rf "$BUILD_DIR"' EXIT
25 CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -ldflags "$LDFLAGS" \ 29 CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -ldflags "$LDFLAGS" \
26 -o "$BUILD_DIR/eitri-server" ./cmd/eitri-server 30 -o "$BUILD_DIR/eitri-server" ./cmd/eitri-server
31 CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -ldflags "$LDFLAGS" \
32 -o "$BUILD_DIR/eitri-oidc" ./cmd/eitri-oidc
27 cp deploy/server/Dockerfile "$BUILD_DIR/" 33 cp deploy/server/Dockerfile "$BUILD_DIR/"
28 34
29 docker build --platform linux/arm64 -t "$SERVER_IMAGE:$VERSION" "$BUILD_DIR" 35 docker build --platform linux/arm64 -t "$SERVER_IMAGE:$VERSION" "$BUILD_DIR"
scripts/ship.env.example
Old New
@@ -0,0 +1,55 @@
1 # eitri pipeline config — sourced by scripts/ship.sh, one file per plane.
2 #
3 # Copy this to ~/eitri-deploy/<target>/ship.env (override the whole path with
4 # $EITRI_SHIP_ENV) and fill it in. This file is site-specific and points at
5 # secrets — keep it OUT of the repo. What the PLANE is (hostnames, host ports,
6 # namespace, which plane gets backups and a purge) is committed instead, in
7 # deploy/server/plane.<target>.env.
8 #
9 # The same layout as scripts/deploy.env.example on purpose: one convention.
10
11 # ── Registries ────────────────────────────────────────────────────────────────
12 # ship.sh points the image builds at THIS file, so these are the only place a
13 # plane's registry is named.
14 SERVER_IMAGE="registry.example.com/eitri-server"
15 SITE_IMAGE="registry.example.com/eitri-site"
16 # Image platform for the site image; match the node arch in the plane file.
17 SITE_PLATFORM="linux/arm64"
18
19 # ── The plane's fleet, for the hosted smoke ───────────────────────────────────
20 # The smoke reads the guest's serial log over ssh to its host and dials that
21 # host's own uplink for the exposure leg, so these are LAN addresses of the
22 # hosts joined to THIS plane — unrelated to where the control plane runs.
23 AGENT_HOSTS="ubuntu@192.168.0.193:2222"
24 AGENT_STATE_DIR="/var/lib/eitri-agent"
25
26 # ── Credentials ───────────────────────────────────────────────────────────────
27 # Name ONE of the two arrangements. ship.sh forwards whatever it finds here and
28 # clears the rest, so the plane decides and the pipeline stays the same script.
29 #
30 # (a) A plane running the bundled eitri-oidc — stg. The smoke signs in through
31 # the real code flow, which proves the whole credential chain, and mints its
32 # own short-lived PAT for the run: no token to paste, none to rotate. The
33 # identity must exist in the issuer (README, "One-time bring-up").
34 # CI_USER="ship@eitri.local"
35 # CI_PASSWORD_FILE="$HOME/eitri-deploy/stg/ship-password"
36 #
37 # (b) A plane fronted by a real identity provider — prod. There is no password
38 # to post at Google, so the credential-chain proof is skipped and an
39 # operator-minted, non-expiring PAT carries the run. The smoke derives its
40 # tenant from the token rather than being told.
41 CI_PAT_FILE="$HOME/eitri-deploy/prod/deploy-pat"
42
43 # The smoke's own user CA, load-or-created here and registered with the tenant.
44 SMOKE_USER_CA_FILE="$HOME/eitri-deploy/prod/smoke_user_ca"
45
46 # ── Optional ──────────────────────────────────────────────────────────────────
47 # The MCP origins the smoke exercises, space-separated. The FIRST gets the full
48 # cycle — including the minutes-long vm_create that a proxy's silent-origin
49 # timeout would cut — and the rest get an unauthenticated-401 and a toolset
50 # check. Defaults to "https://<console host> https://<api host>".
51 # SMOKE_MCP_URL="https://stg.eitri.sh https://api.stg.eitri.sh"
52 # SMOKE_VM_USER="ubuntu"
53 # A local CLOUDHV.fd to mirror into the release (scripts/release.sh); without
54 # one the release ships no firmware and agents cannot bootstrap it.
55 # FIRMWARE_SRC="$HOME/.cache/eitri/CLOUDHV.fd"
scripts/ship.sh
Old New
@@ -0,0 +1,575 @@
1 #!/usr/bin/env bash
2 #
3 # Deploy a tagged tree to one plane. A release is this script run twice: first
4 # against stg with a pre-release tag, then — same script, same stages, the same
5 # artifacts rebuilt from the same tag — against prod. Nothing reaches prod that
6 # a machine has not already done to stg.
7 #
8 # scripts/ship.sh --target <stg|prod> --tag <vX.Y.Z[-pre.N]> [options]
9 #
10 # --from <n> resume at stage n after a partial failure
11 # --skip-smoke stop after the roll (stage 9 still reports)
12 # --render-only print the rendered manifests and exit; touches nothing
13 #
14 # Every stage is idempotent: re-running from the top is always safe and is the
15 # documented default. Every remote or destructive action is gated on --target,
16 # and the plane's values come from deploy/server/plane.<target>.env — the
17 # hostnames, the host-port triple, and which plane gets backups and a purge.
18 #
19 # ONE STEP IS UNREHEARSED, DELIBERATELY: the CDN purge runs at prod and only at
20 # prod. stg is served cache-bypassed, so a purge there would invalidate nothing
21 # and make the prod step look practiced when it is not. Its first real execution
22 # is at promote time. That is why it is kept to a single curl with an explicit
23 # file list — small enough that an unrehearsed run is safe.
24 #
25 # The pipeline reads the plane's config Secret and validates it. It never
26 # writes one: a script that can write config secrets is a script that can
27 # overwrite prod's.
28 #
29 # Site-specific values (registries, credential paths, the plane's fleet) come
30 # from ~/eitri-deploy/<target>/ship.env — see scripts/ship.env.example. No
31 # secrets and no site-specific values live in the repo.
32 set -euo pipefail
33
34 TARGET=""
35 TAG=""
36 FROM=1
37 SKIP_SMOKE=0
38 RENDER_ONLY=0
39
40 usage() {
41 sed -n '2,30p' "$0" | sed 's/^#\{1,2\} \{0,1\}//'
42 exit "${1:-1}"
43 }
44
45 while [[ $# -gt 0 ]]; do
46 case "$1" in
47 --target) TARGET="${2:-}"; shift 2 ;;
48 --tag) TAG="${2:-}"; shift 2 ;;
49 --from) FROM="${2:-}"; shift 2 ;;
50 --skip-smoke) SKIP_SMOKE=1; shift ;;
51 --render-only) RENDER_ONLY=1; shift ;;
52 -h | --help) usage 0 ;;
53 *) echo "ship: unknown argument: $1" >&2; usage ;;
54 esac
55 done
56
57 case "$TARGET" in
58 stg | prod) ;;
59 *) echo "ship: --target must be stg or prod (got '${TARGET}')" >&2; usage ;;
60 esac
61 [[ -n "$TAG" ]] || { echo "ship: --tag is required" >&2; usage; }
62 [[ "$FROM" =~ ^[1-9]$ ]] || { echo "ship: --from must be a stage number 1-9" >&2; exit 1; }
63
64 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
65 cd "$REPO_ROOT"
66
67 bold() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; }
68 fail() { printf '\nship: %s\n' "$*" >&2; exit 1; }
69 warn() { printf 'ship: WARNING — %s\n' "$*" >&2; }
70
71 # ── Plane values (committed) and site values (not) ────────────────────────────
72 PLANE_ENV="$REPO_ROOT/deploy/server/plane.$TARGET.env"
73 [[ -f "$PLANE_ENV" ]] || fail "no plane file for target $TARGET at $PLANE_ENV"
74 # shellcheck disable=SC1090
75 source "$PLANE_ENV"
76
77 SHIP_ENV="${EITRI_SHIP_ENV:-$HOME/eitri-deploy/$TARGET/ship.env}"
78 if [[ ! -f "$SHIP_ENV" ]]; then
79 fail "config not found: $SHIP_ENV
80 cp scripts/ship.env.example \"$SHIP_ENV\" and edit it."
81 fi
82 # shellcheck disable=SC1090
83 source "$SHIP_ENV"
84
85 # The build scripts (release/site-image/server-image) read the registry names
86 # from $EITRI_DEPLOY_ENV. Point them at THIS plane's file so a prod run can
87 # never pick up the dev fleet's deploy.env by accident.
88 export EITRI_DEPLOY_ENV="$SHIP_ENV"
89
90 # The variables the templates may reference. envsubst is called with exactly
91 # this list, so an unrelated $FOO added to a manifest later is passed through
92 # rather than blanked — and every name here is checked non-empty first, because
93 # envsubst's own answer to an unset variable is to substitute nothing at all.
94 RENDER_VARS=(
95 NAMESPACE NODE_NAME IMAGE_ARCH
96 CONSOLE_HOST API_HOST GATE_HOST SYNC_HOST SITE_HOST
97 HTTP_PORT SYNC_PORT GATE_PORT
98 CONFIG_SECRET PVC_NAME PVC_SIZE TLS_SECRET SITE_TLS_SECRET
99 SERVER_IMAGE SITE_IMAGE TAG
100 )
101 # The bundled issuer's values join the list only on a plane that runs one, so a
102 # plane without it never carries them. An ${OIDC_*} reference added to a SHARED
103 # template later would then survive rendering as literal text and fail the apply
104 # loudly, rather than being blanked into a plausible-looking manifest.
105 if [[ "$LOCAL_OIDC" == "1" ]]; then
106 RENDER_VARS+=(OIDC_HOST OIDC_PORT OIDC_TLS_SECRET OIDC_CONFIG_SECRET OIDC_PVC_NAME OIDC_PVC_SIZE)
107 fi
108 export TAG
109
110 require_render_vars() {
111 local missing=() name
112 for name in "${RENDER_VARS[@]}"; do
113 [[ -n "${!name:-}" ]] || missing+=("$name")
114 done
115 if [[ ${#missing[@]} -gt 0 ]]; then
116 fail "unset or empty: ${missing[*]}
117 plane values live in $PLANE_ENV; site values in $SHIP_ENV"
118 fi
119 export "${RENDER_VARS[@]?}"
120 }
121
122 # The manifest sets, in the order they are applied. The shape goes on first; the
123 # two Deployments follow in separate stages because their ORDER is load-bearing
124 # (see stage 6).
125 SHAPE_MANIFESTS=(
126 namespace.yaml middleware.yaml
127 pvc.yaml service.yaml certificate.yaml ingressroute.yaml
128 site-service.yaml site-certificate.yaml site-ingressroute.yaml
129 )
130 if [[ "$BACKUPS" == "1" ]]; then
131 SHAPE_MANIFESTS+=(backup-cronjob.yaml)
132 fi
133 if [[ "$LOCAL_OIDC" == "1" ]]; then
134 SHAPE_MANIFESTS+=(oidc.yaml)
135 fi
136
137 # The Deployments that carry the release tag, rolled in stage 7. The issuer runs
138 # from the server's image so the two are always the same build.
139 SERVER_MANIFESTS=(deployment.yaml)
140 if [[ "$LOCAL_OIDC" == "1" ]]; then
141 SERVER_MANIFESTS+=(oidc-deployment.yaml)
142 fi
143
144 # render prints one manifest with this plane's values substituted.
145 render() {
146 local vars="" name
147 for name in "${RENDER_VARS[@]}"; do vars+="\${$name} "; done
148 envsubst "$vars" <"$REPO_ROOT/deploy/server/$1"
149 }
150
151 render_all() {
152 local f
153 for f in "${SHAPE_MANIFESTS[@]}" site-deployment.yaml "${SERVER_MANIFESTS[@]}"; do
154 echo "---"
155 echo "# deploy/server/$f"
156 render "$f"
157 done
158 }
159
160 # ── --render-only: read the manifests, touch nothing ──────────────────────────
161 if [[ "$RENDER_ONLY" == "1" ]]; then
162 require_render_vars
163 render_all
164 exit 0
165 fi
166
167 bold "Shipping $TAG to $TARGET (namespace $NAMESPACE)"
168 echo "console https://$CONSOLE_HOST api https://$API_HOST site https://$SITE_HOST"
169 echo "gate $GATE_HOST:$GATE_PORT sync $SYNC_HOST:$SYNC_PORT http :$HTTP_PORT"
170 [[ "$FROM" -gt 1 ]] && echo "(resuming at stage $FROM)"
171
172 # ── 1. Verify the tag ─────────────────────────────────────────────────────────
173 if [[ "$FROM" -le 1 ]]; then
174 bold "1. Verify the tag"
175 git diff --quiet || fail "working tree has unstaged changes; ship from a clean tagged tree"
176 git diff --cached --quiet || fail "working tree has staged changes; ship from a clean tagged tree"
177 git rev-parse "$TAG^{commit}" >/dev/null 2>&1 || fail "tag $TAG does not resolve to a commit"
178 described="$(git describe --tags --exact-match 2>/dev/null || true)"
179 [[ "$described" == "$TAG" ]] || fail "HEAD is at '${described:-no tag}', not $TAG — check out the tag first"
180
181 # The version must be parsable by the SAME code the fleet orders versions
182 # with, because an unparsable one silently disables the upgrade button
183 # everywhere at once. Ask that code rather than re-deriving its rule here:
184 # any parsable version sorts below an absurdly high one, and an unparsable
185 # version sorts below nothing.
186 probe="$(mktemp -d "$REPO_ROOT/.shipcheck.XXXXXX")"
187 trap 'rm -rf "$probe"' EXIT
188 cat >"$probe/main.go" <<-'PROBE'
189 package main
190
191 import (
192 "fmt"
193 "os"
194
195 "github.com/a73x/eitri/internal/server/release"
196 )
197
198 func main() {
199 if !release.Less(os.Args[1], "v9999.0.0") {
200 fmt.Println("unparsable")
201 return
202 }
203 fmt.Println("parsable")
204 }
205 PROBE
206 parsable="$(go run "./$(basename "$probe")" "$TAG")"
207 rm -rf "$probe"
208 trap - EXIT
209 if [[ "$parsable" != "parsable" ]]; then
210 fail "$TAG is unparsable to internal/server/release.Less.
211 Every agent orders versions with that code, so shipping this tag would
212 disable the upgrade button fleet-wide. Releases are vX.Y.Z and
213 pre-releases are vX.Y.Z-pre.N."
214 fi
215 echo "tag ok: $TAG at $(git rev-parse --short HEAD)"
216 fi
217
218 # ── 2. Build the release artifacts ────────────────────────────────────────────
219 # The manifest base is the one build input the two planes legitimately differ
220 # on, and it is why the stg run PROVES the download and upgrade paths rather
221 # than merely rehearsing them: the artifacts stg publishes name stg's own /dl.
222 if [[ "$FROM" -le 2 ]]; then
223 bold "2. Build release artifacts -> dist/$TAG"
224 MANIFEST_BASE="https://$SITE_HOST/dl/$TAG" "$REPO_ROOT/scripts/release.sh"
225 fi
226
227 # ── 3. Build and push the images ──────────────────────────────────────────────
228 #
229 # The site image bakes dist/$TAG in, so the artifacts stage 2 built are what
230 # /dl serves — the two image builds are called directly rather than through
231 # `make site-image`, whose own `make release` would rebuild those artifacts
232 # against the default manifest base and undo stage 2.
233 if [[ "$FROM" -le 3 ]]; then
234 bold "3. Build and push images at $TAG"
235 [[ -d "dist/$TAG" ]] || fail "dist/$TAG missing — run from stage 2"
236 make -C "$REPO_ROOT" web
237 "$REPO_ROOT/scripts/server-image.sh"
238 make -C "$REPO_ROOT" site SITE_DIST="dist/$TAG"
239 "$REPO_ROOT/scripts/site-image.sh"
240 fi
241
242 # ── 4. Config schema and value check ──────────────────────────────────────────
243 # The stage that justifies the script. Two checks, both against the Secret this
244 # plane is about to run with: does the tagged tree's schema agree with the
245 # committed contract, and does the Secret agree with the plane's manifests?
246 #
247 # The Secret is read, never written, and never printed. Schema failures name
248 # keys only. The plane-agreement checks below DO print the values they compare —
249 # listen addresses and public hostnames, none of them secret, and the mismatch
250 # is the whole point of the check.
251 if [[ "$FROM" -le 4 ]]; then
252 bold "4. Check the config schema and the plane's values"
253 require_render_vars
254
255 secret_json="$(kubectl -n "$NAMESPACE" get secret "$CONFIG_SECRET" \
256 -o jsonpath='{.data.server\.json}' 2>/dev/null | base64 -d)" ||
257 fail "cannot read secret $CONFIG_SECRET in namespace $NAMESPACE.
258 Create it once by hand (the pipeline never writes config secrets):
259 kubectl -n $NAMESPACE create secret generic $CONFIG_SECRET \\
260 --from-file=server.json=\$HOME/eitri-deploy/$TARGET/server.json"
261 printf '%s' "$secret_json" | jq -e . >/dev/null 2>&1 ||
262 fail "secret $CONFIG_SECRET does not hold a server.json key with valid JSON"
263
264 CONTRACT="$REPO_ROOT/deploy/server/config.required"
265 SCHEMA_SRC="$REPO_ROOT/internal/server/config/config.go"
266
267 # Inventory the keys the TAGGED TREE declares, as JSON paths. Nested structs
268 # reach a path through a prefix declared in the contract, so a new nested
269 # struct cannot slip in unclassified — it is reported here by name.
270 schema_keys="$(awk '
271 FNR == NR {
272 if ($0 ~ /^# struct-prefix:/) {
273 line = $0
274 sub(/^# struct-prefix:[ \t]*/, "", line)
275 eq = index(line, "=")
276 prefix[substr(line, 1, eq - 1)] = substr(line, eq + 1)
277 }
278 next
279 }
280 /^type [A-Za-z_]+ struct \{/ { st = $2; next }
281 st != "" && /^\}/ { st = ""; next }
282 st != "" && match($0, /json:"[^"]+"/) {
283 tag = substr($0, RSTART + 6, RLENGTH - 7)
284 sub(/,.*/, "", tag)
285 if (tag == "" || tag == "-") next
286 if (!(st in prefix)) { print "!" st; next }
287 print prefix[st] tag
288 }
289 ' "$CONTRACT" "$SCHEMA_SRC")"
290
291 undeclared="$(printf '%s\n' "$schema_keys" | grep '^!' | sort -u | tr -d '!' || true)"
292 if [[ -n "$undeclared" ]]; then
293 fail "the tagged tree declares config struct(s) with no JSON path:
294 $(echo "$undeclared" | tr '\n' ' ')
295 Add a '# struct-prefix: <Struct>=<json.path.>' line to
296 deploy/server/config.required and classify the keys it carries."
297 fi
298
299 # The contract's own key list, for the two set comparisons below.
300 contract_keys="$(grep -v '^[[:space:]]*#' "$CONTRACT" | awk 'NF {print $1}')"
301
302 # jq_path turns a contract path into a jq expression. A "*" segment stands
303 # for a map key and iterates every entry.
304 jq_path() { printf '.%s' "${1//.\*./[].}"; }
305 # Collect the path's values and require EVERY one of them: a wildcard path
306 # holds for all the map's entries or for none. A plane with an arm64 image
307 # and a blank amd64 one is not half-configured, it is broken for half its
308 # hosts. An empty collection is a missing key.
309 cfg_has() {
310 printf '%s' "$secret_json" |
311 jq -e "[$(jq_path "$1")] | length > 0 and all(. != null and . != \"\")" >/dev/null 2>&1
312 }
313
314 schema_fail=0
315 while read -r key status; do
316 [[ -z "$key" || "$key" == \#* ]] && continue
317 case "$status" in
318 required)
319 if ! cfg_has "$key"; then
320 echo "FAIL: $CONFIG_SECRET does not set required key '$key'" >&2
321 echo " patch it in (values never come from this repo):" >&2
322 echo " kubectl -n $NAMESPACE get secret $CONFIG_SECRET -o jsonpath='{.data.server\\.json}' | base64 -d > /tmp/server.json" >&2
323 echo " # add \"$key\", then:" >&2
324 echo " kubectl -n $NAMESPACE create secret generic $CONFIG_SECRET --from-file=server.json=/tmp/server.json --dry-run=client -o yaml | kubectl apply -f -" >&2
325 schema_fail=1
326 fi
327 ;;
328 retired)
329 cfg_has "$key" && warn "$CONFIG_SECRET still sets retired key '$key'; the server ignores it"
330 ;;
331 optional) ;;
332 *) fail "$CONTRACT classifies '$key' as '$status'; want required, optional or retired" ;;
333 esac
334 # A contract entry the schema no longer declares is stale documentation,
335 # not a deployment hazard — say so and carry on.
336 grep -qxF "$key" <<<"$schema_keys" ||
337 warn "$CONTRACT classifies '$key', which the tagged tree no longer declares"
338 done < <(grep -v '^[[:space:]]*#' "$CONTRACT" | grep -v '^[[:space:]]*$')
339
340 # The other direction, and the one the v0.0.3 incident needed: a key the tree
341 # grew that nobody classified.
342 while read -r key; do
343 [[ -z "$key" ]] && continue
344 grep -qxF "$key" <<<"$contract_keys" || {
345 echo "FAIL: the tagged tree grew config key '$key'" >&2
346 echo " classify it in deploy/server/config.required and, if it is required," >&2
347 echo " set it in the $NAMESPACE Secret $CONFIG_SECRET before shipping." >&2
348 schema_fail=1
349 }
350 done <<<"$schema_keys"
351
352 # Plane agreement. Reported in the same pass as the schema findings above,
353 # not after a re-run: this stage exists to be hit, and an operator fixing a
354 # config should see everything wrong with it at once. Each of these disagreeing is a hosted-shape failure that
355 # presents as something else entirely: a wrong gate domain reads as "pubkey
356 # denied", a wrong advertise_quic as a host that enrolls and never syncs.
357 agree_fail=0
358 assert_cfg() {
359 local path="$1" want="$2" got
360 got="$(printf '%s' "$secret_json" | jq -r "$(jq_path "$path") // \"\"")"
361 if [[ "$got" != "$want" ]]; then
362 echo "FAIL: $CONFIG_SECRET has $path = '$got', but this plane's manifests say '$want'" >&2
363 agree_fail=1
364 fi
365 }
366 assert_cfg http_listen ":$HTTP_PORT"
367 assert_cfg quic_listen ":$SYNC_PORT"
368 assert_cfg ssh_listen ":$GATE_PORT"
369 assert_cfg ssh_gate_domain "$GATE_HOST"
370 assert_cfg advertise_http "https://$CONSOLE_HOST"
371 assert_cfg advertise_quic "$SYNC_HOST:$SYNC_PORT"
372 assert_cfg oidc.public_url "https://$CONSOLE_HOST"
373
374 # A plane running the bundled issuer has a second config to keep in step, and
375 # the two ways it drifts both present as something else: a server pointed at
376 # the wrong issuer fails discovery ("sign-in temporarily unavailable"), and a
377 # redirect_url that does not match the server's callback exactly is a 400 at
378 # the end of an otherwise working login. Neither says what is actually wrong.
379 if [[ "$LOCAL_OIDC" == "1" ]]; then
380 assert_cfg oidc.issuer "https://$OIDC_HOST"
381 issuer_json="$(kubectl -n "$NAMESPACE" get secret "$OIDC_CONFIG_SECRET" \
382 -o jsonpath='{.data.eitri-oidc\.json}' 2>/dev/null | base64 -d)" ||
383 fail "cannot read secret $OIDC_CONFIG_SECRET in namespace $NAMESPACE.
384 This plane runs the bundled issuer; create its config once by hand
385 (see deploy/server/README.md, 'One-time bring-up of a plane')."
386 assert_issuer() {
387 local path="$1" want="$2" got
388 got="$(printf '%s' "$issuer_json" | jq -r "$path // \"\"")"
389 if [[ "$got" != "$want" ]]; then
390 echo "FAIL: $OIDC_CONFIG_SECRET has $path = '$got', want '$want'" >&2
391 agree_fail=1
392 fi
393 }
394 assert_issuer .issuer "https://$OIDC_HOST"
395 assert_issuer .listen ":$OIDC_PORT"
396 assert_issuer '.clients[0].redirect_url' "https://$CONSOLE_HOST/auth/callback"
397 # The client the server presents must be one the issuer knows.
398 server_client="$(printf '%s' "$secret_json" | jq -r '.oidc.client_id // ""')"
399 printf '%s' "$issuer_json" | jq -e --arg id "$server_client" \
400 '[.clients[].id] | index($id) != null' >/dev/null 2>&1 ||
401 {
402 echo "FAIL: the server signs in as client '$server_client', which $OIDC_CONFIG_SECRET does not register" >&2
403 agree_fail=1
404 }
405 fi
406
407 if [[ "$schema_fail" != "0" || "$agree_fail" != "0" ]]; then
408 fail "$NAMESPACE's config and the $TAG tree do not agree (see above); nothing was deployed"
409 fi
410 echo "config ok: schema classified, required keys present, plane values agree"
411 fi
412
413 # ── 5. Render and apply the shape ─────────────────────────────────────────────
414 # Everything except the two Deployments: namespace, middleware, storage,
415 # service, certificates and routes. Applying these is idempotent and cannot
416 # restart anything.
417 if [[ "$FROM" -le 5 ]]; then
418 bold "5. Apply the plane's shape to $NAMESPACE"
419 require_render_vars
420 for f in "${SHAPE_MANIFESTS[@]}"; do
421 echo " $f"
422 render "$f" | kubectl apply -f -
423 done
424 fi
425
426 # ── 6. Roll the site, then purge ──────────────────────────────────────────────
427 # ORDER IS LOAD-BEARING. The server fetches the release manifest at boot and
428 # pins the answer for 24 hours, so a server rolled ahead of its site serves the
429 # previous release to the whole fleet for a day.
430 if [[ "$FROM" -le 6 ]]; then
431 bold "6. Roll the site to $TAG (before the server)"
432 require_render_vars
433 render site-deployment.yaml | kubectl apply -f -
434 kubectl -n "$NAMESPACE" rollout status deployment/web --timeout=5m
435 echo "site serving $TAG at https://$SITE_HOST/dl/$TAG/"
436
437 PURGED="not attempted"
438 if [[ "$CDN_PURGE" != "1" ]]; then
439 echo "purge: skipped — $TARGET is served cache-bypassed, so there is nothing to invalidate"
440 PURGED="not applicable at $TARGET"
441 else
442 CF_ENV="${EITRI_CF_ENV:-$HOME/eitri-deploy/pipeline/cf-purge.env}"
443 if [[ ! -f "$CF_ENV" ]]; then
444 if [[ "${ALLOW_MANUAL_PURGE:-0}" == "1" ]]; then
445 warn "no $CF_ENV — purge NOT run. Do it by hand before announcing:
446 every URL under https://$SITE_HOST/dl/$TAG/, plus /dl/latest/manifest.json,
447 /dl/ and /docs/releases/. Serving a stale artifact is how a release breaks."
448 PURGED="DEFERRED — run it by hand"
449 else
450 fail "no $CF_ENV, so the CDN purge cannot run.
451 /dl is served immutable: without a purge the CDN keeps serving the
452 previous bytes. Create the file with CF_ZONE_ID and CF_API_TOKEN, or
453 re-run with ALLOW_MANUAL_PURGE=1 to say out loud that you will do it
454 by hand."
455 fi
456 else
457 # shellcheck disable=SC1090
458 source "$CF_ENV"
459 : "${CF_ZONE_ID:?set in $CF_ENV}" "${CF_API_TOKEN:?set in $CF_ENV}"
460 # An explicit file list, never a zone-wide purge: the tempting
461 # simplification would evict the whole site's cache on every release.
462 purge_urls="$(
463 {
464 find "dist/$TAG" -maxdepth 1 -type f -printf "https://$SITE_HOST/dl/$TAG/%f\n"
465 printf 'https://%s/dl/latest/manifest.json\n' "$SITE_HOST"
466 printf 'https://%s/dl/\n' "$SITE_HOST"
467 printf 'https://%s/docs/releases/\n' "$SITE_HOST"
468 } | jq -R . | jq -s .
469 )"
470 # The API takes at most 30 files per call.
471 while read -r batch; do
472 curl -fsS -X POST \
473 "https://api.cloudflare.com/client/v4/zones/$CF_ZONE_ID/purge_cache" \
474 -H "Authorization: Bearer $CF_API_TOKEN" \
475 -H "Content-Type: application/json" \
476 --data "$batch" >/dev/null
477 done < <(jq -c --argjson u "$purge_urls" -n \
478 '[range(0; ($u | length); 30) as $i | {files: $u[$i:$i+30]}][]')
479 echo "purge: $(jq 'length' <<<"$purge_urls") URLs invalidated"
480 PURGED="ran"
481 fi
482 fi
483 fi
484
485 # ── 7. Roll the server ────────────────────────────────────────────────────────
486 # Recreate strategy: a brief control-plane gap. Agents redial on their own.
487 if [[ "$FROM" -le 7 ]]; then
488 bold "7. Roll the control plane to $TAG"
489 require_render_vars
490 for f in "${SERVER_MANIFESTS[@]}"; do
491 echo " $f"
492 render "$f" | kubectl apply -f -
493 done
494 kubectl -n "$NAMESPACE" rollout status deployment/eitri-server --timeout=5m
495 if [[ "$LOCAL_OIDC" == "1" ]]; then
496 kubectl -n "$NAMESPACE" rollout status deployment/eitri-oidc --timeout=5m
497 fi
498 fi
499
500 # ── 8. Hosted smoke ───────────────────────────────────────────────────────────
501 # The same binary the branch gate runs, pointed at this plane's public names.
502 # The host-facing legs are unchanged by where the control plane lives: the
503 # boot proof reads a serial log over ssh to the agent host and the exposure leg
504 # dials that host's own uplink. What IS new is that the gate leg now crosses the
505 # internet, so GATE_HOST:GATE_PORT must be reachable from here.
506 #
507 # Which credentials the run uses is the plane's business, not this script's: it
508 # forwards whatever ship.env names and nothing else. A plane with a password
509 # issuer sets CI_USER and gets the credential-chain proof plus a freshly minted
510 # PAT; a plane fronted by a real identity provider sets an operator PAT instead,
511 # because there is no password to post at Google. No COVER_OUT either way — the
512 # hosted binaries are not coverage-instrumented.
513 if [[ "$FROM" -le 8 && "$SKIP_SMOKE" != "1" ]]; then
514 bold "8. Hosted smoke against $TARGET"
515 : "${AGENT_HOSTS:?set in $SHIP_ENV}" "${AGENT_STATE_DIR:?}"
516 if [[ -z "${CI_USER:-}" && -z "${CI_PAT_FILE:-}" ]]; then
517 fail "$SHIP_ENV names no credential for the smoke.
518 Set CI_USER + CI_PASSWORD_FILE for a plane with a password issuer, or
519 CI_PAT_FILE for one fronted by an external identity provider."
520 fi
521 go build -o "$REPO_ROOT/bin/eitri-smoke" ./cmd/eitri-smoke
522
523 # Both origins by default: the console host is the one a proxy fronts and
524 # carries the long call, and the API host proves the /mcp-only rule routes
525 # and authenticates with nothing in front of it.
526 SMOKE_MCP_URL="${SMOKE_MCP_URL:-https://$CONSOLE_HOST https://$API_HOST}"
527 SMOKE_USER_CA_FILE="${SMOKE_USER_CA_FILE:-$HOME/eitri-deploy/$TARGET/smoke_user_ca}"
528 smoke_env=(
529 SERVER_URL="https://$CONSOLE_HOST"
530 SMOKE_GATE="$GATE_HOST:$GATE_PORT"
531 SMOKE_MCP_URL="$SMOKE_MCP_URL"
532 SMOKE_USER_CA_FILE="$SMOKE_USER_CA_FILE"
533 AGENT_HOSTS="$AGENT_HOSTS"
534 AGENT_STATE_DIR="$AGENT_STATE_DIR"
535 )
536 if [[ -n "${CI_USER:-}" ]]; then
537 : "${CI_PASSWORD_FILE:?set it alongside CI_USER in $SHIP_ENV}"
538 smoke_env+=(CI_USER="$CI_USER" CI_PASSWORD_FILE="$CI_PASSWORD_FILE")
539 fi
540 if [[ -n "${CI_PAT_FILE:-}" ]]; then
541 smoke_env+=(CI_PAT_FILE="$CI_PAT_FILE")
542 fi
543 if [[ -n "${SMOKE_VM_USER:-}" ]]; then
544 smoke_env+=(SMOKE_VM_USER="$SMOKE_VM_USER")
545 fi
546 # Clear the trio first so an ambient value from this shell can never stand in
547 # for one the plane did not name.
548 env -u CI_USER -u CI_PASSWORD_FILE -u CI_PAT_FILE "${smoke_env[@]}" \
549 "$REPO_ROOT/bin/eitri-smoke" ||
550 fail "hosted smoke FAILED against $TARGET — the plane is serving $TAG and is not proven."
551 elif [[ "$SKIP_SMOKE" == "1" ]]; then
552 bold "8. Hosted smoke SKIPPED (--skip-smoke)"
553 fi
554
555 # ── 9. Report ─────────────────────────────────────────────────────────────────
556 bold "9. $TARGET is serving $TAG"
557 report_deployments=(eitri-server web)
558 if [[ "$LOCAL_OIDC" == "1" ]]; then
559 report_deployments+=(eitri-oidc)
560 fi
561 kubectl -n "$NAMESPACE" get deployment "${report_deployments[@]}" \
562 -o custom-columns='DEPLOYMENT:.metadata.name,IMAGE:.spec.template.spec.containers[0].image,READY:.status.readyReplicas' ||
563 true
564 echo "console https://$CONSOLE_HOST"
565 if [[ "$LOCAL_OIDC" == "1" ]]; then
566 echo "issuer https://$OIDC_HOST"
567 fi
568 echo "mcp https://$CONSOLE_HOST/mcp (proxied) and https://$API_HOST/mcp"
569 echo "downloads https://$SITE_HOST/dl/$TAG/"
570 echo "cdn purge: ${PURGED:-not reached this run}"
571 # An if, not a trailing &&: as the script's last command, a false conditional
572 # would be the whole run's exit status.
573 if [[ "$SKIP_SMOKE" == "1" ]]; then
574 echo "smoke: SKIPPED — this plane is deployed, not proven"
575 fi
web/src/routes/+page.svelte
Old New
@@ -175,10 +175,10 @@
175 {#if joinBlob} 175 {#if joinBlob}
176 <p>Download and verify the host bundle, then run on the new host:</p> 176 <p>Download and verify the host bundle, then run on the new host:</p>
177 <div class="enroll"> 177 <div class="enroll">
178 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/eitri_{dlVersion}_linux_amd64.tar.gz"</code> 178 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/eitri-server_{dlVersion}_linux_amd64.tar.gz"</code>
179 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/SHA256SUMS"</code> 179 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/SHA256SUMS"</code>
180 <code>sha256sum -c SHA256SUMS --ignore-missing</code> 180 <code>sha256sum -c SHA256SUMS --ignore-missing</code>
181 <code>tar xzf eitri_{dlVersion}_linux_amd64.tar.gz && cd eitri_{dlVersion}_linux_amd64</code> 181 <code>tar xzf eitri-server_{dlVersion}_linux_amd64.tar.gz && cd eitri-server_{dlVersion}_linux_amd64</code>
182 <code>sudo install -m 0755 eitri-agent /usr/local/bin/eitri-agent</code> 182 <code>sudo install -m 0755 eitri-agent /usr/local/bin/eitri-agent</code>
183 <code>sudo install -m 0644 eitri-agent.service /etc/systemd/system/eitri-agent.service</code> 183 <code>sudo install -m 0644 eitri-agent.service /etc/systemd/system/eitri-agent.service</code>
184 <code>sudo eitri-agent --state-dir /var/lib/eitri-agent join {joinBlob}</code> 184 <code>sudo eitri-agent --state-dir /var/lib/eitri-agent join {joinBlob}</code>
@@ -193,10 +193,10 @@
193 {#if joinBlob} 193 {#if joinBlob}
194 <div class="enroll"> 194 <div class="enroll">
195 Download and verify the host bundle, then run on the new host: 195 Download and verify the host bundle, then run on the new host:
196 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/eitri_{dlVersion}_linux_amd64.tar.gz"</code> 196 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/eitri-server_{dlVersion}_linux_amd64.tar.gz"</code>
197 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/SHA256SUMS"</code> 197 <code>curl -fsSLO "https://eitri.sh/dl/{dlVersion}/SHA256SUMS"</code>
198 <code>sha256sum -c SHA256SUMS --ignore-missing</code> 198 <code>sha256sum -c SHA256SUMS --ignore-missing</code>
199 <code>tar xzf eitri_{dlVersion}_linux_amd64.tar.gz && cd eitri_{dlVersion}_linux_amd64</code> 199 <code>tar xzf eitri-server_{dlVersion}_linux_amd64.tar.gz && cd eitri-server_{dlVersion}_linux_amd64</code>
200 <code>sudo install -m 0755 eitri-agent /usr/local/bin/eitri-agent</code> 200 <code>sudo install -m 0755 eitri-agent /usr/local/bin/eitri-agent</code>
201 <code>sudo install -m 0644 eitri-agent.service /etc/systemd/system/eitri-agent.service</code> 201 <code>sudo install -m 0644 eitri-agent.service /etc/systemd/system/eitri-agent.service</code>
202 <code>sudo eitri-agent --state-dir /var/lib/eitri-agent join {joinBlob}</code> 202 <code>sudo eitri-agent --state-dir /var/lib/eitri-agent join {joinBlob}</code>