deploy/server/README.md
Ref: Size: 31.5 KiB History
# The hosted planes and the release pipeline
Two planes run on one cluster from one set of manifests: **stg**
(`stg.eitri.sh`) and **prod** (`console.eitri.sh`). A release is
`scripts/ship.sh` run twice — first at stg with a pre-release tag, then at prod
with the release tag, same script, same stages, artifacts rebuilt from the same
tree. Nothing reaches prod that a machine has not already done to stg.
Each plane is one container carrying the whole control plane (console, API,
OIDC relying party, QUIC sync, SSH gate) pinned to the cluster's public-IP
node, with state on a local-path PVC and config in a Secret, plus an nginx
deployment serving the static site and its `/dl` artifacts.
## The two planes
| | prod | stg |
|---|---|---|
| namespace | `eitri` | `eitri-stg` |
| console | `console.eitri.sh` | `stg.eitri.sh` |
| MCP host | `api.eitri.sh` | `api.stg.eitri.sh` |
| gate | `gate.eitri.sh:2222` | `gate.stg.eitri.sh:2223` |
| sync | `sync.eitri.sh:8443` | `sync.stg.eitri.sh:8444` |
| site / `/dl` | `eitri.sh` | `dl.stg.eitri.sh` |
| http listener | `:8081` | `:8082` |
| guest CIDR pool | `10.78.0.0/16` | `10.79.0.0/16` |
| sign-in | Google | bundled `eitri-oidc` at `oidc.stg.eitri.sh` |
| backups | nightly | none — the database is disposable |
The pod runs `hostNetwork`, so those ports are the **node's**: the two planes
share a node and cannot share a port triple. Deployment, Service, PVC and
IngressRoute keep the same names in both namespaces — they are namespace-scoped,
so per-plane naming would be noise, and identical names mean `kubectl -n eitri`
and `kubectl -n eitri-stg` take the same commands. Only what is node-scoped or
cluster-visible (host ports, hostnames, TLS secrets) differs.
The guest CIDR pools differ so a host flicked between planes never carries a
colliding guest subnet. The local dev plane is `10.77.0.0/16`.
### Reading a manifest
The manifests are templates. Rendering is how you read them:
scripts/ship.sh --target stg --tag v0.0.4-pre.1 --render-only
That touches nothing — no cluster, no build — and prints exactly what an apply
would send. Values come from `deploy/server/plane.<target>.env` (committed) and
`~/eitri-deploy/<target>/ship.env` (not). `envsubst` is called with an explicit
variable list and every name on it is checked non-empty first, because
`envsubst`'s own answer to an unset variable is to substitute nothing at all.
One template set, two planes, so the planes cannot differ in *shape* — only in
*values*. That is the whole premise: the shape under test at stg is the real one.
Two components are *present or absent* rather than shaped differently, each
gated on a plane value: the backup CronJob (`BACKUPS`) and the bundled issuer
(`LOCAL_OIDC`). Their values join the template variable list only on a plane
that has them, so a plane without one can never render a half-filled manifest
for it.
### Sign-in, and the one place the planes genuinely differ
prod signs in against Google. stg runs `eitri-oidc`, the issuer that already
serves the local plane, so stg is reproducible from nothing but this repo and a
Secret, its identities are ours to create and delete, and the smoke can drive a
real sign-in headlessly — which is what lets the credential chain be *proven* at
stg rather than skipped.
It runs as its own Deployment from the **server's image at the server's tag**,
so the two can never be different builds, and it is an ordinary pod behind a
Service — nothing about it needs a socket on the node, so it costs the shared
node no port. It gets **its own hostname**, `oidc.stg.eitri.sh`, rather than a
path on the console: the login form posts back to the absolute path
`/authorize`, so mounted under a prefix the browser's POST would land on the
console host's catch-all rule instead of the issuer. An issuer only works at the
root of its own origin.
**Its state is not disposable.** `users.json` holds each identity's subject and
the server derives a tenant from that subject, so losing the file signs the
operator into a brand-new tenant while the old one still owns the plane's hosts
— with no error anywhere to say so. Hence its own PVC, pinned to the same node
as the server's.
What this costs: **stg does not exercise prod's relying-party wiring against a
third-party identity provider.** Everything after sign-in is identical; the
Google-specific half of prod's login is proven only at prod.
### Two facts about routing that will surprise someone
**The console host carries every path, so `console.eitri.sh/mcp` is a working
MCP endpoint** — and it is the one a proxy fronts. `api.eitri.sh` is not the
only way in; it is the way in that does not also route the console (routing the
whole API host would make it a second console origin, and OIDC sign-in works at
exactly `oidc.public_url`). The smoke exercises both.
**stg's console is proxied and prod's is not.** stg is *more* fronted than prod,
which inverts the usual staging relationship and has one practical consequence:
the console's SSE streams traverse Cloudflare at stg and go direct at prod. If a
stg run shows a dead live-updating console or a stalled event stream, the first
hypothesis is proxy buffering, not a code defect — and the confirming test is
the same page against `api.stg.eitri.sh`, which is grey-cloud.
## The pipeline
scripts/ship.sh --target <stg|prod> --tag <vX.Y.Z[-pre.N]>
[--from <stage>] [--skip-smoke] [--render-only]
Every stage announces itself and is idempotent; re-running from the top is
always safe and is the documented default. `--from <n>` resumes after a partial
failure.
Before stage 1 touches anything, the run checks that it can finish. Stages 8
and 9 both authenticate to the plane, and the credential they use has to be
named in `ship.env` and readable. Both of them come after the plane is rolled,
so a credential that is missing or unreadable is refused here, with nothing
deployed, rather than at stage 8 with the site and the server already at the
new tag.
1. **Verify the tag** — clean tree, HEAD is exactly this tag, and the version is
parsable by `internal/server/release`, asked by running that code rather than
re-deriving its rule. An unparsable version silently disables the upgrade
button fleet-wide. Releases are `vX.Y.Z`, pre-releases `vX.Y.Z-pre.N`, and
the fleet orders the whole chain:
`v0.0.4-pre.1 < v0.0.4-pre.2 < v0.0.4 < v0.0.4-2-g<hex>`.
2. **Build the artifacts** into `dist/<tag>`, with the manifest base pointed at
the target's own `/dl`. This is the one build input the planes legitimately
differ on, and it is why the stg run *proves* the download and upgrade paths
rather than rehearsing them. The build is byte-reproducible, which stage 3
relies on.
3. **Build and push the images**, server and site, tagged with the version —
after checking that this plane does not already serve this tag as something
else. The site image bakes `dist/<tag>` in, so the check happens here, at
the last moment before those bytes become an immutable URL. Absent means
first publish; identical means an honest re-run and the site image is left
alone; different is a hard failure, and the fix is a new tag. The server
image is rebuilt and pushed either way.
4. **Check the config** (see below).
5. **Apply the plane's shape** — namespace, middleware, storage, service,
certificates, routes. Nothing here restarts anything.
6. **Roll the site.** Order is load-bearing: the server fetches the release
manifest at boot and pins the answer for 24 hours, so a server rolled ahead
of its site serves the previous release to the whole fleet for a day.
7. **Roll the control plane.** Recreate strategy — a brief gap; agents redial.
8. **Converge the fleet's agents** on the tag just rolled. Agents never upgrade
themselves, and the smoke below drives a full VM cycle on whichever host the
plane picks, so every connected host behind the release is offered the
upgrade here and the stage waits (up to 300s) for all of them to report it.
A disconnected host is warned about and skipped — it cannot be offered
anything and the smoke cannot place a VM on it either. A host the plane
declines to offer as *not behind the latest release* converged between the
listing and the offer; the stage drops that offer and lets the version the
host reports settle it on the next poll, and the same timeout still catches
one that never reports the tag. It runs under `--skip-smoke` too: a
fleet running the release is part of shipping it. The exception is a machine
whose `ship.env` names no credential at all: convergence is an authenticated
act, so with `--skip-smoke` the stage is skipped with a warning — here and
again in stage 10's report — naming what the fleet still runs and how to
finish it. Without `--skip-smoke` that run never starts (see the preflight
above).
9. **Hosted smoke**: the same `eitri-smoke` the branch gate runs, against this
plane's public names.
10. **Report** what is actually running.
### A tag publishes once
`site/nginx.conf` states the cache contract the site is served under:
`/dl/v*` is `immutable`, everything else — the `/dl/latest/` alias the fleet
re-polls daily, the pages, the console bundle — is `no-cache`. That is the
right contract only if a versioned URL really never changes, so the pipeline
makes it true rather than assuming it.
`scripts/release.sh` builds byte-reproducibly: the same tagged tree and the
same manifest base yield the same tarballs, the same `SHA256SUMS` and the same
`manifest.json`, run after run. Stage 3 then compares what it is about to
publish against what the plane already serves for that tag. Absent is a first
publish. Identical is a re-run, which is the pipeline's whole idempotence
promise honoured at the one stage that touches the outside world; the site
image stays as it is, because rebuilding it would produce the same webroot
under a new digest and move a released tag's image for no reason. Different is
refused: those bytes are already in the wild and an agent verifies its upgrade
against the sha the manifest named, so the answer is a new tag, not a new
payload behind the old URL.
Each plane is asked about itself. Artifacts embed their own plane's `/dl`
base, so stg and prod hold genuinely different bytes for one tag and neither
is wrong.
### Stage 4, the one that justifies the script
Every incident on v0.0.3 release day was a hosted-shape failure the local gate
cannot see. Stage 4 is where that class dies. It reads the plane's Secret, never
writes it, and never prints its contents.
*Schema drift, both directions.* The keys the **tagged tree** declares are
extracted from `internal/server/config/config.go` and compared against
`deploy/server/config.required`, which classifies each as `required`, `optional`
or `retired`. A key the tree grew that nobody classified is a hard failure
naming the key; a required key missing from the Secret is a hard failure naming
the key and the patch that fixes it; a retired key still set is a warning. Had
this existed, `default_images` would have failed a pre-deploy check the moment
the struct grew the field, instead of becoming a live secret patch.
*Plane agreement.* The Secret's `http_listen`, `quic_listen`, `ssh_listen`,
`ssh_gate_domain`, `advertise_http`, `advertise_quic` and `oidc.public_url` must
match the plane's rendered manifests. Each of these disagreeing is a failure
that presents as something else entirely: a wrong gate domain reads as "pubkey
denied", a wrong `advertise_quic` as a host that enrolls and then never syncs.
### What stays operator-manual
Ordered by when they bite.
1. **The Google OAuth client — prod only.** No API, and a mistake is invisible
until sign-in fails. stg runs the bundled issuer and needs nothing from
Google.
2. **First-time Secret creation.** The pipeline validates config secrets and
never writes them: a script that can write one is a script that can overwrite
prod's. That is two secrets on a plane with a bundled issuer, one on prod.
3. **`regcred` copied into the namespace.**
4. **Registering the plane's identity — stg only**, one `kubectl exec` at
bring-up. The pipeline stays plane-agnostic: it forwards whatever credential
`ship.env` names and never creates one.
5. **Firewall openings** for the plane's sync and gate ports, in the cloud
security list *and* the node's host firewall.
6. **DNS.** Verify rather than create: the `*.eitri.sh` wildcard covers the HTTP
names. `gate` and `sync` are the exception and need explicit A records — see
"Known gaps".
7. **Minting the operator PAT — prod only.** Where the plane's issuer is one of
ours, the smoke signs in and mints its own, so there is no token to paste and
none to rotate.
8. **Tagging.** The pipeline verifies tags; it never creates or pushes them.
## One-time bring-up of a plane
Shown for stg. prod is the same minus steps 5 and 7, plus a Google OAuth client
(Web application, authorized redirect `https://console.eitri.sh/auth/callback`)
whose id and secret go into `oidc` below, and a hand-minted operator PAT at the
end.
1. **`eitri-oidc.json`** — 0600, in `~/eitri-deploy/stg/`, the bundled issuer's
own config. `issuer` must be the URL browsers and the server reach it at, and
`redirect_url` must equal the server's callback exactly; stage 4 asserts both
against the plane rather than letting them drift into a 400 at the end of an
otherwise working login.
```json
{
"listen": ":9111",
"issuer": "https://oidc.stg.eitri.sh",
"users_file": "/var/lib/eitri-oidc/users.json",
"signing_key": "/var/lib/eitri-oidc/signing.key",
"clients": [
{"id": "eitri-console", "redirect_url": "https://stg.eitri.sh/auth/callback"}
]
}
```
2. **`server.json`** — 0600, in `~/eitri-deploy/<target>/`, never committed and
never written by a script:
```json
{
"db_path": "/var/lib/eitri/eitri.db",
"http_listen": ":8082",
"quic_listen": ":8444",
"ssh_listen": ":2223",
"ssh_gate_domain": "gate.stg.eitri.sh",
"ssh_ca_key": "/var/lib/eitri/ssh_ca",
"ssh_host_key": "/var/lib/eitri/ssh_host_key",
"advertise_http": "https://stg.eitri.sh",
"advertise_quic": "sync.stg.eitri.sh:8444",
"cidr_pool": "10.79.0.0/16",
"host_secret": "<openssl rand -hex 32>",
"key_encryption_key": "<openssl rand -hex 32>",
"release_manifest_url": "https://dl.stg.eitri.sh/dl/latest/manifest.json",
"default_images": {
"amd64": {"url": "...", "sha256": "..."},
"arm64": {"url": "...", "sha256": "..."}
},
"oidc": {
"issuer": "https://oidc.stg.eitri.sh",
"client_id": "eitri-console",
"public_url": "https://stg.eitri.sh",
"allowed_identities": ["ship@eitri.local"]
}
}
```
No `client_secret`: the bundled issuer registers public PKCE clients, and the
server pins the token request's client-auth style accordingly. prod's Google
client is confidential and does set one.
`key_encryption_key` encrypts everything the plane signs with: the host CA
and gate host key on the PVC. The volume and its backups therefore hold
ciphertext, and this Secret is the key's only home. Each plane mints its own and keeps
it—it is never rotated in place, and a plane that loses it loses its host CA,
which is the identity every client pins and every VM's host certificate
names.
`allowed_identities` closes the signup gate. Both architectures get a
`default_images` entry: a host handed an image it cannot execute fails at
boot, which is the mistake that map exists to prevent — and stage 4 enforces
it for every entry, not just the first.
3. **Namespace and pull secret**:
kubectl create ns eitri-stg
kubectl -n eitri get secret regcred -o yaml \
| sed 's/namespace: eitri/namespace: eitri-stg/' | kubectl apply -f -
4. **The config Secrets** — the server's, and the issuer's where there is one:
kubectl -n eitri-stg create secret generic eitri-stg-server-config \
--from-file=server.json=$HOME/eitri-deploy/stg/server.json
kubectl -n eitri-stg create secret generic eitri-stg-oidc-config \
--from-file=eitri-oidc.json=$HOME/eitri-deploy/stg/eitri-oidc.json
5. **`ship.env`**: `cp scripts/ship.env.example ~/eitri-deploy/stg/ship.env`
and fill it in — registries and image platform now, the credential in step 8.
For stg the credential is `CI_USER` and `CI_PASSWORD_FILE`; generate the
password here and lock it down, since step 7 registers the identity with it:
umask 077; openssl rand -hex 16 > ~/eitri-deploy/stg/ship-password
6. **First run**: `scripts/ship.sh --target stg --tag <pre-tag> --skip-smoke`
applies everything else and starts both deployments. The server creates its
own SSH CA and gate host key on the PVC at first boot, so the gate comes up
on its own. Skip the smoke on this one run: the identity it signs in as does
not exist yet. With no credential in `ship.env` yet, stage 8 skips itself
and says so — correctly, since no host has joined this plane either.
7. **Register the plane's identity**, once, in the running issuer. It writes to
the issuer's PVC, so it survives every later roll:
kubectl -n eitri-stg exec deploy/eitri-oidc -i -- \
/eitri-oidc user add --config /etc/eitri/eitri-oidc.json \
--password-file /dev/stdin ship@eitri.local \
< ~/eitri-deploy/stg/ship-password
The password arrives on stdin rather than in an argument, which would put it
in the pod's process list. Re-running this changes the password and **keeps
the identity's subject**, so the tenant it owns survives a rotation.
8. **Sign in at exactly `https://stg.eitri.sh`** — anything else gives "invalid
oauth state" — which JIT-provisions the operator tenant that will own the
plane's hosts. Now that the identity exists, put `CI_USER` and
`CI_PASSWORD_FILE` in `ship.env`, and
`scripts/ship.sh --target stg --tag <pre-tag> --from 8` proves the whole
thing.
There is no PAT to mint by hand here. The smoke signs in as that same
identity and mints its own short-lived token for the run, deriving the tenant
from it rather than being told. prod, which cannot sign in headlessly against
Google, still needs a non-expiring PAT named `ship` saved to
`~/eitri-deploy/prod/deploy-pat` (0600) and named in its `ship.env`.
### Adopting prod's site objects
The `site-*.yaml` files describe the objects that have served `eitri.sh` since
before they were in the tree, and they carry those objects' own names —
Deployment and Service `web`, IngressRoutes `web-http` and `web-https`,
Certificate `web-tls`. They adopt the live site; they do not stand a second one
beside it. Confirm that before the first prod run:
scripts/ship.sh --target prod --tag <tag> --render-only > /tmp/prod.yaml
kubectl diff -f /tmp/prod.yaml
Every object should read as an update, never a create. One intentional
difference is expected: `web-http` gains the `redirect-https` middleware the
console's HTTP route already uses, so plain-HTTP `eitri.sh` starts redirecting.
The pipeline never deletes, so anything the live namespace carries that these
manifests do not describe — an unused `web-data` PVC, say — is left alone.
## Runbook: the nested dev fleet
The branch gate's fleet lives on the dev machine, defined by
`scripts/devhost.sh` rather than hand-built, and driven from the workstation.
Bring it up **before** mewtwo moves to stg — until it is green, the project has
no branch gate.
export DEVHOST_REMOTE=ubuntu@192.168.0.193:2222 # the dev machine, mewtwo
scripts/devhost.sh create # from nothing
scripts/devhost.sh recycle # destroy + create; a wedged host is 3 minutes
scripts/devhost.sh destroy
scripts/devhost.sh address # its AGENT_HOSTS line, any time after
`DEVHOST_REMOTE` is the whole of the remote story: the script copies itself and
its cloud-init seed to that machine and re-execs there under `sudo`, so libvirt,
the image pool and the one-time `apt-get install` of the hypervisor toolchain
all happen where the guest does. **The workstation needs neither root nor
libvirt** — it runs the dev plane as userspace binaries and keeps doing so.
Output streams back and the exit status is the dev machine's. Unset
`DEVHOST_REMOTE` and the same four verbs build the host locally against the
libvirt `default` network.
The guest is a macvtap child of the dev machine's LAN uplink (`DEVHOST_UPLINK`,
default `enp4s0`), so the LAN's router leases it an address and the workstation
dials it directly. macvtap isolates a guest from its own host by design: **the
dev machine cannot reach the guest it runs.** That costs nothing here, because
every gate connection — `deploy.sh` over ssh, the smoke's coverage pull —
comes from the workstation anyway. It does mean the address is discovered
rather than assigned: `create` waits for the guest agent to report it, then
prints the `AGENT_HOSTS` and `AGENT_EXTRA_FLAGS` lines for
`~/eitri-deploy/deploy.env` with the leased LAN address in them.
AGENT_HOSTS="ubuntu@<the address create printed>"
**`make deploy` itself does not change** — only which host it points at. The
address is stable across a recycle as long as the router keeps offering the
same lease to the MAC, which the script derives from the host's name; if it
ever moves, `scripts/devhost.sh address` prints the current one. Discovery
depends on the guest agent the seed installs on first boot, so a guest that is
up but slow to install it leaves `create` waiting; giving the router a
reservation for that MAC and setting `DEVHOST_IP` to the reserved address skips
the wait entirely.
**A fresh host must be enrolled before its first gate** — `deploy.sh` rolls
agents, it never joins them. After `create` (and after the first `make deploy`
has installed the agent binary and unit), mint a token on the local plane and
join:
PAT=$(cat ~/eitri-deploy/oidc/deploy-pat)
JOIN=$(curl -s -X POST http://127.0.0.1:8080/api/v1/enroll-tokens \
-H "Authorization: Bearer $PAT" | python3 -c 'import json,sys; print(json.load(sys.stdin)["join"])')
ssh ubuntu@<address> "sudo eitri-agent --state-dir /var/lib/eitri-agent join '$JOIN' \
&& sudo systemctl reset-failed eitri-agent && sudo systemctl restart eitri-agent"
The `reset-failed` matters: an installed-but-unenrolled agent crash-loops until
systemd's start-rate limit trips, and a plain restart is then refused for a
reason that looks nothing like the cause.
**The smoke assumes the plane's only host is the one it reads.** It places its
throwaway VM on the fleet's first host but reads boot evidence over ssh to
`AGENT_HOSTS` — on a two-host plane those can differ, and the gate fails with
"no userspace boot evidence" while the guest boots happily elsewhere. Keep the
local plane single-host; decommission the old host before gating on the new
one.
**A recycle makes a new machine, and the plane must agree.** The proven flow:
`recycle` (the lease may change even with the same MAC — repoint
`AGENT_HOSTS`), then let the old fleet row drain out (it tombstones on
`DELETE /api/v1/hosts/{id}` and clears itself within the grace once empty —
`?force=true` is the escape hatch for hardware that is really gone), then
`make deploy` to install the agent (its start fails unenrolled — expected),
then the join above, then `make deploy` for the PASS.
After mewtwo moves to stg it carries **both** fleets at once: the stg agent on
bare metal, and this dev VM nested inside it, joined to the local plane. They
share a machine and nothing else. Each is a host in its own right with its own
LAN address, so the 30000-32767 exposure range each allocates from is its own
and two published ports can hold the same number without colliding. Their guest
subnets come from different pools — the local plane allocates from
`10.77.0.0/16`, stg from `10.79.0.0/16` — so neither agent's bridge can swallow
the other's guests.
All four of these must pass before mewtwo is allowed to move:
1. `devhost.sh create` from nothing, then `make deploy` → boot gate **PASS**
including `gate SSH: ok`, `BYO cloud-init merge: ok`, `exposed port: ok`, `remote MCP: ok`. A first pass
on a cold host also proves the agent bootstrap — cloud-hypervisor and
`CLOUDHV.fd` downloaded and sha-verified — works nested.
2. A second `make deploy` on the warm host → PASS. Proves the agent-swap path
and VM re-adoption.
3. `devhost.sh recycle`, then `make deploy` → PASS. Proves the recycle story is
real and not a one-time hand-built machine.
4. `coverage/integration` is written, so the gate's by-product survives the move.
If a nested guest cannot boot at all, everything downstream stops. That is why
this goes first.
## Runbook: moving mewtwo from the local plane to stg
Preconditions: the dev fleet's four proofs are green, and stg is up — console
reachable, sign-in works, `/mcp` refuses an unauthenticated caller, certificate
valid. The local plane stays running throughout, so rollback is always a join
away.
1. **Drain.** List mewtwo's VMs through the local plane and delete every one.
Guests survive an agent swap, not a re-enrollment. Confirm nothing on mewtwo
is precious before starting.
2. **Remove mewtwo from `AGENT_HOSTS` in `~/eitri-deploy/deploy.env`, in the
same sitting.** This is the step most likely to be forgotten and the most
damaging to forget: the next `make deploy` would scp a coverage-instrumented
branch binary over mewtwo's release binary, restart it, and quietly drag it
back toward the dev fleet. Do it before the join, not after.
3. **Decommission from the local plane**: `DELETE /api/v1/hosts/{id}` with the
local operator PAT.
4. **Wipe the plane identity.** On mewtwo: `sudo systemctl stop eitri-agent`,
then remove `/var/lib/eitri-agent/{identity.json,epoch,vms}`. Keep `images/`
— the image cache is plane-agnostic and saves a large download. Removing
`epoch` is what avoids the epoch fence violation described below.
5. **Install the release binary, not a build.** Fetch
`eitri-server_<tag>_linux_amd64.tar.gz` from `https://dl.stg.eitri.sh/dl/<tag>/`,
verify it against `SHA256SUMS`, install. From here mewtwo runs release
artifacts — which is the coverage stg exists to provide.
6. **Join stg.** Mint a join token in the stg console as the stg operator, run
`eitri-agent join <blob>` on mewtwo, `systemctl start eitri-agent`. Confirm
the host appears with the right architecture, uplink and guest subnet.
7. **Prove it**: `scripts/ship.sh --target stg --tag <tag> --from 8`. A full
PASS including gate SSH, exposed port and remote MCP is the acceptance
criterion.
**Rollback**, at any point: re-mint a join token on the local plane, repeat step
4's wipe, re-join, restore `AGENT_HOSTS`. The cost is one more wipe and whatever
guests exist at the time.
## Runbook: the first pre-release run
1. Land the pipeline on the version branch and get `make deploy` green on the
nested dev host — that is the gate for the pipeline code itself.
2. Tag `v0.0.4-pre.1` **on the version branch** and push the tag. Pre-tags are
branch tags; main stays untouched until release.
3. `scripts/ship.sh --target stg --tag v0.0.4-pre.1`. Expect stage 4 to fail the
first time — that is the mechanism working. The fix is a Secret patch plus a
`config.required` classification, both recorded.
4. Migrate mewtwo (above) and re-run stage 9.
5. **The proof this run exists to produce**: drive `vm_create` over
`https://stg.eitri.sh/mcp` — the proxied origin — and observe whether progress
tokens keep Cloudflare from 524-ing a ten-minute call. Record the answer in
`docs/assumptions.md` either way. A negative result is exactly as valuable
and changes what the release can claim.
6. Iterate with `-pre.2`, `-pre.3` as findings land. **The second pre-tag
rehearses the agent upgrade**, which is the failure that defined the v0.0.3
release day and the reason the version ordering understands pre-releases:
after `--target stg --tag v0.0.4-pre.2` has rolled, mewtwo is still on
`-pre.1` and the console must offer it the upgrade. Take it from the console,
confirm the agent comes back reporting `v0.0.4-pre.2` with its guests intact,
and that the artifact it fetched came from `dl.stg.eitri.sh`. An upgrade that
is not offered means the manifest, not the ordering — check
`release_manifest_url` and that the site rolled before the server.
7. When stg is clean, tag the release on main and run
`scripts/ship.sh --target prod --tag v0.0.4` — same script, same stages,
nothing new attempted. The last pre-release upgrades to it like any other
version, because it orders below it.
## Known gaps
**`gate` and `sync` do not inherit the wildcard usefully.** Every `*.eitri.sh`
name resolves to the home origin, where Traefik runs, but the control-plane pod
is `hostNetwork` on the cloud node — so its SSH gate and QUIC sync bind *that*
node's interfaces. The HTTP path works through the Service; the raw TCP and UDP
listeners do not. Both planes want explicit A records for `gate.*` and `sync.*`
pointing at the pod's node, and the matching ports opened there. Verify with
`nc -vz gate.eitri.sh 2222` from outside the LAN before trusting a gate leg.
**Nested cannot prove bridged networking or real-hardware quirks.** That
coverage moved from per-branch to per-pre-tag when mewtwo moved to stg —
deliberate, and the reason stg gates releases on real metal.
**stg's sign-in is not prod's.** stg runs the bundled issuer, so what a stg run
proves is the credential chain end to end against an issuer of ours — and what
it does not touch is prod's relying-party wiring against Google. Everything
after sign-in is identical. See "Sign-in, and the one place the planes genuinely
differ".
## Re-homing a host between planes
A host that previously belonged to a different control plane carries that
plane's snapshot epoch in its state dir and will refuse the new server's
lower-numbered snapshots ("epoch fence violation" in the server log). After
`eitri-agent join` against the new plane:
sudo systemctl stop eitri-agent
sudo rm /var/lib/eitri-agent/epoch
sudo systemctl start eitri-agent
VMs from the old plane are absent from the new plane's desired state and are
reaped through the normal quarantine grace — re-home a host only when its
existing guests are disposable.
## Backups
The nightly CronJob writes dated sqlite backups onto prod's PVC; run
`backup-pull.sh` from cron on an off-cluster machine. local-path storage does
not survive the node, so the off-node copy is the DR story. stg has no CronJob:
its database is disposable by design.
The job runs `$BACKUP_IMAGE` — alpine with sqlite baked in, built by
`make backup-image` (see `backup.Dockerfile`). It carries its OWN tag and is not
rebuilt per release: it holds sqlite and nothing of eitri's. Set `BACKUP_IMAGE`
in the plane's `ship.env` where `BACKUPS=1`; ship refuses to render without it.
It is a separate image because the job used to `apk add sqlite` on every run,
which put a package CDN on the critical path of the only protection a database
nobody can rebuild has. Fetch failures took whole nights of backups with them
and left nothing but a CronJob pod in Error — which is worth knowing about:
nothing alerts on a failed backup, so a run of bad nights is silent until a
restore needs one.
A restore needs the plane's `key_encryption_key` as well as its data. The host
CA on the PVC is sealed under it, so data alone rebuilds a plane that cannot
open it — it would come back without the identity every client pins. Your `~/eitri-deploy/<target>/server.json` is
that key's off-cluster custody: it is not in this repo, not in the backups, and
not recoverable from the cluster if the Secret goes with the node. Keep it, and
keep it somewhere the backups are not.