871f04bd

fix(ship): a release that cannot converge says so before it rolls anything

a73x 2026-08-12 14:54

Commit message

fix(ship): a release that cannot converge says so before it rolls anything

Stage 8 authenticates to the plane, and it runs after stages 6 and 7 have
already rolled it. A machine whose ship.env named no credential — the
credential-less --skip-smoke workflow, previously clean because the only check
lived inside the skipped smoke — reached stage 8 and died there, leaving the
site and the server at the new tag with the fleet a release behind: the exact
half-shipped state the stage exists to end.

The run now establishes what it will need before stage 1 touches anything. The
smoke cannot be driven without a credential, so a missing or unreadable one is
refused in the preflight, with nothing deployed. Convergence is different:
under --skip-smoke, with no credential to act as, stage 8 steps aside and says
so — in a warning that names what the fleet still runs and how to finish it,
and again in stage 10's report. With a credential, --skip-smoke still
converges. Deploy and converge, do not prove.

The other way stage 8 failed a release it should not have was the plane's
"agent is not behind the latest release" refusal, which is returned for the
benign race as well: the host converged between the listing this round was
built from and the POST that followed, because someone clicked Upgrade or an
earlier offer landed. That is now converge-pending — the offer is dropped and
the version the host reports settles it on the next poll, exactly as it does
for every other host. A host that never reports the tag is still caught, by
the timeout, with the refusal quoted; a plane offering some other version is
still caught by name, before any of this; and an unexpected 409 still fails
fast.

deploy/server/README.md

Old		New
@@ -108,6 +108,13 @@ Every stage announces itself and is idempotent; re-running from the top is
108	always safe and is the documented default. `--from <n>` resumes after a partial	108	always safe and is the documented default. `--from <n>` resumes after a partial
109	failure.	109	failure.
110		110
		111	Before stage 1 touches anything, the run checks that it can finish. Stages 8
		112	and 9 both authenticate to the plane, and the credential they use has to be
		113	named in `ship.env` and readable. Both of them come after the plane is rolled,
		114	so a credential that is missing or unreadable is refused here, with nothing
		115	deployed, rather than at stage 8 with the site and the server already at the
		116	new tag.
		117
111	1. Verify the tag — clean tree, HEAD is exactly this tag, and the version is	118	1. Verify the tag — clean tree, HEAD is exactly this tag, and the version is
112	parsable by `internal/server/release`, asked by running that code rather than	119	parsable by `internal/server/release`, asked by running that code rather than
113	re-deriving its rule. An unparsable version silently disables the upgrade	120	re-deriving its rule. An unparsable version silently disables the upgrade
@@ -138,8 +145,17 @@ failure.
138	plane picks, so every connected host behind the release is offered the	145	plane picks, so every connected host behind the release is offered the
139	upgrade here and the stage waits (up to 300s) for all of them to report it.	146	upgrade here and the stage waits (up to 300s) for all of them to report it.
140	A disconnected host is warned about and skipped — it cannot be offered	147	A disconnected host is warned about and skipped — it cannot be offered
141	anything and the smoke cannot place a VM on it either. It runs under	148	anything and the smoke cannot place a VM on it either. A host the plane
142	`--skip-smoke` too: a fleet running the release is part of shipping it.	149	declines to offer as not behind the latest release converged between the
		150	listing and the offer; the stage drops that offer and lets the version the
		151	host reports settle it on the next poll, and the same timeout still catches
		152	one that never reports the tag. It runs under `--skip-smoke` too: a
		153	fleet running the release is part of shipping it. The exception is a machine
		154	whose `ship.env` names no credential at all: convergence is an authenticated
		155	act, so with `--skip-smoke` the stage is skipped with a warning — here and
		156	again in stage 10's report — naming what the fleet still runs and how to
		157	finish it. Without `--skip-smoke` that run never starts (see the preflight
		158	above).
143	9. Hosted smoke: the same `eitri-smoke` the branch gate runs, against this	159	9. Hosted smoke: the same `eitri-smoke` the branch gate runs, against this
144	plane's public names.	160	plane's public names.
145	10. Report what is actually running.	161	10. Report what is actually running.
@@ -299,8 +315,9 @@ end.
299	--from-file=eitri-oidc.json=$HOME/eitri-deploy/stg/eitri-oidc.json	315	--from-file=eitri-oidc.json=$HOME/eitri-deploy/stg/eitri-oidc.json
300		316
301	5. `ship.env`: `cp scripts/ship.env.example ~/eitri-deploy/stg/ship.env`	317	5. `ship.env`: `cp scripts/ship.env.example ~/eitri-deploy/stg/ship.env`
302	and fill it in. For stg that means `CI_USER` and `CI_PASSWORD_FILE`; generate	318	and fill it in — registries and image platform now, the credential in step 8.
303	the password once and lock it down:	319	For stg the credential is `CI_USER` and `CI_PASSWORD_FILE`; generate the
		320	password here and lock it down, since step 7 registers the identity with it:
304		321
305	umask 077; openssl rand -hex 16 > ~/eitri-deploy/stg/ship-password	322	umask 077; openssl rand -hex 16 > ~/eitri-deploy/stg/ship-password
306		323
@@ -308,7 +325,8 @@ end.
308	applies everything else and starts both deployments. The server creates its	325	applies everything else and starts both deployments. The server creates its
309	own SSH CA and gate host key on the PVC at first boot, so the gate comes up	326	own SSH CA and gate host key on the PVC at first boot, so the gate comes up
310	on its own. Skip the smoke on this one run: the identity it signs in as does	327	on its own. Skip the smoke on this one run: the identity it signs in as does
311	not exist yet.	328	not exist yet. With no credential in `ship.env` yet, stage 8 skips itself
		329	and says so — correctly, since no host has joined this plane either.
312		330
313	7. Register the plane's identity, once, in the running issuer. It writes to	331	7. Register the plane's identity, once, in the running issuer. It writes to
314	the issuer's PVC, so it survives every later roll:	332	the issuer's PVC, so it survives every later roll:
@@ -324,8 +342,10 @@ end.
324		342
325	8. Sign in at exactly `https://stg.eitri.sh` — anything else gives "invalid	343	8. Sign in at exactly `https://stg.eitri.sh` — anything else gives "invalid
326	oauth state" — which JIT-provisions the operator tenant that will own the	344	oauth state" — which JIT-provisions the operator tenant that will own the
327	plane's hosts. Then `scripts/ship.sh --target stg --tag <pre-tag> --from 8`	345	plane's hosts. Now that the identity exists, put `CI_USER` and
328	proves the whole thing.	346	`CI_PASSWORD_FILE` in `ship.env`, and
		347	`scripts/ship.sh --target stg --tag <pre-tag> --from 8` proves the whole
		348	thing.
329		349
330	There is no PAT to mint by hand here. The smoke signs in as that same	350	There is no PAT to mint by hand here. The smoke signs in as that same
331	identity and mints its own short-lived token for the run, deriving the tenant	351	identity and mints its own short-lived token for the run, deriving the tenant

scripts/ship.env.example

Old		New
@@ -21,7 +21,14 @@ SITE_PLATFORM="linux/arm64"
21	# clears the rest, so the plane decides and the pipeline stays the same script.	21	# clears the rest, so the plane decides and the pipeline stays the same script.
22	# The same credential converges the fleet's agents one stage earlier, so the	22	# The same credential converges the fleet's agents one stage earlier, so the
23	# identity named here must be the one whose tenant owns the fleet's hosts — a	23	# identity named here must be the one whose tenant owns the fleet's hosts — a
24	# credential from another tenant sees no hosts, and stage 8 says so.	24	# credential from another tenant sees no hosts, and stage 8 says so. ship.sh
		25	# checks that whichever of these files you name is readable before it deploys
		26	# anything, because both the stages that use it run after the plane is rolled.
		27	#
		28	# Naming neither is a real arrangement, not a broken one: with --skip-smoke the
		29	# run rolls the plane and skips stage 8 with a warning, leaving the fleet on the
		30	# previous release for whoever holds the credential to converge. Without
		31	# --skip-smoke it is refused before stage 1.
25	#	32	#
26	# (a) A plane running the bundled eitri-oidc — stg. The smoke signs in through	33	# (a) A plane running the bundled eitri-oidc — stg. The smoke signs in through
27	# the real code flow, which proves the whole credential chain, and mints its	34	# the real code flow, which proves the whole credential chain, and mints its

scripts/ship.sh

Old		New
@@ -9,7 +9,9 @@
9	#	9	#
10	# --from <n> resume at stage n after a partial failure	10	# --from <n> resume at stage n after a partial failure
11	# --skip-smoke deploy and converge the fleet, but do not prove it	11	# --skip-smoke deploy and converge the fleet, but do not prove it
12	# (stage 10 still reports)	12	# (stage 10 still reports). On a machine whose ship.env
		13	# names no credential, stage 8 is skipped with a warning
		14	# and the fleet stays on the release before this one.
13	# --render-only print the rendered manifests and exit; touches nothing	15	# --render-only print the rendered manifests and exit; touches nothing
14	#	16	#
15	# Every stage is idempotent: re-running from the top is always safe and is the	17	# Every stage is idempotent: re-running from the top is always safe and is the
@@ -40,7 +42,7 @@ SKIP_SMOKE=0
40	RENDER_ONLY=0	42	RENDER_ONLY=0
41		43
42	usage() {	44	usage() {
43	sed -n '2,33p' "$0" \| sed 's/^#\{1,2\} \{0,1\}//'	45	sed -n '2,35p' "$0" \| sed 's/^#\{1,2\} \{0,1\}//'
44	exit "${1:-1}"	46	exit "${1:-1}"
45	}	47	}
46		48
@@ -171,6 +173,59 @@ echo "console https://$CONSOLE_HOST api https://$API_HOST site https://$SITE_H
171	echo "gate $GATE_HOST:$GATE_PORT sync $SYNC_HOST:$SYNC_PORT http :$HTTP_PORT"	173	echo "gate $GATE_HOST:$GATE_PORT sync $SYNC_HOST:$SYNC_PORT http :$HTTP_PORT"
172	[[ "$FROM" -gt 1 ]] && echo "(resuming at stage $FROM)"	174	[[ "$FROM" -gt 1 ]] && echo "(resuming at stage $FROM)"
173		175
		176	# ── Preflight: what the run will need, before it changes anything ─────────────
		177	#
		178	# Stages 8 and 9 are the two that authenticate to the plane, and both of them
		179	# run AFTER stages 6 and 7 have rolled it. A run that cannot finish has to say
		180	# so while the plane is still serving what it served this morning: the state to
		181	# avoid is a site and a server at $TAG with the fleet a release behind, which is
		182	# the half-shipped state stage 8 exists to end.
		183	#
		184	# What is required depends on which stages this run will actually reach. The
		185	# smoke cannot be driven without a credential at all, so a missing one is fatal
		186	# here. Convergence is not: rolling a plane from a machine that holds no
		187	# credential, with the proof left to someone who does, is a workflow this
		188	# script has always allowed — and stage 8 skips itself and says loudly what the
		189	# fleet is still running, rather than dying with the plane already rolled.
		190	CONVERGE_WILL_RUN=0
		191	SMOKE_WILL_RUN=0
		192	CONVERGE_SKIPPED=0
		193	[[ "$FROM" -le 8 ]] && CONVERGE_WILL_RUN=1
		194	[[ "$FROM" -le 9 && "$SKIP_SMOKE" != "1" ]] && SMOKE_WILL_RUN=1
		195
		196	HAVE_CREDENTIAL=0
		197	[[ -n "${CI_USER:-}" \|\| -n "${CI_PAT_FILE:-}" ]] && HAVE_CREDENTIAL=1
		198
		199	if [[ "$SMOKE_WILL_RUN" == "1" && "$HAVE_CREDENTIAL" == "0" ]]; then
		200	fail "$SHIP_ENV names no credential, so nothing in this run could authenticate
		201	to the plane — the smoke at stage 9 needs one, as does the convergence at
		202	stage 8 before it. Nothing has been deployed.
		203	Set CI_USER + CI_PASSWORD_FILE for a plane with a password issuer, or
		204	CI_PAT_FILE for one fronted by an external identity provider — or pass
		205	--skip-smoke to roll this plane from a machine that holds neither, in
		206	which case stage 8 is skipped and the fleet stays where it is."
		207	fi
		208
		209	# A credential that is named but unreachable fails for the same reason and at
		210	# the same cost, so it is established here too. These are file tests: the
		211	# pipeline never reads a credential it does not have to.
		212	if [[ "$CONVERGE_WILL_RUN" == "1" \|\| "$SMOKE_WILL_RUN" == "1" ]]; then
		213	if [[ -n "${CI_PAT_FILE:-}" ]]; then
		214	[[ -r "$CI_PAT_FILE" ]] \|\|
		215	fail "CI_PAT_FILE names $CI_PAT_FILE, which cannot be read. Nothing has been deployed."
		216	fi
		217	# CI_USER's password is what the smoke signs in with whenever CI_USER is
		218	# named, and what stage 8 signs in with only when there is no PAT for it to
		219	# prefer. Required exactly where it is used.
		220	if [[ -n "${CI_USER:-}" ]] && [[ "$SMOKE_WILL_RUN" == "1" \|\| -z "${CI_PAT_FILE:-}" ]]; then
		221	[[ -n "${CI_PASSWORD_FILE:-}" ]] \|\|
		222	fail "$SHIP_ENV sets CI_USER=$CI_USER but no CI_PASSWORD_FILE, so nothing can
		223	sign in as that identity. Nothing has been deployed."
		224	[[ -r "$CI_PASSWORD_FILE" ]] \|\|
		225	fail "CI_PASSWORD_FILE names $CI_PASSWORD_FILE, which cannot be read. Nothing has been deployed."
		226	fi
		227	fi
		228
174	# ── 1. Verify the tag ─────────────────────────────────────────────────────────	229	# ── 1. Verify the tag ─────────────────────────────────────────────────────────
175	if [[ "$FROM" -le 1 ]]; then	230	if [[ "$FROM" -le 1 ]]; then
176	bold "1. Verify the tag"	231	bold "1. Verify the tag"
@@ -546,8 +601,18 @@ fi
546	#	601	#
547	# The stage runs even under --skip-smoke. A fleet running the release is part of	602	# The stage runs even under --skip-smoke. A fleet running the release is part of
548	# shipping it, not part of proving it — a plane at $TAG with agents a release	603	# shipping it, not part of proving it — a plane at $TAG with agents a release
549	# behind is exactly the half-shipped state this stage exists to end.	604	# behind is exactly the half-shipped state this stage exists to end. Deploy and
550	if [[ "$FROM" -le 8 ]]; then	605	# converge, do not prove: that is what --skip-smoke means here.
		606	#
		607	# The one thing that stops it is having nothing to act as. Convergence is an
		608	# authenticated act, and rolling a plane from a machine that holds no credential
		609	# is a workflow this script allowed before this stage existed. It still does —
		610	# the stage steps aside and says, in the warning and again in stage 10's report,
		611	# exactly what the fleet is still running and how to finish it. What it will not
		612	# do is take the plane to $TAG and then fail for want of a token, which would
		613	# leave behind the very state described above. The preflight has already refused
		614	# the run outright if the smoke was going to need that credential.
		615	if [[ "$FROM" -le 8 && "$HAVE_CREDENTIAL" == "1" ]]; then
551	bold "8. Converge the fleet's agents to $TAG"	616	bold "8. Converge the fleet's agents to $TAG"
552		617
553	# How long a host may take to land the new agent before the ship gives up on	618	# How long a host may take to land the new agent before the ship gives up on
@@ -573,12 +638,7 @@ if [[ "$FROM" -le 8 ]]; then
573	# identity that must offer them an upgrade and the one the smoke will place a	638	# identity that must offer them an upgrade and the one the smoke will place a
574	# VM as. A stage with a credential of its own would be a third token to	639	# VM as. A stage with a credential of its own would be a third token to
575	# issue, rotate and lose.	640	# issue, rotate and lose.
576	if [[ -z "${CI_USER:-}" && -z "${CI_PAT_FILE:-}" ]]; then	641	#
577	fail "$SHIP_ENV names no credential, so the fleet cannot be converged.
578	Set CI_USER + CI_PASSWORD_FILE for a plane with a password issuer, or
579	CI_PAT_FILE for one fronted by an external identity provider."
580	fi
581
582	# Everything the credential touches is a file in one 0700 directory that goes	642	# Everything the credential touches is a file in one 0700 directory that goes
583	# away with the run: the bearer header and the cookie jar are files rather	643	# away with the run: the bearer header and the cookie jar are files rather
584	# than arguments, because an argument is visible in ps to every user on this	644	# than arguments, because an argument is visible in ps to every user on this
@@ -640,6 +700,7 @@ if [[ "$FROM" -le 8 ]]; then
640	offered=" " # hosts this stage has offered the upgrade to	700	offered=" " # hosts this stage has offered the upgrade to
641	announced=" " # hosts already reported as converged	701	announced=" " # hosts already reported as converged
642	skipped=" " # disconnected hosts, warned about once	702	skipped=" " # disconnected hosts, warned about once
		703	notbehind=" " # hosts the plane called not-behind, noted once
643	refusal="" # the last thing the plane refused, for the timeout message	704	refusal="" # the last thing the plane refused, for the timeout message
644	converged=0	705	converged=0
645	while :; do	706	while :; do
@@ -748,14 +809,38 @@ if [[ "$FROM" -le 8 ]]; then
748	waiting="$waiting $name (plane has no release manifest yet)"	809	waiting="$waiting $name (plane has no release manifest yet)"
749	;;	810	;;
750	409)	811	409)
751	# internal/server/api/upgrade.go refuses three ways. "host is	812	# internal/server/api/upgrade.go refuses three ways, and two of
752	# offline" is a race against the listing above and resolves itself	813	# them are races against the listing this round was built from —
753	# — the next round sees the host dark and skips it. The other two	814	# the state changed between the GET and this POST.
754	# (not behind the release, no artifact for this os/arch) need a	815	#
755	# person, and waiting out the timeout would only delay telling them.	816	# "host is offline": the next round sees the host dark and skips
		817	# it.
		818	#
		819	# "not behind the latest release": the host converged in that same
		820	# gap. An operator clicked Upgrade in the console, or an offer from
		821	# an earlier round landed and the agent re-exec'd while this round
		822	# was still walking the list. There is nothing to offer a host that
		823	# is already there, and nothing wrong either — so the offer is
		824	# dropped and the version the host REPORTS decides it, on the next
		825	# round, exactly as it decides it for every other host. If it never
		826	# reports $TAG, the timeout below says so.
		827	#
		828	# That leaves the third refusal, no artifact for this os/arch, for
		829	# which waiting out the timeout would only delay telling a person.
		830	# Nothing here weakens the stale-manifest catch: a plane offering
		831	# some OTHER version is caught by name, above, before any of this.
756	if [[ "$api_body" == "host is offline" ]]; then	832	if [[ "$api_body" == "host is offline" ]]; then
757	refusal="$name: $api_body"	833	refusal="$name: $api_body"
758	waiting="$waiting $name (went offline mid-offer)"	834	waiting="$waiting $name (went offline mid-offer)"
		835	elif [[ "$api_body" == "not behind the latest release" ]]; then
		836	refusal="$name: $api_body"
		837	case "$notbehind" in
		838	" $id ") ;;
		839	*)
		840	echo " $name: not behind the release the plane names — reading back the version it reports"
		841	notbehind="$notbehind$id " ;;
		842	esac
		843	waiting="$waiting $name (offer declined as not-behind; awaiting its reported version)"
759	else	844	else
760	fail "the plane refuses to offer $TAG to host $name ($id): $api_body"	845	fail "the plane refuses to offer $TAG to host $name ($id): $api_body"
761	fi	846	fi
@@ -795,6 +880,19 @@ if [[ "$FROM" -le 8 ]]; then
795	if [[ "$converged" == "0" && "$skipped_n" != "0" ]]; then	880	if [[ "$converged" == "0" && "$skipped_n" != "0" ]]; then
796	warn "no connected host is running $TAG, so the smoke has nowhere to place a VM."	881	warn "no connected host is running $TAG, so the smoke has nowhere to place a VM."
797	fi	882	fi
		883	elif [[ "$FROM" -le 8 ]]; then
		884	bold "8. Converge the fleet's agents SKIPPED (no credential in ship.env)"
		885	CONVERGE_SKIPPED=1
		886	warn "the fleet was NOT converged: its agents still run the release before $TAG,
		887	while this plane now serves $TAG. Nothing offers an agent an upgrade on
		888	its own, so they will stay there until someone does.
		889	Finish it either way:
		890	- the console's fleet page, host by host, from any browser signed in as
		891	the tenant that owns them; or
		892	- this same run from a machine that holds the credential:
		893	scripts/ship.sh --target $TARGET --tag $TAG --from 8
		894	$SHIP_ENV names neither CI_PAT_FILE nor CI_USER, which is why this stage
		895	had nothing to act as."
798	fi	896	fi
799		897
800	# ── 9. Hosted smoke ───────────────────────────────────────────────────────────	898	# ── 9. Hosted smoke ───────────────────────────────────────────────────────────
@@ -866,6 +964,11 @@ echo "mcp https://$CONSOLE_HOST/mcp (proxied) and https://$API_HOST/mcp"
866	echo "downloads https://$SITE_HOST/dl/$TAG/"	964	echo "downloads https://$SITE_HOST/dl/$TAG/"
867	# An if, not a trailing &&: as the script's last command, a false conditional	965	# An if, not a trailing &&: as the script's last command, a false conditional
868	# would be the whole run's exit status.	966	# would be the whole run's exit status.
		967	if [[ "$CONVERGE_SKIPPED" == "1" ]]; then
		968	echo "fleet: NOT CONVERGED — the agents still run the release before $TAG;"
		969	echo " converge them from the console, or re-run with --from 8 from a"
		970	echo " machine whose ship.env names a credential"
		971	fi
869	if [[ "$SKIP_SMOKE" == "1" ]]; then	972	if [[ "$SKIP_SMOKE" == "1" ]]; then
870	echo "smoke: SKIPPED — this plane is deployed, not proven"	973	echo "smoke: SKIPPED — this plane is deployed, not proven"
871	fi	974	fi