82b50b87

feat(devhost): the branch gate's fleet runs on the dev machine

a73x 2026-08-08 14:02

Commit message

feat(devhost): the branch gate's fleet runs on the dev machine

The workstation drives the branch gate's fleet without hosting it: it runs the
dev plane as userspace binaries and holds no root of its own, while a VM needs
libvirt and root. DEVHOST_REMOTE names the machine that has both. The script is
its own payload — it and the cloud-init seed go to a mktemp dir there and
re-exec under sudo, so libvirt, the image pool and the one-time install of the
hypervisor toolchain all happen where the guest does. Output streams back and
the exit status is the dev machine's. The four verbs read the same either way,
and with DEVHOST_REMOTE unset they build the host on this machine against the
libvirt 'default' network.

The guest is a macvtap child of the dev machine's LAN uplink, so the LAN's
router leases it an address and the workstation dials it directly rather than
through a NAT the dev machine owns. macvtap isolates a guest from its own host,
which costs nothing here — every gate connection comes from the workstation
anyway — but it does mean the address is discovered rather than assigned: the
guest agent reports it over its virtio channel, the script waits for it, and
the workstation makes the ssh proof against the address it read back.

Nested KVM boots and gates a full stack, proven on eitri-dev1: all four proofs
in deploy/server/README.md — cold deploy with the agent bootstrap, warm
re-deploy, deploy after a recycle, coverage merged — passed on the nested host,
which is what lets the branch gate leave real hardware for stg to own.

deploy/server/README.md

Old		New
@@ -221,6 +221,7 @@ end.
221	"advertise_quic": "sync.stg.eitri.sh:8444",	221	"advertise_quic": "sync.stg.eitri.sh:8444",
222	"cidr_pool": "10.79.0.0/16",	222	"cidr_pool": "10.79.0.0/16",
223	"host_secret": "<openssl rand -hex 32>",	223	"host_secret": "<openssl rand -hex 32>",
		224	"key_encryption_key": "<openssl rand -hex 32>",
224	"release_manifest_url": "https://dl.stg.eitri.sh/dl/latest/manifest.json",	225	"release_manifest_url": "https://dl.stg.eitri.sh/dl/latest/manifest.json",
225	"default_images": {	226	"default_images": {
226	"amd64": {"url": "...", "sha256": "..."},	227	"amd64": {"url": "...", "sha256": "..."},
@@ -239,6 +240,13 @@ end.
239	server pins the token request's client-auth style accordingly. prod's Google	240	server pins the token request's client-auth style accordingly. prod's Google
240	client is confidential and does set one.	241	client is confidential and does set one.
241		242
		243	`key_encryption_key` encrypts everything the plane signs with: the host CA
		244	and gate host key on the PVC. The volume and its backups therefore hold
		245	ciphertext, and this Secret is the key's only home. Each plane mints its own and keeps
		246	it—it is never rotated in place, and a plane that loses it loses its host CA,
		247	which is the identity every client pins and every VM's host certificate
		248	names.
		249
242	`allowed_identities` closes the signup gate. Both architectures get a	250	`allowed_identities` closes the signup gate. Both architectures get a
243	`default_images` entry: a host handed an image it cannot execute fails at	251	`default_images` entry: a host handed an image it cannot execute fails at
244	boot, which is the mistake that map exists to prevent — and stage 4 enforces	252	boot, which is the mistake that map exists to prevent — and stage 4 enforces
@@ -311,17 +319,85 @@ manifests do not describe — an unused `web-data` PVC, say — is left alone.
311		319
312	## Runbook: the nested dev fleet	320	## Runbook: the nested dev fleet
313		321
314	The branch gate's fleet lives on the workstation, defined by	322	The branch gate's fleet lives on the dev machine, defined by
315	`scripts/devhost.sh` rather than hand-built. Bring it up before mewtwo	323	`scripts/devhost.sh` rather than hand-built, and driven from the workstation.
316	moves to stg — until it is green, the project has no branch gate.	324	Bring it up before mewtwo moves to stg — until it is green, the project has
		325	no branch gate.
317		326
		327	export DEVHOST_REMOTE=ubuntu@192.168.0.193:2222 # the dev machine, mewtwo
318	scripts/devhost.sh create # from nothing	328	scripts/devhost.sh create # from nothing
319	scripts/devhost.sh recycle # destroy + create; a wedged host is 3 minutes	329	scripts/devhost.sh recycle # destroy + create; a wedged host is 3 minutes
320	scripts/devhost.sh destroy	330	scripts/devhost.sh destroy
321		331	scripts/devhost.sh address # its AGENT_HOSTS line, any time after
322	`create` prints the `AGENT_HOSTS` and `AGENT_EXTRA_FLAGS` lines for	332
323	`~/eitri-deploy/deploy.env`. `make deploy` itself does not change — only	333	`DEVHOST_REMOTE` is the whole of the remote story: the script copies itself and
324	which host it points at.	334	its cloud-init seed to that machine and re-execs there under `sudo`, so libvirt,
		335	the image pool and the one-time `apt-get install` of the hypervisor toolchain
		336	all happen where the guest does. **The workstation needs neither root nor
		337	libvirt** — it runs the dev plane as userspace binaries and keeps doing so.
		338	Output streams back and the exit status is the dev machine's. Unset
		339	`DEVHOST_REMOTE` and the same four verbs build the host locally against the
		340	libvirt `default` network.
		341
		342	The guest is a macvtap child of the dev machine's LAN uplink (`DEVHOST_UPLINK`,
		343	default `enp4s0`), so the LAN's router leases it an address and the workstation
		344	dials it directly. macvtap isolates a guest from its own host by design: **the
		345	dev machine cannot reach the guest it runs.** That costs nothing here, because
		346	every gate connection — `deploy.sh` over ssh, the smoke's serial-log reads —
		347	comes from the workstation anyway. It does mean the address is discovered
		348	rather than assigned: `create` waits for the guest agent to report it, then
		349	prints the `AGENT_HOSTS` and `AGENT_EXTRA_FLAGS` lines for
		350	`~/eitri-deploy/deploy.env` with the leased LAN address in them.
		351
		352	AGENT_HOSTS="ubuntu@<the address create printed>"
		353
		354	`make deploy` itself does not change — only which host it points at. The
		355	address is stable across a recycle as long as the router keeps offering the
		356	same lease to the MAC, which the script derives from the host's name; if it
		357	ever moves, `scripts/devhost.sh address` prints the current one. Discovery
		358	depends on the guest agent the seed installs on first boot, so a guest that is
		359	up but slow to install it leaves `create` waiting; giving the router a
		360	reservation for that MAC and setting `DEVHOST_IP` to the reserved address skips
		361	the wait entirely.
		362
		363	A fresh host must be enrolled before its first gate — `deploy.sh` rolls
		364	agents, it never joins them. After `create` (and after the first `make deploy`
		365	has installed the agent binary and unit), mint a token on the local plane and
		366	join:
		367
		368	PAT=$(cat ~/eitri-deploy/oidc/deploy-pat)
		369	JOIN=$(curl -s -X POST http://127.0.0.1:8080/api/v1/enroll-tokens \
		370	-H "Authorization: Bearer $PAT" \| python3 -c 'import json,sys; print(json.load(sys.stdin)["join"])')
		371	ssh ubuntu@<address> "sudo eitri-agent --state-dir /var/lib/eitri-agent join '$JOIN' \
		372	&& sudo systemctl reset-failed eitri-agent && sudo systemctl restart eitri-agent"
		373
		374	The `reset-failed` matters: an installed-but-unenrolled agent crash-loops until
		375	systemd's start-rate limit trips, and a plain restart is then refused for a
		376	reason that looks nothing like the cause.
		377
		378	The smoke assumes the plane's only host is the one it reads. It places its
		379	throwaway VM on the fleet's first host but reads boot evidence over ssh to
		380	`AGENT_HOSTS` — on a two-host plane those can differ, and the gate fails with
		381	"no userspace boot evidence" while the guest boots happily elsewhere. Keep the
		382	local plane single-host; decommission the old host before gating on the new
		383	one.
		384
		385	A recycle makes a new machine, and the plane must agree. The proven flow:
		386	`recycle` (the lease may change even with the same MAC — repoint
		387	`AGENT_HOSTS`), then let the old fleet row drain out (it tombstones on
		388	`DELETE /api/v1/hosts/{id}` and clears itself within the grace once empty —
		389	`?force=true` is the escape hatch for hardware that is really gone), then
		390	`make deploy` to install the agent (its start fails unenrolled — expected),
		391	then the join above, then `make deploy` for the PASS.
		392
		393	After mewtwo moves to stg it carries both fleets at once: the stg agent on
		394	bare metal, and this dev VM nested inside it, joined to the local plane. They
		395	share a machine and nothing else. Each is a host in its own right with its own
		396	LAN address, so the 30000-32767 exposure range each allocates from is its own
		397	and two published ports can hold the same number without colliding. Their guest
		398	subnets come from different pools — the local plane allocates from
		399	`10.77.0.0/16`, stg from `10.79.0.0/16` — so neither agent's bridge can swallow
		400	the other's guests.
325		401
326	All four of these must pass before mewtwo is allowed to move:	402	All four of these must pass before mewtwo is allowed to move:
327		403
@@ -452,3 +528,10 @@ The nightly CronJob writes dated sqlite backups onto prod's PVC; run
452	`backup-pull.sh` from cron on an off-cluster machine. local-path storage does	528	`backup-pull.sh` from cron on an off-cluster machine. local-path storage does
453	not survive the node, so the off-node copy is the DR story. stg has no CronJob:	529	not survive the node, so the off-node copy is the DR story. stg has no CronJob:
454	its database is disposable by design.	530	its database is disposable by design.
		531
		532	A restore needs the plane's `key_encryption_key` as well as its data. The host
		533	CA on the PVC is sealed under it, so data alone rebuilds a plane that cannot
		534	open it — it would come back without the identity every client pins. Your `~/eitri-deploy/<target>/server.json` is
		535	that key's off-cluster custody: it is not in this repo, not in the backups, and
		536	not recoverable from the cluster if the Secret goes with the node. Keep it, and
		537	keep it somewhere the backups are not.

docs/assumptions.md

Old		New
@@ -444,9 +444,10 @@ tag—stays unordered and is never offered an upgrade.
444	A guest of a workstation VM boots under cloud-hypervisor and exercises the same	444	A guest of a workstation VM boots under cloud-hypervisor and exercises the same
445	agent, bootstrap, gate and exposure paths bare metal does. Underpins moving the	445	agent, bootstrap, gate and exposure paths bare metal does. Underpins moving the
446	branch gate off real hardware so a hosted outage cannot fail it.	446	branch gate off real hardware so a hosted outage cannot fail it.
447	Unproven: the four proofs in deploy/server/README.md—cold deploy, warm	447	Proven 2026-08-07: all four proofs in deploy/server/README.md passed on
448	deploy, deploy after a recycle, coverage merged—have not been run. Bridged	448	eitri-dev1 (nested on the dev machine over macvtap) — cold deploy with the
449	networking and real-hardware quirks are known to be out of its reach, which is	449	agent bootstrap, warm re-deploy, deploy after a recycle, coverage merged.
		450	Bridged networking and real-hardware quirks remain out of its reach, which is
450	why real metal gates pre-release tags instead.	451	why real metal gates pre-release tags instead.
451		452
452	### A proxied MCP origin carries long calls	453	### A proxied MCP origin carries long calls

scripts/devhost.sh

Old		New
@@ -1,23 +1,39 @@
1	#!/usr/bin/env bash	1	#!/usr/bin/env bash
2	#	2	#
3	# The branch gate's fleet: a nested KVM host on this workstation, defined here	3	# The branch gate's fleet: a nested KVM host, defined here rather than
4	# rather than hand-built, so a wedged dev host is a three-minute rebuild instead	4	# hand-built, so a wedged dev host is a three-minute rebuild instead of a
5	# of a debugging session.	5	# debugging session.
6	#	6	#
7	# scripts/devhost.sh create [name] build it from nothing	7	# scripts/devhost.sh create [name] build it from nothing
8	# scripts/devhost.sh recycle [name] destroy, then create — the whole point	8	# scripts/devhost.sh recycle [name] destroy, then create — the whole point
9	# scripts/devhost.sh destroy [name]	9	# scripts/devhost.sh destroy [name]
10	# scripts/devhost.sh address [name] print its AGENT_HOSTS line	10	# scripts/devhost.sh address [name] print its AGENT_HOSTS line
11	#	11	#
12	# Workstation-local on purpose. The branch gate is the most-run check in the	12	# The host runs on the dev machine rather than the workstation:
13	# project, and a guest on a remote box would make it fail whenever something
14	# hosted broke — inverting what the gate is for. This host has no network
15	# coupling to any fleet and dies the moment `make deploy` is not running.
16	#	13	#
17	# Nested guests get hardware virtualization through --cpu host-passthrough, so	14	# DEVHOST_REMOTE=ubuntu@192.168.0.193:2222 scripts/devhost.sh create
18	# the host CPU must expose nested KVM (kvm_amd nested=1 / kvm_intel nested=1).	15	#
19	# What nested CANNOT prove is bridged networking and real-hardware quirks —	16	# With DEVHOST_REMOTE set the script copies itself and the seed there and
20	# that coverage moved to stg, which gates pre-release tags on real metal.	17	# re-execs under sudo, so every hypervisor call — libvirt, qemu-img, apt —
		18	# happens on the machine that owns the guest, and the workstation stays free of
		19	# both root and libvirt. Output streams back; the exit status is the remote's.
		20	# Unset, the same four verbs build the host on this machine against the libvirt
		21	# 'default' network.
		22	#
		23	# The gate's fleet has no coupling to any hosted thing and dies the moment
		24	# `make deploy` is not running. Nested guests get hardware virtualization
		25	# through --cpu host-passthrough, so the machine's CPU must expose nested KVM
		26	# (kvm_amd nested=1 / kvm_intel nested=1). What nested CANNOT prove is bridged
		27	# networking and real-hardware quirks — that coverage moved to stg, which gates
		28	# pre-release tags on real metal.
		29	#
		30	# On the dev machine the guest is a macvtap child of the LAN uplink
		31	# (DEVHOST_UPLINK, default enp4s0): the LAN's router leases it an address and
		32	# the workstation dials it directly. macvtap isolates a guest from its own
		33	# host by design, so the dev machine cannot reach the guest it runs — which
		34	# costs nothing here, because every gate connection comes from the workstation
		35	# anyway. It does mean the address is discovered after boot rather than
		36	# assigned: the guest agent reports it, and this script waits for it.
21	#	37	#
22	# The seed installs nothing eitri needs: cloud-hypervisor, the guest firmware	38	# The seed installs nothing eitri needs: cloud-hypervisor, the guest firmware
23	# and the agent all arrive through `make deploy` and the agent's own bootstrap,	39	# and the agent all arrive through `make deploy` and the agent's own bootstrap,
@@ -27,7 +43,12 @@
27	# rm calls below use sudo for that and nothing else.	43	# rm calls below use sudo for that and nothing else.
28	set -euo pipefail	44	set -euo pipefail
29		45
		46	# The pool dir and the 'default' network live on the SYSTEM daemon; a bare
		47	# virsh from a user shell talks to qemu:///session and sees neither.
		48	export LIBVIRT_DEFAULT_URI="${LIBVIRT_DEFAULT_URI:-qemu:///system}"
		49
30	REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"	50	REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
		51	SCRIPT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
31		52
32	# Pinned guest image. Bump deliberately: take the sha from the SHA256SUMS file	53	# Pinned guest image. Bump deliberately: take the sha from the SHA256SUMS file
33	# beside the image in the same dated directory.	54	# beside the image in the same dated directory.
@@ -48,6 +69,26 @@ POOL_DIR="${DEVHOST_POOL_DIR:-/var/lib/libvirt/images}"
48	CACHE_DIR="${DEVHOST_CACHE_DIR:-$HOME/.cache/eitri/devhost}"	69	CACHE_DIR="${DEVHOST_CACHE_DIR:-$HOME/.cache/eitri/devhost}"
49	SSH_KEY="${DEVHOST_SSH_KEY:-$HOME/.ssh/id_ed25519.pub}"	70	SSH_KEY="${DEVHOST_SSH_KEY:-$HOME/.ssh/id_ed25519.pub}"
50	SSH_PORT="${DEVHOST_SSH_PORT:-22}"	71	SSH_PORT="${DEVHOST_SSH_PORT:-22}"
		72	SEED_TEMPLATE="${DEVHOST_SEED:-$REPO_ROOT/scripts/devhost.cloud-init.yaml}"
		73
		74	# Where the host is built. REMOTE empty means this machine. MANAGED is set only
		75	# on the copy that lands on the dev machine, and says two things: this machine
		76	# is ours to prepare, and something upstream is waiting to print the result.
		77	REMOTE="${DEVHOST_REMOTE:-}"
		78	MANAGED="${DEVHOST_MANAGED:-}"
		79
		80	# A non-empty uplink switches the guest onto macvtap over that interface; empty
		81	# keeps it on the libvirt network. Driving a remote implies the former, because
		82	# a dev machine is a LAN citizen and its guest should be one too.
		83	UPLINK="${DEVHOST_UPLINK:-}"
		84	if [[ -n "$REMOTE" && -z "$UPLINK" ]]; then
		85	UPLINK=enp4s0
		86	fi
		87
		88	# The key the guest must admit is the workstation's, wherever the guest is
		89	# built: the driving side reads the file and hands the material over, so the
		90	# dev machine never needs a key of its own.
		91	SSH_PUBKEY="${DEVHOST_SSH_PUBKEY:-}"
51		92
52	bold() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; }	93	bold() { printf '\n\033[1;36m==> %s\033[0m\n' "$*"; }
53	die() { printf 'devhost: %s\n' "$*" >&2; exit 1; }	94	die() { printf 'devhost: %s\n' "$*" >&2; exit 1; }
@@ -58,44 +99,198 @@ NAME="${2:-eitri-dev1}"
58	case "$VERB" in	99	case "$VERB" in
59	create \| recycle \| destroy \| address) ;;	100	create \| recycle \| destroy \| address) ;;
60	*)	101	*)
61	sed -n '2,20p' "$0" \| sed 's/^#\{1,2\} \{0,1\}//' >&2	102	sed -n '2,21p' "$0" \| sed 's/^#\{1,2\} \{0,1\}//' >&2
62	exit 1	103	exit 1
63	;;	104	;;
64	esac	105	esac
65		106
66	# Address and MAC are derived from the name, so a recycled host comes back at	107	RTARGET=""
67	# the same address and AGENT_HOSTS never has to be edited. The index is the	108	RPORT=22
68	# trailing digits of the name ("eitri-dev2" -> 2 -> .51).	109	if [[ -n "$REMOTE" ]]; then
69	index="$(printf '%s' "$NAME" \| grep -oE '[0-9]+$' \|\| echo 1)"	110	RTARGET="${REMOTE%:*}"
70	IP="${DEVHOST_IP:-192.168.122.$((49 + index))}"	111	if [[ "$REMOTE" == : ]]; then
71	# Locally-administered QEMU prefix plus three bytes of the name's digest: stable	112	RPORT="${REMOTE##*:}"
72	# per name, and distinct enough not to collide with anything else on the bridge.	113	fi
		114	[[ "$RTARGET" == @ ]] \|\| die "DEVHOST_REMOTE must read user@host[:port] (got '$REMOTE')"
		115	fi
		116
		117	# Where to look when a host will not answer. Named early because half the
		118	# failure paths below want to say it.
		119	CONSOLE_HINT="virsh console $NAME"
		120	if [[ -n "$REMOTE" ]]; then
		121	CONSOLE_HINT="ssh -p $RPORT $RTARGET sudo virsh console $NAME"
		122	fi
		123
		124	# The MAC is derived from the name, so a recycled host comes back the same
		125	# machine to the router and to libvirt. Locally-administered QEMU prefix plus
		126	# three bytes of the name's digest: stable per name, and distinct enough not to
		127	# collide with anything else on the segment.
73	mac_tail="$(printf '%s' "$NAME" \| sha256sum \| cut -c1-6 \| sed 's/../&:/g; s/:$//')"	128	mac_tail="$(printf '%s' "$NAME" \| sha256sum \| cut -c1-6 \| sed 's/../&:/g; s/:$//')"
74	MAC="${DEVHOST_MAC:-52:54:00:$mac_tail}"	129	MAC="${DEVHOST_MAC:-52:54:00:$mac_tail}"
75	DISK="$POOL_DIR/$NAME.qcow2"	130	DISK="$POOL_DIR/$NAME.qcow2"
76		131
		132	# On the libvirt network the address comes from the name too, via a reservation
		133	# this script owns, so AGENT_HOSTS never has to be edited. The index is the
		134	# trailing digits of the name ("eitri-dev2" -> 2 -> .51). On the LAN uplink the
		135	# router owns DHCP, so the address is whatever it leased and is discovered.
		136	IP="${DEVHOST_IP:-}"
		137	if [[ -z "$UPLINK" && -z "$IP" ]]; then
		138	index="$(printf '%s' "$NAME" \| grep -oE '[0-9]+$' \|\| echo 1)"
		139	IP="192.168.122.$((49 + index))"
		140	fi
		141
77	need() { command -v "$1" >/dev/null 2>&1 \|\| die "$1 is not installed"; }	142	need() { command -v "$1" >/dev/null 2>&1 \|\| die "$1 is not installed"; }
78		143
		144	# The dev machine gets the hypervisor toolchain the first time it is asked for
		145	# it, and nothing at all on every run after that. Only ever on a machine this
		146	# script was copied to: the workstation's package set is the workstation's own
		147	# business.
		148	ensure_prereqs() {
		149	[[ -n "$MANAGED" ]] \|\| return 0
		150
		151	missing=()
		152	for cmd in virsh virt-install qemu-img curl; do
		153	command -v "$cmd" >/dev/null 2>&1 \|\| missing+=("$cmd")
		154	done
		155	if ((${#missing[@]} > 0)); then
		156	# Named rather than pulled in by recommendation, and every one of them
		157	# is used: the emulator, qemu-img, the daemon, virsh, virt-install. A
		158	# guest on the LAN uplink needs no NAT network, so nothing here has to
		159	# drag in dnsmasq or the bridge tooling to serve one.
		160	bold "Installing the hypervisor toolchain (missing: ${missing[*]})"
		161	sudo -n env DEBIAN_FRONTEND=noninteractive apt-get update -qq
		162	sudo -n env DEBIAN_FRONTEND=noninteractive apt-get install -y \
		163	--no-install-recommends qemu-system-x86 qemu-utils \
		164	libvirt-daemon-system libvirt-clients virtinst curl
		165	fi
		166	if systemctl is-active --quiet libvirtd; then
		167	echo "libvirtd is running"
		168	else
		169	bold "Starting libvirtd"
		170	sudo -n systemctl enable --now libvirtd
		171	fi
		172	}
		173
		174	# The script is its own payload. It and the seed go to the dev machine's mktemp
		175	# dir and are re-exec'd there as root, which is what lets a single ssh session
		176	# do libvirt, the pool and apt without the driving side holding any privilege.
		177	# The remote's stdout is teed so the address it discovers can be read back;
		178	# its stderr passes straight through untouched.
		179	run_remote() {
		180	if [[ "$VERB" == create \|\| "$VERB" == recycle ]]; then
		181	[[ -f "$SSH_KEY" ]] \|\| die "no public key at $SSH_KEY (set DEVHOST_SSH_KEY)"
		182	SSH_PUBKEY="$(cat "$SSH_KEY")"
		183	fi
		184
		185	# ServerAliveInterval, because a create is ten quiet minutes of image
		186	# fetching and first boot on the far end, and a session that times out
		187	# during them abandons a guest that is already running.
		188	ssh_opts=(-o BatchMode=yes -o ConnectTimeout=10 -o ServerAliveInterval=30)
		189	tmp="$(ssh -p "$RPORT" "${ssh_opts[@]}" "$RTARGET" 'mktemp -d /tmp/devhost.XXXXXX')" \|\|
		190	die "$REMOTE did not answer ssh"
		191	scp -P "$RPORT" -q "${ssh_opts[@]}" "$SCRIPT" "$SEED_TEMPLATE" "$RTARGET:$tmp/" \|\|
		192	die "could not copy the script to $REMOTE"
		193
		194	envs=(
		195	"DEVHOST_REMOTE="
		196	"DEVHOST_MANAGED=1"
		197	"DEVHOST_SEED=$tmp/$(basename "$SEED_TEMPLATE")"
		198	"DEVHOST_UPLINK=$UPLINK"
		199	"DEVHOST_MAC=$MAC"
		200	"DEVHOST_MEM_MB=$MEM_MB"
		201	"DEVHOST_VCPUS=$VCPUS"
		202	"DEVHOST_DISK_GB=$DISK_GB"
		203	"DEVHOST_IMAGE_URL=$IMAGE_URL"
		204	"DEVHOST_IMAGE_SHA256=$IMAGE_SHA256"
		205	"DEVHOST_OS_VARIANT=$OS_VARIANT"
		206	"DEVHOST_POOL_DIR=$POOL_DIR"
		207	)
		208	if [[ -n "$SSH_PUBKEY" ]]; then
		209	envs+=("DEVHOST_SSH_PUBKEY=$SSH_PUBKEY")
		210	fi
		211	if [[ -n "$IP" ]]; then
		212	envs+=("DEVHOST_IP=$IP")
		213	fi
		214	# The image cache belongs to whichever machine fetches, so it is forwarded
		215	# only when someone has said where they want it.
		216	if [[ -n "${DEVHOST_CACHE_DIR:-}" ]]; then
		217	envs+=("DEVHOST_CACHE_DIR=$DEVHOST_CACHE_DIR")
		218	fi
		219
		220	cmd="sudo -n env"
		221	for assignment in "${envs[@]}"; do
		222	cmd+=" $(printf '%q' "$assignment")"
		223	done
		224	cmd+=" bash $tmp/$(basename "$SCRIPT") $(printf '%q %q' "$VERB" "$NAME")"
		225	# The temp dir goes whatever the run did, and the run's status survives it.
		226	cmd="$cmd; rc=\$?; rm -rf $tmp; exit \$rc"
		227
		228	bold "Driving $VERB of $NAME on $RTARGET"
		229	out="$(mktemp)"
		230	trap 'rm -f "$out"' EXIT
		231	if ssh -p "$RPORT" "${ssh_opts[@]}" "$RTARGET" "$cmd" \| tee "$out"; then
		232	rc=0
		233	else
		234	rc=$?
		235	fi
		236	[[ $rc -eq 0 ]] \|\| exit "$rc"
		237
		238	case "$VERB" in
		239	create \| recycle \| address)
		240	IP="$(sed -n 's/^devhost-address: //p' "$out" \| tail -n1)"
		241	[[ -n "$IP" ]] \|\| die "$RTARGET never reported an address for $NAME"
		242	;;
		243	esac
		244	rm -f "$out"
		245	trap - EXIT
		246
		247	case "$VERB" in
		248	create \| recycle)
		249	wait_for_ssh
		250	print_address
		251	;;
		252	address) print_address ;;
		253	esac
		254	}
		255
79	destroy_host() {	256	destroy_host() {
80	bold "Destroying $NAME"	257	bold "Destroying $NAME"
81	virsh destroy "$NAME" >/dev/null 2>&1 \|\| true	258	if command -v virsh >/dev/null 2>&1; then
82	# --remove-all-storage takes the root disk and the cloud-init seed ISO with	259	virsh destroy "$NAME" >/dev/null 2>&1 \|\| true
83	# the domain; the explicit rm is the belt for a domain that never defined.	260	# --remove-all-storage takes the root disk and the cloud-init seed ISO
84	virsh undefine "$NAME" --nvram --remove-all-storage >/dev/null 2>&1 \|\| true	261	# with the domain; the explicit rm is the belt for a domain that never
		262	# defined.
		263	virsh undefine "$NAME" --nvram --remove-all-storage >/dev/null 2>&1 \|\| true
		264	fi
85	sudo rm -f "$DISK"	265	sudo rm -f "$DISK"
86	# The DHCP reservation deliberately survives: it is what makes the address	266	if [[ -n "$UPLINK" ]]; then
87	# stable across a recycle, and create re-adds it either way.	267	echo "$NAME is gone"
88	echo "$NAME is gone (its reservation at $IP is kept)"	268	else
		269	# The DHCP reservation deliberately survives: it is what makes the
		270	# address stable across a recycle, and create re-adds it either way.
		271	echo "$NAME is gone (its reservation at $IP is kept)"
		272	fi
89	}	273	}
90		274
91	create_host() {	275	create_host() {
		276	ensure_prereqs
92	need virsh	277	need virsh
93	need virt-install	278	need virt-install
94	need qemu-img	279	need qemu-img
95	[[ -e /dev/kvm ]] \|\| die "/dev/kvm is missing — nested guests need hardware virtualization"	280	[[ -e /dev/kvm ]] \|\| die "/dev/kvm is missing — nested guests need hardware virtualization"
96	[[ -f "$SSH_KEY" ]] \|\| die "no public key at $SSH_KEY (set DEVHOST_SSH_KEY)"	281	if [[ -z "$SSH_PUBKEY" ]]; then
97	virsh net-info "$NETWORK" >/dev/null 2>&1 \|\| die "libvirt network '$NETWORK' does not exist"	282	[[ -f "$SSH_KEY" ]] \|\| die "no public key at $SSH_KEY (set DEVHOST_SSH_KEY)"
98	virsh net-info "$NETWORK" \| grep -q 'Active:.*yes' \|\| die "libvirt network '$NETWORK' is not running (virsh net-start $NETWORK)"	283	SSH_PUBKEY="$(cat "$SSH_KEY")"
		284	fi
		285	if [[ -n "$UPLINK" ]]; then
		286	ip link show "$UPLINK" >/dev/null 2>&1 \|\|
		287	die "no interface '$UPLINK' on this machine (set DEVHOST_UPLINK)"
		288	else
		289	virsh net-info "$NETWORK" >/dev/null 2>&1 \|\| die "libvirt network '$NETWORK' does not exist"
		290	# grep consumes all input on purpose: under pipefail, grep -q's early exit
		291	# EPIPEs the writer and fails the pipeline even when the match succeeded.
		292	virsh net-info "$NETWORK" \| grep 'Active:.*yes' >/dev/null \|\| die "libvirt network '$NETWORK' is not running (virsh net-start $NETWORK)"
		293	fi
99	if virsh dominfo "$NAME" >/dev/null 2>&1; then	294	if virsh dominfo "$NAME" >/dev/null 2>&1; then
100	die "$NAME already exists — 'recycle' rebuilds it, 'destroy' removes it"	295	die "$NAME already exists — 'recycle' rebuilds it, 'destroy' removes it"
101	fi	296	fi
@@ -113,11 +308,17 @@ create_host() {
113	}	308	}
114	echo "image ok: $cached"	309	echo "image ok: $cached"
115		310
116	bold "Pinning $NAME to $IP on the '$NETWORK' network"	311	if [[ -n "$UPLINK" ]]; then
117	virsh net-update "$NETWORK" delete ip-dhcp-host "<host mac='$MAC'/>" \	312	netarg="type=direct,source=$UPLINK,source_mode=bridge,model=virtio,mac=$MAC"
118	--live --config >/dev/null 2>&1 \|\| true	313	bold "Putting $NAME on the LAN through $UPLINK ($MAC)"
119	virsh net-update "$NETWORK" add ip-dhcp-host \	314	else
120	"<host mac='$MAC' name='$NAME' ip='$IP'/>" --live --config >/dev/null	315	netarg="network=$NETWORK,mac=$MAC,model=virtio"
		316	bold "Pinning $NAME to $IP on the '$NETWORK' network"
		317	virsh net-update "$NETWORK" delete ip-dhcp-host "<host mac='$MAC'/>" \
		318	--live --config >/dev/null 2>&1 \|\| true
		319	virsh net-update "$NETWORK" add ip-dhcp-host \
		320	"<host mac='$MAC' name='$NAME' ip='$IP'/>" --live --config >/dev/null
		321	fi
121		322
122	bold "Creating $NAME ($VCPUS vCPU, $((MEM_MB / 1024)) GB, $DISK_GB GB)"	323	bold "Creating $NAME ($VCPUS vCPU, $((MEM_MB / 1024)) GB, $DISK_GB GB)"
123	sudo qemu-img convert -O qcow2 "$cached" "$DISK"	324	sudo qemu-img convert -O qcow2 "$cached" "$DISK"
@@ -125,8 +326,8 @@ create_host() {
125		326
126	seed="$(mktemp)"	327	seed="$(mktemp)"
127	trap 'rm -f "$seed"' EXIT	328	trap 'rm -f "$seed"' EXIT
128	SSH_AUTHORIZED_KEY="$(cat "$SSH_KEY")" \	329	template="$(cat "$SEED_TEMPLATE")"
129	envsubst '${SSH_AUTHORIZED_KEY}' <"$REPO_ROOT/scripts/devhost.cloud-init.yaml" >"$seed"	330	printf '%s\n' "${template//'${SSH_AUTHORIZED_KEY}'/$SSH_PUBKEY}" >"$seed"
130		331
131	virt-install \	332	virt-install \
132	--name "$NAME" \	333	--name "$NAME" \
@@ -134,7 +335,7 @@ create_host() {
134	--vcpus "$VCPUS" \	335	--vcpus "$VCPUS" \
135	--cpu host-passthrough \	336	--cpu host-passthrough \
136	--disk "path=$DISK,format=qcow2,bus=virtio" \	337	--disk "path=$DISK,format=qcow2,bus=virtio" \
137	--network "network=$NETWORK,mac=$MAC,model=virtio" \	338	--network "$netarg" \
138	--os-variant "$OS_VARIANT" \	339	--os-variant "$OS_VARIANT" \
139	--cloud-init "user-data=$seed" \	340	--cloud-init "user-data=$seed" \
140	--graphics none \	341	--graphics none \
@@ -143,6 +344,64 @@ create_host() {
143	rm -f "$seed"	344	rm -f "$seed"
144	trap - EXIT	345	trap - EXIT
145		346
		347	if [[ -n "$UPLINK" ]]; then
		348	# Discovery waits on the guest agent, which the seed installs on first
		349	# boot. DEVHOST_IP is the way past it for a guest the router already has
		350	# a reservation for: the address is known before the machine exists.
		351	if [[ -z "$IP" ]]; then
		352	IP="$(await_address)"
		353	fi
		354	# macvtap keeps this machine off its own guest, so the ssh proof belongs
		355	# to whoever drives this script. Report the address and let them make it.
		356	printf 'devhost-address: %s\n' "$IP"
		357	if [[ -z "$MANAGED" ]]; then
		358	print_address
		359	fi
		360	else
		361	wait_for_ssh
		362	print_address
		363	fi
		364	}
		365
		366	# The guest's address, from the first source that knows it. Under macvtap the
		367	# host has no path to its own guest, so the ARP table never learns it and
		368	# libvirt owns no lease to read — the guest agent's virtio channel is the one
		369	# source that answers. The other two are kept for a host whose guest shares its
		370	# L2, where they answer sooner.
		371	domain_ipv4() {
		372	for src in agent arp; do
		373	found="$(virsh domifaddr "$NAME" --source "$src" 2>/dev/null \|
		374	awk '{
		375	for (i = 1; i < NF; i++)
		376	if ($i == "ipv4") {
		377	split($(i + 1), a, "/")
		378	if (a[1] !~ /^(127\.\|169\.254\.)/ && !ip) ip = a[1]
		379	}
		380	} END { if (ip) print ip }' \|\| true)"
		381	if [[ -n "$found" ]]; then
		382	printf '%s' "$found"
		383	return 0
		384	fi
		385	done
		386	ip -4 neigh show 2>/dev/null \|
		387	awk -v mac="$MAC" 'BEGIN { mac = tolower(mac) }
		388	tolower($0) ~ mac && !ip { ip = $1 } END { if (ip) print ip }' \|\| true
		389	}
		390
		391	await_address() {
		392	bold "Waiting for $NAME to report the address the LAN leased it" >&2
		393	for _ in $(seq 1 90); do
		394	found="$(domain_ipv4)"
		395	if [[ -n "$found" ]]; then
		396	printf '%s' "$found"
		397	return 0
		398	fi
		399	sleep 5
		400	done
		401	die "$NAME never reported an address ($CONSOLE_HINT)"
		402	}
		403
		404	wait_for_ssh() {
146	# A rebuilt host presents a new host key at an address ssh already knows,	405	# A rebuilt host presents a new host key at an address ssh already knows,
147	# which otherwise fails the first deploy with a warning that looks like an	406	# which otherwise fails the first deploy with a warning that looks like an
148	# attack rather than a recycle.	407	# attack rather than a recycle.
@@ -152,14 +411,11 @@ create_host() {
152	for _ in $(seq 1 60); do	411	for _ in $(seq 1 60); do
153	if ssh -p "$SSH_PORT" -o BatchMode=yes -o StrictHostKeyChecking=accept-new \	412	if ssh -p "$SSH_PORT" -o BatchMode=yes -o StrictHostKeyChecking=accept-new \
154	-o ConnectTimeout=5 "ubuntu@$IP" true 2>/dev/null; then	413	-o ConnectTimeout=5 "ubuntu@$IP" true 2>/dev/null; then
155	break	414	return 0
156	fi	415	fi
157	sleep 5	416	sleep 5
158	done	417	done
159	ssh -p "$SSH_PORT" -o BatchMode=yes -o ConnectTimeout=5 "ubuntu@$IP" true 2>/dev/null \|\|	418	die "$NAME did not answer ssh at $IP within 5 minutes ($CONSOLE_HINT)"
160	die "$NAME did not answer ssh at $IP within 5 minutes (virsh console $NAME)"
161
162	print_address
163	}	419	}
164		420
165	print_address() {	421	print_address() {
@@ -174,6 +430,11 @@ print_address() {
174	EOF	430	EOF
175	}	431	}
176		432
		433	if [[ -n "$REMOTE" && -z "$MANAGED" ]]; then
		434	run_remote
		435	exit 0
		436	fi
		437
177	case "$VERB" in	438	case "$VERB" in
178	create) create_host ;;	439	create) create_host ;;
179	destroy) destroy_host ;;	440	destroy) destroy_host ;;
@@ -181,5 +442,18 @@ recycle)
181	destroy_host	442	destroy_host
182	create_host	443	create_host
183	;;	444	;;
184	address) print_address ;;	445	address)
		446	if [[ -n "$UPLINK" ]]; then
		447	if [[ -z "$IP" ]]; then
		448	IP="$(domain_ipv4)"
		449	fi
		450	[[ -n "$IP" ]] \|\| die "$NAME has no address to report ($CONSOLE_HINT)"
		451	printf 'devhost-address: %s\n' "$IP"
		452	if [[ -z "$MANAGED" ]]; then
		453	print_address
		454	fi
		455	else
		456	print_address
		457	fi
		458	;;
185	esac	459	esac