docs/upgrade.md

Ref: main v0.0.5   Size: 6.3 KiB   History

# Upgrading

Three things version independently: the per-host agent, the server, and
cloud-hypervisor. None of them touch running VMs.

## Agents, from the console

The fleet overview shows each host's agent version. When the server knows a
newer release (it polls `https://eitri.sh/dl/latest/manifest.json` daily), an
`↑` button appears next to hosts that are behind. Clicking it tells that one
agent to upgrade itself:

1. The agent downloads the host bundle from eitri.sh—the same tarball you
   would install by hand—verifies its sha256 against the release manifest, and
   takes the `eitri-agent` out of it.
2. It swaps the binary in place—the old one is kept next to it as
   `eitri-agent.prev`—and re-execs. The process keeps its PID; running VMs
   are untouched and stay under the agent's care throughout.
3. The host reports its new version on the next sync, and the button
   disappears.

If a download fails, nothing is swapped; click again to retry. To roll back by
hand, stop the agent, move `eitri-agent.prev` back over the binary, and start
it again.

The release pipeline makes this same offer to every connected host after it
rolls a plane, and waits for them to report the new version, so a hosted fleet
arrives at a release without anyone clicking through it host by host
(`deploy/server/README.md`, stage 8).

**Requirements.** The button lights up only when the running agent's version
orders before the published release. Release tags (`vX.Y.Z`), pre-releases
(`vX.Y.Z-pre.N`, which order below the release they lead to), and git-describe
builds (`vX.Y.Z-N-g<hex>`, N commits past the tag) all order, so a
hand-deployed describe build behind the latest release is offered the upgrade,
and one ahead of it is never offered a downgrade. Anything else—`dev`, a
`-dirty` tree—orders before nothing and is never offered anything. The server
needs `release_manifest_url` reachable; set it to `""` in the server config to
disable upgrade checks entirely.

## Agents, by hand

An agent whose version never orders—a `dev` or `-dirty` build—never gets the
button. The manual path is the same swap, done by you.

On Linux, take the current release from `/dl/latest/` and read its version out
of `SHA256SUMS`; on an arm64 host, `_arm64` replaces `_amd64` throughout:

```sh
curl -fsSLO https://eitri.sh/dl/latest/SHA256SUMS
V=$(sed -n 's/.*eitri-server_\(v[^_]*\)_linux_amd64\.tar\.gz$/\1/p' SHA256SUMS)
curl -fsSLO "https://eitri.sh/dl/latest/eitri-server_${V}_linux_amd64.tar.gz"
grep " eitri-server_${V}_linux_amd64.tar.gz$" SHA256SUMS | sha256sum -c -
tar xzf "eitri-server_${V}_linux_amd64.tar.gz"
sudo cp /usr/local/bin/eitri-agent /usr/local/bin/eitri-agent.prev
sudo install -m 0755 "eitri-server_${V}_linux_amd64/eitri-agent" /usr/local/bin/eitri-agent
sudo systemctl restart eitri-agent
```

The host bundle is the artifact—there is one set of bytes per platform per
release, and it is the same one the console's button fetches. Each release also
has an immutable home at `/dl/<version>/`, but only the current one is served,
so `/dl/latest/` is the address that always answers.

Running VMs survive the restart—the unit's `KillMode=process` (see Operations
below) leaves the guests alone. The `.prev` copy mirrors what the button path
keeps, so rollback is the same either way.

On a Mac the binary lives in the running account's own space (e.g.
`~/.local/bin/eitri-agent`). Overwrite it with `eitri-agent` from the darwin
bundle (`eitri-agent_<version>_darwin_arm64.tar.gz`), then restart the
LaunchAgent:

```sh
launchctl kickstart -k gui/$(id -u)/sh.eitri.agent
```

No `sudo` anywhere, same as the install. If guests turn unreachable through
the gate after the swap ("cannot reach VM"), check the agent's Local Network
permission—the quickstart's "Join a Mac" section covers it.

If the swapped-in agent reports it is not enrolled, it is running as a
different account than the one that joined: enrollment state is per-account
(`~/.eitri/agent` on macOS, `/var/lib/eitri-agent` on Linux), so under a new
account the binary is a new host. Join it fresh from the console, then
decommission the old host row it leaves behind. If that old agent is gone for
good, the delete needs force—the dead-hardware escape hatch on host delete.

## The server

The console banner links here when the published release differs from the
running server's version. Server upgrades are manual and downtime is fine—agents keep
reconciling and VMs keep running while it's away:

1. Stop `eitri-server`.
2. Replace the binary with the new release.
3. Start it. The schema is applied idempotently on boot (existing tables are
   left as-is), and agents reconnect on their own.

A newer server with older agents is safe: new fields in the sync protocol are
ignored by agents that predate them.

**One exception, from v0.0.4 on.** A guest's SSH host key is generated by the
host that runs it, and an agent from before v0.0.4 does not know how to do
that. The server will not place a VM on such a host: the create is refused with
the host's name, the version it reports, and the endpoint that upgrades it —
nothing could verify a guest created there. A connected host that has reported
no version at all is read the same way, since nothing says otherwise. Only a
host that is connected is judged: one that is offline has told this server
nothing about the agent that will eventually run the VM, so the create is
accepted as desired state and the guest is verified — or refused — when it
boots. Placement chosen for you skips those hosts entirely. Existing VMs are unaffected — they
keep the host key and certificate they were built with; one created against an
older release with an uncertified key is refused by `eitri ssh` at host
verification, and re-creating it after the upgrade is the fix.

## cloud-hypervisor

Swap `/usr/local/bin/cloud-hypervisor` on the host. New and restarted VMs use
the new binary; running VMs keep their old process until they stop. There is
no live handover for running guests.

## Operations

The agent runs under systemd (`eitri-agent.service`). The unit sets
`KillMode=process`—that line is load-bearing: the default would kill every
cloud-hypervisor guest in the unit's cgroup whenever the agent stops. Logs:
`journalctl -u eitri-agent`.

## Related

- [cert-rotation.md](cert-rotation.md)—rotating the server's QUIC identity
- [credential-revocation.md](credential-revocation.md)—the disaster levers
  when credentials leak