docs/decisions.md

Ref: main v0.0.5   Size: 12.9 KiB   History

# Decisions

Settled ground: the irreversible or contested choices, what they beat, and why.
Three lines each—when a rationale needs more, it lives in the relevant doc.
For *how* we decide, see [ethos.md](ethos.md); for what these rest on, see
[assumptions.md](assumptions.md).

### Desired-state loop, not RPC

The server streams desired state; agents converge toward it and report actual
state back. Instead of imperative RPC per operation. Level-triggered convergence
survives disconnects, crashes, and missed messages—a host re-derives
everything from persisted records plus what it observes. The loop keeps the
adjective; the messages do not—every message in a snapshot is desired and every
message in a report is actual, so they are `Snapshot`/`VMSpec` down and
`Report`/`VMStatus` back. Details in [architecture.md](architecture.md).

### Guests boot their own kernel (UEFI firmware)

VMs boot via UEFI firmware (`CLOUDHV.fd`) into the image's own bootloader.
Instead of a host-supplied kernel/initrd (cloud-hypervisor's direct-boot path).
The guest owns the guest: any disk-only cloud image boots unmodified, and guest
kernel upgrades are the guest's business, not the fleet's.

### Bring-your-own tenant user CA

Each tenant signs its own SSH user certs; eitri stores only CA public keys.
Instead of the server minting user certs from an eitri-held CA. A compromised
eitri server cannot mint access to any tenant's guests—the user-auth trust
root lives with the tenant. Details in [connecting.md](connecting.md).

### Delegated credentials, not a CA eitri holds

A tenant signs an ephemeral key eitri generates, and eitri uses the resulting
certificate until it expires. Instead of eitri holding a signing key on the
tenant's behalf, which was the obvious way to let a token-only caller reach a
guest: it would make the control-plane database worth stealing and give eitri a
power that never lapses. A certificate does lapse, and a restart drops it.
Instead of persisting delegations, too — a credential that survives the process
that was granted it is one nobody remembers granting. The cost is re-delegating
after a redeploy, which is one `ssh-keygen` against a stable public key.
Details in [connecting.md](connecting.md).

### A guest's host key is generated by its host, and certified by the fleet

The host generates the keypair and sends up the public half; the control plane
signs a certificate for the name on the VM's row. Instead of the control plane
generating the key and shipping the private half down in desired state, which
put every guest's host key in one database. The host supplies a key and never a
name, so it cannot obtain a certificate for a VM it does not run. The cost is
one extra tick before a guest boots, against a create that already waits
minutes for cloud-init.

### SSH-CA jump gate, not a mesh

Guest access is plain OpenSSH through a certificate-verified bastion. Instead
of an overlay mesh with per-node identity. Both ends speak stock ssh,
verification is by certificate in both directions, and there is no fabric
dependency to operate or trust.

### Agent-local resource admission

Each host admits vCPU / memory / disk / address through one serialized gate on
the agent. Instead of a server-side scheduling ledger. The host is the source
of truth for its own capacity—admission that consults reality cannot drift
from it, and a partitioned host stays correct on its own.

### Embedded DHCP owns guest addressing

The agent runs a DHCP server handing each VM a sticky, deterministic address,
reserved at create through the same admission gate. Instead of static IP
injection via cloud-init or an external DHCP daemon. Addressing is a host
resource like memory; owning it host-locally keeps images untouched and needs
no external service.

### The host reports its guest network

A host tells the fleet which subnet its guests are on and the fleet records it;
the pool becomes a suggestion for hosts that have no opinion. Instead of the
fleet allocating every host a subnet and validating reported addresses against
it. A host whose OS owns the guest network sits on a subnet no allocation of
ours can contain—so the fleet can only be told, which is also why a value it did
not choose is never recycled back into the pool.

### Networking stays minimal: just enough to be usable

A guest gets an address, a way out, and the ports its tenant chose to publish—
embedded DHCP, the host as NAT gateway, and a userspace port-forward table for
TCP and UDP alike (a spliced connection per caller, a session per client
address; both capped, both deletable, neither routing anything else)—any
of which could be deleted with the VM still booting, syncing, and taking SSH.
Instead of growing the agent toward a virtual router (guest DNS, ACLs,
east-west, in-path policy). Network machinery that basic lifecycle would come
to depend on is the mesh mistake again—a binding point taxing every feature
before the basics stood—and anything the site's network already does better
arrives with bridged networking, not inside the proxy.

### No superuser principal

Every API principal is `{Tenant, Fleet}`; fleet-level endpoints are gated
fleet-only, and no principal crosses tenant boundaries. Instead of a global
admin identity. Cross-tenant capability is the thing multi-tenancy exists to
prevent, so it is unrepresentable rather than merely forbidden.

### One owner per API surface

Each capability has exactly one package that owns its surface, enforced by
fitness functions wherever the import graph can see it: SQLite belongs to
internal/server/store (R10), process execution to the R7 sanctioned set, the
wire types to the leaf plane's one shared contract (R9), the client's ssh to
internal/cli, importable only by its binary (R8), and the server's HTTP API
to internal/server/api/client over the internal/server/api/types leaf
(R11/R12), with the generated OpenAPI spec drift-gated by api-check. Instead
of convention plus review, which let three parallel HTTP clients of the same
endpoints accumulate before anyone noticed. When a second implementation of a
surface appears, the fix is consolidation behind the owner, not a second
owner.

### Agents pin the server cert by fingerprint

Agents trust the server's self-signed QUIC cert by fingerprint from the join
blob—not by CA path, not by expiry. Instead of a CA hierarchy or expiring
trust. An expired cert can never brick a fleet; rotation is deliberate hygiene,
not an emergency. Details in [cert-rotation.md](cert-rotation.md).

### The agent aspires to kubelet

The eitri agent is deliberately kubelet-shaped and stays that way on purpose:
level-triggered reconcile with admission and per-VM workers; a small
runtime-agnostic Provisioner seam in the CRI mold, where cloud-hypervisor and
Virtualization.framework are two runtimes behind one set of verbs; capacity
reported against a user-declared quota the way a node reports capacity against
allocatable; and, when a host runs with no control plane, VMs from local
manifests that a later join adopts — kubelet's static pods. Instead of
inventing a new node-agent shape: a decade of prior art already made these
decisions, in public, under load.

Two disciplines ride the aspiration. The Provisioner seam stays in-process
until a genuinely foreign backend wants in — a versioned plugin wire is a
binding point, and this repo has already paid once for building a binding
point before its parties existed. And the deploy smoke is the seam's
conformance suite: a Provisioner is whatever passes it.

### One MCP surface: `/mcp` on the plane

eitri speaks MCP in exactly one place: `/mcp` on the control plane, authenticated
by a bearer PAT. Not a local binary beside it, not a public SSO front ahead of
it: every added arrangement is another credential story to keep honest — its own
config, its own key material, its own way of naming a tenant — while the plane
already authenticates PATs and holds no signing key at all. A self-hoster gets
an MCP server by self-hosting the plane, and a public front would target this
same endpoint rather than being a second thing. Reversed by a genuine air-gapped
need: guests to drive with no plane to reach.
Details in [mcp.md](mcp.md).

### Per-VM named networks

A host advertises the bridge-backed networks its operator declared
(`--host-network name=bridge`); a VM requests one by name; admission places it
on a host advertising that name or refuses at create. Instead of a per-host
bridged mode, which could not express the actual fleet — one host serving a
LAN pet and NAT'd cattle at once. And never a silent fallback: an agent from
before the field advertises nothing and would quietly NAT the guest, so
silence refuses — a VM that asked for the LAN either gets it or does not
exist.

The operator owns the bridge; eitri uses it and never creates it. The
precedent is unanimous: runtime-created network objects are how everyone
builds NAT bridges and macvlan attachments — docker0, virbr0, eitri's own
eitri0, Docker's macvlan — while a bridge that enslaves the uplink is
persistent host configuration everywhere one exists — Proxmox's vmbr0,
libvirt's bridge mode. eitri copies the only precedent that exists. And the
address is discovered, never allocated: the agent snoops the guest's own tap
for the DHCP ACK keyed to its deterministic MAC — the site's DHCP addresses
the guests, the agent discovers the result.

### A named network adds a NIC, it does not replace one

Every guest keeps its NAT attachment: `eth0` on `eitri0`, its reservation
allocated before boot, the gate's splice target, its egress and its way to the
other guests on that host. A named network is a SECOND NIC on the operator's
bridge, addressed by the site's own DHCP, and the VM reports both addresses —
`assigned_ip` from boot, `network_ip` once the network answers. The default
route stays on the NAT NIC (metric 100 against 200, as the default Ubuntu
image's netplan renders them), so egress is deterministic and either side can
be down without taking the other with it.

This is multipass's shape, which is the shape the daily driver already runs:
NAT primary with the default route, LAN secondary, and which address a service
advertises left to that service. The replace-shaped version — LAN *instead of*
NAT — was built first and rejected: it shipped two networking models, with a
guest's reachability, its gate path and its VM-to-VM story all forking on
which one it was, and it opened a discovery window on the gate path that the
NAT-only guest never had. One model costs a stated blast radius instead (see
[assumptions.md](assumptions.md) — a LAN-facing guest is also on the host's
private L2), and that trade was taken deliberately.

### A named network is called by the operator's name for it

The API and wire already agreed with themselves — `network`, `network_ip`,
`--host-network` — but the console had coined a second word, "LAN", for the
same idea, on a second row for it. Both are gone: one Network row now carries
the operator's own name as its value, joined to the discovered address once
the network's DHCP answers (`lan · 192.168.0.37`). eitri displays the
operator's vocabulary rather than inventing its own.

### A Mac agent is unprivileged, and a LaunchAgent

The macOS agent runs as the logged-in user, installed through
`eitri-agent-launchagent.sh`, with its binary and state in that account's own
directories. Instead of the root-owned systemd shape the Linux agent takes.
On macOS eitri owns none of the host's networking—Virtualization.framework
owns the guest network and vfkit carries the entitlement for it—so there is
nothing to be root for; and a Mac host is nearly always somebody's Mac, where
an agent belongs alongside the person already logged into it. It is also what
lets the agent replace its own binary on upgrade, which a root-owned prefix
would quietly prevent.

### vfkit comes from Homebrew, not from the fleet

The agent fetches cloud-hypervisor and guest firmware itself on Linux, but a
Mac host installs vfkit by hand. Instead of bootstrapping it like the Linux
hypervisor. vfkit only runs carrying Apple's virtualization entitlement, an
entitlement lives in a code signature, and signing is not something a fetched
artifact can carry across—so the signed copy has to come from a distributor
Apple's rules already fit.

### Service users are separate per key they hold

The server runs as `eitri`, the bundled issuer as `eitri-oidc`, each with its
own state directory. Instead of one shared service account. The issuer holds
password hashes and the sign-in signing key; the server holds the KEK, the host
CA and the gate host key. Separate accounts keep either one's key material
unreadable by the other service.

### The published trust story is what you get, not why

Site pages say what a reader can do and what will happen; the reasoning behind
the shape—two CAs in opposite directions, delegation held in memory, an
unprivileged Mac agent—lives here and in [architecture.md](architecture.md),
which never publish. A user-facing page that argues with itself is a page the
reader has to finish before they can start.