d6105e58

fix(placement): the plane's refusal is never stricter than the host's admission

a73x 2026-08-12 14:51

Commit message

fix(placement): the plane's refusal is never stricter than the host's admission

The create-time capacity check refused two things it had no business refusing.

A host with no --max-* flags advertises the machine's raw totals, and its agent
admits whatever it is asked — guest disks are sparse, guest memory is not
preallocated, and running past the paper numbers is what such a host is for.
Treating those totals as a wall refused VMs the fleet is built to run, with no
way for the operator to see why short of an ssh session. The wire carries one
capacity triple and no cap flag, but the report proves the difference anyway:
the clamp only ever lowers, so an advertisement below the size the same
report's metrics imply can only be a cap. A dimension is judged only where that
proof holds. Everything unprovable — a --max-vcpus cap (no report carries a
core count), a failed probe, a cap set at exactly the machine's total — reads
as uncapped and rides to the agent, which was always the authority.

And a refusal that counted a destroy in flight told the caller to delete a VM,
which is what they had just done: the resources holding them out were the ones
that delete has not finished releasing. Tombstones still count — the disk is on
the host until the reap — but when they are what tips the refusal it now says
so, names how many teardowns are running, and gives the remedy that matches:
wait, because that wait ends by itself. CommittedOnHost returns the two halves
so the refusal can tell them apart; the delete remedy stays for a host whose
live VMs fill it on their own.

docs/assumptions.md

+never counted stays distinguishable from a port that counted zero. False if
+anything alerts or bills on these numbers — that needs durable counters with
+an epoch, which these are deliberately not.
+### An advertisement below the machine is a cap, and nothing else is
+An agent advertises the machine's totals clamped to its `--max-*` flags and the
+clamp only ever lowers, so a reported capacity below the size the same report's
+metrics imply can only be a cap its operator set. Underpins the create-time
+capacity refusal firing there and nowhere else: an uncapped agent admits
+whatever it is asked — guest disks are sparse and guest memory is not
+preallocated — so refusing against its raw totals would block the overcommit
+such a host exists to do. **Proven** in code on both platforms and pinned by an
+invariant test in `hostinfo`: memory's used+available is exactly the advertised
+total, and disk's used+free undercounts it by the filesystem's reserved blocks,
+an error that can only fail to prove a cap, never invent one. The wire carries
+no cap flag, so three cases read as uncapped and fall back to the agent
+refusing at boot — a `--max-vcpus` cap (no report carries a core count), a
+failed probe, and a cap set at exactly the machine's total. False the moment
+the wire says which a number is; the refusal should then judge the declared
+capacity directly and this arithmetic goes away.

internal/agent/hostinfo/hostinfo_test.go

@@ @@ -220,6 +220,36 @@ func TestCapacityReportsTotals(t *testing.T) { @@
+// TestMetricsImplyTheCapacityTheSameProbeSees guards a coupling that reaches
+// all the way to the control plane's placement refusal. The agent advertises
+// this machine's totals clamped to its --max-* flags, and the clamp only lowers
+// — so an advertisement BELOW the machine's real size is the server's only
+// proof that an operator set a cap, and the metrics in the same report are
+// where it reads that real size (see api.declaredLimit).
+//
+// Memory must therefore land on the nose: readMetrics computes used as
+// total-minus-available, so used+available is exactly Capacity's mem_mb. Disk
+// may only UNDERCOUNT — used+free omits the filesystem's reserved blocks — and
+// the direction is the safe one: an undercount can fail to prove a cap, never
+// invent one. Break either and the server starts refusing creates on hosts
+// whose agents would have run them.
+func TestMetricsImplyTheCapacityTheSameProbeSees(t *testing.T) {
+	dir := t.TempDir()
+	c, m := Capacity(dir), Metrics(dir)
+	if m.GetMemAvailableMb() == 0 {
+		t.Skip("memory probe unavailable on this machine")
+	}
+	assert.Equal(t, c.GetMemMb(), m.GetMemUsedMb()+m.GetMemAvailableMb(),
+		"mem_used_mb + mem_available_mb must be exactly the advertised total")
+	if m.GetDiskFreeGb() == 0 && m.GetDiskUsedGb() == 0 {
+		t.Skip("disk probe unavailable on this machine")
+	}
+	assert.LessOrEqual(t, m.GetDiskUsedGb()+m.GetDiskFreeGb(), c.GetDiskGb(),
+		"disk_used_gb + disk_free_gb may undercount the total, never overstate it")
+}
+// TestCapacityExactArithmetic and TestCapacityBestEffortOnSyscallFailure live
+// in hostinfo_linux_test.go: they stub sysinfoFn/statfsFn with
+// syscall.Sysinfo_t/Statfs_t literals, which only exist on Linux.

internal/mcpserver/api_test.go

internal/server/api/api.go

Old		New
@@ -853,11 +853,18 @@ func (a API) handleCreateVM(w http.ResponseWriter, r http.Request) {
853	}	853	}
854		854
855	// Capacity precondition, the third refusal of this same shape: the request	855	// Capacity precondition, the third refusal of this same shape: the request
856	// is fine, and the host cannot serve it. A host that is already holding as	856	// is fine, and the host has been told not to serve it. A host whose operator
857	// much as it says it has cannot boot one more guest — the agent refuses it	857	// capped it is already holding as much as it may hold cannot boot one more
858	// at materialization ("host capacity limit reached", see reconcile.Engine)	858	// guest — the agent refuses it at materialization ("host capacity limit
859	// and the VM ends up failed, minutes later, with an answer that existed	859	// reached", see reconcile.Engine) and the VM ends up failed, minutes later,
860	// here. So the create answers 409 now, naming the dimension and the numbers.	860	// with an answer that existed here. So the create answers 409 now, naming
		861	// the dimension and the numbers.
		862	//
		863	// It answers 409 ONLY there. An uncapped host advertises the machine's
		864	// totals and its agent admits past them on purpose (sparse disks, memory
		865	// that is not preallocated), so treating those totals as a wall would refuse
		866	// VMs the fleet is designed to run. overCapacityRefusal judges a dimension
		867	// only when the report proves a cap, and says nothing otherwise.
861	//	868	//
862	// Judged only on a host that has spoken, exactly like the refusal above and	869	// Judged only on a host that has spoken, exactly like the refusal above and
863	// for the same reason: reported capacity is registry state, so an offline or	870	// for the same reason: reported capacity is registry state, so an offline or
@@ -872,7 +879,7 @@ func (a API) handleCreateVM(w http.ResponseWriter, r http.Request) {
872	return	879	return
873	}	880	}
874	want := store.Alloc{VCPUs: req.VCPUs, MemMB: req.MemMB, DiskGB: req.DiskGB}	881	want := store.Alloc{VCPUs: req.VCPUs, MemMB: req.MemMB, DiskGB: req.DiskGB}
875	if msg := overCapacityRefusal(host.Name, req.HostID, want, held, hostState.Capacity); msg != "" {	882	if msg := overCapacityRefusal(host.Name, req.HostID, want, held, hostState.Report); msg != "" {
876	http.Error(w, msg, http.StatusConflict)	883	http.Error(w, msg, http.StatusConflict)
877	return	884	return
878	}	885	}

internal/server/api/capacity.go

@@ @@ -8,39 +8,143 @@ import ( @@
+	"github.com/a73x/eitri/internal/server/store"
+// declaredLimit answers the only question this refusal may rest on: is the
+// number this host advertised for a dimension a limit its operator declared, or
+// is it just the size of the machine? It returns the limit when the report
+// proves the first, and 0 — "nothing proven, do not judge" — otherwise.
+//
+// The proof is arithmetic on the host's own report. An agent advertises the
+// machine's totals clamped to its --max-vcpus/--max-mem-mb/--max-disk-gb flags
+// (syncclient.advertisedCapacity), and the clamp only ever lowers, so an
+// advertisement BELOW the machine's total can only be a configured cap. The
+// machine's total is not on the wire as such, but the metrics in the same
+// report imply it: the agent computes used as total-minus-available, so
+// mem_used_mb + mem_available_mb is exactly the machine's memory, and
+// disk_used_gb + disk_free_gb is the state dir's filesystem minus its reserved
+// blocks — an UNDERCOUNT, which is the safe direction here (it can only fail to
+// prove a cap, never invent one).
+//
+// This is a coupling to hostinfo.readMetrics on both platforms, and an
+// invariant test there guards it (TestMetricsImplyTheCapacityTheSameProbeSees).
+// If that identity is ever broken the failure is a refusal that stops firing,
+// not one that fires wrongly.
+//
+// Everything unprovable reads as uncapped, and the list is worth naming: vCPUs
+// always (no report carries the machine's core count, so a --max-vcpus cap is
+// invisible here), a dimension whose probe failed and reports zeroes, and a cap
+// set at exactly the machine's total. Each of those falls back to the world
+// before this preflight existed — the agent refuses at boot and the VM sits
+// failed — which is the honest cost of not refusing a VM the host would have
+// run. See overCapacityRefusal for why that asymmetry is the whole design.
+func declaredLimit(advertised, machineTotal int64) int64 {
+	if advertised <= 0 || machineTotal <= 0 || advertised >= machineTotal {
+		return 0
+	}
+	return advertised
+}
+// dimension is one resource under judgement: what the VM asked for, what the
+// host is holding (split live vs. still-being-torn-down), and the limit — 0
+// when this dimension is not judged at all.
+type dimension struct {
+	noun, unit          string
+	want, live, pending int64
+	limit               int64
+}
+func (d dimension) overWithPending() bool { return d.limit > 0 && d.live+d.pending+d.want > d.limit }
+func (d dimension) overLiveAlone() bool   { return d.limit > 0 && d.live+d.want > d.limit }
+// held phrases the binding numbers the way an operator has to act on them:
+// what was asked for, what is already there, out of what.
+func (d dimension) held() string {
+	return fmt.Sprintf("%s — needs %d%s, already holds %d of %d%s",
+		d.noun, d.want, d.unit, d.live+d.pending, d.limit, d.unit)
+}
+// heldWithPending is the same line for a refusal a teardown is tipping: the
+// share held by VMs on their way out is the part that is about to go away.
+func (d dimension) heldWithPending() string {
+	return fmt.Sprintf("%s, %d%s of it by the teardown", d.held(), d.pending, d.unit)
+}
+// overCapacityRefusal explains why a host cannot be given a VM: it is already
+// holding as much as its operator said it may. It returns "" when the VM fits,
+// and — deliberately — whenever this server cannot prove the host would refuse.
+//
+// The answer exists at create — the host's capacity and everything already on
+// it are both known here — so the caller learns it in milliseconds instead of
+// watching a VM sit `creating` and then `failed` because the agent reached the
+// same conclusion minutes later, at the far end of an image download.
+//
+// The agent's admission is the AUTHORITY; this is a courtesy that runs first,
+// and a courtesy must never be stricter than the authority it anticipates. The
+// agent refuses a boot only against a configured cap (reconcile.quotaCheckLocked
+// — with no --max-* flag it admits whatever it is asked, and that is the design:
+// guest disks are sparse and memory is not preallocated, so a host deliberately
+// runs more than it has on paper). So this refuses only where declaredLimit can
+// prove a cap exists, and stays silent everywhere else. Refusing a VM the host
+// would have run is the one failure with no remedy short of an ssh session;
+// missing one it will refuse costs the caller the wait it always used to cost.
+//
+// Every judged dimension that binds is named: a VM that is too big in two ways
+// should be resized once, not twice.
+func overCapacityRefusal(hostName, hostID string, want store.Alloc, held store.Commitment, rep registry.Report) string {
+	m := rep.Metrics
+	dims := []dimension{
+		{noun: "vcpus", want: want.VCPUs, live: held.Live.VCPUs, pending: held.Pending.VCPUs,
+			limit: declaredLimit(rep.Capacity.VCPUs, 0)},
+		{noun: "memory", unit: "MB", want: want.MemMB, live: held.Live.MemMB, pending: held.Pending.MemMB,
+			limit: declaredLimit(rep.Capacity.MemMB, m.MemUsedMB+m.MemAvailableMB)},
+		{noun: "disk", unit: "GB", want: want.DiskGB, live: held.Live.DiskGB, pending: held.Pending.DiskGB,
+			limit: declaredLimit(rep.Capacity.DiskGB, m.DiskUsedGB+m.DiskFreeGB)},
+	}
+	var binding []dimension
+	liveAloneFits := true
+	for _, d := range dims {
+		if d.overWithPending() {
+			binding = append(binding, d)
+		}
+		if d.overLiveAlone() {
+			liveAloneFits = false
+	if len(binding) == 0 {
+		return ""
+	// A destroy in flight is the reason, and "delete a VM" would be a lie: the
+	// caller may well have just deleted one, and is being refused by the very
+	// resources that delete has not finished releasing. Say what is actually
+	// happening and give the remedy that matches it — the wait ends by itself.
+	if liveAloneFits {
+		var lines []string
+		for _, d := range binding {
+			lines = append(lines, d.heldWithPending())
+		}
+		return fmt.Sprintf("host %s (%s) has no room for this VM yet: %s. %s, and a deleted VM holds its "+
+			"vcpus, memory and disk until its host finishes tearing it down — the VMs still running there "+
+			"would leave room for this one. Wait for the teardown to finish and create it again, ask for "+
+			"less, or create it on a host with room.",
+			hostName, hostID, strings.Join(lines, "; "), pendingTeardowns(held.PendingVMs))
+	}
+	var lines []string
+	for _, d := range binding {
+		lines = append(lines, d.held())
+	}
+	return fmt.Sprintf("host %s (%s) has no room for this VM: %s. That is a limit its operator set below "+
+		"what the machine has, so the agent there would refuse this VM at boot and it would sit failed. "+
+		"Ask for less, delete a VM on that host to free what it holds, or create it on a host with room.",
+		hostName, hostID, strings.Join(lines, "; "))
+}
+// pendingTeardowns names the deletes still in flight, counted, so the caller
+// can tell "the one I just deleted" from "somebody else is clearing the host".
+func pendingTeardowns(n int) string {
+	if n == 1 {
+		return "1 VM deleted from that host is still being torn down"
+	}
+	return fmt.Sprintf("%d VMs deleted from that host are still being torn down", n)

internal/server/api/capacity_api_test.go

Old		New
@@ -13,12 +13,33 @@ import (
13	"github.com/a73x/eitri/internal/server/registry"	13	"github.com/a73x/eitri/internal/server/registry"
14	)	14	)
15		15
16	// agentReportsCapacity is the report of a host that has told this server what	16	// agentReportsCappedCapacity is the report of a host whose operator held some
17	// machine it is: the create path judges placement only on a host that has	17	// of it back: it advertises less than the machine it also measures. That gap is
18	// spoken, and this is what speaking sounds like.	18	// the only proof this server gets that a cap exists — and a cap is the only
19	func agentReportsCapacity(hostID string, vcpus, memMB, diskGB int64) {	19	// thing the agent will refuse a boot against — so it is the setup for every
		20	// refusal below. The metrics are a machine an order of magnitude bigger than
		21	// what it offers, which is what --max-mem-mb/--max-disk-gb produce.
		22	func agentReportsCappedCapacity(hostID string, vcpus, memMB, diskGB int64) {
20	testReg.UpdateReport(hostID, registry.Report{	23	testReg.UpdateReport(hostID, registry.Report{
21	Capacity: registry.Capacity{VCPUs: vcpus, MemMB: memMB, DiskGB: diskGB},	24	Capacity: registry.Capacity{VCPUs: vcpus, MemMB: memMB, DiskGB: diskGB},
		25	Metrics: registry.Metrics{
		26	MemUsedMB: memMB, MemAvailableMB: memMB * 10,
		27	DiskUsedGB: diskGB, DiskFreeGB: diskGB * 10,
		28	},
		29	})
		30	}
		31
		32	// agentReportsUncappedCapacity is the other host: no --max-* flags, so it
		33	// advertises exactly the machine its own metrics describe. Nothing here says
		34	// "limit" — an uncapped agent admits whatever it is asked — so nothing here may
		35	// refuse a create.
		36	func agentReportsUncappedCapacity(hostID string, vcpus, memMB, diskGB int64) {
		37	testReg.UpdateReport(hostID, registry.Report{
		38	Capacity: registry.Capacity{VCPUs: vcpus, MemMB: memMB, DiskGB: diskGB},
		39	Metrics: registry.Metrics{
		40	MemUsedMB: memMB / 4, MemAvailableMB: memMB - memMB/4,
		41	DiskUsedGB: diskGB / 4, DiskFreeGB: diskGB - diskGB/4,
		42	},
22	})	43	})
23	}	44	}
24		45
@@ -40,23 +61,23 @@ func bodyOf(t testing.T, resp http.Response) string {
40	}	61	}
41		62
42	// TestCreateVMRefusesAHostWithNoRoom pins the third refusal of this shape: the	63	// TestCreateVMRefusesAHostWithNoRoom pins the third refusal of this shape: the
43	// request is fine and the host cannot serve it. Each dimension binds on its	64	// request is fine and the host has been told not to serve it. Each dimension
44	// own, and the refusal has to be actionable on its own terms — which dimension,	65	// binds on its own, and the refusal has to be actionable on its own terms —
45	// what the VM asked for, what the host is already holding out of what it has.	66	// which dimension, what the VM asked for, what the host is already holding out
		67	// of what it may hold.
46	func TestCreateVMRefusesAHostWithNoRoom(t *testing.T) {	68	func TestCreateVMRefusesAHostWithNoRoom(t *testing.T) {
47	for _, tc := range []struct {	69	for _, tc := range []struct {
48	name string	70	name string
49	vcpus, memMB, diskGB int64	71	vcpus, memMB, diskGB int64
50	want string	72	want string
51	}{	73	}{
52	{"vcpus", 8, 1024, 10, "vcpus — needs 8, already holds 2 of 6"},
53	{"memory", 1, 8192, 10, "memory — needs 8192MB, already holds 2048 of 4096MB"},	74	{"memory", 1, 8192, 10, "memory — needs 8192MB, already holds 2048 of 4096MB"},
54	{"disk", 1, 1024, 200, "disk — needs 200GB, already holds 10 of 100GB"},	75	{"disk", 1, 1024, 200, "disk — needs 200GB, already holds 10 of 100GB"},
55	} {	76	} {
56	t.Run(tc.name, func(t *testing.T) {	77	t.Run(tc.name, func(t *testing.T) {
57	ts, st, _ := testServer(t)	78	ts, st, _ := testServer(t)
58	out := enroll(t, ts)	79	out := enroll(t, ts)
59	agentReportsCapacity(out["host_id"], 6, 4096, 100)	80	agentReportsCappedCapacity(out["host_id"], 6, 4096, 100)
60		81
61	// One VM already there, so the refusal has a "holds" to report.	82	// One VM already there, so the refusal has a "holds" to report.
62	require.Equal(t, 201, createVM(t, ts, out["host_id"], "sitting", 2, 2048, 10).StatusCode)	83	require.Equal(t, 201, createVM(t, ts, out["host_id"], "sitting", 2, 2048, 10).StatusCode)
@@ -66,6 +87,7 @@ func TestCreateVMRefusesAHostWithNoRoom(t *testing.T) {
66	msg := bodyOf(t, resp)	87	msg := bodyOf(t, resp)
67	assert.Contains(t, msg, tc.want)	88	assert.Contains(t, msg, tc.want)
68	assert.Contains(t, msg, "host host-a ("+out["host_id"]+")", "the refusal must name the host")	89	assert.Contains(t, msg, "host host-a ("+out["host_id"]+")", "the refusal must name the host")
		90	assert.Contains(t, msg, "delete a VM on that host", "live VMs fill it: the remedy is to free one")
69	assert.Contains(t, msg, "create it on a host with room", "the refusal must name a way out")	91	assert.Contains(t, msg, "create it on a host with room", "the refusal must name a way out")
70		92
71	vms, err := st.ListVMs()	93	vms, err := st.ListVMs()
@@ -80,31 +102,68 @@ func TestCreateVMRefusesAHostWithNoRoom(t *testing.T) {
80	func TestCreateVMRefusalNamesEveryBindingDimension(t *testing.T) {	102	func TestCreateVMRefusalNamesEveryBindingDimension(t *testing.T) {
81	ts, _, _ := testServer(t)	103	ts, _, _ := testServer(t)
82	out := enroll(t, ts)	104	out := enroll(t, ts)
83	agentReportsCapacity(out["host_id"], 6, 4096, 100)	105	agentReportsCappedCapacity(out["host_id"], 6, 4096, 100)
84		106
85	resp := createVM(t, ts, out["host_id"], "huge", 8, 8192, 10)	107	resp := createVM(t, ts, out["host_id"], "huge", 1, 8192, 200)
86	require.Equal(t, 409, resp.StatusCode)	108	require.Equal(t, 409, resp.StatusCode)
87	msg := bodyOf(t, resp)	109	msg := bodyOf(t, resp)
88	assert.Contains(t, msg, "vcpus — needs 8, already holds 0 of 6")
89	assert.Contains(t, msg, "memory — needs 8192MB, already holds 0 of 4096MB")	110	assert.Contains(t, msg, "memory — needs 8192MB, already holds 0 of 4096MB")
90	assert.NotContains(t, msg, "disk —", "a dimension that fits is not the caller's problem")	111	assert.Contains(t, msg, "disk — needs 200GB, already holds 0 of 100GB")
91	}	112	}
92		113
93	// TestCreateVMFillsAHostExactly is the boundary: capacity is a limit, not a	114	// TestCreateVMFillsAHostExactly is the boundary: a declared capacity is a
94	// threshold. A VM that fits the last of the host is placed, and the next one —	115	// limit, not a threshold. A VM that fits the last of the host is placed, and
95	// however small — is refused, because there is nothing left.	116	// the next one — however small — is refused, because there is nothing left.
96	func TestCreateVMFillsAHostExactly(t *testing.T) {	117	func TestCreateVMFillsAHostExactly(t *testing.T) {
97	ts, _, _ := testServer(t)	118	ts, _, _ := testServer(t)
98	out := enroll(t, ts)	119	out := enroll(t, ts)
99	agentReportsCapacity(out["host_id"], 4, 4096, 40)	120	agentReportsCappedCapacity(out["host_id"], 4, 4096, 40)
100		121
101	require.Equal(t, 201, createVM(t, ts, out["host_id"], "half", 2, 2048, 20).StatusCode)	122	require.Equal(t, 201, createVM(t, ts, out["host_id"], "half", 1, 2048, 20).StatusCode)
102	require.Equal(t, 201, createVM(t, ts, out["host_id"], "rest", 2, 2048, 20).StatusCode,	123	require.Equal(t, 201, createVM(t, ts, out["host_id"], "rest", 1, 2048, 20).StatusCode,
103	"a VM that exactly fills the host still fits")	124	"a VM that exactly fills the host still fits")
104		125
105	resp := createVM(t, ts, out["host_id"], "onemore", 1, 1, 1)	126	resp := createVM(t, ts, out["host_id"], "onemore", 1, 1, 1)
106	require.Equal(t, 409, resp.StatusCode)	127	require.Equal(t, 409, resp.StatusCode)
107	assert.Contains(t, bodyOf(t, resp), "vcpus — needs 1, already holds 4 of 4")	128	assert.Contains(t, bodyOf(t, resp), "memory — needs 1MB, already holds 4096 of 4096MB")
		129	}
		130
		131	// TestCreateVMNeverRefusesAnUncappedHost is the rule this whole preflight is
		132	// bounded by: the agent's admission is the authority, and an agent with no
		133	// --max-* flag admits whatever it is asked. A host that advertises the machine
		134	// it measures has declared no limit, so its totals are not a wall — guest disks
		135	// are sparse and guest memory is not preallocated, and running well past the
		136	// paper numbers is what such a host is for. A courtesy check that refused here
		137	// would block VMs the fleet is designed to run, and no operator could see why
		138	// without an ssh session.
		139	func TestCreateVMNeverRefusesAnUncappedHost(t *testing.T) {
		140	ts, st, _ := testServer(t)
		141	out := enroll(t, ts)
		142	agentReportsUncappedCapacity(out["host_id"], 4, 4096, 100)
		143
		144	require.Equal(t, 201, createVM(t, ts, out["host_id"], "big", 8, 8192, 400).StatusCode,
		145	"one VM may exceed the whole machine")
		146	require.Equal(t, 201, createVM(t, ts, out["host_id"], "bigger", 8, 8192, 400).StatusCode,
		147	"and so may the next one")
		148
		149	vms, err := st.ListVMs()
		150	require.NoError(t, err)
		151	assert.Len(t, vms, 2)
		152	}
		153
		154	// TestCreateVMNeverJudgesVCPUs is the honest edge of the proof. A cap is proven
		155	// by an advertisement below the machine's own measurements, and no report
		156	// carries the machine's core count — so a --max-vcpus cap is invisible here and
		157	// a vCPU overflow is never refused, even on a host proven capped in memory. The
		158	// agent still enforces it at boot; this server just cannot say so first, and
		159	// guessing would refuse every oversubscribed VM on the fleet.
		160	func TestCreateVMNeverJudgesVCPUs(t *testing.T) {
		161	ts, _, _ := testServer(t)
		162	out := enroll(t, ts)
		163	agentReportsCappedCapacity(out["host_id"], 2, 4096, 100)
		164
		165	resp := createVM(t, ts, out["host_id"], "manycpus", 64, 1024, 10)
		166	assert.Equal(t, 201, resp.StatusCode)
108	}	167	}
109		168
110	// TestCreateVMJudgesOnlyAHostThatHasReportedCapacity: reported capacity is	169	// TestCreateVMJudgesOnlyAHostThatHasReportedCapacity: reported capacity is
@@ -140,6 +199,17 @@ func TestCreateVMJudgesOnlyAHostThatHasReportedCapacity(t *testing.T) {
140	vms, _ := dbst.ListVMs()	199	vms, _ := dbst.ListVMs()
141	assert.Len(t, vms, 1)	200	assert.Len(t, vms, 1)
142	})	201	})
		202
		203	t.Run("capacity but no metrics to measure it against", func(t *testing.T) {
		204	ts, dbst, _ := testServer(t)
		205	out := enroll(t, ts)
		206	testReg.UpdateReport(out["host_id"], registry.Report{
		207	Capacity: registry.Capacity{VCPUs: 4, MemMB: 4096, DiskGB: 40}})
		208	require.Equal(t, 201, createVM(t, ts, out["host_id"], "unjudged", 8, 99999, 999).StatusCode,
		209	"without the machine's size, nothing proves that capacity is a limit")
		210	vms, _ := dbst.ListVMs()
		211	assert.Len(t, vms, 1)
		212	})
143	}	213	}
144		214
145	// TestCreateVMCountsATombstonedVMUntilItIsReaped is the predicate the whole	215	// TestCreateVMCountsATombstonedVMUntilItIsReaped is the predicate the whole
@@ -147,24 +217,59 @@ func TestCreateVMJudgesOnlyAHostThatHasReportedCapacity(t *testing.T) {
147	// the host and the guest may still be shutting down, so its resources are not	217	// the host and the guest may still be shutting down, so its resources are not
148	// free to promise to somebody else. Only the reap — the agent's ack, or the	218	// free to promise to somebody else. Only the reap — the agent's ack, or the
149	// abandoned sweep — frees them, and it frees them by removing the row.	219	// abandoned sweep — frees them, and it frees them by removing the row.
		220	//
		221	// It is also the case where the obvious remedy is a lie. The caller is being
		222	// refused BY the VM they just deleted, so "delete a VM" would send them looking
		223	// for one that is already gone; the refusal has to name the teardown and say to
		224	// wait for it, because that wait ends by itself.
150	func TestCreateVMCountsATombstonedVMUntilItIsReaped(t *testing.T) {	225	func TestCreateVMCountsATombstonedVMUntilItIsReaped(t *testing.T) {
151	ts, dbst, _ := testServer(t)	226	ts, dbst, _ := testServer(t)
152	out := enroll(t, ts)	227	out := enroll(t, ts)
153	agentReportsCapacity(out["host_id"], 4, 4096, 40)	228	agentReportsCappedCapacity(out["host_id"], 4, 4096, 40)
154		229
155	resp := createVM(t, ts, out["host_id"], "first", 4, 4096, 40)	230	resp := createVM(t, ts, out["host_id"], "first", 1, 4096, 40)
156	require.Equal(t, 201, resp.StatusCode)	231	require.Equal(t, 201, resp.StatusCode)
157	var created map[string]string	232	var created map[string]string
158	require.NoError(t, json.NewDecoder(resp.Body).Decode(&created))	233	require.NoError(t, json.NewDecoder(resp.Body).Decode(&created))
159		234
160	require.Equal(t, 204, do(t, "DELETE", ts.URL+"/api/v1/vms/"+created["id"], testPAT, nil).StatusCode)	235	require.Equal(t, 204, do(t, "DELETE", ts.URL+"/api/v1/vms/"+created["id"], testPAT, nil).StatusCode)
161		236
162	refused := createVM(t, ts, out["host_id"], "second", 4, 4096, 40)	237	refused := createVM(t, ts, out["host_id"], "second", 1, 4096, 40)
163	require.Equal(t, 409, refused.StatusCode, "a deleting VM still occupies the host")	238	require.Equal(t, 409, refused.StatusCode, "a deleting VM still occupies the host")
164	assert.Contains(t, bodyOf(t, refused), "already holds 4 of 4")	239	msg := bodyOf(t, refused)
		240	assert.Contains(t, msg, "memory — needs 4096MB, already holds 4096 of 4096MB, 4096MB of it by the teardown")
		241	assert.Contains(t, msg, "1 VM deleted from that host is still being torn down",
		242	"the refusal must name how many teardowns are holding the room")
		243	assert.Contains(t, msg, "Wait for the teardown to finish", "the wait is the remedy")
		244	assert.NotContains(t, msg, "delete a VM on that host",
		245	"the caller already deleted one; sending them to delete another is the lie")
165		246
166	// The agent acks the destroy: the row goes, and with it the commitment.	247	// The agent acks the destroy: the row goes, and with it the commitment.
167	require.NoError(t, dbst.HardDeleteVM(created["id"]))	248	require.NoError(t, dbst.HardDeleteVM(created["id"]))
168	assert.Equal(t, 201, createVM(t, ts, out["host_id"], "second", 4, 4096, 40).StatusCode,	249	assert.Equal(t, 201, createVM(t, ts, out["host_id"], "second", 1, 4096, 40).StatusCode,
169	"a reaped VM holds nothing")	250	"a reaped VM holds nothing")
170	}	251	}
		252
		253	// TestCreateVMKeepsTheDeleteRemedyWhenLiveVMsFillTheHost is the other half of
		254	// the pair: a teardown in flight does not soften a host that is full without
		255	// it. The VMs holding the room are really there, so the remedy is really to
		256	// free one.
		257	func TestCreateVMKeepsTheDeleteRemedyWhenLiveVMsFillTheHost(t *testing.T) {
		258	ts, _, _ := testServer(t)
		259	out := enroll(t, ts)
		260	agentReportsCappedCapacity(out["host_id"], 8, 8192, 80)
		261
		262	require.Equal(t, 201, createVM(t, ts, out["host_id"], "living", 1, 6144, 60).StatusCode)
		263	dying := createVM(t, ts, out["host_id"], "dying", 1, 2048, 20)
		264	require.Equal(t, 201, dying.StatusCode)
		265	var created map[string]string
		266	require.NoError(t, json.NewDecoder(dying.Body).Decode(&created))
		267	require.Equal(t, 204, do(t, "DELETE", ts.URL+"/api/v1/vms/"+created["id"], testPAT, nil).StatusCode)
		268
		269	refused := createVM(t, ts, out["host_id"], "next", 1, 4096, 40)
		270	require.Equal(t, 409, refused.StatusCode)
		271	msg := bodyOf(t, refused)
		272	assert.Contains(t, msg, "delete a VM on that host",
		273	"the live VMs alone leave no room: waiting for the teardown will not help")
		274	assert.NotContains(t, msg, "still being torn down")
		275	}

internal/server/store/allocation_test.go

Old		New
@@ -72,16 +72,35 @@ func TestCommittedOnHostCountsTombstonesUntilTheyAreReaped(t *testing.T) {
72		72
73	held, err := s.CommittedOnHost(h.ID)	73	held, err := s.CommittedOnHost(h.ID)
74	require.NoError(t, err)	74	require.NoError(t, err)
75	assert.Equal(t, Alloc{VCPUs: 6, MemMB: 6144, DiskGB: 30}, held,	75	assert.Equal(t, Alloc{VCPUs: 6, MemMB: 6144, DiskGB: 30}, held.Held(),
76	"a VM being destroyed still occupies the host")	76	"a VM being destroyed still occupies the host")
77		77
78	require.NoError(t, s.HardDeleteVM(dying.ID))	78	require.NoError(t, s.HardDeleteVM(dying.ID))
79	held, err = s.CommittedOnHost(h.ID)	79	held, err = s.CommittedOnHost(h.ID)
80	require.NoError(t, err)	80	require.NoError(t, err)
81	assert.Equal(t, Alloc{VCPUs: 2, MemMB: 2048, DiskGB: 10}, held,	81	assert.Equal(t, Alloc{VCPUs: 2, MemMB: 2048, DiskGB: 10}, held.Held(),
82	"the reap is what frees it")	82	"the reap is what frees it")
83	}	83	}
84		84
		85	// TestCommittedOnHostSplitsWhatIsRunningFromWhatIsLeaving is the half of the
		86	// commitment a refusal has to be able to name. Both halves are held, and only
		87	// one of them can be freed by deleting something — the other frees itself.
		88	func TestCommittedOnHostSplitsWhatIsRunningFromWhatIsLeaving(t *testing.T) {
		89	s := newStore(t)
		90	h := enrollHost(t, s)
		91	vmWithResources(t, s, h, "a", 2, 2048, 10)
		92	for _, name := range []string{"b", "c"} {
		93	dying := vmWithResources(t, s, h, name, 4, 4096, 20)
		94	require.NoError(t, s.TombstoneVM(dying.ID))
		95	}
		96
		97	held, err := s.CommittedOnHost(h.ID)
		98	require.NoError(t, err)
		99	assert.Equal(t, Alloc{VCPUs: 2, MemMB: 2048, DiskGB: 10}, held.Live)
		100	assert.Equal(t, Alloc{VCPUs: 8, MemMB: 8192, DiskGB: 40}, held.Pending)
		101	assert.Equal(t, 2, held.PendingVMs, "the count is what a refusal names")
		102	}
		103
85	// TestCommittedOnHostSeparatesHosts: one host's commitment says nothing about	104	// TestCommittedOnHostSeparatesHosts: one host's commitment says nothing about
86	// another's, and a host with nothing on it holds nothing rather than erroring.	105	// another's, and a host with nothing on it holds nothing rather than erroring.
87	func TestCommittedOnHostSeparatesHosts(t *testing.T) {	106	func TestCommittedOnHostSeparatesHosts(t *testing.T) {
@@ -91,7 +110,7 @@ func TestCommittedOnHostSeparatesHosts(t *testing.T) {
91		110
92	held, err := s.CommittedOnHost("no-such-host")	111	held, err := s.CommittedOnHost("no-such-host")
93	require.NoError(t, err)	112	require.NoError(t, err)
94	assert.Equal(t, Alloc{}, held)	113	assert.Equal(t, Commitment{}, held)
95	}	114	}
96		115
97	func (s Store) mustAllocated(t testing.T) map[string]Alloc {	116	func (s Store) mustAllocated(t testing.T) map[string]Alloc {

internal/server/store/store.go

Old		New
@@ -978,6 +978,27 @@ func allocatedByHost(q querier) (map[string]Alloc, error) {
978	return out, rows.Err()	978	return out, rows.Err()
979	}	979	}
980		980
		981	// Commitment is everything a host is holding, split by why it is holding it.
		982	// Live is the VMs it is meant to be running; Pending is the tombstoned rows —
		983	// deletes it has been asked for and not yet finished — and PendingVMs is how
		984	// many rows those are. Held() is the sum, which is the number a placement has
		985	// to beat; the split exists so a refusal can say which half is binding, because
		986	// the two have different remedies (delete something, or wait).
		987	type Commitment struct {
		988	Live Alloc
		989	Pending Alloc
		990	PendingVMs int
		991	}
		992
		993	// Held is what the host is holding in total: live VMs plus destroys in flight.
		994	func (c Commitment) Held() Alloc {
		995	return Alloc{
		996	VCPUs: c.Live.VCPUs + c.Pending.VCPUs,
		997	MemMB: c.Live.MemMB + c.Pending.MemMB,
		998	DiskGB: c.Live.DiskGB + c.Pending.DiskGB,
		999	}
		1000	}
		1001
981	// CommittedOnHost sums the specs of every VM row still on a host — what the	1002	// CommittedOnHost sums the specs of every VM row still on a host — what the
982	// machine is holding, which is the number a placement decision has to beat.	1003	// machine is holding, which is the number a placement decision has to beat.
983	//	1004	//
@@ -989,16 +1010,31 @@ func allocatedByHost(q querier) (map[string]Alloc, error) {
989	// may still be shutting down, so its resources are not free to promise to	1010	// may still be shutting down, so its resources are not free to promise to
990	// somebody else. Reaped VMs need no predicate — their rows are gone.	1011	// somebody else. Reaped VMs need no predicate — their rows are gone.
991	//	1012	//
		1013	// The two halves come back separately because counting a tombstone is right and
		1014	// blaming the caller for it is not: a create refused by resources a delete has
		1015	// not finished releasing has a different answer ("wait") than one refused by
		1016	// VMs that are actually there ("delete something"), and only the split can tell
		1017	// them apart.
		1018	//
992	// allocatedByHost, which feeds the console's "allocated", counts live rows	1019	// allocatedByHost, which feeds the console's "allocated", counts live rows
993	// only, so during a teardown this reads higher. Erring that way is the safe	1020	// only, so during a teardown Held() reads higher. Erring that way is the safe
994	// direction for admission: refusing a VM for a bed that is being stripped costs	1021	// direction for admission: refusing a VM for a bed that is being stripped costs
995	// a retry, admitting one into it costs a failed VM.	1022	// a retry, admitting one into it costs a failed VM.
996	func (s *Store) CommittedOnHost(hostID string) (Alloc, error) {	1023	func (s *Store) CommittedOnHost(hostID string) (Commitment, error) {
997	var a Alloc	1024	var c Commitment
998	err := s.db.QueryRow(`	1025	err := s.db.QueryRow(`
999	SELECT COALESCE(SUM(vcpus),0), COALESCE(SUM(mem_mb),0), COALESCE(SUM(disk_gb),0)	1026	SELECT COALESCE(SUM(CASE WHEN deleted_at IS NULL THEN vcpus END),0),
1000	FROM vms WHERE host_id=?`, hostID).Scan(&a.VCPUs, &a.MemMB, &a.DiskGB)	1027	COALESCE(SUM(CASE WHEN deleted_at IS NULL THEN mem_mb END),0),
1001	return a, err	1028	COALESCE(SUM(CASE WHEN deleted_at IS NULL THEN disk_gb END),0),
		1029	COALESCE(SUM(CASE WHEN deleted_at IS NOT NULL THEN vcpus END),0),
		1030	COALESCE(SUM(CASE WHEN deleted_at IS NOT NULL THEN mem_mb END),0),
		1031	COALESCE(SUM(CASE WHEN deleted_at IS NOT NULL THEN disk_gb END),0),
		1032	COUNT(deleted_at)
		1033	FROM vms WHERE host_id=?`, hostID).Scan(
		1034	&c.Live.VCPUs, &c.Live.MemMB, &c.Live.DiskGB,
		1035	&c.Pending.VCPUs, &c.Pending.MemMB, &c.Pending.DiskGB,
		1036	&c.PendingVMs)
		1037	return c, err
1002	}	1038	}
1003		1039
1004	// AuditEntry is one row of the append-only audit trail.	1040	// AuditEntry is one row of the append-only audit trail.

web/src/lib/fleet.svelte.ts

Old		New
@@ -714,11 +714,14 @@ export type CapacityReading = {
714	* gauge needs. Past 90% the level turns, the same wall the fleet table's load	714	* gauge needs. Past 90% the level turns, the same wall the fleet table's load
715	* meter marks.	715	* meter marks.
716	*	716	*
717	* Over-allocation is a state the console has to be able to draw. Creating a VM	717	* Over-allocation is a state the console has to be able to draw, and a normal
718	* a host has no room for is refused now, but a host that re-reports a smaller	718	* one. A host that declares no cap advertises the machine's totals and its
719	* capacity (a disk shrank, an operator lowered the agent's cap) puts VMs that	719	* agent admits past them on purpose — sparse disks, memory that is not
720	* were placed honestly over the line, as do rows made before the refusal	720	* preallocated — so allocation over capacity is what deliberate overcommit
721	* existed. Rendering that as "-2 free" states it as a negative amount of room,	721	* looks like from here. A capped host is placed against its cap, but one that
		722	* re-reports a smaller capacity (a disk shrank, an operator lowered the
		723	* agent's cap) puts VMs that were placed honestly over the line just the same.
		724	* Rendering that as "-2 free" states it as a negative amount of room,
722	* which is not a thing; "over by 2" is the same number said truthfully.	725	* which is not a thing; "over by 2" is the same number said truthfully.
723	*	726	*
724	* A total of 0 is a host that has reported no capacity, not a host with none —	727	* A total of 0 is a host that has reported no capacity, not a host with none —

Old		New
@@ -149,8 +149,9 @@ func TestVMCreateSurfacesThePreCSRRefusal(t *testing.T) {
149	// remedy. A model told only "409" would retry the same too-large VM; one told	149	// remedy. A model told only "409" would retry the same too-large VM; one told
150	// which dimension bound, and by how much, can ask for a size that fits.	150	// which dimension bound, and by how much, can ask for a size that fits.
151	func TestVMCreateSurfacesTheCapacityRefusal(t *testing.T) {	151	func TestVMCreateSurfacesTheCapacityRefusal(t *testing.T) {
152	const refusal = "host mewtwo (h1) has no room for this VM: vcpus — needs 8, already holds 4 of 6. " +	152	const refusal = "host mewtwo (h1) has no room for this VM: memory — needs 8192MB, already holds 4096 of 6144MB. " +
153	"A host runs no more than it reports. Ask for less, delete a VM on that host to free what it holds, " +	153	"That is a limit its operator set below what the machine has, so the agent there would refuse this VM " +
		154	"at boot and it would sit failed. Ask for less, delete a VM on that host to free what it holds, " +
154	"or create it on a host with room."	155	"or create it on a host with room."
155	c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {	156	c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {
156	if r.URL.Path == "/api/v1/hosts" {	157	if r.URL.Path == "/api/v1/hosts" {
@@ -162,10 +163,10 @@ func TestVMCreateSurfacesTheCapacityRefusal(t *testing.T) {
162	http.Error(w, refusal, http.StatusConflict)	163	http.Error(w, refusal, http.StatusConflict)
163	})	164	})
164	tools := &Tools{API: c}	165	tools := &Tools{API: c}
165	_, err := tools.VMCreate(t.Context(), VMCreateIn{Host: "mewtwo", VCPUs: 8})	166	_, err := tools.VMCreate(t.Context(), VMCreateIn{Host: "mewtwo", MemMB: 8192})
166	require.Error(t, err)	167	require.Error(t, err)
167	assert.Contains(t, err.Error(), "has no room for this VM")	168	assert.Contains(t, err.Error(), "has no room for this VM")
168	assert.Contains(t, err.Error(), "vcpus — needs 8, already holds 4 of 6")	169	assert.Contains(t, err.Error(), "memory — needs 8192MB, already holds 4096 of 6144MB")
169	assert.Contains(t, err.Error(), "create it on a host with room")	170	assert.Contains(t, err.Error(), "create it on a host with room")
170	}	171	}
171		172