a73x

55e7bb7f

feat(server): a full host is not a place to put one

a73x   2026-08-11 10:38

Commit message
feat(server): a full host is not a place to put one

Create has always admitted any placement as desired state, and the host has
always been the one to find out. An agent that cannot fit a VM refuses it at
materialization — "host capacity limit reached: needs 8 vcpus, host cap 6" —
so the caller waits through an image download to be told something that was
knowable the instant they asked, and is left with a failed VM rather than a
refused request. It cost the fleet a ten-minute smoke timeout twice.

So the create answers it, as the third refusal of a shape this handler already
carries twice: the certified-host-key 409 and the tenant-no-CA 409. All three
say the same thing — the request is fine, this host cannot serve it — and this
one says which dimension bound, what the VM asked for, and what the host is
already holding out of what it has, because with capacity the numbers are the
remedy. Every binding dimension is named at once: a VM too big in two ways
should be resized once. The words reach an MCP caller intact, like its
siblings', since the client carries the control plane's own words out.

Only a host that has spoken is judged, the rule the certified-host-key refusal
arrived at and for the same reason. Capacity is registry state, filled by agent
reports, so a host that is offline, or has never reported, or reports a zero
where a probe failed has said nothing about the size of the machine — and a
zero read as "no room" would refuse every create on the fleet for the moment
after a server restart. Such a host takes the create as desired state. The
agent's own admission check is untouched: it remains the authority for the
races this cannot close — two creates in flight, a capacity re-reported smaller
between check and boot — and the only judge a host with no control plane has.

What counts as held is every VM row still on the host, tombstones included.
A tombstone is a destroy in progress, not a destroy: the disk is there and the
guest may still be shutting down until the agent acks and the row is reaped.
Reaped VMs need no predicate, their rows are gone. That reads higher than the
console's "allocated", which counts live rows only, and erring that way is the
safe direction — refusing a VM for a bed being stripped costs a retry,
admitting one into it costs a VM.

The console stops printing "-2 free". Free clamps at zero and over-allocation
states itself as what it is ("8 / 6 · over by 2"), bold like a host past 1.0
load, in the panel bars and in the fleet table's allocation columns. New
creates are refused, but over-allocation still happens without anyone being at
fault: a host that re-reports a smaller machine puts VMs placed honestly over
the line, as do rows made before this existed. The host page's Resources note
said vCPU was commonly oversubscribed while the agent was already enforcing it
as a cap; it now says all three are limits at placement, which is true.

One unrelated line goes with it: the console test fixture still set the
`persistent` field that v0.0.6 dropped, which `npm run check` has been red
about since. web-test alone did not catch it.
internal/mcpserver/api_test.go
Old New
@@ -145,6 +145,30 @@ func TestVMCreateSurfacesThePreCSRRefusal(t *testing.T) {
145 assert.Contains(t, err.Error(), "Upgrade that host's agent") 145 assert.Contains(t, err.Error(), "Upgrade that host's agent")
146 } 146 }
147 147
148 // The same is true of the capacity refusal, and more so: the numbers ARE the
149 // remedy. A model told only "409" would retry the same too-large VM; one told
150 // which dimension bound, and by how much, can ask for a size that fits.
151 func TestVMCreateSurfacesTheCapacityRefusal(t *testing.T) {
152 const refusal = "host mewtwo (h1) has no room for this VM: vcpus — needs 8, already holds 4 of 6. " +
153 "A host runs no more than it reports. Ask for less, delete a VM on that host to free what it holds, " +
154 "or create it on a host with room."
155 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {
156 if r.URL.Path == "/api/v1/hosts" {
157 json.NewEncoder(w).Encode([]map[string]any{
158 {"id": "h1", "name": "mewtwo", "online": true, "agent_version": "v0.0.5"},
159 })
160 return
161 }
162 http.Error(w, refusal, http.StatusConflict)
163 })
164 tools := &Tools{API: c}
165 _, err := tools.VMCreate(t.Context(), VMCreateIn{Host: "mewtwo", VCPUs: 8})
166 require.Error(t, err)
167 assert.Contains(t, err.Error(), "has no room for this VM")
168 assert.Contains(t, err.Error(), "vcpus — needs 8, already holds 4 of 6")
169 assert.Contains(t, err.Error(), "create it on a host with room")
170 }
171
148 func TestAPIErrorSurfacesBodyNotToken(t *testing.T) { 172 func TestAPIErrorSurfacesBodyNotToken(t *testing.T) {
149 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) { 173 c := fakeAPI(t, func(w http.ResponseWriter, r *http.Request) {
150 http.Error(w, "invalid name", http.StatusBadRequest) 174 http.Error(w, "invalid name", http.StatusBadRequest)
internal/server/api/api.go
Old New
@@ -829,11 +829,39 @@ func (a *API) handleCreateVM(w http.ResponseWriter, r *http.Request) {
829 // serve one right now: the row is desired state, the agent that materializes 829 // serve one right now: the row is desired state, the agent that materializes
830 // it may well be a newer one, and vmssh still refuses the dial if the guest 830 // it may well be a newer one, and vmssh still refuses the dial if the guest
831 // it eventually boots has no certificate. 831 // it eventually boots has no certificate.
832 if st, ok := a.reg.Get(req.HostID); ok && st.Online && !release.CertifiesGuestHostKeys(st.AgentVersion) { 832 hostState, hostHasSpoken := a.reg.Get(req.HostID)
833 http.Error(w, precsrRefusal(host.Name, req.HostID, st.AgentVersion, a.URL(upgradeAgentPath(req.HostID))), http.StatusConflict) 833 hostHasSpoken = hostHasSpoken && hostState.Online
834 if hostHasSpoken && !release.CertifiesGuestHostKeys(hostState.AgentVersion) {
835 http.Error(w, precsrRefusal(host.Name, req.HostID, hostState.AgentVersion, a.URL(upgradeAgentPath(req.HostID))), http.StatusConflict)
834 return 836 return
835 } 837 }
836 838
839 // Capacity precondition, the third refusal of this same shape: the request
840 // is fine, and the host cannot serve it. A host that is already holding as
841 // much as it says it has cannot boot one more guest — the agent refuses it
842 // at materialization ("host capacity limit reached", see reconcile.Engine)
843 // and the VM ends up failed, minutes later, with an answer that existed
844 // here. So the create answers 409 now, naming the dimension and the numbers.
845 //
846 // Judged only on a host that has spoken, exactly like the refusal above and
847 // for the same reason: reported capacity is registry state, so an offline or
848 // never-reporting host has zeroes that say nothing about the machine. Such a
849 // host takes the create as desired state, and the agent's own admission
850 // check — which is anyway the authority for the races this cannot close —
851 // remains the backstop.
852 if hostHasSpoken {
853 held, err := a.st.CommittedOnHost(req.HostID)
854 if err != nil {
855 http.Error(w, "internal error", http.StatusInternalServerError)
856 return
857 }
858 want := store.Alloc{VCPUs: req.VCPUs, MemMB: req.MemMB, DiskGB: req.DiskGB}
859 if msg := overCapacityRefusal(host.Name, req.HostID, want, held, hostState.Capacity); msg != "" {
860 http.Error(w, msg, http.StatusConflict)
861 return
862 }
863 }
864
837 // Install the SSH key into user-supplied cloud-init. When only one of the 865 // Install the SSH key into user-supplied cloud-init. When only one of the
838 // two is set the seed builder handles it (verbatim user-data, or the 866 // two is set the seed builder handles it (verbatim user-data, or the
839 // generated default template); it's the BOTH case that used to silently 867 // generated default template); it's the BOTH case that used to silently
internal/server/api/capacity.go
Old New
@@ -0,0 +1,46 @@
1 package api
2
3 import (
4 "fmt"
5 "strings"
6
7 "github.com/a73x/eitri/internal/server/registry"
8 "github.com/a73x/eitri/internal/server/store"
9 )
10
11 // overCapacityRefusal explains why a host cannot be given a VM: it is already
12 // holding as much as it says it has. It returns "" when the VM fits.
13 //
14 // The answer exists at create — the host's capacity and everything already on
15 // it are both known here — so the caller learns it in milliseconds instead of
16 // watching a VM sit `creating` and then `failed` because the agent reached the
17 // same conclusion minutes later, at the far end of an image download. The
18 // agent's own admission check stays where it is: it is the authority for the
19 // race this one cannot close (two creates in flight, capacity re-reported
20 // smaller between check and boot) and the only judge a serverless host has.
21 //
22 // Every dimension the host has spoken about is judged, and each binding one is
23 // named: a VM that is too big in two ways should be resized once, not twice. A
24 // dimension whose reported capacity is 0 is skipped — an agent reports totals,
25 // and a zero is a failed probe (or a report that predates the field), not a
26 // host with no memory. Judging on it would refuse every VM on the fleet.
27 func overCapacityRefusal(hostName, hostID string, want, held store.Alloc, cap registry.Capacity) string {
28 var over []string
29 dim := func(noun, unit string, want, held, capacity int64) {
30 if capacity <= 0 || held+want <= capacity {
31 return
32 }
33 over = append(over, fmt.Sprintf("%s — needs %d%s, already holds %d of %d%s",
34 noun, want, unit, held, capacity, unit))
35 }
36 dim("vcpus", "", want.VCPUs, held.VCPUs, cap.VCPUs)
37 dim("memory", "MB", want.MemMB, held.MemMB, cap.MemMB)
38 dim("disk", "GB", want.DiskGB, held.DiskGB, cap.DiskGB)
39 if len(over) == 0 {
40 return ""
41 }
42 return fmt.Sprintf("host %s (%s) has no room for this VM: %s. A host runs no more than it reports, so the "+
43 "agent there would refuse this VM at boot and it would sit failed. Ask for less, delete a VM on that "+
44 "host to free what it holds, or create it on a host with room.",
45 hostName, hostID, strings.Join(over, "; "))
46 }
internal/server/api/capacity_api_test.go
Old New
@@ -0,0 +1,170 @@
1 package api
2
3 import (
4 "encoding/json"
5 "io"
6 "net/http"
7 "net/http/httptest"
8 "testing"
9
10 "github.com/stretchr/testify/assert"
11 "github.com/stretchr/testify/require"
12
13 "github.com/a73x/eitri/internal/server/registry"
14 )
15
16 // agentReportsCapacity is the report of a host that has told this server what
17 // machine it is: the create path judges placement only on a host that has
18 // spoken, and this is what speaking sounds like.
19 func agentReportsCapacity(hostID string, vcpus, memMB, diskGB int64) {
20 testReg.UpdateReport(hostID, registry.Report{
21 Capacity: registry.Capacity{VCPUs: vcpus, MemMB: memMB, DiskGB: diskGB},
22 })
23 }
24
25 // createVM posts one create and returns the response, so a test reads as the
26 // sequence of placements it is really making.
27 func createVM(t *testing.T, ts *httptest.Server, hostID, name string, vcpus, memMB, diskGB int64) *http.Response {
28 t.Helper()
29 return do(t, "POST", ts.URL+"/api/v1/vms", testPAT, map[string]any{
30 "host_id": hostID, "name": name,
31 "vcpus": vcpus, "mem_mb": memMB, "disk_gb": diskGB,
32 })
33 }
34
35 func bodyOf(t *testing.T, resp *http.Response) string {
36 t.Helper()
37 b, err := io.ReadAll(resp.Body)
38 require.NoError(t, err)
39 return string(b)
40 }
41
42 // TestCreateVMRefusesAHostWithNoRoom pins the third refusal of this shape: the
43 // request is fine and the host cannot serve it. Each dimension binds on its
44 // own, and the refusal has to be actionable on its own terms — which dimension,
45 // what the VM asked for, what the host is already holding out of what it has.
46 func TestCreateVMRefusesAHostWithNoRoom(t *testing.T) {
47 for _, tc := range []struct {
48 name string
49 vcpus, memMB, diskGB int64
50 want string
51 }{
52 {"vcpus", 8, 1024, 10, "vcpus — needs 8, already holds 2 of 6"},
53 {"memory", 1, 8192, 10, "memory — needs 8192MB, already holds 2048 of 4096MB"},
54 {"disk", 1, 1024, 200, "disk — needs 200GB, already holds 10 of 100GB"},
55 } {
56 t.Run(tc.name, func(t *testing.T) {
57 ts, st, _ := testServer(t)
58 out := enroll(t, ts)
59 agentReportsCapacity(out["host_id"], 6, 4096, 100)
60
61 // One VM already there, so the refusal has a "holds" to report.
62 require.Equal(t, 201, createVM(t, ts, out["host_id"], "sitting", 2, 2048, 10).StatusCode)
63
64 resp := createVM(t, ts, out["host_id"], "toobig", tc.vcpus, tc.memMB, tc.diskGB)
65 require.Equal(t, 409, resp.StatusCode)
66 msg := bodyOf(t, resp)
67 assert.Contains(t, msg, tc.want)
68 assert.Contains(t, msg, "host host-a ("+out["host_id"]+")", "the refusal must name the host")
69 assert.Contains(t, msg, "create it on a host with room", "the refusal must name a way out")
70
71 vms, err := st.ListVMs()
72 require.NoError(t, err)
73 require.Len(t, vms, 1, "a refused create must leave no row behind")
74 })
75 }
76 }
77
78 // TestCreateVMRefusalNamesEveryBindingDimension: a VM too big in two ways
79 // should be resized once, not discovered twice.
80 func TestCreateVMRefusalNamesEveryBindingDimension(t *testing.T) {
81 ts, _, _ := testServer(t)
82 out := enroll(t, ts)
83 agentReportsCapacity(out["host_id"], 6, 4096, 100)
84
85 resp := createVM(t, ts, out["host_id"], "huge", 8, 8192, 10)
86 require.Equal(t, 409, resp.StatusCode)
87 msg := bodyOf(t, resp)
88 assert.Contains(t, msg, "vcpus — needs 8, already holds 0 of 6")
89 assert.Contains(t, msg, "memory — needs 8192MB, already holds 0 of 4096MB")
90 assert.NotContains(t, msg, "disk —", "a dimension that fits is not the caller's problem")
91 }
92
93 // TestCreateVMFillsAHostExactly is the boundary: capacity is a limit, not a
94 // threshold. A VM that fits the last of the host is placed, and the next one —
95 // however small — is refused, because there is nothing left.
96 func TestCreateVMFillsAHostExactly(t *testing.T) {
97 ts, _, _ := testServer(t)
98 out := enroll(t, ts)
99 agentReportsCapacity(out["host_id"], 4, 4096, 40)
100
101 require.Equal(t, 201, createVM(t, ts, out["host_id"], "half", 2, 2048, 20).StatusCode)
102 require.Equal(t, 201, createVM(t, ts, out["host_id"], "rest", 2, 2048, 20).StatusCode,
103 "a VM that exactly fills the host still fits")
104
105 resp := createVM(t, ts, out["host_id"], "onemore", 1, 1, 1)
106 require.Equal(t, 409, resp.StatusCode)
107 assert.Contains(t, bodyOf(t, resp), "vcpus — needs 1, already holds 4 of 4")
108 }
109
110 // TestCreateVMJudgesOnlyAHostThatHasReportedCapacity: reported capacity is
111 // registry state. A host that has never connected, one that has said hello and
112 // nothing since, and one whose report carries no capacity at all are all hosts
113 // this server knows nothing about the size of — and a zero read as "no room"
114 // would refuse every create on the fleet for the moment after a restart. All
115 // three take the create as desired state; the agent's own admission check is
116 // the backstop.
117 func TestCreateVMJudgesOnlyAHostThatHasReportedCapacity(t *testing.T) {
118 t.Run("never connected", func(t *testing.T) {
119 ts, st, _ := testServer(t)
120 out := enrollSilent(t, ts)
121 require.Equal(t, 201, createVM(t, ts, out["host_id"], "pending", 64, 999999, 9999).StatusCode)
122 vms, _ := st.ListVMs()
123 assert.Len(t, vms, 1, "the create must land as desired state")
124 })
125
126 t.Run("said hello, never reported", func(t *testing.T) {
127 ts, dbst, _ := testServer(t)
128 out := enroll(t, ts) // a Hello and nothing since: the registry knows the
129 // agent's version and not one thing about the machine
130 require.Equal(t, 201, createVM(t, ts, out["host_id"], "pending", 64, 999999, 9999).StatusCode)
131 vms, _ := dbst.ListVMs()
132 assert.Len(t, vms, 1)
133 })
134
135 t.Run("online but reporting no capacity", func(t *testing.T) {
136 ts, dbst, _ := testServer(t)
137 out := enroll(t, ts)
138 agentReports(out["host_id"]) // a report with no capacity in it
139 require.Equal(t, 201, createVM(t, ts, out["host_id"], "unjudged", 64, 999999, 9999).StatusCode)
140 vms, _ := dbst.ListVMs()
141 assert.Len(t, vms, 1)
142 })
143 }
144
145 // TestCreateVMCountsATombstonedVMUntilItIsReaped is the predicate the whole
146 // refusal rests on. A tombstone is a destroy in progress: the disk is still on
147 // the host and the guest may still be shutting down, so its resources are not
148 // free to promise to somebody else. Only the reap — the agent's ack, or the
149 // abandoned sweep — frees them, and it frees them by removing the row.
150 func TestCreateVMCountsATombstonedVMUntilItIsReaped(t *testing.T) {
151 ts, dbst, _ := testServer(t)
152 out := enroll(t, ts)
153 agentReportsCapacity(out["host_id"], 4, 4096, 40)
154
155 resp := createVM(t, ts, out["host_id"], "first", 4, 4096, 40)
156 require.Equal(t, 201, resp.StatusCode)
157 var created map[string]string
158 require.NoError(t, json.NewDecoder(resp.Body).Decode(&created))
159
160 require.Equal(t, 204, do(t, "DELETE", ts.URL+"/api/v1/vms/"+created["id"], testPAT, nil).StatusCode)
161
162 refused := createVM(t, ts, out["host_id"], "second", 4, 4096, 40)
163 require.Equal(t, 409, refused.StatusCode, "a deleting VM still occupies the host")
164 assert.Contains(t, bodyOf(t, refused), "already holds 4 of 4")
165
166 // The agent acks the destroy: the row goes, and with it the commitment.
167 require.NoError(t, dbst.HardDeleteVM(created["id"]))
168 assert.Equal(t, 201, createVM(t, ts, out["host_id"], "second", 4, 4096, 40).StatusCode,
169 "a reaped VM holds nothing")
170 }
internal/server/store/allocation_test.go
Old New
@@ -58,6 +58,42 @@ func TestAllocatedByHostSeparatesHosts(t *testing.T) {
58 assert.Equal(t, int64(8), alloc[h2.ID].VCPUs) 58 assert.Equal(t, int64(8), alloc[h2.ID].VCPUs)
59 } 59 }
60 60
61 // TestCommittedOnHostCountsTombstonesUntilTheyAreReaped is the other half of
62 // the pair above, and the difference between them is the point. What the
63 // console calls "allocated" is what is live; what a placement has to beat is
64 // what the machine is still holding, and a tombstoned VM's disk is on the host
65 // until the reap deletes the row.
66 func TestCommittedOnHostCountsTombstonesUntilTheyAreReaped(t *testing.T) {
67 s := newStore(t)
68 h := enrollHost(t, s)
69 vmWithResources(t, s, h, "a", 2, 2048, 10)
70 dying := vmWithResources(t, s, h, "b", 4, 4096, 20)
71 require.NoError(t, s.TombstoneVM(dying.ID))
72
73 held, err := s.CommittedOnHost(h.ID)
74 require.NoError(t, err)
75 assert.Equal(t, Alloc{VCPUs: 6, MemMB: 6144, DiskGB: 30}, held,
76 "a VM being destroyed still occupies the host")
77
78 require.NoError(t, s.HardDeleteVM(dying.ID))
79 held, err = s.CommittedOnHost(h.ID)
80 require.NoError(t, err)
81 assert.Equal(t, Alloc{VCPUs: 2, MemMB: 2048, DiskGB: 10}, held,
82 "the reap is what frees it")
83 }
84
85 // TestCommittedOnHostSeparatesHosts: one host's commitment says nothing about
86 // another's, and a host with nothing on it holds nothing rather than erroring.
87 func TestCommittedOnHostSeparatesHosts(t *testing.T) {
88 s := newStore(t)
89 h := enrollHost(t, s)
90 vmWithResources(t, s, h, "a", 2, 2048, 10)
91
92 held, err := s.CommittedOnHost("no-such-host")
93 require.NoError(t, err)
94 assert.Equal(t, Alloc{}, held)
95 }
96
61 func (s *Store) mustAllocated(t *testing.T) map[string]Alloc { 97 func (s *Store) mustAllocated(t *testing.T) map[string]Alloc {
62 t.Helper() 98 t.Helper()
63 _, alloc, _, err := s.Snapshot() 99 _, alloc, _, err := s.Snapshot()
internal/server/store/store.go
Old New
@@ -963,6 +963,29 @@ func allocatedByHost(q querier) (map[string]Alloc, error) {
963 return out, rows.Err() 963 return out, rows.Err()
964 } 964 }
965 965
966 // CommittedOnHost sums the specs of every VM row still on a host — what the
967 // machine is holding, which is the number a placement decision has to beat.
968 //
969 // It deliberately does NOT filter deleted_at, and that is the whole point of
970 // having it beside allocatedByHost. A tombstone is a destroy in progress, not a
971 // destroy: the row is marked, the agent is asked to tear the guest down, and
972 // only when it acks (or the abandoned sweep gives up on an offline host) is the
973 // row hard-deleted. Until that reap the disk is still on the host and the guest
974 // may still be shutting down, so its resources are not free to promise to
975 // somebody else. Reaped VMs need no predicate — their rows are gone.
976 //
977 // allocatedByHost, which feeds the console's "allocated", counts live rows
978 // only, so during a teardown this reads higher. Erring that way is the safe
979 // direction for admission: refusing a VM for a bed that is being stripped costs
980 // a retry, admitting one into it costs a failed VM.
981 func (s *Store) CommittedOnHost(hostID string) (Alloc, error) {
982 var a Alloc
983 err := s.db.QueryRow(`
984 SELECT COALESCE(SUM(vcpus),0), COALESCE(SUM(mem_mb),0), COALESCE(SUM(disk_gb),0)
985 FROM vms WHERE host_id=?`, hostID).Scan(&a.VCPUs, &a.MemMB, &a.DiskGB)
986 return a, err
987 }
988
966 // AuditEntry is one row of the append-only audit trail. 989 // AuditEntry is one row of the append-only audit trail.
967 type AuditEntry struct { 990 type AuditEntry struct {
968 At time.Time 991 At time.Time
web/src/lib/ResourceBar.svelte
Old New
@@ -1,4 +1,6 @@
1 <script lang="ts"> 1 <script lang="ts">
2 import { capacityReading } from '$lib/fleet.svelte';
3
2 let { 4 let {
3 label, 5 label,
4 used, 6 used,
@@ -6,22 +8,27 @@
6 unit = '' 8 unit = ''
7 }: { label: string; used: number; total: number; unit?: string } = $props(); 9 }: { label: string; used: number; total: number; unit?: string } = $props();
8 10
9 const pct = $derived(total > 0 ? Math.min(100, Math.round((used / total) * 100)) : 0); 11 const reading = $derived(capacityReading(used, total));
10 const free = $derived(total > 0 ? total - used : 0);
11 const level = $derived(pct >= 90 ? 'hot' : pct >= 70 ? 'warm' : 'ok');
12 </script> 12 </script>
13 13
14 <div class="bar"> 14 <div class="bar">
15 <div class="head"> 15 <div class="head">
16 <span class="label">{label}</span> 16 <span class="label">{label}</span>
17 {#if total > 0} 17 {#if total > 0}
18 <span class="nums">{used} / {total}{unit} <span class="free">· {free}{unit} free</span></span> 18 <span class="nums">
19 {used} / {total}{unit}
20 {#if reading.over > 0}
21 <span class="over">· over by {reading.over}{unit}</span>
22 {:else}
23 <span class="free">· {reading.free}{unit} free</span>
24 {/if}
25 </span>
19 {:else} 26 {:else}
20 <span class="nums dim">— (offline)</span> 27 <span class="nums dim">— (offline)</span>
21 {/if} 28 {/if}
22 </div> 29 </div>
23 <div class="track"> 30 <div class="track">
24 <div class="fill {level}" style="width: {pct}%"></div> 31 <div class="fill {reading.level}" style="width: {reading.pct}%"></div>
25 </div> 32 </div>
26 </div> 33 </div>
27 34
@@ -45,6 +52,12 @@
45 .dim { 52 .dim {
46 color: var(--faint); 53 color: var(--faint);
47 } 54 }
55 /* Over-allocation is not a quieter kind of "free": it is the one reading on
56 this panel worth looking at, so it sets in the same bold the fleet
57 table's load meter uses past 1.0. */
58 .over {
59 font-weight: bold;
60 }
48 /* The same meter the fleet table uses, stretched to the panel: a wash track 61 /* The same meter the fleet table uses, stretched to the panel: a wash track
49 with an ink fill. The numbers above it are the reading; the bar is the 62 with an ink fill. The numbers above it are the reading; the bar is the
50 glance. Past 90% the fill turns, because that is a wall you can hit. */ 63 glance. Past 90% the fill turns, because that is a wall you can hit. */
web/src/lib/fleet.svelte.ts
Old New
@@ -504,6 +504,44 @@ export function formatHostPort(addr: string, port: number): string {
504 return addr.includes(':') ? `[${addr}]:${port}` : `${addr}:${port}`; 504 return addr.includes(':') ? `[${addr}]:${port}` : `${addr}:${port}`;
505 } 505 }
506 506
507 /** CapacityReading is one allocation read against the capacity its host
508 * reports: what to draw, and what to say about the difference. */
509 export type CapacityReading = {
510 /** pct fills the meter, clamped to 0–100 — a bar cannot overflow its track. */
511 pct: number;
512 /** free is what is left, and it stops at zero. Below that is not less room;
513 * it is a different fact, and `over` is the one that states it. */
514 free: number;
515 /** over is by how much the allocation exceeds capacity, 0 when it fits. */
516 over: number;
517 level: 'ok' | 'warm' | 'hot';
518 };
519
520 /** capacityReading folds an allocation and a reported capacity into what a
521 * gauge needs. Past 90% the level turns, the same wall the fleet table's load
522 * meter marks.
523 *
524 * Over-allocation is a state the console has to be able to draw. Creating a VM
525 * a host has no room for is refused now, but a host that re-reports a smaller
526 * capacity (a disk shrank, an operator lowered the agent's cap) puts VMs that
527 * were placed honestly over the line, as do rows made before the refusal
528 * existed. Rendering that as "-2 free" states it as a negative amount of room,
529 * which is not a thing; "over by 2" is the same number said truthfully.
530 *
531 * A total of 0 is a host that has reported no capacity, not a host with none —
532 * the reading is empty and the caller says "offline" rather than drawing a
533 * full bar over an unknown. */
534 export function capacityReading(used: number, total: number): CapacityReading {
535 if (total <= 0) return { pct: 0, free: 0, over: 0, level: 'ok' };
536 const pct = Math.min(100, Math.round((used / total) * 100));
537 return {
538 pct,
539 free: Math.max(0, total - used),
540 over: Math.max(0, used - total),
541 level: pct >= 90 ? 'hot' : pct >= 70 ? 'warm' : 'ok'
542 };
543 }
544
507 /** eventLabel maps a lifecycle VMEvent to a human label, folding in the one 545 /** eventLabel maps a lifecycle VMEvent to a human label, folding in the one
508 * detail that matters per action. detail arrives already decoded (raw JSON on 546 * detail that matters per action. detail arrives already decoded (raw JSON on
509 * the wire); guard the shape defensively rather than parsing. */ 547 * the wire); guard the shape defensively rather than parsing. */
web/src/lib/fleet.test.ts
Old New
@@ -1,5 +1,5 @@
1 import { beforeEach, describe, expect, test } from 'vitest'; 1 import { beforeEach, describe, expect, test } from 'vitest';
2 import { fleet, vmTrustStale, type UserCA, type VM } from './fleet.svelte'; 2 import { capacityReading, fleet, vmTrustStale, type UserCA, type VM } from './fleet.svelte';
3 import type { components } from './api-types'; 3 import type { components } from './api-types';
4 4
5 type TrustedCA = components['schemas']['TrustedCA']; 5 type TrustedCA = components['schemas']['TrustedCA'];
@@ -28,7 +28,6 @@ function vmTrusting(trusted_cas: TrustedCA[] | null): VM {
28 lifecycle: 'ready', 28 lifecycle: 'ready',
29 mem_mb: 1024, 29 mem_mb: 1024,
30 name: 'guest', 30 name: 'guest',
31 persistent: false,
32 phase: 'ready', 31 phase: 'ready',
33 power_state: 'running', 32 power_state: 'running',
34 status: 'ready', 33 status: 'ready',
@@ -88,3 +87,30 @@ describe('vmTrustStale', () => {
88 ).toBe(false); 87 ).toBe(false);
89 }); 88 });
90 }); 89 });
90
91 describe('capacityReading', () => {
92 test('a half-full host has room and a calm level', () => {
93 expect(capacityReading(2, 4)).toEqual({ pct: 50, free: 2, over: 0, level: 'ok' });
94 });
95
96 test('past 90% the level turns, the same wall the load meter marks', () => {
97 expect(capacityReading(7, 10).level).toBe('warm');
98 expect(capacityReading(9, 10).level).toBe('hot');
99 });
100
101 test('a full host is full, not over', () => {
102 expect(capacityReading(6, 6)).toEqual({ pct: 100, free: 0, over: 0, level: 'hot' });
103 });
104
105 test('an over-allocated host says by how much, and never a negative free', () => {
106 expect(capacityReading(8, 6)).toEqual({ pct: 100, free: 0, over: 2, level: 'hot' });
107 });
108
109 test('the meter cannot overflow its track', () => {
110 expect(capacityReading(600, 6).pct).toBe(100);
111 });
112
113 test('a host that has reported no capacity reads empty, not full', () => {
114 expect(capacityReading(4, 0)).toEqual({ pct: 0, free: 0, over: 0, level: 'ok' });
115 });
116 });
web/src/routes/+page.svelte
Old New
@@ -16,6 +16,7 @@
16 deleteConfirm, 16 deleteConfirm,
17 upgradeAgent, 17 upgradeAgent,
18 refreshUserCAs, 18 refreshUserCAs,
19 capacityReading,
19 type CreateVMRequest, 20 type CreateVMRequest,
20 type VM 21 type VM
21 } from '$lib/fleet.svelte'; 22 } from '$lib/fleet.svelte';
@@ -281,9 +282,15 @@
281 </td> 282 </td>
282 <td class="num">{vmCountByHost.get(h.id) ?? 0}</td> 283 <td class="num">{vmCountByHost.get(h.id) ?? 0}</td>
283 <td>{h.bridge_cidr || '—'}</td> 284 <td>{h.bridge_cidr || '—'}</td>
284 <td class="num">{h.allocated.vcpus}/{h.capacity.vcpus || '?'}</td> 285 <td class="num" class:over={capacityReading(h.allocated.vcpus, h.capacity.vcpus).over > 0}
285 <td class="num">{h.allocated.mem_mb}/{h.capacity.mem_mb || '?'}</td> 286 >{h.allocated.vcpus}/{h.capacity.vcpus || '?'}</td
286 <td class="num">{h.allocated.disk_gb}/{h.capacity.disk_gb || '?'}</td> 287 >
288 <td class="num" class:over={capacityReading(h.allocated.mem_mb, h.capacity.mem_mb).over > 0}
289 >{h.allocated.mem_mb}/{h.capacity.mem_mb || '?'}</td
290 >
291 <td class="num" class:over={capacityReading(h.allocated.disk_gb, h.capacity.disk_gb).over > 0}
292 >{h.allocated.disk_gb}/{h.capacity.disk_gb || '?'}</td
293 >
287 <td> 294 <td>
288 {#if h.status !== 'decommissioning'} 295 {#if h.status !== 'decommissioning'}
289 <button class="danger" onclick={() => decommission(h.id, h.name)}>Decommission</button> 296 <button class="danger" onclick={() => decommission(h.id, h.name)}>Decommission</button>
@@ -464,7 +471,12 @@
464 display: inline-block; 471 display: inline-block;
465 font-variant-numeric: tabular-nums; 472 font-variant-numeric: tabular-nums;
466 } 473 }
467 .load.over { 474 /* One mark for "past the line", wherever a line exists: a host past 1.0 load,
475 and an allocation column past the capacity its host reports. The latter
476 can happen without a create being at fault — a host that re-reports a
477 smaller machine puts VMs placed honestly over it. */
478 .load.over,
479 td.num.over {
468 font-weight: bold; 480 font-weight: bold;
469 } 481 }
470 .meter { 482 .meter {
web/src/routes/hosts/[id]/+page.svelte
Old New
@@ -100,8 +100,8 @@
100 <ResourceBar label="Disk" used={host.allocated.disk_gb} total={host.capacity.disk_gb} unit="GB" /> 100 <ResourceBar label="Disk" used={host.allocated.disk_gb} total={host.capacity.disk_gb} unit="GB" />
101 </div> 101 </div>
102 <p class="note"> 102 <p class="note">
103 Allocated = sum of live VM specs. Capacity is reported by the agent (shown when online). 103 Allocated = sum of live VM specs. Capacity is reported by the agent (shown when online). All
104 vCPU is commonly oversubscribed; memory and disk are hard limits. 104 three are limits at placement: a create that would put any of them over capacity is refused.
105 </p> 105 </p>
106 106
107 {#if host.online && host.metrics} 107 {#if host.online && host.metrics}