a73x

7d929b91

feat(console): an offered upgrade shows itself

a73x   2026-08-11 10:56

Commit message
feat(console): an offered upgrade shows itself

Clicking "Upgrade agent" used to be silent. The offer rides the host's next
snapshot, the agent downloads, verifies, swaps and re-execs, and only when the
new version arrives in a Hello does the row change — so for the seconds in
between, and for good when the updater fails on the host, the console showed
exactly what it shows for a click that never happened. On the v0.0.5 rollout
that cost a real host: the first click on onyx looked like nothing, and the
second one, minutes later, worked.

The server already held the answer and never said it. A pending offer now
carries the moment it was made, and GET /api/v1/hosts (and the SSE snapshot,
which is the same builder) reports it as pending_upgrade: the version on offer
and age_s, the wait derived at read time. The age rather than the instant is
the same choice seconds_since_last_seen makes, and for the same reason — a
browser subtracting our timestamp from its own clock would be reporting the
gap between two clocks along with the wait. It reaches the API through the
AgentUpgrader seam that already carries the offer the other way, and it clears
exactly where offers clear: when the agent's Hello names the version, and when
a host leaves the fleet. A second click re-offers and restarts the clock; the
wait an operator reads is the wait since the ask they remember making.

Offers stay in memory, beside the snapshot stream that carries them, and this
does not change that. A restarted server reports nothing pending and the button
comes back — which is the truth, because that server has also forgotten to put
the offer in the next snapshot. Writing offers down to make the console prettier
would make it lie.

In the console the button steps aside while an offer stands: the Agent cell
reads "upgrading → v0.0.6…" in the faint used for things that are merely true,
and the fleet table shows the same in miniature. Past three minutes on a host
that is plainly online and still reporting the old version, the cell stops
being patient and says so in full, in the colour errors already speak in, and
sends the operator to the agent's log on the host — the failure happened there
and nothing the control plane knows will add to it. Which log depends on the
host: systemd's journal on Linux, ~/Library/Logs/eitri-agent.log on a Mac,
where the agent runs as a LaunchAgent and there is no journal to read. A host
whose OS we have not been told is sent to its log with no command named at all,
because sending someone to a command their machine does not have wastes exactly
the time this message exists to save.

Three minutes because the work is a download, a checksum, a swap and a re-exec
and the offer rides one ~10s tick: on a LAN it lands inside a minute, so the
threshold is an order of magnitude clear of a slow link while still catching
the operator before they have walked away for good. An offline host is never
stuck — it cannot have failed an upgrade it has not been handed.
docs/openapi.json
Old New
@@ -412,6 +412,16 @@
412 "os_version": { 412 "os_version": {
413 "type": "string" 413 "type": "string"
414 }, 414 },
415 "pending_upgrade": {
416 "anyOf": [
417 {
418 "$ref": "#/components/schemas/PendingUpgrade"
419 },
420 {
421 "type": "null"
422 }
423 ]
424 },
415 "provisioner": { 425 "provisioner": {
416 "type": "string" 426 "type": "string"
417 }, 427 },
@@ -548,6 +558,21 @@
548 }, 558 },
549 "type": "object" 559 "type": "object"
550 }, 560 },
561 "PendingUpgrade": {
562 "properties": {
563 "age_s": {
564 "type": "integer"
565 },
566 "version": {
567 "type": "string"
568 }
569 },
570 "required": [
571 "age_s",
572 "version"
573 ],
574 "type": "object"
575 },
551 "RevokeSSHCertRequest": { 576 "RevokeSSHCertRequest": {
552 "properties": { 577 "properties": {
553 "certificate": { 578 "certificate": {
internal/server/api/api.go
Old New
@@ -60,13 +60,17 @@ type ReleaseSource interface {
60 Latest() (release.Manifest, bool) 60 Latest() (release.Manifest, bool)
61 } 61 }
62 62
63 // AgentUpgrader records a pending per-host agent self-upgrade. 63 // AgentUpgrader records a pending per-host agent self-upgrade — and answers for
64 // *syncsvc.Service satisfies it. 64 // the one it is holding, which is how the offer becomes visible to an operator
65 // instead of vanishing into a snapshot. *syncsvc.Service satisfies it.
65 type AgentUpgrader interface { 66 type AgentUpgrader interface {
66 OfferAgentUpgrade(hostID, version, url, sha256 string) 67 OfferAgentUpgrade(hostID, version, url, sha256 string)
67 // ClearAgentUpgrade drops any pending offer for hostID — called when the 68 // ClearAgentUpgrade drops any pending offer for hostID — called when the
68 // host leaves the fleet so a decommission cannot strand a stale offer. 69 // host leaves the fleet so a decommission cannot strand a stale offer.
69 ClearAgentUpgrade(hostID string) 70 ClearAgentUpgrade(hostID string)
71 // PendingAgentUpgrade reports the version offered to hostID and how long
72 // the offer has stood; ok is false when none is outstanding.
73 PendingAgentUpgrade(hostID string) (version string, age time.Duration, ok bool)
70 } 74 }
71 75
72 // API is the HTTP handler container. 76 // API is the HTTP handler container.
@@ -536,6 +540,16 @@ func (a *API) buildHostResponses(hosts []store.Host, alloc map[string]store.Allo
536 out[i] = toHostResponse(h, rs.st, rs.ok, alloc[h.ID]) 540 out[i] = toHostResponse(h, rs.st, rs.ok, alloc[h.ID])
537 out[i].AgentUpdateAvailable = latest != "" && out[i].Online && 541 out[i].AgentUpdateAvailable = latest != "" && out[i].Online &&
538 out[i].AgentVersion != "" && release.Less(out[i].AgentVersion, latest) 542 out[i].AgentVersion != "" && release.Less(out[i].AgentVersion, latest)
543 // An offer the agent has not taken yet. The upgrader is the only place
544 // it exists — offers are held in memory beside the snapshot stream that
545 // carries them, never written down — so a restarted server reports none
546 // and the console offers the button again, which is exactly right: the
547 // restart dropped the offer too.
548 if a.upgrader != nil {
549 if v, age, ok := a.upgrader.PendingAgentUpgrade(h.ID); ok {
550 out[i].PendingUpgrade = &types.PendingUpgrade{Version: v, AgeS: int64(age.Seconds())}
551 }
552 }
539 } 553 }
540 return out 554 return out
541 } 555 }
internal/server/api/testdata/host.golden.json
Old New
@@ -24,6 +24,10 @@
24 }, 24 },
25 "agent_version": "v0.0.1-agent", 25 "agent_version": "v0.0.1-agent",
26 "agent_update_available": true, 26 "agent_update_available": true,
27 "pending_upgrade": {
28 "version": "v0.0.2-agent",
29 "age_s": 12
30 },
27 "os_id": "arch", 31 "os_id": "arch",
28 "os_pretty": "Arch Linux", 32 "os_pretty": "Arch Linux",
29 "os_version": "rolling", 33 "os_version": "rolling",
internal/server/api/testdata/snapshot.golden.json
Old New
@@ -26,6 +26,10 @@
26 }, 26 },
27 "agent_version": "v0.0.1-agent", 27 "agent_version": "v0.0.1-agent",
28 "agent_update_available": true, 28 "agent_update_available": true,
29 "pending_upgrade": {
30 "version": "v0.0.2-agent",
31 "age_s": 12
32 },
29 "os_id": "arch", 33 "os_id": "arch",
30 "os_pretty": "Arch Linux", 34 "os_pretty": "Arch Linux",
31 "os_version": "rolling", 35 "os_version": "rolling",
internal/server/api/types/types.go
Old New
@@ -38,6 +38,17 @@ type Metrics struct {
38 DiskFreeGB int64 `json:"disk_free_gb"` 38 DiskFreeGB int64 `json:"disk_free_gb"`
39 } 39 }
40 40
41 // PendingUpgrade is an agent self-upgrade that has been offered to a host and
42 // not yet converged: the version on offer, and how long the offer has stood.
43 //
44 // The wait is served as an age rather than the instant the offer was made,
45 // matching seconds_since_last_seen: the server derives it on read, so a client
46 // whose clock disagrees with the server's still reads the true wait.
47 type PendingUpgrade struct {
48 Version string `json:"version"`
49 AgeS int64 `json:"age_s"`
50 }
51
41 // Host is the explicit snake_case wire representation of a host, served by 52 // Host is the explicit snake_case wire representation of a host, served by
42 // GET /api/v1/hosts and the SSE snapshot. Every field is spelled out — no 53 // GET /api/v1/hosts and the SSE snapshot. Every field is spelled out — no
43 // struct embedding — to prevent PascalCase field leakage. 54 // struct embedding — to prevent PascalCase field leakage.
@@ -68,6 +79,10 @@ type Host struct {
68 // online (offerable). 79 // online (offerable).
69 AgentVersion string `json:"agent_version"` 80 AgentVersion string `json:"agent_version"`
70 AgentUpdateAvailable bool `json:"agent_update_available"` 81 AgentUpdateAvailable bool `json:"agent_update_available"`
82 // PendingUpgrade is the upgrade this host's agent has been offered and has
83 // not yet taken, null when none is outstanding. It is what the seconds
84 // between the click and the new version landing look like from outside.
85 PendingUpgrade *PendingUpgrade `json:"pending_upgrade"`
71 // Host OS facts (persisted; refreshed from each Hello). 86 // Host OS facts (persisted; refreshed from each Hello).
72 OSID string `json:"os_id"` 87 OSID string `json:"os_id"`
73 OSPretty string `json:"os_pretty"` 88 OSPretty string `json:"os_pretty"`
internal/server/api/upgrade_test.go
Old New
@@ -4,6 +4,7 @@ import (
4 "encoding/json" 4 "encoding/json"
5 "net/http" 5 "net/http"
6 "testing" 6 "testing"
7 "time"
7 8
8 "github.com/a73x/eitri/internal/server/api/types" 9 "github.com/a73x/eitri/internal/server/api/types"
9 "github.com/a73x/eitri/internal/server/registry" 10 "github.com/a73x/eitri/internal/server/registry"
@@ -23,7 +24,12 @@ type fakeRelease struct {
23 func (f fakeRelease) Latest() (release.Manifest, bool) { return f.m, f.ok } 24 func (f fakeRelease) Latest() (release.Manifest, bool) { return f.m, f.ok }
24 25
25 // fakeUpgrader is an AgentUpgrader test double that captures the last offer. 26 // fakeUpgrader is an AgentUpgrader test double that captures the last offer.
26 type fakeUpgrader struct{ host, version, url, sha string } 27 // age is what PendingAgentUpgrade reports for it, so a test can stand an offer
28 // for as long as it likes without waiting.
29 type fakeUpgrader struct {
30 host, version, url, sha string
31 age time.Duration
32 }
27 33
28 func (f *fakeUpgrader) OfferAgentUpgrade(hostID, version, url, sha256 string) { 34 func (f *fakeUpgrader) OfferAgentUpgrade(hostID, version, url, sha256 string) {
29 f.host, f.version, f.url, f.sha = hostID, version, url, sha256 35 f.host, f.version, f.url, f.sha = hostID, version, url, sha256
@@ -35,6 +41,13 @@ func (f *fakeUpgrader) ClearAgentUpgrade(hostID string) {
35 } 41 }
36 } 42 }
37 43
44 func (f *fakeUpgrader) PendingAgentUpgrade(hostID string) (string, time.Duration, bool) {
45 if f.host != hostID || f.host == "" {
46 return "", 0, false
47 }
48 return f.version, f.age, true
49 }
50
38 // upgradeManifest is the happy-path manifest: a newer release with an 51 // upgradeManifest is the happy-path manifest: a newer release with an
39 // eitri-agent artifact for linux/amd64 (matching the enrolled test host). 52 // eitri-agent artifact for linux/amd64 (matching the enrolled test host).
40 func upgradeManifest() release.Manifest { 53 func upgradeManifest() release.Manifest {
@@ -208,6 +221,44 @@ func TestHostResponseAgentUpdateAvailable(t *testing.T) {
208 }) 221 })
209 } 222 }
210 223
224 // TestHostResponsePendingUpgrade pins that an offer is visible while it stands.
225 // Before this the click was silent: the offer rode the next snapshot and
226 // nothing said so, which reads exactly like a button that did nothing.
227 func TestHostResponsePendingUpgrade(t *testing.T) {
228 ts, _, _, reg, a := newServer(t)
229 out := enroll(t, ts)
230 hostID := out["host_id"]
231 reg.SetAgentVersion(hostID, "v0.0.1")
232 reg.UpdateReport(hostID, registry.Report{})
233 a.SetReleaseSource(fakeRelease{m: upgradeManifest(), ok: true})
234 up := &fakeUpgrader{}
235 a.SetAgentUpgrader(up)
236
237 hosts := func() map[string]any {
238 resp := do(t, "GET", ts.URL+"/api/v1/hosts", testPAT, nil)
239 require.Equal(t, http.StatusOK, resp.StatusCode)
240 items := decodeJSONKeys(t, resp)
241 require.Len(t, items, 1)
242 return items[0]
243 }
244
245 assert.Nil(t, hosts()["pending_upgrade"], "no offer, nothing pending")
246
247 resp := do(t, "POST", ts.URL+"/api/v1/hosts/"+hostID+"/upgrade-agent", testPAT, nil)
248 require.Equal(t, http.StatusAccepted, resp.StatusCode)
249
250 up.age = 90 * time.Second
251 pending, ok := hosts()["pending_upgrade"].(map[string]any)
252 require.True(t, ok, "an outstanding offer is on the wire")
253 assert.Equal(t, "v0.0.2", pending["version"])
254 assert.Equal(t, float64(90), pending["age_s"])
255
256 // The agent takes it: the offer converges and the pending state goes away
257 // on its own, without the console having to decide when to stop showing it.
258 up.ClearAgentUpgrade(hostID)
259 assert.Nil(t, hosts()["pending_upgrade"], "a converged offer leaves nothing pending")
260 }
261
211 // TestMarshalSnapshotCarriesVersions pins that the SSE snapshot payload 262 // TestMarshalSnapshotCarriesVersions pins that the SSE snapshot payload
212 // itself (not just GET /hosts) carries the server's own build version and the 263 // itself (not just GET /hosts) carries the server's own build version and the
213 // latest known release version. marshalSnapshots is called directly (in-package) 264 // latest known release version. marshalSnapshots is called directly (in-package)
internal/server/api/wire_golden_test.go
Old New
@@ -69,6 +69,7 @@ func TestWireGolden(t *testing.T) {
69 Allocated: types.Capacity{VCPUs: 4, MemMB: 8192, DiskGB: 100}, 69 Allocated: types.Capacity{VCPUs: 4, MemMB: 8192, DiskGB: 100},
70 AgentVersion: "v0.0.1-agent", 70 AgentVersion: "v0.0.1-agent",
71 AgentUpdateAvailable: true, 71 AgentUpdateAvailable: true,
72 PendingUpgrade: &types.PendingUpgrade{Version: "v0.0.2-agent", AgeS: 12},
72 OSID: "arch", 73 OSID: "arch",
73 OSPretty: "Arch Linux", 74 OSPretty: "Arch Linux",
74 OSVersion: "rolling", 75 OSVersion: "rolling",
internal/server/syncsvc/syncsvc.go
Old New
@@ -76,7 +76,19 @@ type Service struct {
76 // Hello reports the target version. In-memory only — a restart forgets 76 // Hello reports the target version. In-memory only — a restart forgets
77 // pending offers and the operator clicks again (idempotent). 77 // pending offers and the operator clicks again (idempotent).
78 offersMu sync.Mutex 78 offersMu sync.Mutex
79 offers map[string]*pb.AgentUpgrade 79 offers map[string]offer
80 // now is the clock the offer timestamps are read from, injected the way
81 // registry.New takes one so a test can age an offer without sleeping.
82 now func() time.Time
83 }
84
85 // offer is one pending agent self-upgrade and the moment it was made. The
86 // timestamp is what lets a reader tell an upgrade in flight from one that has
87 // failed on the host: the work is a download, a checksum, a swap and a re-exec,
88 // so an offer still standing minutes later is not slow, it is stuck.
89 type offer struct {
90 up *pb.AgentUpgrade
91 offeredAt time.Time
80 } 92 }
81 93
82 // New constructs a Service with the production-default down-stream write 94 // New constructs a Service with the production-default down-stream write
@@ -94,7 +106,7 @@ func newWithWriteTimeout(st *store.Store, reg *registry.Registry, h *hub.Hub, se
94 } 106 }
95 return &Service{st: st, reg: reg, hub: h, secret: secret, maxCredAge: maxCredAge, writeTimeout: writeTimeout, 107 return &Service{st: st, reg: reg, hub: h, secret: secret, maxCredAge: maxCredAge, writeTimeout: writeTimeout,
96 conns: map[string]quic.Connection{}, recorder: st, tracker: newStatusTracker(), netTrack: newNetTracker(), 108 conns: map[string]quic.Connection{}, recorder: st, tracker: newStatusTracker(), netTrack: newNetTracker(),
97 uplinkTrack: newNetTracker(), certTrack: newNetTracker(), offers: map[string]*pb.AgentUpgrade{}} 109 uplinkTrack: newNetTracker(), certTrack: newNetTracker(), offers: map[string]offer{}, now: time.Now}
98 } 110 }
99 111
100 // Serve accepts QUIC connections until ctx is cancelled. 112 // Serve accepts QUIC connections until ctx is cancelled.
@@ -404,18 +416,45 @@ func (s *Service) pushSnapshot(down quic.Stream, hostID string) error {
404 } 416 }
405 417
406 // OfferAgentUpgrade records a pending agent self-upgrade for hostID; the 418 // OfferAgentUpgrade records a pending agent self-upgrade for hostID; the
407 // host's next snapshot carries it (callers poke the host via the hub). 419 // host's next snapshot carries it (callers poke the host via the hub). A second
420 // click re-offers and restarts the clock: the operator has asked again, and the
421 // wait they are being told about is the wait since that ask.
408 func (s *Service) OfferAgentUpgrade(hostID, version, url, sha256 string) { 422 func (s *Service) OfferAgentUpgrade(hostID, version, url, sha256 string) {
409 s.offersMu.Lock() 423 s.offersMu.Lock()
410 defer s.offersMu.Unlock() 424 defer s.offersMu.Unlock()
411 s.offers[hostID] = &pb.AgentUpgrade{Version: version, Url: url, Sha256: sha256} 425 s.offers[hostID] = offer{
426 up: &pb.AgentUpgrade{Version: version, Url: url, Sha256: sha256},
427 offeredAt: s.now(),
428 }
412 } 429 }
413 430
414 // offerFor returns hostID's pending upgrade (nil when none). 431 // offerFor returns hostID's pending upgrade (nil when none).
415 func (s *Service) offerFor(hostID string) *pb.AgentUpgrade { 432 func (s *Service) offerFor(hostID string) *pb.AgentUpgrade {
416 s.offersMu.Lock() 433 s.offersMu.Lock()
417 defer s.offersMu.Unlock() 434 defer s.offersMu.Unlock()
418 return s.offers[hostID] 435 return s.offers[hostID].up
436 }
437
438 // PendingAgentUpgrade reports hostID's outstanding offer: the version offered
439 // and how long it has been standing. ok is false when there is none.
440 //
441 // The age is derived here, at read time, rather than served as an absolute
442 // instant — the same choice registry.HostState makes for SinceLastSeen, and for
443 // the same reason: a browser comparing our timestamp against its own clock
444 // would be reporting the difference between two clocks as well as the wait.
445 //
446 // Offers live in memory only. A server restart forgets them, so the pending
447 // state disappears and the button comes back — which is the honest answer,
448 // because a restarted server has also forgotten to put the offer in the next
449 // snapshot. The cure is the same click as before.
450 func (s *Service) PendingAgentUpgrade(hostID string) (version string, age time.Duration, ok bool) {
451 s.offersMu.Lock()
452 defer s.offersMu.Unlock()
453 o, ok := s.offers[hostID]
454 if !ok {
455 return "", 0, false
456 }
457 return o.up.GetVersion(), s.now().Sub(o.offeredAt), true
419 } 458 }
420 459
421 // ClearAgentUpgrade drops any pending offer for hostID unconditionally — 460 // ClearAgentUpgrade drops any pending offer for hostID unconditionally —
@@ -432,7 +471,7 @@ func (s *Service) ClearAgentUpgrade(hostID string) {
432 func (s *Service) clearOfferIfDone(hostID, reportedVersion string) { 471 func (s *Service) clearOfferIfDone(hostID, reportedVersion string) {
433 s.offersMu.Lock() 472 s.offersMu.Lock()
434 defer s.offersMu.Unlock() 473 defer s.offersMu.Unlock()
435 if up, ok := s.offers[hostID]; ok && up.Version == reportedVersion { 474 if o, ok := s.offers[hostID]; ok && o.up.GetVersion() == reportedVersion {
436 delete(s.offers, hostID) 475 delete(s.offers, hostID)
437 } 476 }
438 } 477 }
internal/server/syncsvc/syncsvc_test.go
Old New
@@ -826,7 +826,7 @@ func TestReportMetricsLandInRegistry(t *testing.T) {
826 } 826 }
827 827
828 func TestUpgradeOffers(t *testing.T) { 828 func TestUpgradeOffers(t *testing.T) {
829 s := &Service{offers: map[string]*pb.AgentUpgrade{}} 829 s := &Service{offers: map[string]offer{}, now: time.Now}
830 830
831 s.OfferAgentUpgrade("h1", "v0.0.2", "https://eitri.sh/dl/v0.0.2/a.tar.gz", "ab") 831 s.OfferAgentUpgrade("h1", "v0.0.2", "https://eitri.sh/dl/v0.0.2/a.tar.gz", "ab")
832 up := s.offerFor("h1") 832 up := s.offerFor("h1")
@@ -856,6 +856,43 @@ func TestUpgradeOffers(t *testing.T) {
856 } 856 }
857 } 857 }
858 858
859 // TestPendingAgentUpgradeAges pins the fact the console reads: an offer knows
860 // how long it has been standing, so "in flight" and "not landing" are
861 // distinguishable from outside the host.
862 func TestPendingAgentUpgradeAges(t *testing.T) {
863 clock := time.Date(2026, 8, 11, 9, 0, 0, 0, time.UTC)
864 s := &Service{offers: map[string]offer{}, now: func() time.Time { return clock }}
865
866 if _, _, ok := s.PendingAgentUpgrade("h1"); ok {
867 t.Fatal("a host with no offer has nothing pending")
868 }
869
870 s.OfferAgentUpgrade("h1", "v0.0.6", "https://eitri.sh/dl/v0.0.6/a.tar.gz", "ab")
871 v, age, ok := s.PendingAgentUpgrade("h1")
872 if !ok || v != "v0.0.6" || age != 0 {
873 t.Fatalf("fresh offer = %q %v %v", v, age, ok)
874 }
875
876 // The wait is measured at read time, not at offer time.
877 clock = clock.Add(4 * time.Minute)
878 if _, age, _ = s.PendingAgentUpgrade("h1"); age != 4*time.Minute {
879 t.Fatalf("age after 4m = %v", age)
880 }
881
882 // A second click is a fresh ask, and the wait it reports is the wait since
883 // that ask — otherwise the operator reads the age of an offer they replaced.
884 s.OfferAgentUpgrade("h1", "v0.0.6", "https://eitri.sh/dl/v0.0.6/a.tar.gz", "ab")
885 if _, age, _ = s.PendingAgentUpgrade("h1"); age != 0 {
886 t.Fatalf("re-offer did not restart the clock: age = %v", age)
887 }
888
889 // Convergence ends it: the agent reports the version it was offered.
890 s.clearOfferIfDone("h1", "v0.0.6")
891 if _, _, ok = s.PendingAgentUpgrade("h1"); ok {
892 t.Fatal("offer still pending after the agent reported the target version")
893 }
894 }
895
859 // TestHelloRefreshesProvisioner pins that a host which changes backend stops 896 // TestHelloRefreshesProvisioner pins that a host which changes backend stops
860 // lying about itself on its next reconnect. The provisioner is recorded at 897 // lying about itself on its next reconnect. The provisioner is recorded at
861 // enrollment and was only ever LOGGED on Hello, so a Mac enrolled before its 898 // enrollment and was only ever LOGGED on Hello, so a Mac enrolled before its
web/src/lib/api-types.ts
Old New
@@ -1603,6 +1603,7 @@ export interface components {
1603 os_id: string; 1603 os_id: string;
1604 os_pretty: string; 1604 os_pretty: string;
1605 os_version: string; 1605 os_version: string;
1606 pending_upgrade?: components["schemas"]["PendingUpgrade"] | null;
1606 provisioner: string; 1607 provisioner: string;
1607 seconds_since_last_seen?: number | null; 1608 seconds_since_last_seen?: number | null;
1608 sessions: number; 1609 sessions: number;
@@ -1634,6 +1635,10 @@ export interface components {
1634 PatchVMRequest: { 1635 PatchVMRequest: {
1635 power_state?: string; 1636 power_state?: string;
1636 }; 1637 };
1638 PendingUpgrade: {
1639 age_s: number;
1640 version: string;
1641 };
1637 RevokeSSHCertRequest: { 1642 RevokeSSHCertRequest: {
1638 certificate?: string; 1643 certificate?: string;
1639 reason?: string; 1644 reason?: string;
web/src/lib/fleet.svelte.ts
Old New
@@ -453,6 +453,76 @@ export function vmTrustStale(vm: VM): boolean {
453 return fleet.userCAs.some((ca) => ca.fingerprint && !trusted.has(ca.fingerprint)); 453 return fleet.userCAs.some((ca) => ca.fingerprint && !trusted.has(ca.fingerprint));
454 } 454 }
455 455
456 /** UPGRADE_STUCK_S is how long an offered agent upgrade may stand before the
457 * console stops calling it "in flight" and starts calling it stuck.
458 *
459 * The work behind an offer is a download, a checksum, a binary swap and a
460 * re-exec, and the offer itself rides the host's next snapshot — one agent
461 * tick, ~10s. On a LAN the whole thing is done inside a few tens of seconds,
462 * and the new version is in the following Hello. Three minutes is an order of
463 * magnitude past that: far enough out that a slow link or a large artifact is
464 * never accused, close enough that an operator who clicked and looked away
465 * learns it failed while they are still the person who clicked. */
466 export const UPGRADE_STUCK_S = 180;
467
468 /** upgradeStuck reports that a host's pending upgrade is not going to land on
469 * its own.
470 *
471 * Three things have to be true together. The offer has stood past
472 * UPGRADE_STUCK_S. The host is online — it is reporting NOW, which means it
473 * has reported since the offer was made, so the agent has seen the offer and
474 * is still running. And it is still on the old version, so what it saw it did
475 * not take. That combination is not a wait; it is a failure, and it lives in
476 * the host's own log.
477 *
478 * An offline host is deliberately never stuck. It cannot have failed an
479 * upgrade it has not been handed: the offer waits in memory for the agent to
480 * come back, and the honest reading of a dark host is that nobody has heard
481 * from it, not that something broke. */
482 export function upgradeStuck(h: Host): boolean {
483 const pending = h.pending_upgrade;
484 if (!pending || !h.online) return false;
485 if (h.agent_version === pending.version) return false;
486 return pending.age_s >= UPGRADE_STUCK_S;
487 }
488
489 /** upgradeAge renders how long an offer has stood, coarsely — "40s", "4m",
490 * "1h 5m". The number is evidence that something is wrong, not a stopwatch,
491 * so it rounds down to the unit that makes the point. */
492 export function upgradeAge(seconds: number): string {
493 if (seconds < 60) return `${Math.max(0, Math.floor(seconds))}s`;
494 if (seconds < 3600) return `${Math.floor(seconds / 60)}m`;
495 return `${Math.floor(seconds / 3600)}h ${Math.floor((seconds % 3600) / 60)}m`;
496 }
497
498 /** agentLogHint is where this host keeps the agent's log, in the exact form you
499 * would type or open. A Linux host runs the agent under systemd and its log is
500 * in the journal; a Mac runs it as a LaunchAgent, which has no journal and
501 * writes to a file (docs/quickstart.md). A host whose OS we have not been told
502 * gets no hint at all rather than a guess: sending an operator to a command
503 * their machine does not have wastes exactly the time this message exists to
504 * save. Returns '' in that case, and callers drop the parenthetical. */
505 function agentLogHint(os: string): string {
506 if (os === 'darwin') return '~/Library/Logs/eitri-agent.log';
507 if (os === 'linux') return 'journalctl -u eitri-agent';
508 return '';
509 }
510
511 /** upgradeStuckNote is what the console says about a stuck upgrade, or '' when
512 * the host has none. One sentence of fact and one of where to look: the
513 * failure happened on the host, so the host's log is the only place holding
514 * the reason, and nothing the control plane knows will add to it. */
515 export function upgradeStuckNote(h: Host): string {
516 if (!upgradeStuck(h)) return '';
517 const pending = h.pending_upgrade!;
518 const hint = agentLogHint(h.os);
519 return (
520 `Upgrade to ${pending.version} was offered ${upgradeAge(pending.age_s)} ago and this host ` +
521 `is still reporting ${h.agent_version || 'no version'}—the agent is not taking it. ` +
522 `Read the agent's log on the host${hint ? ` (${hint})` : ''} for the reason, then offer it again.`
523 );
524 }
525
456 /** vmPowerAction is the power flip a VM's controls may offer, or null when 526 /** vmPowerAction is the power flip a VM's controls may offer, or null when
457 * offering one would be a lie. 527 * offering one would be a lie.
458 * 528 *
web/src/lib/fleet.test.ts
Old New
@@ -1,5 +1,16 @@
1 import { beforeEach, describe, expect, test } from 'vitest'; 1 import { beforeEach, describe, expect, test } from 'vitest';
2 import { capacityReading, fleet, vmTrustStale, type UserCA, type VM } from './fleet.svelte'; 2 import {
3 capacityReading,
4 fleet,
5 upgradeAge,
6 upgradeStuck,
7 upgradeStuckNote,
8 vmTrustStale,
9 UPGRADE_STUCK_S,
10 type Host,
11 type UserCA,
12 type VM
13 } from './fleet.svelte';
3 import type { components } from './api-types'; 14 import type { components } from './api-types';
4 15
5 type TrustedCA = components['schemas']['TrustedCA']; 16 type TrustedCA = components['schemas']['TrustedCA'];
@@ -88,6 +99,125 @@ describe('vmTrustStale', () => {
88 }); 99 });
89 }); 100 });
90 101
102 /** hostUpgrading is a host as the console sees one mid-upgrade: online unless
103 * told otherwise, running `agent_version`, and holding an offer of `offered`
104 * made `age_s` seconds ago. Pass offered null for a host with no offer, and an
105 * os other than linux for a host that keeps its agent's log somewhere else. */
106 function hostUpgrading(
107 agent_version: string,
108 offered: { version: string; age_s: number } | null,
109 online = true,
110 os = 'linux'
111 ): Host {
112 return {
113 agent_update_available: true,
114 agent_version,
115 allocated: { vcpus: 1, mem_mb: 1024, disk_gb: 10 },
116 arch: 'amd64',
117 bridge_cidr: '10.77.1.0/24',
118 capacity: { vcpus: 8, mem_mb: 16384, disk_gb: 256 },
119 cpu_model: 'AMD Ryzen 9 7950X',
120 enrolled_at: '2026-08-01T09:00:00Z',
121 id: 'host-1',
122 kernel: '6.15.4-arch1-1',
123 last_seen: '2026-08-11T09:00:00Z',
124 metrics: null,
125 name: 'onyx',
126 online,
127 os,
128 os_id: 'arch',
129 os_pretty: 'Arch Linux',
130 os_version: 'rolling',
131 pending_upgrade: offered,
132 provisioner: 'cloudhypervisor',
133 seconds_since_last_seen: 2,
134 sessions: 1,
135 stale: false,
136 status: 'active',
137 uplink_addr: '192.168.0.190',
138 virt: 'kvm'
139 };
140 }
141
142 describe('upgradeStuck', () => {
143 test('a host with no offer outstanding is not stuck', () => {
144 expect(upgradeStuck(hostUpgrading('v0.0.5', null))).toBe(false);
145 });
146
147 test('a fresh offer is in flight, not stuck', () => {
148 expect(upgradeStuck(hostUpgrading('v0.0.5', { version: 'v0.0.6', age_s: 12 }))).toBe(false);
149 });
150
151 test('an offer still standing at the threshold, on a reporting host, is stuck', () => {
152 expect(
153 upgradeStuck(hostUpgrading('v0.0.5', { version: 'v0.0.6', age_s: UPGRADE_STUCK_S }))
154 ).toBe(true);
155 });
156
157 test('an offline host is never stuck—it has not been handed the offer yet', () => {
158 expect(upgradeStuck(hostUpgrading('v0.0.5', { version: 'v0.0.6', age_s: 3600 }, false))).toBe(
159 false
160 );
161 });
162
163 test('a host already reporting the offered version has taken it, however old the offer', () => {
164 expect(upgradeStuck(hostUpgrading('v0.0.6', { version: 'v0.0.6', age_s: 3600 }))).toBe(false);
165 });
166 });
167
168 describe('upgradeStuckNote', () => {
169 test('a stuck upgrade names both versions, the wait, and the log to read', () => {
170 const note = upgradeStuckNote(hostUpgrading('v0.0.5', { version: 'v0.0.6', age_s: 300 }));
171 expect(note).toContain('v0.0.6');
172 expect(note).toContain('v0.0.5');
173 expect(note).toContain('5m');
174 expect(note).toContain('journalctl -u eitri-agent');
175 });
176
177 test('an upgrade still in flight has nothing to complain about', () => {
178 expect(upgradeStuckNote(hostUpgrading('v0.0.5', { version: 'v0.0.6', age_s: 12 }))).toBe('');
179 });
180
181 test('a Mac host is sent to its log file—it runs no systemd to have a journal', () => {
182 const note = upgradeStuckNote(
183 hostUpgrading('v0.0.5', { version: 'v0.0.6', age_s: 300 }, true, 'darwin')
184 );
185 expect(note).toContain('~/Library/Logs/eitri-agent.log');
186 expect(note).not.toContain('journalctl');
187 });
188
189 test('a host of unnamed OS is sent to its log, without a command it may not have', () => {
190 const note = upgradeStuckNote(
191 hostUpgrading('v0.0.5', { version: 'v0.0.6', age_s: 300 }, true, '')
192 );
193 expect(note).toContain("Read the agent's log on the host for the reason");
194 expect(note).not.toContain('(');
195 });
196
197 test('a host that never named its version is described, not blanked', () => {
198 expect(upgradeStuckNote(hostUpgrading('', { version: 'v0.0.6', age_s: 300 }))).toContain(
199 'no version'
200 );
201 });
202 });
203
204 describe('upgradeAge', () => {
205 test('under a minute counts seconds', () => {
206 expect(upgradeAge(0)).toBe('0s');
207 expect(upgradeAge(59)).toBe('59s');
208 });
209
210 test('past a minute rounds down to whole minutes', () => {
211 expect(upgradeAge(60)).toBe('1m');
212 expect(upgradeAge(299)).toBe('4m');
213 });
214
215 test('past an hour says hours and minutes', () => {
216 expect(upgradeAge(3600)).toBe('1h 0m');
217 expect(upgradeAge(3900)).toBe('1h 5m');
218 });
219 });
220
91 describe('capacityReading', () => { 221 describe('capacityReading', () => {
92 test('a half-full host has room and a calm level', () => { 222 test('a half-full host has room and a calm level', () => {
93 expect(capacityReading(2, 4)).toEqual({ pct: 50, free: 2, over: 0, level: 'ok' }); 223 expect(capacityReading(2, 4)).toEqual({ pct: 50, free: 2, over: 0, level: 'ok' });
web/src/routes/+page.svelte
Old New
@@ -15,6 +15,8 @@
15 vmTrustStale, 15 vmTrustStale,
16 deleteConfirm, 16 deleteConfirm,
17 upgradeAgent, 17 upgradeAgent,
18 upgradeStuck,
19 upgradeStuckNote,
18 refreshUserCAs, 20 refreshUserCAs,
19 capacityReading, 21 capacityReading,
20 type CreateVMRequest, 22 type CreateVMRequest,
@@ -261,7 +263,16 @@
261 <td title={osTitle(h)}>{osLabel(h)}</td> 263 <td title={osTitle(h)}>{osLabel(h)}</td>
262 <td> 264 <td>
263 {h.agent_version || '—'} 265 {h.agent_version || '—'}
264 {#if h.agent_update_available} 266 <!-- The host page's three readings, at table scale: a version
267 cell has no room for the sentence, so a stuck offer shows
268 the word and carries the whole of it in the tooltip. -->
269 {#if h.pending_upgrade}
270 {#if upgradeStuck(h)}
271 <span class="stuck" title={upgradeStuckNote(h)}>stuck → {h.pending_upgrade.version}</span>
272 {:else}
273 <span class="pending">…→ {h.pending_upgrade.version}</span>
274 {/if}
275 {:else if h.agent_update_available}
265 <button class="ghost upgrade" onclick={() => upgrade(h)}>↑ {fleet.latest_version}</button> 276 <button class="ghost upgrade" onclick={() => upgrade(h)}>↑ {fleet.latest_version}</button>
266 {/if} 277 {/if}
267 </td> 278 </td>
@@ -495,6 +506,18 @@
495 .upgrade { 506 .upgrade {
496 margin-left: 0.5em; 507 margin-left: 0.5em;
497 } 508 }
509 /* Both sit where the button was. In flight is faint, like every other row
510 fact; stuck borrows the colour errors already speak in and hangs the
511 explanation off the title, since the fleet table is scanned, not read. */
512 .pending {
513 margin-left: 0.5em;
514 color: var(--faint);
515 }
516 .stuck {
517 margin-left: 0.5em;
518 color: var(--bad);
519 cursor: help;
520 }
498 .actions { 521 .actions {
499 display: flex; 522 display: flex;
500 gap: 0.5em; 523 gap: 0.5em;
web/src/routes/hosts/[id]/+page.svelte
Old New
@@ -5,6 +5,8 @@
5 action, 5 action,
6 decommissionHost, 6 decommissionHost,
7 upgradeAgent, 7 upgradeAgent,
8 upgradeStuck,
9 upgradeStuckNote,
8 vmsForHost, 10 vmsForHost,
9 vmPhase, 11 vmPhase,
10 vmPower, 12 vmPower,
@@ -58,7 +60,19 @@
58 <th>Agent</th> 60 <th>Agent</th>
59 <td> 61 <td>
60 {host.agent_version || '—'} 62 {host.agent_version || '—'}
61 {#if host.agent_update_available} 63 <!-- One cell, three readings. An offer standing means the click
64 landed and the work is under way, so the button steps aside
65 rather than inviting a second click that changes nothing.
66 Once the offer is old on a host that is plainly reporting,
67 the same cell says so at length: the failure is on the host
68 and only its journal knows why. -->
69 {#if host.pending_upgrade}
70 {#if upgradeStuck(host)}
71 <p class="stuck">{upgradeStuckNote(host)}</p>
72 {:else}
73 <span class="pending">upgrading → {host.pending_upgrade.version}…</span>
74 {/if}
75 {:else if host.agent_update_available}
62 <button class="ghost upgrade" onclick={upgrade}>↑ {fleet.latest_version}</button> 76 <button class="ghost upgrade" onclick={upgrade}>↑ {fleet.latest_version}</button>
63 {/if} 77 {/if}
64 </td> 78 </td>
@@ -202,4 +216,19 @@
202 .upgrade { 216 .upgrade {
203 margin-left: 0.5em; 217 margin-left: 0.5em;
204 } 218 }
219 /* An upgrade in flight is a fact about the next few seconds, not a control:
220 it sits where the button was, in the faint the rest of the page uses for
221 things that are merely true. */
222 .pending {
223 margin-left: 0.5em;
224 color: var(--faint);
225 }
226 /* A stuck one is the exception—it is the only thing on this page telling an
227 operator that something they did failed—so it takes its own line and the
228 colour errors already speak in. No new colour, just the loud one. */
229 .stuck {
230 color: var(--bad);
231 max-width: 60ch;
232 margin: 0.3em 0 0;
233 }
205 </style> 234 </style>