Teardown observability: a quarantined VM reports power_state=running with all actual-state fields blank #74

New issue

Open

opened 2026-08-21 18:31:45 +00:00 by a73x · 0 comments

a73x commented

2026-08-21 18:31:45 +00:00

(Migrated from git.a73x.sh)

[claude 2026-08-21] Found while tearing down the demo VMs on onyx. A tombstoned
VM spends 5 minutes (-tombstone-grace, internal/agent/run/cli.go:138) in
quarantine before its disk is destroyed. During that window the API reports the
VM in a way that actively misleads:

power_state still reads running — that is stale DESIRED state. The agent
has already called Prov.Shutdown (internal/agent/reconcile/reconcile.go:481);
the guest is off within seconds of the DELETE.
phase, actual_power and status_detail all go EMPTY. reconcile.go:350-352
puts quarantined VMs in Quarantined[] rather than Vms[], so the server has
no ActualVM for them and every actual-state field blanks — precisely during the
one window where the operator most wants to know what teardown is doing.

Net effect: for 5 minutes the console and API say "running", with no phase and no
detail, for a guest that is already stopped and counting down to disk destruction.

Repro

DELETE /api/v1/vms/{id}
GET /api/v1/vms      # power_state=running, phase="", actual_power="", status_detail=""
                     # lifecycle=deleting, destroy_at=<now+5min>

Why it matters

I misread this as a stalled teardown and went looking for agent logs. The only
in-band signal that the agent had actually acted was the host's allocated
dropping (releaseCompute runs at quarantine) — which requires knowing to check a
different endpoint and knowing each VM's shape to do the arithmetic.

destroy_at is present and correct, so the countdown IS on the wire; the
surrounding fields just contradict it.

Suggested shape (not decided)

A phase=quarantined (or stopping/draining) with status_detail carrying the
countdown would answer the question directly. Whether that means synthesising an
ActualVM for quarantined VMs server-side, or widening the Quarantined[] entry, is
the design question. Note the wire has a field-NUMBER lock (#32) — a new field
needs a fresh number.

Migrated from git-collab issue 5a39478b47db30b9a632f01af37bfb01f498951e.

Original author: a73x dev@a73x.sh.

[claude 2026-08-21] Found while tearing down the demo VMs on onyx. A tombstoned VM spends 5 minutes (`-tombstone-grace`, internal/agent/run/cli.go:138) in quarantine before its disk is destroyed. During that window the API reports the VM in a way that actively misleads: - `power_state` still reads `running` — that is stale DESIRED state. The agent has already called `Prov.Shutdown` (internal/agent/reconcile/reconcile.go:481); the guest is off within seconds of the DELETE. - `phase`, `actual_power` and `status_detail` all go EMPTY. reconcile.go:350-352 puts quarantined VMs in `Quarantined[]` rather than `Vms[]`, so the server has no ActualVM for them and every actual-state field blanks — precisely during the one window where the operator most wants to know what teardown is doing. Net effect: for 5 minutes the console and API say "running", with no phase and no detail, for a guest that is already stopped and counting down to disk destruction. ## Repro DELETE /api/v1/vms/{id} GET /api/v1/vms # power_state=running, phase="", actual_power="", status_detail="" # lifecycle=deleting, destroy_at=<now+5min> ## Why it matters I misread this as a stalled teardown and went looking for agent logs. The only in-band signal that the agent had actually acted was the host's `allocated` dropping (releaseCompute runs at quarantine) — which requires knowing to check a different endpoint and knowing each VM's shape to do the arithmetic. `destroy_at` is present and correct, so the countdown IS on the wire; the surrounding fields just contradict it. ## Suggested shape (not decided) A `phase=quarantined` (or `stopping`/`draining`) with `status_detail` carrying the countdown would answer the question directly. Whether that means synthesising an ActualVM for quarantined VMs server-side, or widening the Quarantined[] entry, is the design question. Note the wire has a field-NUMBER lock (#32) — a new field needs a fresh number. --- Migrated from git-collab issue `5a39478b47db30b9a632f01af37bfb01f498951e`. Original author: a73x <dev@a73x.sh>.