Smoke VMs leak: smoke-c546aac6 survived 11 days on onyx #75

New issue

Open

opened 2026-08-21 18:32:15 +00:00 by a73x · 1 comment

a73x commented

2026-08-21 18:32:15 +00:00

(Migrated from git.a73x.sh)

[claude 2026-08-21] While tearing down the demo VMs on onyx I found
smoke-c546aac6 (2 vCPU / 2048 MB / 10 GB) still running on onyx, created
2026-08-10 16:28:14Z — alive for 11 days. Smoke VMs are meant to be torn down by
the run that created them.

Two sandbox-* VMs from the same afternoon (sandbox-a609d8 15:10, sandbox-03e9c8
16:06) were also still up, so whatever happened on 2026-08-10 may have stranded
several at once rather than being specific to the smoke.

All three had zero exposures and zero sessions; nothing was using them. I deleted
them today, so the evidence on the host is gone — this issue is to fix the
cleanup path, not to recover that state.

Why it matters

A leaked smoke VM holds compute against the host quota forever. onyx was at
15/16 vCPU allocated before this cleanup and 8/16 after; 5 of those 7 reclaimed
vCPU were leaked test VMs. A future smoke run could be refused for capacity
because of its own leaked predecessors.

What to check

Does the smoke delete its VM on the failure path, or only on success? A run
that dies mid-way (or is killed) plausibly skips teardown entirely.
Is teardown deferred in a way that a non-zero exit or a signal skips?
Worth considering a belt-and-braces reaper: smoke VMs are named smoke-<hex>,
so a run could delete any pre-existing smoke-* older than some age before it
starts — self-healing without needing the failure path to be perfect.

Migrated from git-collab issue cf4a75da1dd43b8fb434d01b231456aa2268408c.

Original author: a73x dev@a73x.sh.

[claude 2026-08-21] While tearing down the demo VMs on onyx I found `smoke-c546aac6` (2 vCPU / 2048 MB / 10 GB) still running on onyx, created 2026-08-10 16:28:14Z — alive for 11 days. Smoke VMs are meant to be torn down by the run that created them. Two `sandbox-*` VMs from the same afternoon (sandbox-a609d8 15:10, sandbox-03e9c8 16:06) were also still up, so whatever happened on 2026-08-10 may have stranded several at once rather than being specific to the smoke. All three had zero exposures and zero sessions; nothing was using them. I deleted them today, so the evidence on the host is gone — this issue is to fix the cleanup path, not to recover that state. ## Why it matters A leaked smoke VM holds compute against the host quota forever. onyx was at 15/16 vCPU allocated before this cleanup and 8/16 after; 5 of those 7 reclaimed vCPU were leaked test VMs. A future smoke run could be refused for capacity because of its own leaked predecessors. ## What to check - Does the smoke delete its VM on the failure path, or only on success? A run that dies mid-way (or is killed) plausibly skips teardown entirely. - Is teardown deferred in a way that a non-zero exit or a signal skips? - Worth considering a belt-and-braces reaper: smoke VMs are named `smoke-<hex>`, so a run could delete any pre-existing `smoke-*` older than some age before it starts — self-healing without needing the failure path to be perfect. --- Migrated from git-collab issue `cf4a75da1dd43b8fb434d01b231456aa2268408c`. Original author: a73x <dev@a73x.sh>.

a73x commented

2026-09-02 09:23:53 +00:00

(Migrated from git.a73x.sh)

[claude 2026-09-02] Another instance, on the gate fleet: a boot gate that FAILS at the gate-SSH leg never reaches the reap, so smoke-15bf25a8 (2 vCPU) stayed on the devhost; the next make deploy then failed at placement with 'needs 8 vcpus, host cap 6' — a capacity error that reads nothing like 'the previous run left a VM behind'. Two things worth doing: reap on any scenario exit (defer), and have the smoke name any leftover smoke-* VMs it finds before creating its own.