Smoke VMs leak: smoke-c546aac6 survived 11 days on onyx

open   by a73x

Labels: backlog

[claude 2026-08-21] While tearing down the demo VMs on onyx I found
`smoke-c546aac6` (2 vCPU / 2048 MB / 10 GB) still running on onyx, created
2026-08-10 16:28:14Z — alive for 11 days. Smoke VMs are meant to be torn down by
the run that created them.

Two `sandbox-*` VMs from the same afternoon (sandbox-a609d8 15:10, sandbox-03e9c8
16:06) were also still up, so whatever happened on 2026-08-10 may have stranded
several at once rather than being specific to the smoke.

All three had zero exposures and zero sessions; nothing was using them. I deleted
them today, so the evidence on the host is gone — this issue is to fix the
cleanup path, not to recover that state.

## Why it matters

A leaked smoke VM holds compute against the host quota forever. onyx was at
15/16 vCPU allocated before this cleanup and 8/16 after; 5 of those 7 reclaimed
vCPU were leaked test VMs. A future smoke run could be refused for capacity
because of its own leaked predecessors.

## What to check

- Does the smoke delete its VM on the failure path, or only on success? A run
  that dies mid-way (or is killed) plausibly skips teardown entirely.
- Is teardown deferred in a way that a non-zero exit or a signal skips?
- Worth considering a belt-and-braces reaper: smoke VMs are named `smoke-<hex>`,
  so a run could delete any pre-existing `smoke-*` older than some age before it
  starts — self-healing without needing the failure path to be perfect.

Comments

[claude 2026-09-02] Another instance, on the gate fleet: a boot gate that FAILS at the gate-SSH leg never reaches the reap, so smoke-15bf25a8 (2 vCPU) stayed on the devhost; the next make deploy then failed at placement with 'needs 8 vcpus, host cap 6' — a capacity error that reads nothing like 'the previous run left a VM behind'. Two things worth doing: reap on any scenario exit (defer), and have the smoke name any leftover smoke-* VMs it finds before creating its own.