a73x

e4d75234

smoke: a host nobody could converge cannot fail a release

a73x   2026-09-05 19:06

Commit message
smoke: a host nobody could converge cannot fail a release

The version gate this scenario grew for #33 judged whichever host the
listing put first, which on the v0.0.8 prod ship was a machine that was
switched off. Two connected hosts had converged and were ready to prove
the release; the run failed on the third, and no retry could have passed
while it stayed off — the ship had already warned and skipped that host
one stage earlier, for the same reason.

The scenario now places on the first CONNECTED host. This is not shopping
for a host on the release: every connected host is still judged, so a
fleet that is up and behind fails exactly as it did. Only hosts nobody
could have converged are skipped, and a fleet with none left connected
fails saying which half is at fault.

The image scanner moves to trivy 0.74 in the same breath, because the
same ship printed the evidence: 0.58 cannot parse go1.27.0-X:nodwarf5, so
it matched no Go module against any advisory and the gobinary half of
every scan was doing nothing.
RETRO.md
Old New
@@ -99,3 +99,10 @@ One line per push to `main`: what slowed the work down. Enforced by
99 ImagePullBackOff nobody was watching, which is the exact failure the change 99 ImagePullBackOff nobody was watching, which is the exact failure the change
100 set out to end. A workload that changes registries changes its auth, and only 100 set out to end. A workload that changes registries changes its auth, and only
101 running the job proves it. 101 running the job proves it.
102 - Scanner blindness: the flood of "Version matching error" lines during the
103 v0.0.8 ship read as cosmetic noise and was the finding. trivy 0.58 could not
104 parse the toolchain's go1.27.0-X:nodwarf5 stamp, so it matched no Go module
105 against any advisory and said so once per module rather than once per image —
106 loud enough to be ignored, quiet enough to look like a warning. A scanner that
107 cannot read a version passes; only reading the summary and noticing the
108 gobinary targets were missing would have caught it.
internal/smoke/scenario.go
Old New
@@ -5,6 +5,7 @@ import (
5 "errors" 5 "errors"
6 "fmt" 6 "fmt"
7 "regexp" 7 "regexp"
8 "strings"
8 "time" 9 "time"
9 10
10 "github.com/a73x/eitri/internal/server/api/client" 11 "github.com/a73x/eitri/internal/server/api/client"
@@ -106,6 +107,36 @@ func proveHostRelease(h client.Host, want string) error {
106 h.Name, reported, want, want) 107 h.Name, reported, want, want)
107 } 108 }
108 109
110 // placeableHost picks the host this run will place its VM on: the first one
111 // that is connected.
112 //
113 // A dark host is skipped rather than judged. The ship has already dealt with it
114 // one stage earlier — it cannot take an upgrade offer while it is disconnected,
115 // so stage 8 names it, warns, and carries on — and a host that cannot take an
116 // offer cannot take a VM either. Letting it reach proveHostRelease meant a
117 // machine that was merely switched off could fail a release two other hosts had
118 // converged and were ready to prove, with no retry that could ever pass while it
119 // stayed off.
120 //
121 // This is deliberately not "search for a host on the release". Every connected
122 // host is still judged by proveHostRelease, so a fleet that is up and behind
123 // fails exactly as it did — which is the #33 defect this whole check exists for.
124 // The only hosts skipped are the ones nobody could have converged.
125 func placeableHost(hosts []client.Host) (client.Host, error) {
126 for _, h := range hosts {
127 if h.Online {
128 return h, nil
129 }
130 }
131 names := make([]string, 0, len(hosts))
132 for _, h := range hosts {
133 names = append(names, h.Name)
134 }
135 return client.Host{}, fmt.Errorf("FAIL: no connected host to place a VM on — every host is dark (%s). "+
136 "The plane is up, so this is the fleet, not the release: bring a host back and resume "+
137 "with scripts/ship.sh --from 8", strings.Join(names, ", "))
138 }
139
109 // gateHooks bundles the optional SSH-CA gate steps. nil means "skip the gate". 140 // gateHooks bundles the optional SSH-CA gate steps. nil means "skip the gate".
110 type gateHooks struct { 141 type gateHooks struct {
111 register func(ctx context.Context) error // upload the smoke user CA to the tenant (before create) 142 register func(ctx context.Context) error // upload the smoke user CA to the tenant (before create)
@@ -175,12 +206,15 @@ func runScenario(ctx context.Context, vmName string, c vmAPI, dialConsole consol
175 if len(hostList) == 0 { 206 if len(hostList) == 0 {
176 return "", errors.New("no hosts available") 207 return "", errors.New("no hosts available")
177 } 208 }
178 host := hostList[0] 209 host, err := placeableHost(hostList)
210 if err != nil {
211 return "", err
212 }
179 // Before anything is created: there is no reason to boot-prove a VM on an 213 // Before anything is created: there is no reason to boot-prove a VM on an
180 // agent this run was never meant to prove. The list is not searched for a 214 // agent this run was never meant to prove. The list is still not searched
181 // host that does match, either — a fleet where only some hosts converged is 215 // for a host that does match — a CONNECTED fleet where only some hosts
182 // itself the defect, and shopping for a working host would hide exactly the 216 // converged is itself the defect, and shopping among those would hide
183 // bug this check exists to expose. 217 // exactly the bug this check exists to expose.
184 if err := proveHostRelease(host, expectAgentVersion); err != nil { 218 if err := proveHostRelease(host, expectAgentVersion); err != nil {
185 return "", err 219 return "", err
186 } 220 }
internal/smoke/scenario_test.go
Old New
@@ -166,7 +166,7 @@ func TestRunScenarioSuccess(t *testing.T) {
166 deleted := false 166 deleted := false
167 api := &testAPI{ 167 api := &testAPI{
168 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 168 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
169 return []client.Host{{ID: "host-1", UplinkAddr: "192.168.0.190"}}, nil 169 return []client.Host{{ID: "host-1", Online: true, UplinkAddr: "192.168.0.190"}}, nil
170 }, 170 },
171 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) { 171 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
172 if req.HostID != "host-1" { 172 if req.HostID != "host-1" {
@@ -241,7 +241,7 @@ func TestRunScenarioRebootDeathFails(t *testing.T) {
241 power := "running" 241 power := "running"
242 api := &testAPI{ 242 api := &testAPI{
243 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 243 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
244 return []client.Host{{ID: "host-1", UplinkAddr: "192.168.0.190"}}, nil 244 return []client.Host{{ID: "host-1", Online: true, UplinkAddr: "192.168.0.190"}}, nil
245 }, 245 },
246 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) { 246 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
247 return client.CreateVMResponse{ID: "vm-1"}, nil 247 return client.CreateVMResponse{ID: "vm-1"}, nil
@@ -305,7 +305,7 @@ func TestRunScenarioExposureFailureFails(t *testing.T) {
305 func TestRunScenarioSerialPanicFails(t *testing.T) { 305 func TestRunScenarioSerialPanicFails(t *testing.T) {
306 api := &testAPI{ 306 api := &testAPI{
307 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 307 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
308 return []client.Host{{ID: "host-1", UplinkAddr: "192.168.0.190"}}, nil 308 return []client.Host{{ID: "host-1", Online: true, UplinkAddr: "192.168.0.190"}}, nil
309 }, 309 },
310 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) { 310 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
311 return client.CreateVMResponse{ID: "vm-1"}, nil 311 return client.CreateVMResponse{ID: "vm-1"}, nil
@@ -335,7 +335,7 @@ func TestRunScenarioSerialPanicFails(t *testing.T) {
335 func TestRunScenarioNeverReadyTimesOut(t *testing.T) { 335 func TestRunScenarioNeverReadyTimesOut(t *testing.T) {
336 api := &testAPI{ 336 api := &testAPI{
337 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 337 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
338 return []client.Host{{ID: "host-1", UplinkAddr: "192.168.0.190"}}, nil 338 return []client.Host{{ID: "host-1", Online: true, UplinkAddr: "192.168.0.190"}}, nil
339 }, 339 },
340 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) { 340 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
341 return client.CreateVMResponse{ID: "vm-1"}, nil 341 return client.CreateVMResponse{ID: "vm-1"}, nil
@@ -376,7 +376,7 @@ func TestRunScenarioNeverReapedTimesOut(t *testing.T) {
376 power := "running" 376 power := "running"
377 api := &testAPI{ 377 api := &testAPI{
378 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 378 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
379 return []client.Host{{ID: "host-1", UplinkAddr: "192.168.0.190"}}, nil 379 return []client.Host{{ID: "host-1", Online: true, UplinkAddr: "192.168.0.190"}}, nil
380 }, 380 },
381 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) { 381 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
382 return client.CreateVMResponse{ID: "vm-1"}, nil 382 return client.CreateVMResponse{ID: "vm-1"}, nil
@@ -445,7 +445,7 @@ func happyPathAPI(t *testing.T, calls *[]string) *testAPI {
445 created := 0 445 created := 0
446 return &testAPI{ 446 return &testAPI{
447 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 447 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
448 return []client.Host{{ID: "host-1", UplinkAddr: "192.168.0.190"}}, nil 448 return []client.Host{{ID: "host-1", Online: true, UplinkAddr: "192.168.0.190"}}, nil
449 }, 449 },
450 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) { 450 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
451 *calls = append(*calls, "createVM") 451 *calls = append(*calls, "createVM")
@@ -731,7 +731,7 @@ func TestRunScenarioFailsWhenTheMCPLegFails(t *testing.T) {
731 func TestRunScenarioRefusesAHostBehindTheRelease(t *testing.T) { 731 func TestRunScenarioRefusesAHostBehindTheRelease(t *testing.T) {
732 api := &testAPI{ 732 api := &testAPI{
733 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 733 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
734 return []client.Host{{ID: "host-1", Name: "onyx", AgentVersion: "v0.0.6"}}, nil 734 return []client.Host{{ID: "host-1", Name: "onyx", Online: true, AgentVersion: "v0.0.6"}}, nil
735 }, 735 },
736 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) { 736 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
737 t.Fatal("CreateVM must not be called on a host that is not running the release") 737 t.Fatal("CreateVM must not be called on a host that is not running the release")
@@ -769,7 +769,7 @@ func TestRunScenarioAcceptsTheHostRunningTheRelease(t *testing.T) {
769 listHostsCalls := 0 769 listHostsCalls := 0
770 api.listHostsFunc = func(ctx context.Context) ([]client.Host, error) { 770 api.listHostsFunc = func(ctx context.Context) ([]client.Host, error) {
771 listHostsCalls++ 771 listHostsCalls++
772 return []client.Host{{ID: "host-1", Name: "onyx", UplinkAddr: "192.168.0.190", AgentVersion: "v0.0.7"}}, nil 772 return []client.Host{{ID: "host-1", Name: "onyx", Online: true, UplinkAddr: "192.168.0.190", AgentVersion: "v0.0.7"}}, nil
773 } 773 }
774 774
775 clock := &fakeClock{t: time.Unix(0, 0)} 775 clock := &fakeClock{t: time.Unix(0, 0)}
@@ -794,7 +794,7 @@ func TestRunScenarioWithoutAnExpectationJudgesNoVersion(t *testing.T) {
794 listHostsCalls := 0 794 listHostsCalls := 0
795 api.listHostsFunc = func(ctx context.Context) ([]client.Host, error) { 795 api.listHostsFunc = func(ctx context.Context) ([]client.Host, error) {
796 listHostsCalls++ 796 listHostsCalls++
797 return []client.Host{{ID: "host-1", Name: "onyx", UplinkAddr: "192.168.0.190", AgentVersion: "dev"}}, nil 797 return []client.Host{{ID: "host-1", Name: "onyx", Online: true, UplinkAddr: "192.168.0.190", AgentVersion: "dev"}}, nil
798 } 798 }
799 799
800 clock := &fakeClock{t: time.Unix(0, 0)} 800 clock := &fakeClock{t: time.Unix(0, 0)}
@@ -822,3 +822,99 @@ func TestProveHostReleaseNamesAnAgentTooOldToNameItself(t *testing.T) {
822 t.Errorf("error = %q, want it to say the host reports no version at all", err.Error()) 822 t.Errorf("error = %q, want it to say the host reports no version at all", err.Error())
823 } 823 }
824 } 824 }
825
826 // TestRunScenarioSkipsADarkHostAndProvesOnAConnectedOne is the other half of
827 // the #33 story, learned on the first prod ship of v0.0.8: a host that is
828 // simply switched off must not fail a release that two connected hosts have
829 // converged and are ready to prove. The ship already said its piece about that
830 // host one stage earlier — it cannot take an upgrade offer while it is dark —
831 // and no retry could ever pass while it stayed off.
832 func TestRunScenarioSkipsADarkHostAndProvesOnAConnectedOne(t *testing.T) {
833 var calls []string
834 api := happyPathAPI(t, &calls)
835 var placedOn string
836 api.listHostsFunc = func(ctx context.Context) ([]client.Host, error) {
837 return []client.Host{
838 // Dark, never reported a version: exactly onyx on 2026-09-05.
839 {ID: "host-dark", Name: "onyx", Online: false, AgentVersion: ""},
840 {ID: "host-up", Name: "charizard", Online: true, AgentVersion: "v0.0.8"},
841 }, nil
842 }
843 create := api.createVMFunc
844 api.createVMFunc = func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
845 placedOn = req.HostID
846 return create(ctx, req)
847 }
848
849 clock := &fakeClock{t: time.Unix(0, 0)}
850 msg, err := runScenario(context.Background(), "smoke-test", api, powerCycleConsole(t, api, bootedGuest, "ubuntu-vm login: ").dial, nil, nil, "v0.0.8", clock.now, clock.sleep, noopReadPubKey, okBanner)
851 if err != nil {
852 t.Fatalf("runScenario: want the dark host skipped, got %v", err)
853 }
854 if placedOn != "host-up" {
855 t.Errorf("placed on %q, want the connected host host-up", placedOn)
856 }
857 if !strings.Contains(msg, "SMOKE COMPLETE") {
858 t.Errorf("message = %q, want a complete run", msg)
859 }
860 }
861
862 // TestRunScenarioStillRefusesAConnectedHostBehindTheRelease guards the line the
863 // fix above must not cross. Skipping dark hosts is not "shop for a host on the
864 // release": a host that is UP and behind is the #33 defect itself, and it still
865 // fails even when a converged host sits beside it in the same listing.
866 func TestRunScenarioStillRefusesAConnectedHostBehindTheRelease(t *testing.T) {
867 api := &testAPI{
868 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
869 return []client.Host{
870 {ID: "host-old", Name: "onyx", Online: true, AgentVersion: "v0.0.7"},
871 {ID: "host-up", Name: "charizard", Online: true, AgentVersion: "v0.0.8"},
872 }, nil
873 },
874 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
875 t.Fatal("CreateVM must not be called while a connected host is behind the release")
876 return client.CreateVMResponse{}, nil
877 },
878 }
879
880 clock := &fakeClock{t: time.Unix(0, 0)}
881 _, err := runScenario(context.Background(), "smoke-test", api, nil, nil, nil, "v0.0.8", clock.now, clock.sleep, noopReadPubKey, okBanner)
882 if err == nil {
883 t.Fatal("runScenario: want a failure on a connected host a release behind, got nil")
884 }
885 for _, want := range []string{"onyx", "v0.0.7", "v0.0.8", "did not converge"} {
886 if !strings.Contains(err.Error(), want) {
887 t.Errorf("error = %q, want it to mention %q", err.Error(), want)
888 }
889 }
890 }
891
892 // TestRunScenarioFailsWhenEveryHostIsDark: skipping dark hosts cannot become
893 // skipping the proof. With nobody left to place on, the run fails — and says
894 // this is the fleet rather than the release, because a reader who has just been
895 // told the plane is serving the tag deserves to know which half is at fault.
896 func TestRunScenarioFailsWhenEveryHostIsDark(t *testing.T) {
897 api := &testAPI{
898 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {
899 return []client.Host{
900 {ID: "host-a", Name: "onyx", Online: false, AgentVersion: "v0.0.8"},
901 {ID: "host-b", Name: "Squirtle.local", Online: false, AgentVersion: "v0.0.8"},
902 }, nil
903 },
904 createVMFunc: func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
905 t.Fatal("CreateVM must not be called when no host is connected")
906 return client.CreateVMResponse{}, nil
907 },
908 }
909
910 clock := &fakeClock{t: time.Unix(0, 0)}
911 _, err := runScenario(context.Background(), "smoke-test", api, nil, nil, nil, "v0.0.8", clock.now, clock.sleep, noopReadPubKey, okBanner)
912 if err == nil {
913 t.Fatal("runScenario: want a failure when every host is dark, got nil")
914 }
915 for _, want := range []string{"onyx", "Squirtle.local", "dark", "--from 8"} {
916 if !strings.Contains(err.Error(), want) {
917 t.Errorf("error = %q, want it to mention %q", err.Error(), want)
918 }
919 }
920 }
scripts/scan-image.sh
Old New
@@ -21,7 +21,14 @@ set -euo pipefail
21 21
22 IMAGE="${1:?scan-image: usage: scan-image.sh <image:tag>}" 22 IMAGE="${1:?scan-image: usage: scan-image.sh <image:tag>}"
23 SEVERITY="${SCAN_SEVERITY:-HIGH,CRITICAL}" 23 SEVERITY="${SCAN_SEVERITY:-HIGH,CRITICAL}"
24 TRIVY_IMAGE="${TRIVY_IMAGE:-docker.io/aquasec/trivy:0.58.1}" 24 # 0.74, not 0.58: the older scanner could not parse the version string this
25 # toolchain stamps into every Go binary — go1.27.0-X:nodwarf5, the experiment
26 # tag baked into the packaged toolchain — and answered by printing one
27 # "Version matching error" per module and matching NONE of them. The Go half of
28 # every image scan was quietly doing nothing while the OS half carried the gate.
29 # A scanner that cannot read a version does not fail; it passes. Bump this when
30 # the toolchain moves again, and check the summary names each gobinary target.
31 TRIVY_IMAGE="${TRIVY_IMAGE:-docker.io/aquasec/trivy:0.74.0}"
25 DOCKER="$(command -v docker || command -v podman)" 32 DOCKER="$(command -v docker || command -v podman)"
26 33
27 if [ "${SKIP_IMAGE_SCAN:-}" = "1" ]; then 34 if [ "${SKIP_IMAGE_SCAN:-}" = "1" ]; then