a73x

ef7319c9

seed: a tenant's own cloud-init keeps eitri's trust file and disk grow

a73x   2026-09-02 09:42

Commit message
seed: a tenant's own cloud-init keeps eitri's trust file and disk grow

A guest booted from a tenant cloud_init carrying write_files or runcmd lost
eitri's own: no /etc/ssh/eitri_user_ca.pub, so every CA-signed certificate was
refused, or no online root grow, so a 10G disk booted with the image's 3.5G
root. cloud-init merges user-data over vendor-data first-wins per key, lists
not appended, and its part handler pops merge_how before that cross-source
merge ever runs, so no directive in vendor-data can change it.

The seed now composes user-data as a MIME archive: the tenant's document
verbatim first, eitri's cloud-config last with
merge_how "list(prepend)+dict(recurse_array,no_replace)+str()" inside it.
Within one source the handler merges each part under that part's own mergers,
so eitri's write_files and runcmd are prepended to the tenant's and every
scalar the tenant set still wins, growpart included. No vendor-data is written.
cloudinit.AppendCloudConfig shares the wrap-or-append path with AddSSHKey.

The boot gate proves it: the smoke's VM is created with a tenant document
carrying both keys, and the gate leg, inside its retry window, reads back a
root above half the default disk (DefaultDiskGB re-exported through the API
client, R11) plus the tenant's own marker files. assumptions.md records the
mechanism, the falsified vendor-data route, and the falsifier.

Patch: aba61eb929696b4bb08206e78f47f0a387cba0e5
deploy/server/README.md
Old New
@@ -455,7 +455,7 @@ the other's guests.
455 All four of these must pass before mewtwo is allowed to move: 455 All four of these must pass before mewtwo is allowed to move:
456 456
457 1. `devhost.sh create` from nothing, then `make deploy` → boot gate **PASS** 457 1. `devhost.sh create` from nothing, then `make deploy` → boot gate **PASS**
458 including `gate SSH: ok`, `exposed port: ok`, `remote MCP: ok`. A first pass 458 including `gate SSH: ok`, `BYO cloud-init merge: ok`, `exposed port: ok`, `remote MCP: ok`. A first pass
459 on a cold host also proves the agent bootstrap — cloud-hypervisor and 459 on a cold host also proves the agent bootstrap — cloud-hypervisor and
460 `CLOUDHV.fd` downloaded and sha-verified — works nested. 460 `CLOUDHV.fd` downloaded and sha-verified — works nested.
461 2. A second `make deploy` on the warm host → PASS. Proves the agent-swap path 461 2. A second `make deploy` on the warm host → PASS. Proves the agent-swap path
docs/assumptions.md
Old New
@@ -702,3 +702,32 @@ sweeps the whole import graph for `protojson`, and `depguard` denies it in the
702 editor. Falsified the moment that import lands anywhere, because the names it 702 editor. Falsified the moment that import lands anywhere, because the names it
703 writes to a log or a row become a contract retroactively — and the rename after 703 writes to a log or a row become a contract retroactively — and the rename after
704 that one breaks it silently. 704 that one breaks it silently.
705
706 ### cloud-init honours `merge_how` from the part being merged in
707
708 eitri's own cloud-config — the CA trust file and sshd drop-in, the clock
709 drop-in, the online root grow, the sshd reload — reaches a guest as the LAST
710 part of a multipart user-data, after the tenant's own document. Within one data
711 source, cloud-init's cloud-config handler folds each part into what the earlier
712 parts built, using the mergers the incoming part names, and a part that names
713 none falls back to `dict(replace)+list()+str()`. So eitri's part carries
714 `merge_how: "list(prepend)+dict(recurse_array,no_replace)+str()"`, and that one
715 line is why a tenant's `write_files` and eitri's both land, why eitri's grow
716 runs before the tenant's commands, and why every scalar the tenant set still
717 wins. Underpins the whole seed contract for BYO cloud-init.
718 **Traced** 2026-09-02 in cloud-init 26.1's own source (`handlers/cloud_config.py`
719 and `mergers/__init__.py`) on the Ubuntu resolute image, where the merger string
720 itself was also run against the guest's copy. Held to the boot gate from here:
721 the smoke's VM is created with a tenant document carrying both keys, and the
722 gate leg reads back a CA-signed login, a grown root, and the tenant's own
723 markers, so a run that goes green has proven this and a run that does not fails.
724 Falsified by a guest whose BYO `write_files` leaves `/etc/ssh/eitri_user_ca.pub`
725 absent — it boots, runs sshd, and refuses only CA-signed certificates, which is
726 why the boot gate has to ask rather than assume.
727
728 The same claim was first made for vendor-data and was **falsified by the boot
729 gate on 2026-09-02**: `mergers.dict_extract_mergers` pops `merge_how` out of a
730 document the first time the part handler reads it and the stripped result is
731 what gets written, so the later user-data-versus-vendor-data merge never sees
732 one and its first-wins default stands. Vendor-data was dropped from the seed
733 entirely rather than left as a second mechanism.
docs/mcp.md
Old New
@@ -15,7 +15,7 @@ self-hosted eitri is a self-hosted MCP server, with nothing extra to install.
15 15
16 | Tool | Purpose | 16 | Tool | Purpose |
17 |---|---| 17 |---|---|
18 | `vm_create` | Create a VM; by default waits for `lifecycle=ready` + IP, then for guest SSH and `cloud-init status --wait` to finish. Takes an optional `network`, the named host network to give the guest a second NIC on; a host serves only the names it advertises, and naming another refuses the create. Returns `network` and `network_ip`—the address that network's own DHCP granted, empty at ready when it has not answered yet. Refused while the tenant has no registered SSH user CA—call `ca_upload` first, or the guest would trust nothing. | 18 | `vm_create` | Create a VM; by default waits for `lifecycle=ready` + IP, then for guest SSH and `cloud-init status --wait` to finish. Takes an optional `network`, the named host network to give the guest a second NIC on; a host serves only the names it advertises, and naming another refuses the create. Returns `network` and `network_ip`—the address that network's own DHCP granted, empty at ready when it has not answered yet. Refused while the tenant has no registered SSH user CA—call `ca_upload` first, or the guest would trust nothing. An optional `cloud_init` is the guest's own user-data, taken verbatim: eitri's own `write_files` and `runcmd` (the CA trust file, the sshd drop-in, the clock drop-in, the root grow) are **prepended** to yours rather than replaced by them, and every scalar you set—`growpart` included—still wins. |
19 | `vm_list` | List all VMs on the fleet (id, name, lifecycle, IP, size, `network` and `network_ip`). | 19 | `vm_list` | List all VMs on the fleet (id, name, lifecycle, IP, size, `network` and `network_ip`). |
20 | `vm_info` | Show one VM's state, both addresses, plus a ready-to-use `ssh` command. Where to read `network_ip` back when it was still empty at create. | 20 | `vm_info` | Show one VM's state, both addresses, plus a ready-to-use `ssh` command. Where to read `network_ip` back when it was still empty at create. |
21 | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. | 21 | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. |
@@ -101,7 +101,8 @@ three-step recipe rather than a generic authentication failure.
101 101
102 VMs are addressed by their namespaced connect name, not IP, so recycled IPs and 102 VMs are addressed by their namespaced connect name, not IP, so recycled IPs and
103 host-key churn are not a concern. The guest trusts the tenant's registered user 103 host-key churn are not a concern. The guest trusts the tenant's registered user
104 CAs via `TrustedUserCAKeys` (provisioned through vendor-data), so no per-VM 104 CAs via `TrustedUserCAKeys` (provisioned by eitri's own cloud-config, merged
105 with yours), so no per-VM
105 `authorized_key` injection is needed. The PAT authenticates API calls only; SSH 106 `authorized_key` injection is needed. The PAT authenticates API calls only; SSH
106 traffic uses the certificate, and the token itself is never surfaced in a tool 107 traffic uses the certificate, and the token itself is never surfaced in a tool
107 result or error. 108 result or error.
docs/shape.html
Old New
@@ -269,6 +269,7 @@
269 "plane": "data", 269 "plane": "data",
270 "synopsis": "Package seed builds the cloud-init NoCloud config-drive ISO (label CIDATA).", 270 "synopsis": "Package seed builds the cloud-init NoCloud config-drive ISO (label CIDATA).",
271 "imports": [ 271 "imports": [
272 "internal/cloudinit",
272 "internal/guest" 273 "internal/guest"
273 ] 274 ]
274 }, 275 },
docs/shape.json
Old New
@@ -218,6 +218,7 @@
218 "plane": "data", 218 "plane": "data",
219 "synopsis": "Package seed builds the cloud-init NoCloud config-drive ISO (label CIDATA).", 219 "synopsis": "Package seed builds the cloud-init NoCloud config-drive ISO (label CIDATA).",
220 "imports": [ 220 "imports": [
221 "internal/cloudinit",
221 "internal/guest" 222 "internal/guest"
222 ] 223 ]
223 }, 224 },
internal/agent/seed/seed.go
Old New
@@ -6,6 +6,7 @@ import (
6 "os" 6 "os"
7 "strings" 7 "strings"
8 8
9 "github.com/a73x/eitri/internal/cloudinit"
9 "github.com/a73x/eitri/internal/guest" 10 "github.com/a73x/eitri/internal/guest"
10 diskfs "github.com/diskfs/go-diskfs" 11 diskfs "github.com/diskfs/go-diskfs"
11 "github.com/diskfs/go-diskfs/disk" 12 "github.com/diskfs/go-diskfs/disk"
@@ -112,12 +113,14 @@ func sshdDropIn(p Params) string {
112 // growRuncmd grows the root partition to fill disk_gb and resizes the 113 // growRuncmd grows the root partition to fill disk_gb and resizes the
113 // filesystem, online, on the mounted root — eitri's side of the contract 114 // filesystem, online, on the mounted root — eitri's side of the contract
114 // (eitri delivers disk_gb to the guest; the guest owns everything above it). 115 // (eitri delivers disk_gb to the guest; the guest owns everything above it).
115 // It lives in vendor-data, not the default user-data, so it applies even when a 116 // It lives in eitri's own cloud-config part, not the default user-data, so it
116 // tenant brings their own cloud-init. 117 // applies even when a tenant brings their own cloud-init — including one with a
118 // `runcmd` of its own, which without the merge directive (see mergeHow) would
119 // replace this.
117 // 120 //
118 // The image ships a GPT sized for its built-in ~3.5G disk; the agent extends the 121 // The image ships a GPT sized for its built-in ~3.5G disk; the agent extends the
119 // raw file to disk_gb, which strands the GPT's backup header mid-device. The 122 // raw file to disk_gb, which strands the GPT's backup header mid-device. The
120 // image's own cloud-init growpart is therefore turned OFF (see vendorDataDoc): 123 // image's own cloud-init growpart is therefore turned OFF (see eitriConfigDoc):
121 // its sfdisk resizer treats sfdisk's non-zero exit on the "GPT backup not at the 124 // its sfdisk resizer treats sfdisk's non-zero exit on the "GPT backup not at the
122 // end / PMBR size mismatch" auto-correction as a failure and REVERTS the grow, 125 // end / PMBR size mismatch" auto-correction as a failure and REVERTS the grow,
123 // so the root never grows and cloud-init reports degraded. This runcmd does the 126 // so the root never grows and cloud-init reports degraded. This runcmd does the
@@ -173,20 +176,56 @@ makestep 1 -1
173 // failing the runcmd and reporting cloud-init degraded. 176 // failing the runcmd and reporting cloud-init degraded.
174 const chronyRuncmd = ` - ["sh", "-c", "systemctl try-restart chrony.service 2>/dev/null || :"]` + "\n" 177 const chronyRuncmd = ` - ["sh", "-c", "systemctl try-restart chrony.service 2>/dev/null || :"]` + "\n"
175 178
176 // vendorDataDoc renders the vendor-data #cloud-config for a seed. It always 179 // mergeHow is the merge directive eitri's cloud-config carries, and the only
177 // carries the disk-grow (growpart disabled + the online grow runcmd, see 180 // thing that makes the rest of it survive a tenant's own cloud-init.
178 // growRuncmd) and the clock drop-in (see chronyDropIn); it additionally carries 181 //
179 // the eitri user-CA trust drop-in when the CA key is set and the per-VM host key 182 // cloud-init's cloud-config part handler folds the parts of one data source
180 // + cert when those are set. It is never empty, so a vendor-data file is always 183 // into a single buffer, one at a time, merging each part into what the earlier
181 // written. 184 // parts built. The mergers it uses are the ones named by the part being merged
185 // IN — a `merge_how` key in that part's document, or a `Merge-Type` MIME header
186 // — and a part that names neither falls back to `dict(replace)+list()+str()`:
187 // dict keys REPLACED, lists NOT appended. That fallback is the defect. A tenant
188 // document carrying `write_files:` or `runcmd:` and eitri's document carrying
189 // its own would leave the guest with whichever was folded in last and nothing
190 // of the other — no /etc/ssh/eitri_user_ca.pub and no CA-signed certificate
191 // accepted, or the image's ~3.5G root still sitting on a 10G disk.
192 //
193 // So eitri's document names its own mergers and is the LAST part appended (see
194 // composeUserData): `list(prepend)` puts eitri's list items in front of the
195 // tenant's instead of discarding them; `dict(recurse_array,no_replace)` merges
196 // mappings key by key, recursing into a list a key holds rather than treating
197 // it as one opaque value, and never overwriting a key the tenant set; `str()`
198 // leaves a scalar the tenant set alone. eitri's entries are ADDED to the
199 // tenant's, and every scalar the tenant chose still wins — `growpart: mode`
200 // included, which is what the userData comment below promises them.
201 //
202 // prepend rather than append because the order is load-bearing: the grow
203 // runcmd has to run before whatever the tenant put in runcmd, so their commands
204 // see the disk they asked for and not the image's.
205 //
206 // This is the ONE layer that reads merge_how. The handler pops the key out of
207 // each part as it folds it in and writes the stripped result, so the merge that
208 // happens afterwards — user-data against vendor-data — never sees a merge_how
209 // from either. That is why eitri's config is a part of user-data and there is
210 // no vendor-data in the seed at all.
211 const mergeHow = `merge_how: "list(prepend)+dict(recurse_array,no_replace)+str()"` + "\n"
212
213 // eitriConfigDoc renders eitri's own #cloud-config for a seed — the part
214 // appended after the tenant's document (see composeUserData). It always carries
215 // the disk-grow (growpart disabled + the online grow runcmd, see growRuncmd) and
216 // the clock drop-in (see chronyDropIn); it additionally carries the eitri
217 // user-CA trust drop-in when the CA key is set and the per-VM host key + cert
218 // when those are set. Its first line is the merge directive (see mergeHow) that
219 // keeps all of it from being discarded by a tenant's own write_files or runcmd.
182 // 220 //
183 // The keys, cert, and drop-in conf are embedded as YAML block scalars (`|`), so 221 // The keys, cert, and drop-in conf are embedded as YAML block scalars (`|`), so
184 // their bytes appear verbatim in the guest files. Block scalars are 222 // their bytes appear verbatim in the guest files. Block scalars are
185 // injection-safe: every indented line is literal content. The sshd reload is 223 // injection-safe: every indented line is literal content. The sshd reload is
186 // gated behind `sshd -t` so a malformed config can never lock anyone out. 224 // gated behind `sshd -t` so a malformed config can never lock anyone out.
187 func vendorDataDoc(p Params) string { 225 func eitriConfigDoc(p Params) string {
188 var b strings.Builder 226 var b strings.Builder
189 b.WriteString("#cloud-config\n") 227 b.WriteString("#cloud-config\n")
228 b.WriteString(mergeHow)
190 // Disable the image's growpart: its sfdisk resizer reverts on this disk (see 229 // Disable the image's growpart: its sfdisk resizer reverts on this disk (see
191 // growRuncmd). Quoted so YAML reads the string "off", not the boolean false. 230 // growRuncmd). Quoted so YAML reads the string "off", not the boolean false.
192 b.WriteString("growpart:\n mode: \"off\"\n") 231 b.WriteString("growpart:\n mode: \"off\"\n")
@@ -253,10 +292,11 @@ func writeFileBlock(b *strings.Builder, path, content string) {
253 // default is generated with an SSH key and a default ubuntu user. 292 // default is generated with an SSH key and a default ubuntu user.
254 // 293 //
255 // The default does NOT configure growpart or disk_setup: growing the root to 294 // The default does NOT configure growpart or disk_setup: growing the root to
256 // disk_gb is eitri's job and lives in vendor-data (see vendorDataDoc/growRuncmd), 295 // disk_gb is eitri's job and lives in eitri's own cloud-config part (see
257 // which applies to BYO user-data too. Setting growpart here would also override 296 // eitriConfigDoc/growRuncmd), which applies to BYO user-data too. Setting
258 // vendor-data's `growpart: off` (user-data wins the cloud-init merge), re-arming 297 // growpart here would also win over that part's `growpart: off` — a scalar the
259 // the image growpart that reverts on this disk. 298 // tenant's document set is never overwritten (see mergeHow) — re-arming the
299 // image growpart that reverts on this disk.
260 func userData(p Params) string { 300 func userData(p Params) string {
261 if p.UserData != "" { 301 if p.UserData != "" {
262 return p.UserData 302 return p.UserData
@@ -278,6 +318,30 @@ users:
278 `, p.Hostname, guest.LoginUser, sshKeys) 318 `, p.Hostname, guest.LoginUser, sshKeys)
279 } 319 }
280 320
321 // composeUserData builds the guest's whole user-data: the tenant's document
322 // (or eitri's default when they brought none) first, then eitri's own
323 // cloud-config appended as the last part of a MIME multipart archive. Both
324 // halves reach the guest, and the tenant's bytes are never edited.
325 //
326 // One mechanism, not two. eitri's config used to ride in a separate vendor-data
327 // file, which cloud-init merges against user-data with mergers that no document
328 // can influence — first-wins per key, so a tenant list quietly replaced eitri's
329 // (see mergeHow). Inside ONE data source the parts are folded together under the
330 // mergers the incoming part names, so this is the only arrangement in which
331 // eitri's merge directive is read at all.
332 //
333 // It can fail: a jinja-templated or gzipped tenant document cannot be given an
334 // honest MIME part header, so there is nowhere to put eitri's config. Refusing
335 // to build the seed is the point — the guest that would boot instead is one
336 // with no CA trust file, reachable by nobody and silent about why.
337 func composeUserData(p Params) (string, error) {
338 out, err := cloudinit.AppendCloudConfig(userData(p), eitriConfigDoc(p))
339 if err != nil {
340 return "", fmt.Errorf("seed: %w", err)
341 }
342 return out, nil
343 }
344
281 // metaData returns the cloud-init meta-data content. 345 // metaData returns the cloud-init meta-data content.
282 // InstanceID is used when set; falls back to Hostname for backward compatibility. 346 // InstanceID is used when set; falls back to Hostname for backward compatibility.
283 func metaData(p Params) string { 347 func metaData(p Params) string {
@@ -347,11 +411,19 @@ func networkConfig(p Params) string {
347 411
348 // Build creates a cloud-init NoCloud seed ISO at outPath. 412 // Build creates a cloud-init NoCloud seed ISO at outPath.
349 // The ISO uses volume label "cidata" as required by cloud-init's NoCloud source. 413 // The ISO uses volume label "cidata" as required by cloud-init's NoCloud source.
350 // Files written: /user-data, /meta-data, /network-config. 414 // Files written: /user-data, /meta-data, /network-config. eitri's own config is
415 // the last part of /user-data rather than a /vendor-data file (see
416 // composeUserData), so a seed carries three files and no fourth.
351 func Build(outPath string, p Params) error { 417 func Build(outPath string, p Params) error {
352 if err := validateParams(p); err != nil { 418 if err := validateParams(p); err != nil {
353 return err 419 return err
354 } 420 }
421 // Composed before anything is created, so a tenant document eitri cannot
422 // compose with fails without leaving an artifact behind.
423 userDataDoc, err := composeUserData(p)
424 if err != nil {
425 return err
426 }
355 427
356 // 1 MiB covers the text files plus slack for ISO9660 structures. 428 // 1 MiB covers the text files plus slack for ISO9660 structures.
357 isoSize := int64(1 * 1024 * 1024) 429 isoSize := int64(1 * 1024 * 1024)
@@ -411,16 +483,14 @@ func Build(outPath string, p Params) error {
411 return fmt.Errorf("seed: unexpected filesystem type %T", fsi) 483 return fmt.Errorf("seed: unexpected filesystem type %T", fsi)
412 } 484 }
413 485
486 // No vendor-data: eitri's own config rides as the last part of user-data,
487 // which is the only place cloud-init reads a merge directive from (see
488 // composeUserData).
414 files := map[string]string{ 489 files := map[string]string{
415 "/user-data": userData(p), 490 "/user-data": userDataDoc,
416 "/meta-data": metaData(p), 491 "/meta-data": metaData(p),
417 "/network-config": networkConfig(p), 492 "/network-config": networkConfig(p),
418 } 493 }
419 // Vendor-data carries eitri's own config (disk-grow, and the SSH user-CA
420 // drop-in + host cert when set); it is always non-empty, so always written.
421 if vd := vendorDataDoc(p); vd != "" {
422 files["/vendor-data"] = vd
423 }
424 for name, content := range files { 494 for name, content := range files {
425 f, err := fs.OpenFile(name, os.O_CREATE|os.O_RDWR) 495 f, err := fs.OpenFile(name, os.O_CREATE|os.O_RDWR)
426 if err != nil { 496 if err != nil {
internal/agent/seed/seed_test.go
Old New
@@ -2,6 +2,9 @@ package seed
2 2
3 import ( 3 import (
4 "io" 4 "io"
5 "mime"
6 "mime/multipart"
7 "net/mail"
5 "os" 8 "os"
6 "strings" 9 "strings"
7 "testing" 10 "testing"
@@ -52,10 +55,11 @@ func TestUserDataDefaultInjectsKeyNotGrowpart(t *testing.T) {
52 assert.True(t, strings.HasPrefix(ud, "#cloud-config\n")) 55 assert.True(t, strings.HasPrefix(ud, "#cloud-config\n"))
53 assert.Contains(t, ud, "ssh-ed25519 KEY") 56 assert.Contains(t, ud, "ssh-ed25519 KEY")
54 assert.Contains(t, ud, "name: "+guest.LoginUser) 57 assert.Contains(t, ud, "name: "+guest.LoginUser)
55 // Growing the root to disk_gb is eitri's job and lives in vendor-data, not 58 // Growing the root to disk_gb is eitri's job and lives in eitri's own
56 // here: a growpart in user-data would override vendor-data's `growpart: off` 59 // cloud-config part, not here: a growpart in the tenant's document wins over
57 // and re-arm the image growpart that reverts on this disk. 60 // that part's `growpart: off` and re-arms the image growpart, which reverts
58 assert.NotContains(t, ud, "growpart", "growpart belongs in vendor-data, not user-data") 61 // on this disk.
62 assert.NotContains(t, ud, "growpart", "growpart belongs in eitri's own cloud-config part, not this document")
59 assert.NotContains(t, ud, "disk_setup", "the no-op disk_setup block is gone") 63 assert.NotContains(t, ud, "disk_setup", "the no-op disk_setup block is gone")
60 } 64 }
61 65
@@ -220,9 +224,38 @@ func readISOFile(t *testing.T, isoPath, name string) []byte {
220 return b 224 return b
221 } 225 }
222 226
223 func TestBuildAlwaysCarriesVendorDataForDiskGrow(t *testing.T) { 227 // seedParts reads a finished seed's /user-data and returns the two documents it
224 // Even with no CA/host key, the seed carries vendor-data — it holds the 228 // composes: the tenant's part first, eitri's own last. Splitting here keeps
225 // disk-grow that delivers disk_gb to the guest. 229 // every assertion below about content rather than about MIME framing, and the
230 // length check is itself the contract — a seed is those two parts and no others.
231 func seedParts(t *testing.T, isoPath string) (tenant, eitri string) {
232 t.Helper()
233 raw := string(readISOFile(t, isoPath, "/user-data"))
234 msg, err := mail.ReadMessage(strings.NewReader(raw))
235 require.NoError(t, err, "user-data must be a MIME archive")
236 mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
237 require.NoError(t, err)
238 require.Equal(t, "multipart/mixed", mediaType)
239 mr := multipart.NewReader(msg.Body, params["boundary"])
240 var bodies []string
241 for {
242 part, err := mr.NextPart()
243 if err != nil {
244 break
245 }
246 b, err := io.ReadAll(part)
247 require.NoError(t, err)
248 bodies = append(bodies, string(b))
249 }
250 require.Len(t, bodies, 2, "a seed composes the tenant's document and eitri's, in that order")
251 return bodies[0], bodies[1]
252 }
253
254 func TestEverySeedCarriesTheDiskGrowAndWritesNoVendorData(t *testing.T) {
255 // Even with no CA/host key, eitri's part holds the disk-grow that delivers
256 // disk_gb to the guest — and it is a part of user-data, not a vendor-data
257 // file. There is one mechanism, and a stray vendor-data would be a second
258 // one whose mergers no document can influence.
226 out := t.TempDir() + "/seed.iso" 259 out := t.TempDir() + "/seed.iso"
227 require.NoError(t, Build(out, Params{ 260 require.NoError(t, Build(out, Params{
228 Hostname: "plain", 261 Hostname: "plain",
@@ -239,11 +272,12 @@ func TestBuildAlwaysCarriesVendorDataForDiskGrow(t *testing.T) {
239 for _, e := range entries { 272 for _, e := range entries {
240 names[strings.ToLower(e.Name())] = true 273 names[strings.ToLower(e.Name())] = true
241 } 274 }
242 assert.True(t, names["vendor-data"], "every seed carries vendor-data for the disk-grow") 275 assert.False(t, names["vendor-data"], "eitri's config rides in user-data; a vendor-data file is the mechanism that lost it")
243 vd := string(readISOFile(t, out, "/vendor-data")) 276
244 assert.Contains(t, vd, `mode: "off"`, "image growpart disabled") 277 _, eitri := seedParts(t, out)
245 assert.Contains(t, vd, "resize2fs /dev/vda1", "online grow runcmd present") 278 assert.Contains(t, eitri, `mode: "off"`, "image growpart disabled")
246 assert.NotContains(t, vd, "TrustedUserCAKeys", "no CA ⇒ no CA trust line") 279 assert.Contains(t, eitri, "resize2fs /dev/vda1", "online grow runcmd present")
280 assert.NotContains(t, eitri, "TrustedUserCAKeys", "no CA ⇒ no CA trust line")
247 st, err := os.Stat(out) 281 st, err := os.Stat(out)
248 require.NoError(t, err) 282 require.NoError(t, err)
249 assert.Equal(t, int64(1*1024*1024), st.Size(), "ISO size unchanged (1 MiB)") 283 assert.Equal(t, int64(1*1024*1024), st.Size(), "ISO size unchanged (1 MiB)")
@@ -253,8 +287,8 @@ func TestBuildAlwaysCarriesVendorDataForDiskGrow(t *testing.T) {
253 287
254 const testUserCAKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITESTCAKEYaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa eitri-user-ca\n" 288 const testUserCAKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITESTCAKEYaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa eitri-user-ca\n"
255 289
256 func TestVendorDataInjectsUserCATrust(t *testing.T) { 290 func TestEitriPartInjectsUserCATrust(t *testing.T) {
257 // A VM with a CA key gets a vendor-data carrying the sshd drop-in. 291 // A VM with a CA key gets an eitri part carrying the sshd drop-in.
258 out := t.TempDir() + "/seed.iso" 292 out := t.TempDir() + "/seed.iso"
259 require.NoError(t, Build(out, Params{ 293 require.NoError(t, Build(out, Params{
260 Hostname: "plain", 294 Hostname: "plain",
@@ -262,7 +296,7 @@ func TestVendorDataInjectsUserCATrust(t *testing.T) {
262 SSHUserCAAuthorizedKey: testUserCAKey, 296 SSHUserCAAuthorizedKey: testUserCAKey,
263 })) 297 }))
264 298
265 vd := string(readISOFile(t, out, "/vendor-data")) 299 tenant, vd := seedParts(t, out)
266 assert.True(t, strings.HasPrefix(vd, "#cloud-config\n")) 300 assert.True(t, strings.HasPrefix(vd, "#cloud-config\n"))
267 assert.Contains(t, vd, "/etc/ssh/eitri_user_ca.pub") 301 assert.Contains(t, vd, "/etc/ssh/eitri_user_ca.pub")
268 assert.Contains(t, vd, "/etc/ssh/sshd_config.d/eitri-ca.conf") 302 assert.Contains(t, vd, "/etc/ssh/sshd_config.d/eitri-ca.conf")
@@ -272,10 +306,9 @@ func TestVendorDataInjectsUserCATrust(t *testing.T) {
272 // A bad config must NOT lock anyone out: -t gates the reload. 306 // A bad config must NOT lock anyone out: -t gates the reload.
273 assert.Contains(t, vd, "sshd -t && systemctl reload sshd") 307 assert.Contains(t, vd, "sshd -t && systemctl reload sshd")
274 308
275 // user-data is never touched by CA injection. 309 // The tenant's part is never touched by CA injection.
276 ud := string(readISOFile(t, out, "/user-data")) 310 assert.Contains(t, tenant, "hostname: plain")
277 assert.Contains(t, ud, "hostname: plain") 311 assert.NotContains(t, tenant, "eitri-ca.conf")
278 assert.NotContains(t, ud, "eitri-ca.conf")
279 } 312 }
280 313
281 const ( 314 const (
@@ -283,7 +316,7 @@ const (
283 testHostCert = "ssh-ed25519-cert-v01@openssh.com AAAATESTHOSTCERTdata with-hostcert\n" 316 testHostCert = "ssh-ed25519-cert-v01@openssh.com AAAATESTHOSTCERTdata with-hostcert\n"
284 ) 317 )
285 318
286 func TestVendorDataInjectsHostKeyAndCert(t *testing.T) { 319 func TestEitriPartInjectsHostKeyAndCert(t *testing.T) {
287 // A VM with a host key + cert gets them installed and sshd pointed at them. 320 // A VM with a host key + cert gets them installed and sshd pointed at them.
288 out := t.TempDir() + "/seed.iso" 321 out := t.TempDir() + "/seed.iso"
289 require.NoError(t, Build(out, Params{ 322 require.NoError(t, Build(out, Params{
@@ -294,7 +327,7 @@ func TestVendorDataInjectsHostKeyAndCert(t *testing.T) {
294 SSHHostCert: testHostCert, 327 SSHHostCert: testHostCert,
295 })) 328 }))
296 329
297 vd := string(readISOFile(t, out, "/vendor-data")) 330 _, vd := seedParts(t, out)
298 // The host key + cert are handed to cloud-init via its native ssh_keys map, 331 // The host key + cert are handed to cloud-init via its native ssh_keys map,
299 // and sshd's HostCertificate directive is asserted via the drop-in. 332 // and sshd's HostCertificate directive is asserted via the drop-in.
300 assert.Contains(t, vd, "ssh_keys:") 333 assert.Contains(t, vd, "ssh_keys:")
@@ -310,12 +343,12 @@ func TestVendorDataInjectsHostKeyAndCert(t *testing.T) {
310 assert.Contains(t, vd, "sshd -t && systemctl reload sshd") 343 assert.Contains(t, vd, "sshd -t && systemctl reload sshd")
311 } 344 }
312 345
313 func TestVendorDataInjectsHostKeyViaSSHKeys(t *testing.T) { 346 func TestEitriPartInjectsHostKeyViaSSHKeys(t *testing.T) {
314 // The per-VM ed25519 host key + cert are handed to cloud-init via its native 347 // The per-VM ed25519 host key + cert are handed to cloud-init via its native
315 // ssh_keys map — NOT write_files — and we do NOT set ssh_deletekeys: false. 348 // ssh_keys map — NOT write_files — and we do NOT set ssh_deletekeys: false.
316 // So cloud-init (default ssh_deletekeys: true) deletes the image's baked host 349 // So cloud-init (default ssh_deletekeys: true) deletes the image's baked host
317 // keys, installs OUR ed25519 key+cert, and regenerates rsa/ecdsa fresh per VM. 350 // keys, installs OUR ed25519 key+cert, and regenerates rsa/ecdsa fresh per VM.
318 vd := vendorDataDoc(Params{ 351 vd := eitriConfigDoc(Params{
319 SSHHostKeyPEM: testHostKeyPEM, 352 SSHHostKeyPEM: testHostKeyPEM,
320 SSHHostCert: testHostCert, 353 SSHHostCert: testHostCert,
321 }) 354 })
@@ -332,19 +365,19 @@ func TestVendorDataInjectsHostKeyViaSSHKeys(t *testing.T) {
332 "host key must not be injected via write_files") 365 "host key must not be injected via write_files")
333 } 366 }
334 367
335 func TestVendorDataOmitsSSHKeysWithoutHostCert(t *testing.T) { 368 func TestEitriPartOmitsSSHKeysWithoutHostCert(t *testing.T) {
336 // No host-cert injection ⇒ no ssh_keys map (and never the ssh_deletekeys hack). 369 // No host-cert injection ⇒ no ssh_keys map (and never the ssh_deletekeys hack).
337 vd := vendorDataDoc(Params{SSHUserCAAuthorizedKey: testUserCAKey}) 370 vd := eitriConfigDoc(Params{SSHUserCAAuthorizedKey: testUserCAKey})
338 assert.NotContains(t, vd, "ssh_keys:", 371 assert.NotContains(t, vd, "ssh_keys:",
339 "ssh_keys must only appear when injecting a host key") 372 "ssh_keys must only appear when injecting a host key")
340 assert.NotContains(t, vd, "ssh_deletekeys") 373 assert.NotContains(t, vd, "ssh_deletekeys")
341 } 374 }
342 375
343 func TestVendorDataHostCertWithoutCAKey(t *testing.T) { 376 func TestEitriPartHostCertWithoutCAKey(t *testing.T) {
344 // Host cert injection is independent of user-CA trust: a seed with only a 377 // Host cert injection is independent of user-CA trust: a seed with only a
345 // host key still produces vendor-data with the host directives (and no 378 // host key still produces an eitri part with the host directives (and no
346 // TrustedUserCAKeys line). 379 // TrustedUserCAKeys line).
347 vd := vendorDataDoc(Params{ 380 vd := eitriConfigDoc(Params{
348 SSHHostKeyPEM: testHostKeyPEM, 381 SSHHostKeyPEM: testHostKeyPEM,
349 SSHHostCert: testHostCert, 382 SSHHostCert: testHostCert,
350 }) 383 })
@@ -353,10 +386,10 @@ func TestVendorDataHostCertWithoutCAKey(t *testing.T) {
353 assert.NotContains(t, vd, "TrustedUserCAKeys") 386 assert.NotContains(t, vd, "TrustedUserCAKeys")
354 } 387 }
355 388
356 func TestVendorDataDiskGrowAlwaysPresent(t *testing.T) { 389 func TestEitriPartDiskGrowAlwaysPresent(t *testing.T) {
357 // Even with neither CA key nor host key, vendor-data carries the disk-grow: 390 // Even with neither CA key nor host key, eitri's part carries the disk-grow:
358 // growpart disabled + the proven online grow runcmd. No SSH material leaks in. 391 // growpart disabled + the proven online grow runcmd. No SSH material leaks in.
359 vd := vendorDataDoc(Params{}) 392 vd := eitriConfigDoc(Params{})
360 assert.True(t, strings.HasPrefix(vd, "#cloud-config\n")) 393 assert.True(t, strings.HasPrefix(vd, "#cloud-config\n"))
361 assert.Contains(t, vd, "growpart:\n mode: \"off\"\n", 394 assert.Contains(t, vd, "growpart:\n mode: \"off\"\n",
362 "the image growpart is disabled (it reverts on this disk)") 395 "the image growpart is disabled (it reverts on this disk)")
@@ -375,10 +408,10 @@ func TestVendorDataDiskGrowAlwaysPresent(t *testing.T) {
375 assert.NotContains(t, vd, "reload sshd") 408 assert.NotContains(t, vd, "reload sshd")
376 } 409 }
377 410
378 func TestVendorDataClockAlwaysPresent(t *testing.T) { 411 func TestEitriPartClockAlwaysPresent(t *testing.T) {
379 // The clock drop-in is fleet infrastructure like the disk-grow, not SSH 412 // The clock drop-in is fleet infrastructure like the disk-grow, not SSH
380 // material: it lands with no CA key, no host key, and no user-data. 413 // material: it lands with no CA key, no host key, and no user-data.
381 vd := vendorDataDoc(Params{}) 414 vd := eitriConfigDoc(Params{})
382 assert.Contains(t, vd, " - path: /etc/chrony/conf.d/eitri-clock.conf\n", 415 assert.Contains(t, vd, " - path: /etc/chrony/conf.d/eitri-clock.conf\n",
383 "the drop-in path is the guest's, not eitri's: chronyd reads /etc/chrony/conf.d, and a path asserted against the "+ 416 "the drop-in path is the guest's, not eitri's: chronyd reads /etc/chrony/conf.d, and a path asserted against the "+
384 "constant that wrote it lands anywhere at all while the guest's clock quietly keeps drifting") 417 "constant that wrote it lands anywhere at all while the guest's clock quietly keeps drifting")
@@ -387,18 +420,18 @@ func TestVendorDataClockAlwaysPresent(t *testing.T) {
387 assert.Contains(t, vd, chronyRuncmd, "the drop-in reaches the running chronyd, not just the next boot") 420 assert.Contains(t, vd, chronyRuncmd, "the drop-in reaches the running chronyd, not just the next boot")
388 } 421 }
389 422
390 func TestVendorDataClockSurvivesBYOUserData(t *testing.T) { 423 func TestEitriPartClockSurvivesBYOUserData(t *testing.T) {
391 // vendor-data is the reason the clock config is not optional: a tenant who 424 // eitri's own part is the reason the clock config is not optional: a tenant
392 // brings their own user-data still gets it, exactly as they get the grow. 425 // who brings their own user-data still gets it, exactly as they get the grow.
393 vd := vendorDataDoc(Params{UserData: "#cloud-config\npackages: [nginx]\n"}) 426 vd := eitriConfigDoc(Params{UserData: "#cloud-config\npackages: [nginx]\n"})
394 assert.Contains(t, vd, chronyDropInPath) 427 assert.Contains(t, vd, chronyDropInPath)
395 assert.Contains(t, vd, " makestep 1 -1\n") 428 assert.Contains(t, vd, " makestep 1 -1\n")
396 } 429 }
397 430
398 func TestVendorDataDiskGrowCoexistsWithSSHMaterial(t *testing.T) { 431 func TestEitriPartDiskGrowCoexistsWithSSHMaterial(t *testing.T) {
399 // The disk-grow sits alongside the CA trust + host cert, and its runcmd 432 // The disk-grow sits alongside the CA trust + host cert, and its runcmd
400 // entry is a sibling of the sshd-reload — both run. 433 // entry is a sibling of the sshd-reload — both run.
401 vd := vendorDataDoc(Params{ 434 vd := eitriConfigDoc(Params{
402 SSHUserCAAuthorizedKey: testUserCAKey, 435 SSHUserCAAuthorizedKey: testUserCAKey,
403 SSHHostKeyPEM: testHostKeyPEM, 436 SSHHostKeyPEM: testHostKeyPEM,
404 SSHHostCert: testHostCert, 437 SSHHostCert: testHostCert,
@@ -410,21 +443,57 @@ func TestVendorDataDiskGrowCoexistsWithSSHMaterial(t *testing.T) {
410 assert.Contains(t, vd, "sshd -t && systemctl reload sshd") 443 assert.Contains(t, vd, "sshd -t && systemctl reload sshd")
411 } 444 }
412 445
413 func TestUserSuppliedCloudInitKeepsVerbatimWhileVendorGrows(t *testing.T) { 446 func TestBYOCloudInitCannotDropTheCATrustOrTheDiskGrow(t *testing.T) {
414 // A BYO user-data is passed through byte-for-byte (eitri never edits it), yet 447 // A BYO user-data rides byte-for-byte as its own part (eitri never edits it),
415 // the seed's vendor-data still delivers the disk-grow — the contract survives 448 // yet the seed's CA trust and disk-grow still reach the guest — even when the
416 // custom cloud-init. 449 // tenant's document carries the two keys that used to replace them wholesale.
450 // Without a merge directive cloud-init folds a later part in with
451 // dict(replace)+list(), so eitri's `write_files` would take the tenant's with
452 // it or the tenant's `runcmd` would take the grow.
417 out := t.TempDir() + "/seed.iso" 453 out := t.TempDir() + "/seed.iso"
418 custom := "#cloud-config\npackages: [htop]\n" 454 custom := "#cloud-config\npackages: [htop]\n" +
455 "write_files:\n - path: /etc/tenant\n content: mine\n" +
456 "runcmd:\n - [\"sh\", \"-c\", \"echo tenant\"]\n"
419 require.NoError(t, Build(out, Params{ 457 require.NoError(t, Build(out, Params{
420 Hostname: "byo", 458 Hostname: "byo",
421 UserData: custom, 459 UserData: custom,
422 SSHUserCAAuthorizedKey: testUserCAKey, 460 SSHUserCAAuthorizedKey: testUserCAKey,
423 })) 461 }))
424 ud := string(readISOFile(t, out, "/user-data")) 462 tenant, eitri := seedParts(t, out)
425 assert.Equal(t, custom, ud, "user-data is passed through verbatim") 463 assert.Equal(t, custom, tenant, "the tenant's document is passed through verbatim")
426 assert.NotContains(t, ud, "resize2fs", "eitri does not inject the grow into user-data") 464 assert.NotContains(t, tenant, "resize2fs", "eitri does not inject the grow into the tenant's document")
427 vd := string(readISOFile(t, out, "/vendor-data")) 465 assert.NotContains(t, tenant, "merge_how", "the merge directive rides in eitri's part, never in the tenant's")
428 assert.Contains(t, vd, `mode: "off"`) 466
429 assert.Contains(t, vd, "resize2fs /dev/vda1") 467 assert.Contains(t, eitri, `merge_how: "list(prepend)+dict(recurse_array,no_replace)+str()"`,
468 "without this, the handler's dict(replace)+list() fallback drops whichever list was folded in first")
469 assert.Contains(t, eitri, userCAPath, "the CA trust file must survive a tenant write_files")
470 assert.Contains(t, eitri, "resize2fs /dev/vda1", "the online grow must survive a tenant runcmd")
471 assert.Contains(t, eitri, `mode: "off"`)
472 }
473
474 func TestEitrisPartIsMergedInLastOrItsDirectiveGovernsNothing(t *testing.T) {
475 // Part order is the whole mechanism: cloud-init folds each part into what the
476 // earlier parts built, under the mergers the incoming part names. eitri's
477 // document carries merge_how, so it has to be the one arriving last.
478 out := t.TempDir() + "/seed.iso"
479 custom := "#cloud-config\npackages: [htop]\n"
480 require.NoError(t, Build(out, Params{Hostname: "byo", UserData: custom}))
481
482 tenant, eitri := seedParts(t, out)
483 assert.Equal(t, custom, tenant, "the tenant's document is the part folded in first")
484 assert.Contains(t, eitri, "merge_how:", "the part folded in last is the one that names the mergers")
485 }
486
487 func TestBuildRefusesUserDataItCannotComposeWith(t *testing.T) {
488 // A jinja template renders to a type nobody knows yet and a gzip blob is
489 // opaque, so neither can carry an honest MIME part header and eitri's config
490 // has nowhere to go. Refusing beats booting a guest whose CA trust file never
491 // lands: that guest comes up, runs sshd, and is reachable by nobody.
492 for _, ud := range []string{
493 "## template: jinja\n#cloud-config\nhostname: {{ v1.local_hostname }}\n",
494 "\x1f\x8b\x08 gzipped",
495 } {
496 err := Build(t.TempDir()+"/seed.iso", Params{Hostname: "h", UserData: ud})
497 assert.Error(t, err, "Build must refuse user-data it cannot compose eitri's config with")
498 }
430 } 499 }
internal/cloudinit/append_test.go
Old New
@@ -0,0 +1,108 @@
1 package cloudinit
2
3 import (
4 "io"
5 "mime"
6 "mime/multipart"
7 "net/mail"
8 "strings"
9 "testing"
10
11 "github.com/stretchr/testify/assert"
12 "github.com/stretchr/testify/require"
13 )
14
15 // eitriDoc stands in for the cloud-config the agent's seed appends. Its
16 // merge_how line is what the ordering below exists to serve.
17 const eitriDoc = "#cloud-config\nmerge_how: \"list(prepend)+dict(recurse_array,no_replace)+str()\"\nruncmd:\n - [\"sh\", \"-c\", \"resize2fs /dev/vda1\"]\n"
18
19 // orderedParts parses an archive into (contentType, body) pairs IN ORDER.
20 // mimeParts keys by content type, which cannot see the difference between two
21 // cloud-config parts — and their order is the whole contract here.
22 func orderedParts(t *testing.T, s string) []struct{ CT, Body string } {
23 t.Helper()
24 msg, err := mail.ReadMessage(strings.NewReader(s))
25 require.NoError(t, err)
26 mediaType, params, err := mime.ParseMediaType(msg.Header.Get("Content-Type"))
27 require.NoError(t, err)
28 require.Equal(t, "multipart/mixed", mediaType)
29 mr := multipart.NewReader(msg.Body, params["boundary"])
30 var out []struct{ CT, Body string }
31 for {
32 p, err := mr.NextPart()
33 if err != nil {
34 break
35 }
36 body, err := io.ReadAll(p)
37 require.NoError(t, err)
38 mt, _, _ := mime.ParseMediaType(p.Header.Get("Content-Type"))
39 out = append(out, struct{ CT, Body string }{mt, string(body)})
40 }
41 return out
42 }
43
44 func TestAppendCloudConfigPutsTheTenantsDocumentFirstAndEitrisLast(t *testing.T) {
45 // Order is the contract: cloud-init's cloud-config handler merges each part
46 // into the buffer the earlier parts built, using THAT part's mergers. eitri's
47 // part must therefore be merged in last, or its merge_how governs nothing.
48 tenant := "#cloud-config\npackages: [htop]\nruncmd:\n - [\"sh\", \"-c\", \"echo tenant\"]\n"
49 out, err := AppendCloudConfig(tenant, eitriDoc)
50 require.NoError(t, err)
51
52 parts := orderedParts(t, out)
53 require.Len(t, parts, 2)
54 assert.Equal(t, "text/cloud-config", parts[0].CT)
55 assert.Equal(t, tenant, parts[0].Body, "the tenant's document rides byte-for-byte")
56 assert.Equal(t, "text/cloud-config", parts[1].CT)
57 assert.Equal(t, eitriDoc, parts[1].Body, "eitri's document is the last part merged")
58 }
59
60 func TestAppendCloudConfigLeavesANonCloudConfigPayloadIntact(t *testing.T) {
61 // A tenant script is not something eitri can merge into, so it rides as its
62 // own part under the type cloud-init dispatches it with, untouched.
63 script := "#!/bin/bash\necho hello > /tmp/marker\n"
64 out, err := AppendCloudConfig(script, eitriDoc)
65 require.NoError(t, err)
66
67 parts := orderedParts(t, out)
68 require.Len(t, parts, 2)
69 assert.Equal(t, "text/x-shellscript", parts[0].CT)
70 assert.Equal(t, script, parts[0].Body)
71 assert.Equal(t, "text/cloud-config", parts[1].CT)
72 assert.Equal(t, eitriDoc, parts[1].Body)
73 }
74
75 func TestAppendCloudConfigAppendsToAnExistingArchiveRatherThanNesting(t *testing.T) {
76 existing, err := wrapMultipart([]part{
77 typedPart("text/cloud-config", "#cloud-config\npackages:\n - git\n"),
78 typedPart("text/x-shellscript", "#!/bin/bash\necho hi\n"),
79 })
80 require.NoError(t, err)
81
82 out, err := AppendCloudConfig(existing, eitriDoc)
83 require.NoError(t, err)
84
85 parts := orderedParts(t, out)
86 require.Len(t, parts, 3)
87 for _, p := range parts {
88 assert.NotContains(t, p.CT, "multipart", "must append, not nest")
89 }
90 assert.Equal(t, "#cloud-config\npackages:\n - git\n", parts[0].Body, "the tenant's own parts keep their order")
91 assert.Equal(t, "#!/bin/bash\necho hi\n", parts[1].Body)
92 assert.Equal(t, eitriDoc, parts[2].Body, "eitri's part is last, after every part the tenant brought")
93 }
94
95 func TestAppendCloudConfigRefusesAPayloadItCannotGiveAPartHeader(t *testing.T) {
96 // A jinja template's real type is unknown until it renders and a gzip blob is
97 // opaque, so neither can be labelled honestly. Refusing is the only safe
98 // answer: composing them wrong would hand the guest a seed whose CA trust
99 // never lands, which is the failure this whole mechanism exists to prevent.
100 for _, in := range []string{
101 "## template: jinja\n#cloud-config\nhostname: {{ v1.local_hostname }}\n",
102 "\x1f\x8b\x08 gzipped",
103 "random text that is not user-data",
104 } {
105 _, err := AppendCloudConfig(in, eitriDoc)
106 assert.Error(t, err, "must refuse %q rather than drop eitri's config", in[:min(20, len(in))])
107 }
108 }
internal/cloudinit/multipart.go
Old New
@@ -122,22 +122,87 @@ func isMIMEMultipart(s string) bool {
122 // generated #cloud-config, so a newline would produce a garbage key. It is 122 // generated #cloud-config, so a newline would produce a garbage key. It is
123 // added as a YAML scalar node, so it cannot inject structure regardless. 123 // added as a YAML scalar node, so it cannot inject structure regardless.
124 func AddSSHKey(userData, key string) (string, error) { 124 func AddSSHKey(userData, key string) (string, error) {
125 switch f := detectFormat(userData); f { 125 f := detectFormat(userData)
126 case formatCloudConfig: 126 if f == formatCloudConfig {
127 // The common case is a single clean document, so it stays one: merging
128 // the key in beats handing cloud-init an archive to reassemble.
127 return mergeSSHKey(userData, key) 129 return mergeSSHKey(userData, key)
130 }
131 out, err := appendPart(userData, f, keyPart(key))
132 if err != nil {
133 return "", fmt.Errorf("cannot add an ssh key to %s user-data; include the key in the user-data itself", f)
134 }
135 return out, nil
136 }
137
138 // AppendCloudConfig returns userData with doc — a #cloud-config document eitri
139 // generated — appended as the LAST part of a MIME multipart archive, leaving
140 // whatever the tenant supplied byte-for-byte intact as the earlier part(s).
141 //
142 // Last is the contract, not a detail. cloud-init's cloud-config part handler
143 // merges each part into the buffer the earlier parts built, using the mergers
144 // named by THAT part (its own `merge_how`, or a `Merge-Type` MIME header), and
145 // falling back to `dict(replace)+list()+str()` for a part that names none. So a
146 // document that carries a merge_how governs only what is merged into it — it
147 // has to arrive after the document it means to merge with. This is the layer
148 // where merge_how is read at all: the cloud-config handler POPS the key out of
149 // each part as it processes it and writes the stripped result, so the later
150 // user-data-versus-vendor-data merge never sees one (see mergeHow in
151 // internal/agent/seed).
152 //
153 // A tenant document is never edited to make room. A plain cloud-config becomes
154 // its own part; a script, boothook, include or part-handler payload rides under
155 // the type cloud-init dispatches it with; an archive the tenant already built
156 // gets the part appended to it rather than nested inside a new one, keeping
157 // every original part's body AND headers.
158 //
159 // Jinja, gzip and unrecognised payloads are refused. Their real content-type is
160 // not knowable without rendering or decompressing them, and a part labelled
161 // wrongly is a part cloud-init runs wrongly — where the alternative is handing
162 // a guest a seed whose CA trust silently never lands, an error at build time is
163 // the only honest answer.
164 func AppendCloudConfig(userData, doc string) (string, error) {
165 f := detectFormat(userData)
166 out, err := appendPart(userData, f, typedPart("text/cloud-config", doc))
167 if err != nil {
168 return "", fmt.Errorf("cannot compose eitri's cloud-config with %s user-data: %w", f, err)
169 }
170 return out, nil
171 }
172
173 // appendPart returns userData with last appended as the final part of a
174 // multipart archive: an archive gains a part, anything else is wrapped with
175 // itself first. f is userData's already-detected format, so callers that
176 // special-case one of them (AddSSHKey) classify only once.
177 func appendPart(userData string, f format, last part) (string, error) {
178 if f == formatMultipart {
179 return appendToMultipart(userData, last)
180 }
181 ct, ok := partType(f)
182 if !ok {
183 return "", fmt.Errorf("%s payloads cannot be given a MIME part header", f)
184 }
185 return wrapMultipart([]part{typedPart(ct, userData), last})
186 }
187
188 // partType maps a detected format to the content-type cloud-init dispatches
189 // that payload with. The bool reports whether the payload can ride in an
190 // archive at all: jinja renders to a type nobody knows yet, gzip is opaque
191 // bytes, and an unrecognised payload names no handler to run it.
192 func partType(f format) (string, bool) {
193 switch f {
194 case formatCloudConfig:
195 return "text/cloud-config", true
128 case formatShellScript: 196 case formatShellScript:
129 return wrapMultipart([]part{typedPart("text/x-shellscript", userData), keyPart(key)}) 197 return "text/x-shellscript", true
130 case formatBoothook: 198 case formatBoothook:
131 return wrapMultipart([]part{typedPart("text/cloud-boothook", userData), keyPart(key)}) 199 return "text/cloud-boothook", true
132 case formatInclude: 200 case formatInclude:
133 return wrapMultipart([]part{typedPart("text/x-include-url", userData), keyPart(key)}) 201 return "text/x-include-url", true
134 case formatPartHandler: 202 case formatPartHandler:
135 return wrapMultipart([]part{typedPart("text/part-handler", userData), keyPart(key)}) 203 return "text/part-handler", true
136 case formatMultipart:
137 return appendToMultipart(userData, key)
138 default:
139 return "", fmt.Errorf("cannot add an ssh key to %s user-data; include the key in the user-data itself", f)
140 } 204 }
205 return "", false
141 } 206 }
142 207
143 // part is one MIME sub-document: its headers and its (already-decoded) body. 208 // part is one MIME sub-document: its headers and its (already-decoded) body.
@@ -199,9 +264,9 @@ func wrapMultipart(parts []part) (string, error) {
199 // appendToMultipart parses an existing multipart archive, keeps every original 264 // appendToMultipart parses an existing multipart archive, keeps every original
200 // part's body AND headers (cloud-init uses part headers like 265 // part's body AND headers (cloud-init uses part headers like
201 // Content-Disposition/filename to name and order scripts, and Merge-Type to 266 // Content-Disposition/filename to name and order scripts, and Merge-Type to
202 // control cloud-config merging), and re-emits with the key part appended — 267 // control cloud-config merging), and re-emits with last appended —
203 // deliberately NOT nesting the user's archive inside a new one. 268 // deliberately NOT nesting the user's archive inside a new one.
204 func appendToMultipart(userData, key string) (string, error) { 269 func appendToMultipart(userData string, last part) (string, error) {
205 msg, err := mail.ReadMessage(strings.NewReader(userData)) 270 msg, err := mail.ReadMessage(strings.NewReader(userData))
206 if err != nil { 271 if err != nil {
207 return "", fmt.Errorf("parse multipart headers: %w", err) 272 return "", fmt.Errorf("parse multipart headers: %w", err)
@@ -246,7 +311,7 @@ func appendToMultipart(userData, key string) (string, error) {
246 if len(parts) == 0 { 311 if len(parts) == 0 {
247 return "", errors.New("multipart archive has no parts") 312 return "", errors.New("multipart archive has no parts")
248 } 313 }
249 return wrapMultipart(append(parts, keyPart(key))) 314 return wrapMultipart(append(parts, last))
250 } 315 }
251 316
252 // firstNonBlankLine returns the first line with non-whitespace content, trimmed. 317 // firstNonBlankLine returns the first line with non-whitespace content, trimmed.
internal/mcpserver/tools.go
Old New
@@ -173,7 +173,7 @@ type VMCreateIn struct {
173 VCPUs int64 `json:"vcpus,omitempty" jsonschema:"default 2"` 173 VCPUs int64 `json:"vcpus,omitempty" jsonschema:"default 2"`
174 MemMB int64 `json:"mem_mb,omitempty" jsonschema:"default 2048"` 174 MemMB int64 `json:"mem_mb,omitempty" jsonschema:"default 2048"`
175 DiskGB int64 `json:"disk_gb,omitempty" jsonschema:"default 10"` 175 DiskGB int64 `json:"disk_gb,omitempty" jsonschema:"default 10"`
176 CloudInit string `json:"cloud_init,omitempty" jsonschema:"optional user cloud-init"` 176 CloudInit string `json:"cloud_init,omitempty" jsonschema:"optional user cloud-init, taken verbatim; eitri's own write_files and runcmd are prepended to yours, not replaced by them, and every scalar you set wins"`
177 Network string `json:"network,omitempty" jsonschema:"named host network to give the guest a SECOND NIC on, addressed by that network's own DHCP; omit for the private NAT underlay every guest has anyway. A host serves only the names it advertises: naming another refuses the create and names the ones it does serve"` 177 Network string `json:"network,omitempty" jsonschema:"named host network to give the guest a SECOND NIC on, addressed by that network's own DHCP; omit for the private NAT underlay every guest has anyway. A host serves only the names it advertises: naming another refuses the create and names the ones it does serve"`
178 Wait *bool `json:"wait,omitempty" jsonschema:"wait for ready+cloud-init (default true)"` 178 Wait *bool `json:"wait,omitempty" jsonschema:"wait for ready+cloud-init (default true)"`
179 } 179 }
internal/server/api/client/client.go
Old New
@@ -42,6 +42,13 @@ type (
42 Delegation = types.Delegation 42 Delegation = types.Delegation
43 ) 43 )
44 44
45 // DefaultDiskGB is the wire contract's default disk size, re-exported for the
46 // reason the aliases above exist: the client is the only door to the contract
47 // package (R11), so a consumer that needs one of the API's defaults reaches it
48 // here rather than keeping a copy of the number. Its siblings, DefaultVCPUs and
49 // DefaultMemMB, join it when something asks for them.
50 const DefaultDiskGB = types.DefaultDiskGB
51
45 // Client calls the eitri API at BaseURL, authenticating with Token (sent as a 52 // Client calls the eitri API at BaseURL, authenticating with Token (sent as a
46 // Bearer header when non-empty). The zero value plus a BaseURL is a working 53 // Bearer header when non-empty). The zero value plus a BaseURL is a working
47 // client; a nil HTTP falls back to a 30s-timeout http.Client. UserCALabel, if 54 // client; a nil HTTP falls back to a 30s-timeout http.Client. UserCALabel, if
internal/smoke/cloudinit.go
Old New
@@ -0,0 +1,110 @@
1 package smoke
2
3 import (
4 "fmt"
5 "strconv"
6 "strings"
7
8 "github.com/a73x/eitri/internal/server/api/client"
9 )
10
11 // Guest paths the tenant document below creates, one per cloud-init key. They
12 // are read back by seedProbeCmd, and they live under /etc rather than /run
13 // because the gate proof runs again after the power cycle and cloud-init's
14 // write_files and runcmd only ever run once per instance.
15 const (
16 byoWriteFilesMarker = "/etc/eitri-smoke-byo-write-files"
17 byoRuncmdMarker = "/etc/eitri-smoke-byo-runcmd"
18 )
19
20 // byoCloudInit is the tenant cloud-init the scenario's VM is created with. It
21 // exists to make the smoke's one VM a guest of the shape that used to lose
22 // eitri's own seed config: cloud-init folds documents together with mergers
23 // that discard one side's lists, so a tenant document carrying `write_files:`
24 // or `runcmd:` cost the guest its CA trust file or its online root grow. The
25 // seed answers that with a merge directive on its own cloud-config part
26 // (internal/agent/seed, mergeHow), and this document is what makes the boot
27 // gate exercise it rather than take it on faith.
28 //
29 // It carries both keys and nothing else, so the guest still gets its default
30 // user from the image (and the create request's SSH key, which the control
31 // plane merges into whatever document it is given). Each key drops one marker
32 // file, so the proof can see the tenant's own entries ran too — a merge that
33 // silently kept only eitri's side would be the same bug pointing the other way.
34 const byoCloudInit = `#cloud-config
35 write_files:
36 - path: ` + byoWriteFilesMarker + `
37 permissions: '0644'
38 content: |
39 the tenant's own write_files entry ran
40 runcmd:
41 - ["sh", "-c", "echo 'the tenant runcmd entry ran' > ` + byoRuncmdMarker + `"]
42 `
43
44 // seedProbeCmd reads the three facts the merge proof needs out of a booted
45 // guest, one labelled line each, so proveSeedSurvivedBYO can stay pure. The
46 // redirections keep a missing file from failing the whole command: an absent
47 // marker must arrive as an empty value to be reported, not as a shell error
48 // that reads like an unreachable guest.
49 const seedProbeCmd = `printf 'root_bytes=%s\n' "$(findmnt -no SIZE -b / 2>/dev/null)"; ` +
50 `printf 'byo_write_files=%s\n' "$(cat ` + byoWriteFilesMarker + ` 2>/dev/null)"; ` +
51 `printf 'byo_runcmd=%s\n' "$(cat ` + byoRuncmdMarker + ` 2>/dev/null)"`
52
53 // minGrownRootBytes is the floor a grown root must clear: half the disk the
54 // scenario's VM is created with, which is the control plane's default because
55 // the create request names no size. The default image ships a root of about
56 // 3.5G, so half of a default disk sits above what the image brings and below
57 // what the grow produces — a root over the floor can only be one the seed's
58 // grow runcmd resized, and a root under it can only be one that never ran.
59 // Nothing lands between the two, which is why one number is enough.
60 //
61 // It is derived rather than written down so that moving the default disk moves
62 // the floor with it. A literal here would be this package's private copy of a
63 // number it does not own, and the first shrink of the default would turn this
64 // leg's pass into a coincidence.
65 const minGrownRootBytes int64 = client.DefaultDiskGB * (1 << 30) / 2
66
67 // proveSeedSurvivedBYO reads seedProbeCmd's output and reports whether both
68 // sides of the cloud-init merge reached the guest: eitri's grow runcmd (the
69 // root is bigger than the image's) and the tenant's own write_files and runcmd
70 // (their marker files exist). Unreadable output is a failure — a probe that
71 // cannot be parsed proves nothing, and must never read as a pass.
72 func proveSeedSurvivedBYO(out string) error {
73 fields := parseProbe(out)
74
75 raw, ok := fields["root_bytes"]
76 if !ok || raw == "" {
77 return fmt.Errorf("FAIL: the guest reported no root filesystem size, so the seed's disk grow cannot be judged (probe output: %q)", out)
78 }
79 size, err := strconv.ParseInt(raw, 10, 64)
80 if err != nil {
81 return fmt.Errorf("FAIL: the guest reported root filesystem size %q, which is not a number of bytes", raw)
82 }
83 if size < minGrownRootBytes {
84 return fmt.Errorf("FAIL: the guest's root filesystem is %d bytes, under the %d it must exceed once grown — "+
85 "the seed's grow runcmd did not run, which is what a tenant runcmd displacing eitri's looks like", size, minGrownRootBytes)
86 }
87
88 if fields["byo_write_files"] == "" {
89 return fmt.Errorf("FAIL: %s is missing, so the tenant's own write_files never ran — the merge kept eitri's list and dropped theirs", byoWriteFilesMarker)
90 }
91 if fields["byo_runcmd"] == "" {
92 return fmt.Errorf("FAIL: %s is missing, so the tenant's own runcmd never ran — the merge kept eitri's list and dropped theirs", byoRuncmdMarker)
93 }
94 return nil
95 }
96
97 // parseProbe splits seedProbeCmd's `key=value` lines into a map. Lines without
98 // a `=` are ignored: a guest is free to print a login banner or a warning ahead
99 // of the command's own output, and none of it is an answer either way.
100 func parseProbe(out string) map[string]string {
101 fields := make(map[string]string)
102 for line := range strings.SplitSeq(out, "\n") {
103 key, value, ok := strings.Cut(strings.TrimSpace(line), "=")
104 if !ok {
105 continue
106 }
107 fields[key] = strings.TrimSpace(value)
108 }
109 return fields
110 }
internal/smoke/cloudinit_test.go
Old New
@@ -0,0 +1,137 @@
1 package smoke
2
3 import (
4 "strconv"
5 "strings"
6 "testing"
7
8 "github.com/a73x/eitri/internal/cloudinit"
9 "github.com/a73x/eitri/internal/server/api/client"
10 )
11
12 // grownRootBytes is what a guest reports once the seed's runcmd has grown its
13 // root to the disk the scenario asked for: the whole default disk, less the
14 // filesystem's own overhead, which is nowhere near half of it.
15 const grownRootBytes = client.DefaultDiskGB * (1 << 30)
16
17 // imageRootBytes is the root the default Ubuntu image ships — about 3.5G, the
18 // size an ungrown guest reports. Written as a literal because it is a fact
19 // about somebody else's image, not a number eitri gets to choose.
20 const imageRootBytes int64 = 3650722816
21
22 func TestGrownRootFloorSitsBetweenTheImageAndTheDefaultDisk(t *testing.T) {
23 // The floor is derived from the default disk, so it moves when that moves.
24 // What must stay true either way is that it separates the two roots a guest
25 // can actually have: a floor under the image's root passes an ungrown guest,
26 // and one above the disk fails every guest.
27 if minGrownRootBytes <= imageRootBytes {
28 t.Errorf("floor %d does not clear the image's own root %d — an ungrown guest would pass", minGrownRootBytes, imageRootBytes)
29 }
30 if minGrownRootBytes >= grownRootBytes {
31 t.Errorf("floor %d is at or above the whole default disk %d — no guest could clear it", minGrownRootBytes, grownRootBytes)
32 }
33 }
34
35 // probeOutput renders what seedProbeCmd prints in a guest, so each test states
36 // the three facts it is about and nothing else.
37 func probeOutput(rootBytes, writeFiles, runcmd string) string {
38 return "root_bytes=" + rootBytes + "\n" +
39 "byo_write_files=" + writeFiles + "\n" +
40 "byo_runcmd=" + runcmd + "\n"
41 }
42
43 func TestSeedProbeAcceptsAGuestWhereBothSidesOfTheMergeLanded(t *testing.T) {
44 out := probeOutput(strconv.FormatInt(grownRootBytes, 10), "tenant write_files ran", "tenant runcmd ran")
45 if err := proveSeedSurvivedBYO(out); err != nil {
46 t.Fatalf("proveSeedSurvivedBYO: %v", err)
47 }
48 }
49
50 func TestSeedProbeRejectsARootTheGrowNeverReached(t *testing.T) {
51 // The image's own ~3.5G root on a 10G disk: the seed's runcmd was replaced
52 // by the tenant's, which is the half of the defect that costs disk.
53 out := probeOutput(strconv.FormatInt(imageRootBytes, 10), "tenant write_files ran", "tenant runcmd ran")
54 err := proveSeedSurvivedBYO(out)
55 if err == nil {
56 t.Fatal("proveSeedSurvivedBYO: want an error for an ungrown root, got nil")
57 }
58 if !strings.Contains(err.Error(), "runcmd") {
59 t.Errorf("error = %q, want it to name the runcmd that was dropped", err)
60 }
61 }
62
63 func TestSeedProbeRejectsAGuestThatLostTheTenantsWriteFiles(t *testing.T) {
64 // The mirror image: eitri's lists won and the tenant's were discarded. A
65 // merge that only ever keeps eitri's side is not a merge.
66 out := probeOutput(strconv.FormatInt(grownRootBytes, 10), "", "tenant runcmd ran")
67 err := proveSeedSurvivedBYO(out)
68 if err == nil {
69 t.Fatal("proveSeedSurvivedBYO: want an error when the tenant's write_files is gone, got nil")
70 }
71 if !strings.Contains(err.Error(), "write_files") {
72 t.Errorf("error = %q, want it to name the tenant's write_files", err)
73 }
74 }
75
76 func TestSeedProbeRejectsAGuestThatLostTheTenantsRuncmd(t *testing.T) {
77 out := probeOutput(strconv.FormatInt(grownRootBytes, 10), "tenant write_files ran", "")
78 err := proveSeedSurvivedBYO(out)
79 if err == nil {
80 t.Fatal("proveSeedSurvivedBYO: want an error when the tenant's runcmd is gone, got nil")
81 }
82 if !strings.Contains(err.Error(), "runcmd") {
83 t.Errorf("error = %q, want it to name the tenant's runcmd", err)
84 }
85 }
86
87 func TestSeedProbeRejectsOutputItCannotRead(t *testing.T) {
88 // A probe whose root_bytes is missing or not a number proves nothing, and
89 // must never be mistaken for a pass.
90 for _, out := range []string{
91 "",
92 probeOutput("", "w", "r"),
93 probeOutput("not-a-number", "w", "r"),
94 } {
95 if err := proveSeedSurvivedBYO(out); err == nil {
96 t.Errorf("proveSeedSurvivedBYO(%q): want an error, got nil", out)
97 }
98 }
99 }
100
101 func TestBYOCloudInitCarriesTheTwoKeysThatUsedToClobberTheSeed(t *testing.T) {
102 // The document is the whole point of the leg: without both keys the guest
103 // never exercises the merge the seed's merge_how exists for.
104 if !strings.Contains(byoCloudInit, "\nwrite_files:") {
105 t.Error("byoCloudInit must carry a write_files key")
106 }
107 if !strings.Contains(byoCloudInit, "\nruncmd:") {
108 t.Error("byoCloudInit must carry a runcmd key")
109 }
110 if !strings.HasPrefix(byoCloudInit, "#cloud-config\n") {
111 t.Error("byoCloudInit must be a #cloud-config document")
112 }
113 for _, path := range []string{byoWriteFilesMarker, byoRuncmdMarker} {
114 if !strings.Contains(byoCloudInit, path) {
115 t.Errorf("byoCloudInit does not write %s, so the probe reads a file nothing creates", path)
116 }
117 if !strings.Contains(seedProbeCmd, path) {
118 t.Errorf("seedProbeCmd does not read %s", path)
119 }
120 }
121 }
122
123 func TestBYOCloudInitSurvivesTheControlPlanesOwnSSHKeyMerge(t *testing.T) {
124 // The create request carries an SSH key alongside the document, and the
125 // control plane merges the one into the other before the seed ever sees it.
126 // A document that merge refuses is a 400 at create, so the leg would fail
127 // before it proved anything about cloud-init.
128 merged, err := cloudinit.AddSSHKey(byoCloudInit, "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAISMOKE smoke@gate")
129 if err != nil {
130 t.Fatalf("AddSSHKey(byoCloudInit): %v", err)
131 }
132 for _, want := range []string{"write_files", "runcmd", byoWriteFilesMarker, byoRuncmdMarker, "ssh_authorized_keys"} {
133 if !strings.Contains(merged, want) {
134 t.Errorf("merged document lost %q:\n%s", want, merged)
135 }
136 }
137 }
internal/smoke/gatecheck.go
Old New
@@ -23,10 +23,22 @@ func realGateHooks(cfg Config, tenant string, ca gateclient.CertAuthority, userC
23 } 23 }
24 } 24 }
25 25
26 // gateExec proves the guest is reachable through the SSH-CA gate: it retries 26 // gateExec proves the guest is reachable through the SSH-CA gate, and that the
27 // dial+login over the guest's pre-sshd boot window (a dial or session error 27 // seed the guest booted from survived the tenant cloud-init it was created with.
28 // mid-boot is expected, not fatal) until it logs in as cfg.SmokeVMUser and 28 // It retries dial+login over the guest's pre-sshd boot window (a dial or session
29 // confirms `id -un` echoes that same user, or the 120s deadline expires. 29 // error mid-boot is expected, not fatal) until it logs in as cfg.SmokeVMUser,
30 // confirms `id -un` echoes that same user, and reads back a guest where both
31 // sides of the cloud-init merge landed — or the 120s deadline expires.
32 //
33 // The login itself is half the merge proof: a CA-signed certificate is accepted
34 // only because the seed's write_files put the trust file and the sshd drop-in on
35 // the guest, which is exactly what a tenant `write_files` used to replace. The
36 // probe supplies the other half, the root the seed's runcmd grew.
37 //
38 // The probe is inside the retry rather than after it because cloud-init writes
39 // guest files before it runs runcmd, so a login can land in the window between
40 // the two: a root that is not grown YET is an unfinished boot, not a verdict.
41 // Only the deadline decides, and lastErr carries the reason it ran out.
30 func gateExec(ctx context.Context, cfg Config, auth gateclient.Credentials, vmName string, now func() time.Time, sleep func(time.Duration)) error { 42 func gateExec(ctx context.Context, cfg Config, auth gateclient.Credentials, vmName string, now func() time.Time, sleep func(time.Duration)) error {
31 var lastErr error 43 var lastErr error
32 err := pollLoop(ctx, now, sleep, 120*time.Second, 5*time.Second, func() (bool, error) { 44 err := pollLoop(ctx, now, sleep, 120*time.Second, 5*time.Second, func() (bool, error) {
@@ -50,11 +62,21 @@ func gateExec(ctx context.Context, cfg Config, auth gateclient.Credentials, vmNa
50 if got != cfg.SmokeVMUser { 62 if got != cfg.SmokeVMUser {
51 return false, fmt.Errorf("gate SSH logged into %q as %q, want %q", vmName, got, cfg.SmokeVMUser) 63 return false, fmt.Errorf("gate SSH logged into %q as %q, want %q", vmName, got, cfg.SmokeVMUser)
52 } 64 }
65
66 probe, probeErr := runGuestCommand(client, seedProbeCmd)
67 if probeErr != nil {
68 lastErr = probeErr
69 return false, nil
70 }
71 if seedErr := proveSeedSurvivedBYO(probe); seedErr != nil {
72 lastErr = seedErr
73 return false, nil
74 }
53 return true, nil 75 return true, nil
54 }) 76 })
55 if err != nil { 77 if err != nil {
56 if errors.Is(err, errPollTimeout) { 78 if errors.Is(err, errPollTimeout) {
57 return fmt.Errorf("FAIL: could not reach guest %q through the gate within 120s: %w", vmName, lastErr) 79 return fmt.Errorf("FAIL: guest %q did not pass the gate proof within 120s: %w", vmName, lastErr)
58 } 80 }
59 return err 81 return err
60 } 82 }
internal/smoke/scenario.go
Old New
@@ -144,8 +144,9 @@ func pollLoop(ctx context.Context, now func() time.Time, sleep func(time.Duratio
144 // source), and dialBanner (the published-port transport). vmName is the 144 // source), and dialBanner (the published-port transport). vmName is the
145 // pre-generated name for the throwaway VM. gate, when non-nil, registers the 145 // pre-generated name for the throwaway VM. gate, when non-nil, registers the
146 // smoke's user CA with the tenant before create (the guest bakes its trusted 146 // smoke's user CA with the tenant before create (the guest bakes its trusted
147 // CAs at boot, so registration MUST happen first) and proves gate SSH access 147 // CAs at boot, so registration MUST happen first) and proves gate SSH access —
148 // after the boot-proof — a hard gate, so a failure there fails the scenario. 148 // plus the cloud-init merge this VM's tenant document exercises — after the
149 // boot-proof; a hard gate, so a failure there fails the scenario.
149 // mcp, when non-nil, drives a second VM's whole life through the remote MCP 150 // mcp, when non-nil, drives a second VM's whole life through the remote MCP
150 // endpoint — that leg owns its own VM, created after it registers its own CA 151 // endpoint — that leg owns its own VM, created after it registers its own CA
151 // with the tenant, so the guest trusts the certificates eitri presents. 152 // with the tenant, so the guest trusts the certificates eitri presents.
@@ -182,7 +183,12 @@ func runScenario(ctx context.Context, vmName string, c vmAPI, dialConsole consol
182 sshKey := readPubKey() 183 sshKey := readPubKey()
183 184
184 start := now() 185 start := now()
185 created, err := c.CreateVM(ctx, client.CreateVMRequest{HostID: hostID, Name: vmName, SSHAuthorizedKey: sshKey}) 186 // The VM carries a tenant cloud-init (see byoCloudInit) so this one guest is
187 // also the guest the merge proof needs: the gate leg below reads back both
188 // eitri's grown root and the tenant's own marker files. No extra boot buys
189 // that, and a VM created with an empty document would never exercise the
190 // merge at all.
191 created, err := c.CreateVM(ctx, client.CreateVMRequest{HostID: hostID, Name: vmName, SSHAuthorizedKey: sshKey, CloudInit: byoCloudInit})
186 if err != nil { 192 if err != nil {
187 return "", fmt.Errorf("create vm: %w", err) 193 return "", fmt.Errorf("create vm: %w", err)
188 } 194 }
@@ -326,7 +332,10 @@ func runScenario(ctx context.Context, vmName string, c vmAPI, dialConsole consol
326 332
327 msg := fmt.Sprintf("SMOKE COMPLETE — booted under UEFI, cold_start=%ds, reboot: ok, reaped OK", int64(coldStart.Seconds())) 333 msg := fmt.Sprintf("SMOKE COMPLETE — booted under UEFI, cold_start=%ds, reboot: ok, reaped OK", int64(coldStart.Seconds()))
328 if gateOK { 334 if gateOK {
329 msg += ", gate SSH: ok" 335 // One clause per thing the gate leg proved: reaching the guest with a
336 // CA-signed certificate, and the tenant cloud-init the guest booted from
337 // leaving eitri's own write_files and runcmd intact (see gateExec).
338 msg += ", gate SSH: ok, BYO cloud-init merge: ok"
330 } 339 }
331 if exposureOK { 340 if exposureOK {
332 msg += ", exposed port: ok" 341 msg += ", exposed port: ok"
internal/smoke/scenario_test.go
Old New
@@ -474,6 +474,12 @@ func TestRunScenarioGateRegistersBeforeCreateAndExecsAfterBoot(t *testing.T) {
474 if !strings.Contains(msg, "gate SSH: ok") { 474 if !strings.Contains(msg, "gate SSH: ok") {
475 t.Errorf("message = %q, want it to mention gate SSH: ok", msg) 475 t.Errorf("message = %q, want it to mention gate SSH: ok", msg)
476 } 476 }
477 // The gate leg is also where the cloud-init merge is proven, and a run that
478 // proved it says so — a report that only says "gate SSH: ok" leaves a reader
479 // to guess whether the tenant document was exercised at all.
480 if !strings.Contains(msg, "BYO cloud-init merge: ok") {
481 t.Errorf("message = %q, want it to mention BYO cloud-init merge: ok", msg)
482 }
477 483
478 // exec appears twice: once after the first boot proof, once after the 484 // exec appears twice: once after the first boot proof, once after the
479 // power-cycle proof — SSH through the gate must survive a reboot too. 485 // power-cycle proof — SSH through the gate must survive a reboot too.
@@ -489,6 +495,27 @@ func TestRunScenarioGateRegistersBeforeCreateAndExecsAfterBoot(t *testing.T) {
489 } 495 }
490 } 496 }
491 497
498 func TestRunScenarioCreatesItsVMWithATenantCloudInit(t *testing.T) {
499 // The scenario's own VM carries the tenant document, so the gate leg proves
500 // the cloud-init merge on a guest that already exists rather than on a
501 // second boot added for the purpose.
502 var calls []string
503 api := happyPathAPI(t, &calls)
504 var got client.CreateVMRequest
505 api.createVMFunc = func(ctx context.Context, req client.CreateVMRequest) (client.CreateVMResponse, error) {
506 got = req
507 return client.CreateVMResponse{ID: "vm-1"}, nil
508 }
509
510 clock := &fakeClock{t: time.Unix(0, 0)}
511 if _, err := runScenario(context.Background(), "smoke-test", api, powerCycleConsole(t, api, bootedGuest, "ubuntu-vm login: ").dial, nil, nil, "", clock.now, clock.sleep, noopReadPubKey, okBanner); err != nil {
512 t.Fatalf("runScenario: %v", err)
513 }
514 if got.CloudInit != byoCloudInit {
515 t.Errorf("CreateVM CloudInit = %q, want the tenant document the merge proof reads back", got.CloudInit)
516 }
517 }
518
492 func TestRunScenarioGateRegisterErrorAbortsBeforeCreate(t *testing.T) { 519 func TestRunScenarioGateRegisterErrorAbortsBeforeCreate(t *testing.T) {
493 api := &testAPI{ 520 api := &testAPI{
494 listHostsFunc: func(ctx context.Context) ([]client.Host, error) { 521 listHostsFunc: func(ctx context.Context) ([]client.Host, error) {