a73x

18d96cfa

feat(seed): guests own their full disk from first boot

a73x   2026-07-29 19:23

Commit message
feat(seed): guests own their full disk from first boot

The agent extends each VM's raw disk to disk_gb by truncating the base
image's file, which strands the cloud image's GPT: its backup header stays
mid-device and its protective MBR still describes the ~3.5G built-in disk.
The image's cloud-init growpart then fails and reverts — its sfdisk resizer
writes the grown table but returns non-zero on the "GPT backup not at the
end / PMBR size mismatch" auto-correction, which growpart treats as failure
and rolls back. The root never grows, guests wedge at ~3.5G, and cloud-init
reports degraded.

Growing to disk_gb is eitri's side of the contract, so deliver it from
vendor-data: disable the image growpart and run the grow ourselves —
`sfdisk -N 1` writes the full-disk partition table (ignoring the harmless
non-zero exit; the write persists), `partx -u` makes the kernel pick up the
new size online on the mounted root, and resize2fs grows the filesystem.
Vendor-data applies even when a tenant brings their own cloud-init, and its
`growpart: off` is no longer overridden now that the default user-data
carries no growpart. Also drop the anomalous no-op disk_setup block.
internal/agent/seed/seed.go
Old New
@@ -91,21 +91,43 @@ func sshdDropIn(p Params) string {
91 return b.String() 91 return b.String()
92 } 92 }
93 93
94 // vendorDataDoc renders the vendor-data #cloud-config for a seed: the eitri 94 // growRuncmd grows the root partition to fill disk_gb and resizes the
95 // user-CA trust drop-in when the CA key is set, and the per-VM host key + cert 95 // filesystem, online, on the mounted root — eitri's side of the contract
96 // when those are set. Returns "" when neither is set, meaning no vendor-data 96 // (eitri delivers disk_gb to the guest; the guest owns everything above it).
97 // file is written (unchanged TOFU behaviour). 97 // It lives in vendor-data, not the default user-data, so it applies even when a
98 // tenant brings their own cloud-init.
99 //
100 // The image ships a GPT sized for its built-in ~3.5G disk; the agent extends the
101 // raw file to disk_gb, which strands the GPT's backup header mid-device. The
102 // image's own cloud-init growpart is therefore turned OFF (see vendorDataDoc):
103 // its sfdisk resizer treats sfdisk's non-zero exit on the "GPT backup not at the
104 // end / PMBR size mismatch" auto-correction as a failure and REVERTS the grow,
105 // so the root never grows and cloud-init reports degraded. This runcmd does the
106 // same sfdisk write but ignores that exit — the write persists — then completes
107 // the grow online: partx -u makes the kernel pick up the new partition size
108 // without unmounting, and resize2fs grows the mounted root filesystem.
109 //
110 // `printf ',+\n' | sfdisk -N 1` keeps partition 1's start and grows it to fill
111 // the disk; --no-reread/--no-tell-kernel avoid the BLKRRPART that fails on a
112 // mounted root (partx -u supplies the online update instead).
113 const growRuncmd = ` - ["sh", "-c", "printf ',+\\n' | sfdisk --no-reread --no-tell-kernel -N 1 /dev/vda; partx -u /dev/vda; resize2fs /dev/vda1"]` + "\n"
114
115 // vendorDataDoc renders the vendor-data #cloud-config for a seed. It always
116 // carries the disk-grow (growpart disabled + the online grow runcmd, see
117 // growRuncmd); it additionally carries the eitri user-CA trust drop-in when the
118 // CA key is set and the per-VM host key + cert when those are set. It is never
119 // empty, so a vendor-data file is always written.
98 // 120 //
99 // The keys, cert, and drop-in conf are embedded as YAML block scalars (`|`), so 121 // The keys, cert, and drop-in conf are embedded as YAML block scalars (`|`), so
100 // their bytes appear verbatim in the guest files. Block scalars are 122 // their bytes appear verbatim in the guest files. Block scalars are
101 // injection-safe: every indented line is literal content. The runcmd gates the 123 // injection-safe: every indented line is literal content. The sshd reload is
102 // reload behind `sshd -t` so a malformed config can never lock anyone out. 124 // gated behind `sshd -t` so a malformed config can never lock anyone out.
103 func vendorDataDoc(p Params) string { 125 func vendorDataDoc(p Params) string {
104 if p.SSHUserCAAuthorizedKey == "" && p.SSHHostKeyPEM == "" {
105 return ""
106 }
107 var b strings.Builder 126 var b strings.Builder
108 b.WriteString("#cloud-config\n") 127 b.WriteString("#cloud-config\n")
128 // Disable the image's growpart: its sfdisk resizer reverts on this disk (see
129 // growRuncmd). Quoted so YAML reads the string "off", not the boolean false.
130 b.WriteString("growpart:\n mode: \"off\"\n")
109 if p.SSHHostKeyPEM != "" { 131 if p.SSHHostKeyPEM != "" {
110 // Hand the ed25519 host key + cert to cloud-init's native ssh_keys map. 132 // Hand the ed25519 host key + cert to cloud-init's native ssh_keys map.
111 // cc_ssh installs ed25519_private/ed25519_certificate at the default paths 133 // cc_ssh installs ed25519_private/ed25519_certificate at the default paths
@@ -119,14 +141,22 @@ func vendorDataDoc(p Params) string {
119 writeBlockScalar(&b, " ", "ed25519_private", p.SSHHostKeyPEM) 141 writeBlockScalar(&b, " ", "ed25519_private", p.SSHHostKeyPEM)
120 writeBlockScalar(&b, " ", "ed25519_certificate", p.SSHHostCert) 142 writeBlockScalar(&b, " ", "ed25519_certificate", p.SSHHostCert)
121 } 143 }
122 b.WriteString("write_files:\n") 144 dropIn := sshdDropIn(p)
123 if p.SSHUserCAAuthorizedKey != "" { 145 if p.SSHUserCAAuthorizedKey != "" || dropIn != "" {
124 writeFileBlock(&b, userCAPath, p.SSHUserCAAuthorizedKey) 146 b.WriteString("write_files:\n")
147 if p.SSHUserCAAuthorizedKey != "" {
148 writeFileBlock(&b, userCAPath, p.SSHUserCAAuthorizedKey)
149 }
150 if dropIn != "" {
151 writeFileBlock(&b, dropInPath, dropIn)
152 }
125 } 153 }
126 writeFileBlock(&b, dropInPath, sshdDropIn(p))
127 b.WriteString("runcmd:\n") 154 b.WriteString("runcmd:\n")
128 // -t gates the reload: a bad config won't reload, so no lockout. 155 b.WriteString(growRuncmd)
129 b.WriteString(" - [\"sh\", \"-c\", \"sshd -t && systemctl reload sshd\"]\n") 156 if dropIn != "" {
157 // -t gates the reload: a bad config won't reload, so no lockout.
158 b.WriteString(" - [\"sh\", \"-c\", \"sshd -t && systemctl reload sshd\"]\n")
159 }
130 return b.String() 160 return b.String()
131 } 161 }
132 162
@@ -156,7 +186,13 @@ func writeFileBlock(b *strings.Builder, path, content string) {
156 186
157 // userData returns the cloud-config string. If p.UserData is non-empty it is 187 // userData returns the cloud-config string. If p.UserData is non-empty it is
158 // returned verbatim (advanced users own their user-data). Otherwise a sensible 188 // returned verbatim (advanced users own their user-data). Otherwise a sensible
159 // default is generated with an SSH key, growpart, and a default ubuntu user. 189 // default is generated with an SSH key and a default ubuntu user.
190 //
191 // The default does NOT configure growpart or disk_setup: growing the root to
192 // disk_gb is eitri's job and lives in vendor-data (see vendorDataDoc/growRuncmd),
193 // which applies to BYO user-data too. Setting growpart here would also override
194 // vendor-data's `growpart: off` (user-data wins the cloud-init merge), re-arming
195 // the image growpart that reverts on this disk.
160 func userData(p Params) string { 196 func userData(p Params) string {
161 if p.UserData != "" { 197 if p.UserData != "" {
162 return p.UserData 198 return p.UserData
@@ -171,14 +207,6 @@ func userData(p Params) string {
171 } 207 }
172 return fmt.Sprintf(`#cloud-config 208 return fmt.Sprintf(`#cloud-config
173 hostname: %s 209 hostname: %s
174 disk_setup:
175 /dev/vda:
176 table_type: gpt
177 layout: true
178 overwrite: false
179 growpart:
180 mode: auto
181 devices: ["/"]
182 users: 210 users:
183 - name: ubuntu 211 - name: ubuntu
184 sudo: ALL=(ALL) NOPASSWD:ALL 212 sudo: ALL=(ALL) NOPASSWD:ALL
@@ -282,7 +310,8 @@ func Build(outPath string, p Params) error {
282 "/meta-data": metaData(p), 310 "/meta-data": metaData(p),
283 "/network-config": networkConfig(), 311 "/network-config": networkConfig(),
284 } 312 }
285 // Vendor-data carries the eitri user-CA sshd drop-in (empty doc => no file). 313 // Vendor-data carries eitri's own config (disk-grow, and the SSH user-CA
314 // drop-in + host cert when set); it is always non-empty, so always written.
286 if vd := vendorDataDoc(p); vd != "" { 315 if vd := vendorDataDoc(p); vd != "" {
287 files["/vendor-data"] = vd 316 files["/vendor-data"] = vd
288 } 317 }
internal/agent/seed/seed_test.go
Old New
@@ -46,11 +46,16 @@ func TestBuildProducesISOWithNoCloudFiles(t *testing.T) {
46 assert.True(t, names["network-config"], "network-config must be present in ISO") 46 assert.True(t, names["network-config"], "network-config must be present in ISO")
47 } 47 }
48 48
49 func TestUserDataDefaultInjectsKeyAndGrowpart(t *testing.T) { 49 func TestUserDataDefaultInjectsKeyNotGrowpart(t *testing.T) {
50 ud := userData(Params{Hostname: "h", SSHAuthorizedKey: "ssh-ed25519 KEY"}) 50 ud := userData(Params{Hostname: "h", SSHAuthorizedKey: "ssh-ed25519 KEY"})
51 assert.True(t, strings.HasPrefix(ud, "#cloud-config\n")) 51 assert.True(t, strings.HasPrefix(ud, "#cloud-config\n"))
52 assert.Contains(t, ud, "ssh-ed25519 KEY") 52 assert.Contains(t, ud, "ssh-ed25519 KEY")
53 assert.Contains(t, ud, "growpart") // disk_gb resize completes in-guest (spec) 53 assert.Contains(t, ud, "name: ubuntu")
54 // Growing the root to disk_gb is eitri's job and lives in vendor-data, not
55 // here: a growpart in user-data would override vendor-data's `growpart: off`
56 // and re-arm the image growpart that reverts on this disk.
57 assert.NotContains(t, ud, "growpart", "growpart belongs in vendor-data, not user-data")
58 assert.NotContains(t, ud, "disk_setup", "the no-op disk_setup block is gone")
54 } 59 }
55 60
56 func TestUserDataOmitsSSHAuthorizedKeysWhenNoKey(t *testing.T) { 61 func TestUserDataOmitsSSHAuthorizedKeysWhenNoKey(t *testing.T) {
@@ -62,7 +67,6 @@ func TestUserDataOmitsSSHAuthorizedKeysWhenNoKey(t *testing.T) {
62 assert.NotContains(t, ud, "- \n", "no null list item may be rendered") 67 assert.NotContains(t, ud, "- \n", "no null list item may be rendered")
63 // The rest of the default user-data is intact. 68 // The rest of the default user-data is intact.
64 assert.Contains(t, ud, "hostname: h") 69 assert.Contains(t, ud, "hostname: h")
65 assert.Contains(t, ud, "growpart")
66 assert.Contains(t, ud, "name: ubuntu") 70 assert.Contains(t, ud, "name: ubuntu")
67 } 71 }
68 72
@@ -145,7 +149,9 @@ func readISOFile(t *testing.T, isoPath, name string) []byte {
145 return b 149 return b
146 } 150 }
147 151
148 func TestBuildWithoutCAKeepsThreeFileLayout(t *testing.T) { 152 func TestBuildAlwaysCarriesVendorDataForDiskGrow(t *testing.T) {
153 // Even with no CA/host key, the seed carries vendor-data — it holds the
154 // disk-grow that delivers disk_gb to the guest.
149 out := t.TempDir() + "/seed.iso" 155 out := t.TempDir() + "/seed.iso"
150 require.NoError(t, Build(out, Params{ 156 require.NoError(t, Build(out, Params{
151 Hostname: "plain", 157 Hostname: "plain",
@@ -162,7 +168,11 @@ func TestBuildWithoutCAKeepsThreeFileLayout(t *testing.T) {
162 for _, e := range entries { 168 for _, e := range entries {
163 names[strings.ToLower(e.Name())] = true 169 names[strings.ToLower(e.Name())] = true
164 } 170 }
165 assert.False(t, names["vendor-data"], "seeds with no CA key must not carry vendor-data") 171 assert.True(t, names["vendor-data"], "every seed carries vendor-data for the disk-grow")
172 vd := string(readISOFile(t, out, "/vendor-data"))
173 assert.Contains(t, vd, `mode: "off"`, "image growpart disabled")
174 assert.Contains(t, vd, "resize2fs /dev/vda1", "online grow runcmd present")
175 assert.NotContains(t, vd, "TrustedUserCAKeys", "no CA ⇒ no CA trust line")
166 st, err := os.Stat(out) 176 st, err := os.Stat(out)
167 require.NoError(t, err) 177 require.NoError(t, err)
168 assert.Equal(t, int64(1*1024*1024), st.Size(), "ISO size unchanged (1 MiB)") 178 assert.Equal(t, int64(1*1024*1024), st.Size(), "ISO size unchanged (1 MiB)")
@@ -272,30 +282,55 @@ func TestVendorDataHostCertWithoutCAKey(t *testing.T) {
272 assert.NotContains(t, vd, "TrustedUserCAKeys") 282 assert.NotContains(t, vd, "TrustedUserCAKeys")
273 } 283 }
274 284
275 func TestNoVendorDataWhenHostAndCAEmpty(t *testing.T) { 285 func TestVendorDataDiskGrowAlwaysPresent(t *testing.T) {
276 // Neither CA key nor host key ⇒ no vendor-data at all. 286 // Even with neither CA key nor host key, vendor-data carries the disk-grow:
277 assert.Equal(t, "", vendorDataDoc(Params{})) 287 // growpart disabled + the proven online grow runcmd. No SSH material leaks in.
288 vd := vendorDataDoc(Params{})
289 assert.True(t, strings.HasPrefix(vd, "#cloud-config\n"))
290 assert.Contains(t, vd, "growpart:\n mode: \"off\"\n",
291 "the image growpart is disabled (it reverts on this disk)")
292 // The exact proven runcmd: sfdisk grows partition 1, partx -u picks it up
293 // online on the mounted root, resize2fs grows the fs. `\\n` is two literal
294 // bytes here (Go), so the guest shell's printf emits one newline to sfdisk.
295 assert.Contains(t, vd,
296 ` - ["sh", "-c", "printf ',+\\n' | sfdisk --no-reread --no-tell-kernel -N 1 /dev/vda; partx -u /dev/vda; resize2fs /dev/vda1"]`+"\n")
297 // Nothing SSH: no host key, no CA trust, no write_files, no sshd reload.
298 assert.NotContains(t, vd, "ssh_keys:")
299 assert.NotContains(t, vd, "write_files:")
300 assert.NotContains(t, vd, "TrustedUserCAKeys")
301 assert.NotContains(t, vd, "reload sshd")
302 }
303
304 func TestVendorDataDiskGrowCoexistsWithSSHMaterial(t *testing.T) {
305 // The disk-grow sits alongside the CA trust + host cert, and its runcmd
306 // entry is a sibling of the sshd-reload — both run.
307 vd := vendorDataDoc(Params{
308 SSHUserCAAuthorizedKey: testUserCAKey,
309 SSHHostKeyPEM: testHostKeyPEM,
310 SSHHostCert: testHostCert,
311 })
312 assert.Contains(t, vd, `mode: "off"`)
313 assert.Contains(t, vd, "resize2fs /dev/vda1")
314 assert.Contains(t, vd, "TrustedUserCAKeys /etc/ssh/eitri_user_ca.pub")
315 assert.Contains(t, vd, "ed25519_certificate: |")
316 assert.Contains(t, vd, "sshd -t && systemctl reload sshd")
278 } 317 }
279 318
280 func TestNoVendorDataWhenCAKeyEmpty(t *testing.T) { 319 func TestUserSuppliedCloudInitKeepsVerbatimWhileVendorGrows(t *testing.T) {
281 // Empty CA key: no drop-in, no vendor-data at all. 320 // A BYO user-data is passed through byte-for-byte (eitri never edits it), yet
321 // the seed's vendor-data still delivers the disk-grow — the contract survives
322 // custom cloud-init.
282 out := t.TempDir() + "/seed.iso" 323 out := t.TempDir() + "/seed.iso"
324 custom := "#cloud-config\npackages: [htop]\n"
283 require.NoError(t, Build(out, Params{ 325 require.NoError(t, Build(out, Params{
284 Hostname: "plain", 326 Hostname: "byo",
285 SSHAuthorizedKey: "ssh-ed25519 AAAA user@host", 327 UserData: custom,
328 SSHUserCAAuthorizedKey: testUserCAKey,
286 })) 329 }))
287 d, err := diskfs.Open(out, diskfs.WithSectorSize(2048))
288 require.NoError(t, err)
289 defer d.Close()
290 fsi, err := d.GetFilesystem(0)
291 require.NoError(t, err)
292 entries, err := fsi.ReadDir(".")
293 require.NoError(t, err)
294 for _, e := range entries {
295 assert.NotEqual(t, "vendor-data", strings.ToLower(e.Name()),
296 "no CA key must not produce vendor-data")
297 }
298 // user-data untouched (default form).
299 ud := string(readISOFile(t, out, "/user-data")) 330 ud := string(readISOFile(t, out, "/user-data"))
300 assert.Contains(t, ud, "hostname: plain") 331 assert.Equal(t, custom, ud, "user-data is passed through verbatim")
332 assert.NotContains(t, ud, "resize2fs", "eitri does not inject the grow into user-data")
333 vd := string(readISOFile(t, out, "/vendor-data"))
334 assert.Contains(t, vd, `mode: "off"`)
335 assert.Contains(t, vd, "resize2fs /dev/vda1")
301 } 336 }