a73x

692afb3b

feat(agent): a guest steps its clock back into line after its host sleeps

a73x   2026-08-05 13:11

Commit message
feat(agent): a guest steps its clock back into line after its host sleeps

A guest's vCPUs freeze with the host that runs them, so a host suspend
stops the guest's wall clock for its duration and leaves nothing inside
the guest to notice: it wakes hours behind, its own monotonic clock
agreeing that no time passed.

The image's chrony would slew that offset away rather than step it —
its shipped `makestep 1 3` steps only for the first three updates after
chronyd starts — and slewing is rate-capped, so an hour of suspend takes
the better part of a day to walk back. For most of that day the guest is
also unreachable: SSH user certs carry a 30-minute validity window, so a
guest whose clock is further behind than that rejects every certificate
the gate mints.

The seed writes a chrony drop-in that disables the step limit, so any
update with a real offset is stepped. It lands in vendor-data beside the
disk-grow, for the same reason: a true clock is eitri's side of the
contract, so a tenant who brings their own cloud-init gets it too. The
runcmd restarts the chronyd already running from the image's config, so
the setting applies on the boot that writes it rather than the next one.

Time still comes from the image's own sources over the host's NAT; the
drop-in changes how a correction is applied, not where it comes from.
internal/agent/seed/seed.go
Old New
@@ -112,11 +112,55 @@ func sshdDropIn(p Params) string {
112 // mounted root (partx -u supplies the online update instead). 112 // mounted root (partx -u supplies the online update instead).
113 const growRuncmd = ` - ["sh", "-c", "printf ',+\\n' | sfdisk --no-reread --no-tell-kernel -N 1 /dev/vda; partx -u /dev/vda; resize2fs /dev/vda1"]` + "\n" 113 const growRuncmd = ` - ["sh", "-c", "printf ',+\\n' | sfdisk --no-reread --no-tell-kernel -N 1 /dev/vda; partx -u /dev/vda; resize2fs /dev/vda1"]` + "\n"
114 114
115 // chronyDropInPath is the guest file the clock config lands in. The image's
116 // chrony reads /etc/chrony/conf.d via a confdir directive, so a drop-in there
117 // overrides the shipped defaults without editing chrony.conf — the same
118 // drop-in idiom as the sshd config (see dropInPath).
119 const chronyDropInPath = "/etc/chrony/conf.d/eitri-clock.conf"
120
121 // chronyDropIn keeps the guest's clock true across a host suspend. A guest's
122 // vCPUs freeze with the host that runs them, so wall time stands still for the
123 // length of the suspend and nothing inside the guest observes the gap: the
124 // guest simply wakes up hours behind, with its own monotonic clock agreeing
125 // that no time passed. Only an NTP correction can find it.
126 //
127 // The image's chrony ships `makestep 1 3` — step the clock for the first three
128 // updates after chronyd starts, and slew every correction after that. Slewing
129 // is capped at maxslewrate, so an hour of suspend takes the better part of a day
130 // to walk back. A negative limit disables the limit: step at any update, which
131 // is the only setting that recovers a suspend at all.
132 //
133 // chrony's manual prefers stepping only at boot, to spare programs that assume
134 // time advances monotonically. eitri takes the other side of that trade. A step
135 // moves CLOCK_REALTIME and leaves CLOCK_MONOTONIC alone, so the programs that
136 // care are already insulated, and the alternative is worse than a jump: SSH
137 // user certs carry a 30-minute validity window, so a guest whose clock is
138 // further behind than that rejects every certificate the gate mints and is
139 // unreachable until the slew finishes.
140 //
141 // The sources are the image's own (chrony's sources.d carries the Ubuntu NTP
142 // pools), reached over the host's NAT like any other guest egress. Nothing here
143 // adds a server: the drop-in changes how a correction is applied, not where
144 // time comes from.
145 const chronyDropIn = `# Step the clock at any update, not just the first few after boot: this guest's
146 # host can suspend, and the guest cannot observe the gap on its own.
147 makestep 1 -1
148 `
149
150 // chronyRuncmd applies the drop-in to the chronyd that is already running from
151 // the image's own config: cloud-init writes guest files after chrony.service
152 // has started, so without this the setting would first take effect a boot late
153 // — on the boot after the one that needs it. try-restart does nothing on an
154 // image that does not run chrony, and the redirect and `:` keep that image from
155 // failing the runcmd and reporting cloud-init degraded.
156 const chronyRuncmd = ` - ["sh", "-c", "systemctl try-restart chrony.service 2>/dev/null || :"]` + "\n"
157
115 // vendorDataDoc renders the vendor-data #cloud-config for a seed. It always 158 // vendorDataDoc renders the vendor-data #cloud-config for a seed. It always
116 // carries the disk-grow (growpart disabled + the online grow runcmd, see 159 // carries the disk-grow (growpart disabled + the online grow runcmd, see
117 // growRuncmd); it additionally carries the eitri user-CA trust drop-in when the 160 // growRuncmd) and the clock drop-in (see chronyDropIn); it additionally carries
118 // CA key is set and the per-VM host key + cert when those are set. It is never 161 // the eitri user-CA trust drop-in when the CA key is set and the per-VM host key
119 // empty, so a vendor-data file is always written. 162 // + cert when those are set. It is never empty, so a vendor-data file is always
163 // written.
120 // 164 //
121 // The keys, cert, and drop-in conf are embedded as YAML block scalars (`|`), so 165 // The keys, cert, and drop-in conf are embedded as YAML block scalars (`|`), so
122 // their bytes appear verbatim in the guest files. Block scalars are 166 // their bytes appear verbatim in the guest files. Block scalars are
@@ -142,21 +186,23 @@ func vendorDataDoc(p Params) string {
142 writeBlockScalar(&b, " ", "ed25519_certificate", p.SSHHostCert) 186 writeBlockScalar(&b, " ", "ed25519_certificate", p.SSHHostCert)
143 } 187 }
144 dropIn := sshdDropIn(p) 188 dropIn := sshdDropIn(p)
145 if p.SSHUserCAAuthorizedKey != "" || dropIn != "" { 189 // write_files is unconditional: the clock drop-in is always written.
146 b.WriteString("write_files:\n") 190 b.WriteString("write_files:\n")
147 if p.SSHUserCAAuthorizedKey != "" { 191 if p.SSHUserCAAuthorizedKey != "" {
148 writeFileBlock(&b, userCAPath, p.SSHUserCAAuthorizedKey) 192 writeFileBlock(&b, userCAPath, p.SSHUserCAAuthorizedKey)
149 }
150 if dropIn != "" {
151 writeFileBlock(&b, dropInPath, dropIn)
152 }
153 } 193 }
194 if dropIn != "" {
195 writeFileBlock(&b, dropInPath, dropIn)
196 }
197 writeFileBlock(&b, chronyDropInPath, chronyDropIn)
198
154 b.WriteString("runcmd:\n") 199 b.WriteString("runcmd:\n")
155 b.WriteString(growRuncmd) 200 b.WriteString(growRuncmd)
156 if dropIn != "" { 201 if dropIn != "" {
157 // -t gates the reload: a bad config won't reload, so no lockout. 202 // -t gates the reload: a bad config won't reload, so no lockout.
158 b.WriteString(" - [\"sh\", \"-c\", \"sshd -t && systemctl reload sshd\"]\n") 203 b.WriteString(" - [\"sh\", \"-c\", \"sshd -t && systemctl reload sshd\"]\n")
159 } 204 }
205 b.WriteString(chronyRuncmd)
160 return b.String() 206 return b.String()
161 } 207 }
162 208
internal/agent/seed/seed_test.go
Old New
@@ -310,13 +310,34 @@ func TestVendorDataDiskGrowAlwaysPresent(t *testing.T) {
310 // bytes here (Go), so the guest shell's printf emits one newline to sfdisk. 310 // bytes here (Go), so the guest shell's printf emits one newline to sfdisk.
311 assert.Contains(t, vd, 311 assert.Contains(t, vd,
312 ` - ["sh", "-c", "printf ',+\\n' | sfdisk --no-reread --no-tell-kernel -N 1 /dev/vda; partx -u /dev/vda; resize2fs /dev/vda1"]`+"\n") 312 ` - ["sh", "-c", "printf ',+\\n' | sfdisk --no-reread --no-tell-kernel -N 1 /dev/vda; partx -u /dev/vda; resize2fs /dev/vda1"]`+"\n")
313 // Nothing SSH: no host key, no CA trust, no write_files, no sshd reload. 313 // Nothing SSH: no host key, no CA trust, no sshd reload. write_files is
314 // present regardless — it carries the clock drop-in — so the assertion is
315 // on the SSH paths themselves, not on the block that would hold them.
314 assert.NotContains(t, vd, "ssh_keys:") 316 assert.NotContains(t, vd, "ssh_keys:")
315 assert.NotContains(t, vd, "write_files:") 317 assert.NotContains(t, vd, userCAPath)
318 assert.NotContains(t, vd, dropInPath)
316 assert.NotContains(t, vd, "TrustedUserCAKeys") 319 assert.NotContains(t, vd, "TrustedUserCAKeys")
317 assert.NotContains(t, vd, "reload sshd") 320 assert.NotContains(t, vd, "reload sshd")
318 } 321 }
319 322
323 func TestVendorDataClockAlwaysPresent(t *testing.T) {
324 // The clock drop-in is fleet infrastructure like the disk-grow, not SSH
325 // material: it lands with no CA key, no host key, and no user-data.
326 vd := vendorDataDoc(Params{})
327 assert.Contains(t, vd, " - path: "+chronyDropInPath+"\n")
328 assert.Contains(t, vd, " makestep 1 -1\n",
329 "a negative limit disables the step limit, so a host suspend is stepped away rather than slewed")
330 assert.Contains(t, vd, chronyRuncmd, "the drop-in reaches the running chronyd, not just the next boot")
331 }
332
333 func TestVendorDataClockSurvivesBYOUserData(t *testing.T) {
334 // vendor-data is the reason the clock config is not optional: a tenant who
335 // brings their own user-data still gets it, exactly as they get the grow.
336 vd := vendorDataDoc(Params{UserData: "#cloud-config\npackages: [nginx]\n"})
337 assert.Contains(t, vd, chronyDropInPath)
338 assert.Contains(t, vd, " makestep 1 -1\n")
339 }
340
320 func TestVendorDataDiskGrowCoexistsWithSSHMaterial(t *testing.T) { 341 func TestVendorDataDiskGrowCoexistsWithSSHMaterial(t *testing.T) {
321 // The disk-grow sits alongside the CA trust + host cert, and its runcmd 342 // The disk-grow sits alongside the CA trust + host cert, and its runcmd
322 // entry is a sibling of the sshd-reload — both run. 343 // entry is a sibling of the sshd-reload — both run.