[P2] No watchdog/timeout around reconcile Step() #4

Closed
opened 2026-07-02 16:43:44 +00:00 by a73x · 1 comment
a73x commented 2026-07-02 16:43:44 +00:00 (Migrated from git.a73x.sh)

[claude 2026-07-02] From architecture review (data plane). syncclient worker calls Engine.Step(ctx,snap) serially (internal/agent/syncclient/client.go:231-249). A slow op inside Step — disk prep, seed ISO build — can block the worker indefinitely; only imagecache has its own 10m timeout. A stall backs up the ticker and stops all reports for that host with no signal. Add a per-Step deadline (context.WithTimeout) or a watchdog that logs/aborts a Step exceeding N minutes so one wedged VM op cannot silently freeze the agent's whole reconcile loop.


Migrated from git-collab issue af3fa8873b0028952608362336924e2edc17fdc6.

Original author: a73x dev@a73x.sh.

Close reason: Fixed by patch 9dafc985

[claude 2026-07-02] From architecture review (data plane). syncclient worker calls Engine.Step(ctx,snap) serially (internal/agent/syncclient/client.go:231-249). A slow op inside Step — disk prep, seed ISO build — can block the worker indefinitely; only imagecache has its own 10m timeout. A stall backs up the ticker and stops all reports for that host with no signal. Add a per-Step deadline (context.WithTimeout) or a watchdog that logs/aborts a Step exceeding N minutes so one wedged VM op cannot silently freeze the agent's whole reconcile loop. --- Migrated from git-collab issue `af3fa8873b0028952608362336924e2edc17fdc6`. Original author: a73x <dev@a73x.sh>. Close reason: Fixed by patch 9dafc985
a73x commented 2026-07-03 12:27:47 +00:00 (Migrated from git.a73x.sh)

[claude 2026-07-03] Picking this up. Plan: reconcile.Engine gains a StepTimeout field — Step() wraps its ctx with context.WithTimeout at entry (zero = disabled, so existing tests unaffected); main.go wires a 15m default (deliberately above imagecache's own 10m HTTP timeout so a legitimate first-time image download is never aborted). A wedged disk-prep/seed op then fails the create with DeadlineExceeded and retries next cycle instead of freezing the agent forever.

[claude 2026-07-03] Picking this up. Plan: reconcile.Engine gains a StepTimeout field — Step() wraps its ctx with context.WithTimeout at entry (zero = disabled, so existing tests unaffected); main.go wires a 15m default (deliberately above imagecache's own 10m HTTP timeout so a legitimate first-time image download is never aborted). A wedged disk-prep/seed op then fails the create with DeadlineExceeded and retries next cycle instead of freezing the agent forever.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
a73x/eitri#4
No description provided.