a1533a4f
feat: a host proposes its guest subnet, and the fleet only suggests one
a73x 2026-08-06 09:12
Commit message
docs/architecture.md
| Old | New | ||
|---|---|---|---|
| @@ -26,8 +26,11 @@ live in the agent. | |||
| 26 | Real SSH into a guest goes through the **SSH jump gate** | 26 | Real SSH into a guest goes through the **SSH jump gate** |
| 27 | (`internal/server/sshgate` + `internal/server/sshca`): the server holds an SSH | 27 | (`internal/server/sshgate` + `internal/server/sshca`): the server holds an SSH |
| 28 | CA, mints short-lived user certs, and tunnels TCP:22 to the VM over the existing | 28 | CA, mints short-lived user certs, and tunnels TCP:22 to the VM over the existing |
| 29 | server↔agent sync channel. There is no user network—VMs are reachable at their | 29 | server↔agent sync channel. There is no user network—a VM is reachable at the |
| 30 | bridge IP (`assigned_ip`) via the agent. | 30 | guest address its host reports (`assigned_ip`), dialled by the agent on that |
| 31 | host. How the guest came by that address is the host's business: a Linux host | ||
| 32 | allocates it from a bridge it builds, a Mac reads what vmnet's own DHCP handed | ||
| 33 | out. | ||
| 31 | 34 | ||
| 32 | ## Invariants | 35 | ## Invariants |
| 33 | 36 | ||
| @@ -37,7 +40,7 @@ bridge IP (`assigned_ip`) via the agent. | |||
| 37 | | **R2** | No `internal/server` package shells out—the server is pure control plane. Checked transitively: an internal wrapper around `os/exec` cannot smuggle a shell-out in. | `internal/arch` `TestServerNeverShellsOut` (transitive) + `depguard` `server-no-exec` (direct, fast in-editor). | | 40 | | **R2** | No `internal/server` package shells out—the server is pure control plane. Checked transitively: an internal wrapper around `os/exec` cannot smuggle a shell-out in. | `internal/arch` `TestServerNeverShellsOut` (transitive) + `depguard` `server-no-exec` (direct, fast in-editor). | |
| 38 | | **R3** | The wire contract (`pb`, `transport`) imports no other internal package, so a heavy dependency can't leak across the boundary into both binaries. | `internal/arch` `TestWireContractIsLeaf`. Behavior pinned by `transport` round-trip contract tests. | | 41 | | **R3** | The wire contract (`pb`, `transport`) imports no other internal package, so a heavy dependency can't leak across the boundary into both binaries. | `internal/arch` `TestWireContractIsLeaf`. Behavior pinned by `transport` round-trip contract tests. | |
| 39 | | **R4** | Pure domain packages (`agent/state`, `agent/seed`, `agent/ipalloc`, `server/registry`) don't depend on the transport stack (HTTP/QUIC/`transport`). `server/store` may use `transport` (cert helpers) but not HTTP/QUIC. | `internal/arch` `TestDomainDoesNotImportTransportStack` + `depguard` `domain-no-transport`. | | 42 | | **R4** | Pure domain packages (`agent/state`, `agent/seed`, `agent/ipalloc`, `server/registry`) don't depend on the transport stack (HTTP/QUIC/`transport`). `server/store` may use `transport` (cert helpers) but not HTTP/QUIC. | `internal/arch` `TestDomainDoesNotImportTransportStack` + `depguard` `domain-no-transport`. | |
| 40 | | **R5** | The reconcile boundary interface (`Provisioner`) stays consumer-owned and small. A VM's network attachment lives inside it, because on every backend we ship the NIC is a launch argument rather than a separately-lifecycled resource; what crosses the seam is data (the VM's address) and not mechanism (taps, reservations). | Convention (below) + `ireturn` allow-list keeps the seam's interface returns honest. | | 43 | | **R5** | The reconcile boundary interface (`Provisioner`) stays consumer-owned and small. A VM's network attachment lives inside it, because on every backend we ship the NIC is a launch argument rather than a separately-lifecycled resource; what crosses the seam is data (the VM's address) and not mechanism (taps, reservations). The host-wide counterpart crosses the *sync* seam the same way: a host reports the subnet its guests are on (`guest_cidr`) rather than being told which one to build, so the fleet records an observation and never infers a mechanism from it. | Convention (below) + `ireturn` allow-list keeps the seam's interface returns honest. | |
| 41 | | **R6** | All external process execution in the data plane funnels through `agent/exec.Runner`. The exceptions are the provisioners — `agent/cloudhv` and `agent/vfkit` — which launch the long-lived VMM process directly, and the composition root `agent/run`, which builds the Runner it injects. Checked transitively (reaching `os/exec` via a sanctioned package is fine). | `internal/arch` `TestOnlyProvisionersAndRootImportOsExecInDataPlane` (transitive). | | 44 | | **R6** | All external process execution in the data plane funnels through `agent/exec.Runner`. The exceptions are the provisioners — `agent/cloudhv` and `agent/vfkit` — which launch the long-lived VMM process directly, and the composition root `agent/run`, which builds the Runner it injects. Checked transitively (reaching `os/exec` via a sanctioned package is fine). | `internal/arch` `TestOnlyProvisionersAndRootImportOsExecInDataPlane` (transitive). | |
| 42 | | **R13** | The OIDC issuer and the relying party stay separate binaries. `eitri-server` is a pure relying party: the bundled issuer (`internal/oidcprovider`) is importable only by its own binary `cmd/eitri-oidc`, and the `go-oidc` verifier module only by `internal/server/api` (the RP). A server import of the issuer would silently rebuild the embedded-IdP coupling; `go-oidc` anywhere but the RP means a second relying party is being hand-rolled. | `internal/arch` `TestIssuerAndRelyingPartyAreSeparate`. | | 45 | | **R13** | The OIDC issuer and the relying party stay separate binaries. `eitri-server` is a pure relying party: the bundled issuer (`internal/oidcprovider`) is importable only by its own binary `cmd/eitri-oidc`, and the `go-oidc` verifier module only by `internal/server/api` (the RP). A server import of the issuer would silently rebuild the embedded-IdP coupling; `go-oidc` anywhere but the RP means a second relying party is being hand-rolled. | `internal/arch` `TestIssuerAndRelyingPartyAreSeparate`. | |
| 43 | 46 | ||
docs/decisions.md
| Old | New | ||
|---|---|---|---|
| @@ -49,6 +49,15 @@ injection via cloud-init or an external DHCP daemon. Addressing is a host | |||
| 49 | resource like memory; owning it host-locally keeps images untouched and needs | 49 | resource like memory; owning it host-locally keeps images untouched and needs |
| 50 | no external service. | 50 | no external service. |
| 51 | 51 | ||
| 52 | ### The host reports its guest network | ||
| 53 | |||
| 54 | A host tells the fleet which subnet its guests are on and the fleet records it; | ||
| 55 | the pool becomes a suggestion for hosts that have no opinion. Instead of the | ||
| 56 | fleet allocating every host a subnet and validating reported addresses against | ||
| 57 | it. A host whose OS owns the guest network sits on a subnet no allocation of | ||
| 58 | ours can contain—so the fleet can only be told, which is also why a value it did | ||
| 59 | not choose is never recycled back into the pool. | ||
| 60 | |||
| 52 | ### No superuser principal | 61 | ### No superuser principal |
| 53 | 62 | ||
| 54 | Every API principal is `{Tenant, Fleet}`; fleet-level endpoints are gated | 63 | Every API principal is `{Tenant, Fleet}`; fleet-level endpoints are gated |
docs/openapi.json
| Old | New | ||
|---|---|---|---|
| @@ -161,6 +161,12 @@ | |||
| 161 | "arch": { | 161 | "arch": { |
| 162 | "type": "string" | 162 | "type": "string" |
| 163 | }, | 163 | }, |
| 164 | "bridge_cidr": { | ||
| 165 | "type": [ | ||
| 166 | "string", | ||
| 167 | "null" | ||
| 168 | ] | ||
| 169 | }, | ||
| 164 | "name": { | 170 | "name": { |
| 165 | "type": "string" | 171 | "type": "string" |
| 166 | }, | 172 | }, |
internal/agent/enrollclient/enrollclient.go
| Old | New | ||
|---|---|---|---|
| @@ -25,6 +25,10 @@ type Request struct { | |||
| 25 | OS string `json:"os"` | 25 | OS string `json:"os"` |
| 26 | Arch string `json:"arch"` | 26 | Arch string `json:"arch"` |
| 27 | Provisioner string `json:"provisioner"` | 27 | Provisioner string `json:"provisioner"` |
| 28 | // BridgeCIDR is this host's proposal, if it has one. Nil means "no | ||
| 29 | // opinion" and takes the fleet's suggestion; a non-nil "" means "none of my | ||
| 30 | // own", which is what a host whose OS owns the guest network says. | ||
| 31 | BridgeCIDR *string `json:"bridge_cidr,omitempty"` | ||
| 28 | } | 32 | } |
| 29 | 33 | ||
| 30 | // Response carries the fields the agent consumes from a successful enroll. The | 34 | // Response carries the fields the agent consumes from a successful enroll. The |
internal/agent/run/cli.go
| Old | New | ||
|---|---|---|---|
| @@ -80,7 +80,7 @@ func RunCLI(args []string) error { | |||
| 80 | if len(rest) > 1 { | 80 | if len(rest) > 1 { |
| 81 | blob = rest[1] | 81 | blob = rest[1] |
| 82 | } | 82 | } |
| 83 | return join(st, blob) | 83 | return join(st, cfg, blob) |
| 84 | } | 84 | } |
| 85 | 85 | ||
| 86 | return serve(st, cfg) | 86 | return serve(st, cfg) |
| @@ -161,7 +161,7 @@ func parseConfig(args []string) (Config, []string, error) { | |||
| 161 | // join handles the "join <blob>" subcommand: decode the join blob, enroll, | 161 | // join handles the "join <blob>" subcommand: decode the join blob, enroll, |
| 162 | // and persist identity — pinning the server cert from the blob (the enroll | 162 | // and persist identity — pinning the server cert from the blob (the enroll |
| 163 | // response's fingerprint is ignored, so the blob is the sole trust root). | 163 | // response's fingerprint is ignored, so the blob is the sole trust root). |
| 164 | func join(st *state.Store, blob string) error { | 164 | func join(st *state.Store, cfg Config, blob string) error { |
| 165 | if blob == "" { | 165 | if blob == "" { |
| 166 | return errors.New("usage: eitri-agent join <join-blob>") | 166 | return errors.New("usage: eitri-agent join <join-blob>") |
| 167 | } | 167 | } |
| @@ -182,6 +182,7 @@ func join(st *state.Store, blob string) error { | |||
| 182 | OS: runtime.GOOS, | 182 | OS: runtime.GOOS, |
| 183 | Arch: runtime.GOARCH, | 183 | Arch: runtime.GOARCH, |
| 184 | Provisioner: platformProvisioner, | 184 | Provisioner: platformProvisioner, |
| 185 | BridgeCIDR: proposeGuestCIDR(cfg), | ||
| 185 | }) | 186 | }) |
| 186 | if errors.Is(err, enrollclient.ErrTokenRejected) { | 187 | if errors.Is(err, enrollclient.ErrTokenRejected) { |
| 187 | return errors.New("enroll rejected: token already used or expired — mint a new join token") | 188 | return errors.New("enroll rejected: token already used or expired — mint a new join token") |
| @@ -190,17 +191,20 @@ func join(st *state.Store, blob string) error { | |||
| 190 | return fmt.Errorf("enroll failed: %w", err) | 191 | return fmt.Errorf("enroll failed: %w", err) |
| 191 | } | 192 | } |
| 192 | 193 | ||
| 194 | // The response's bridge_cidr is a SUGGESTION, not a value to store: the | ||
| 195 | // resolver owns that field. Copying it here would reinstate server-wins on | ||
| 196 | // every re-enroll, which is the whole thing being removed. | ||
| 193 | id := state.Identity{ | 197 | id := state.Identity{ |
| 194 | HostID: result.HostID, | 198 | HostID: result.HostID, |
| 195 | Credential: result.Credential, | 199 | Credential: result.Credential, |
| 196 | BridgeCIDR: result.BridgeCIDR, | ||
| 197 | ServerQUICAddr: f.QUICAddr, | 200 | ServerQUICAddr: f.QUICAddr, |
| 198 | ServerCertSHA256: f.CertFP, // authoritative; response fingerprint ignored | 201 | ServerCertSHA256: f.CertFP, // authoritative; response fingerprint ignored |
| 199 | } | 202 | } |
| 200 | if err := st.SaveIdentity(id); err != nil { | 203 | if err := st.SaveIdentity(id); err != nil { |
| 201 | return fmt.Errorf("save identity: %w", err) | 204 | return fmt.Errorf("save identity: %w", err) |
| 202 | } | 205 | } |
| 203 | fmt.Printf("Enrolled: host_id=%s bridge_cidr=%s\n", result.HostID, result.BridgeCIDR) | 206 | guestCIDR := resolveGuestCIDR(st, cfg, result.BridgeCIDR) |
| 207 | fmt.Printf("Enrolled: host_id=%s guest_cidr=%s\n", result.HostID, guestCIDR) | ||
| 204 | return nil | 208 | return nil |
| 205 | } | 209 | } |
| 206 | 210 | ||
internal/agent/run/cli_test.go
| Old | New | ||
|---|---|---|---|
| @@ -111,7 +111,7 @@ func TestParseConfigBadFlag(t *testing.T) { | |||
| 111 | // TestJoinEmptyBlob rejects a missing blob before any decode or network call; | 111 | // TestJoinEmptyBlob rejects a missing blob before any decode or network call; |
| 112 | // st is never touched, so nil is safe. | 112 | // st is never touched, so nil is safe. |
| 113 | func TestJoinEmptyBlob(t *testing.T) { | 113 | func TestJoinEmptyBlob(t *testing.T) { |
| 114 | err := join(nil, "") | 114 | err := join(nil, Config{}, "") |
| 115 | require.Error(t, err) | 115 | require.Error(t, err) |
| 116 | assert.Contains(t, err.Error(), "usage: eitri-agent join") | 116 | assert.Contains(t, err.Error(), "usage: eitri-agent join") |
| 117 | } | 117 | } |
| @@ -120,7 +120,7 @@ func TestJoinEmptyBlob(t *testing.T) { | |||
| 120 | // itself (it carries a bearer token). Decode fails before st is used. | 120 | // itself (it carries a bearer token). Decode fails before st is used. |
| 121 | func TestJoinInvalidBlob(t *testing.T) { | 121 | func TestJoinInvalidBlob(t *testing.T) { |
| 122 | const secret = "not-a-valid-join-blob-with-secret" | 122 | const secret = "not-a-valid-join-blob-with-secret" |
| 123 | err := join(nil, secret) | 123 | err := join(nil, Config{}, secret) |
| 124 | require.Error(t, err) | 124 | require.Error(t, err) |
| 125 | assert.Contains(t, err.Error(), "invalid join blob") | 125 | assert.Contains(t, err.Error(), "invalid join blob") |
| 126 | assert.NotContains(t, err.Error(), secret, "the blob carries a bearer token and must never appear in errors") | 126 | assert.NotContains(t, err.Error(), secret, "the blob carries a bearer token and must never appear in errors") |
internal/agent/run/wire_darwin.go
| Old | New | ||
|---|---|---|---|
| @@ -18,6 +18,12 @@ import ( | |||
| 18 | // time — an opaque label the server stores and never interprets. | 18 | // time — an opaque label the server stores and never interprets. |
| 19 | const platformProvisioner = "vfkit" | 19 | const platformProvisioner = "vfkit" |
| 20 | 20 | ||
| 21 | // proposeGuestCIDR is what this platform offers the fleet at enrollment. A Mac | ||
| 22 | // proposes an explicit NOTHING rather than no opinion: vmnet owns the guest | ||
| 23 | // network, so this host has neither a subnet of its own to state nor any use | ||
| 24 | // for one the fleet would allocate. It reports what it observes later instead. | ||
| 25 | func proposeGuestCIDR(Config) *string { none := ""; return &none } | ||
| 26 | |||
| 21 | // defaultStateDir is where a Mac agent keeps its state when nothing says | 27 | // defaultStateDir is where a Mac agent keeps its state when nothing says |
| 22 | // otherwise: a dotdir in the running account's home, which is what the tools a | 28 | // otherwise: a dotdir in the running account's home, which is what the tools a |
| 23 | // Mac user already runs do (colima keeps ~/.colima, lima ~/.lima). Not | 29 | // Mac user already runs do (colima keeps ~/.colima, lima ~/.lima). Not |
internal/agent/run/wire_linux.go
| Old | New | ||
|---|---|---|---|
| @@ -47,6 +47,21 @@ type platform struct { | |||
| 47 | GuestCIDR func() string | 47 | GuestCIDR func() string |
| 48 | } | 48 | } |
| 49 | 49 | ||
| 50 | // proposeGuestCIDR is what this platform offers the fleet at enrollment: the | ||
| 51 | // operator's flag, or no opinion at all. Nil takes the fleet's suggestion, | ||
| 52 | // which is what a Linux host without --bridge-cidr wants. | ||
| 53 | // | ||
| 54 | // Per-platform because parseConfig is deliberately platform-neutral — every | ||
| 55 | // flag is accepted everywhere and ignored by hosts that do not run them — so a | ||
| 56 | // shared reader of cfg.BridgeCIDR would have a Mac propose a fabricated Linux | ||
| 57 | // subnet. | ||
| 58 | func proposeGuestCIDR(cfg Config) *string { | ||
| 59 | if cfg.BridgeCIDR == "" { | ||
| 60 | return nil | ||
| 61 | } | ||
| 62 | return &cfg.BridgeCIDR | ||
| 63 | } | ||
| 64 | |||
| 50 | // defaultStateDir is where a Linux agent keeps its state when nothing says | 65 | // defaultStateDir is where a Linux agent keeps its state when nothing says |
| 51 | // otherwise. The unit runs as root and the agent creates the directory itself, | 66 | // otherwise. The unit runs as root and the agent creates the directory itself, |
| 52 | // so there is nothing for an operator to pre-create. | 67 | // so there is nothing for an operator to pre-create. |
internal/agent/syncclient/client_test.go
| Old | New | ||
|---|---|---|---|
| @@ -201,7 +201,7 @@ func (h *serverHarness) stop() { | |||
| 201 | // enroll creates a host and returns a valid credential for it. | 201 | // enroll creates a host and returns a valid credential for it. |
| 202 | func (h *serverHarness) enroll() (hostID, cred string) { | 202 | func (h *serverHarness) enroll() (hostID, cred string) { |
| 203 | tok, _ := h.st.CreateEnrollmentToken(testTenant) | 203 | tok, _ := h.st.CreateEnrollmentToken(testTenant) |
| 204 | host, err := h.st.RedeemEnrollmentToken(tok, "h", "linux", "amd64", "cloudhv", "") | 204 | host, err := h.st.RedeemEnrollmentToken(tok, store.EnrollFacts{Name: "h", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 205 | require.NoError(h.t, err) | 205 | require.NoError(h.t, err) |
| 206 | return host.ID, hosttoken.Mint(h.secret, host.ID, host.CredGeneration, time.Now()) | 206 | return host.ID, hosttoken.Mint(h.secret, host.ID, host.CredGeneration, time.Now()) |
| 207 | } | 207 | } |
internal/server/api/api.go
| Old | New | ||
|---|---|---|---|
| @@ -338,7 +338,10 @@ func (a *API) handleEnroll(w http.ResponseWriter, r *http.Request) { | |||
| 338 | if !decodeJSON(w, r, &req) { | 338 | if !decodeJSON(w, r, &req) { |
| 339 | return | 339 | return |
| 340 | } | 340 | } |
| 341 | host, err := a.st.RedeemEnrollmentToken(req.Token, req.Name, req.OS, req.Arch, req.Provisioner, clientIP(r)) | 341 | host, err := a.st.RedeemEnrollmentToken(req.Token, store.EnrollFacts{ |
| 342 | Name: req.Name, OS: req.OS, Arch: req.Arch, Provisioner: req.Provisioner, | ||
| 343 | Remote: clientIP(r), BridgeCIDR: req.BridgeCIDR, | ||
| 344 | }) | ||
| 342 | if err != nil { | 345 | if err != nil { |
| 343 | // Unauthenticated enroll attempt: no tenant is resolvable (the token was | 346 | // Unauthenticated enroll attempt: no tenant is resolvable (the token was |
| 344 | // rejected), so the denied row is filed under the system audit scope. | 347 | // rejected), so the denied row is filed under the system audit scope. |
internal/server/api/testdata/enroll-request.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -3,5 +3,6 @@ | |||
| 3 | "name": "host-nine", | 3 | "name": "host-nine", |
| 4 | "os": "linux", | 4 | "os": "linux", |
| 5 | "arch": "arm64", | 5 | "arch": "arm64", |
| 6 | "provisioner": "cloudhypervisor" | 6 | "provisioner": "cloudhypervisor", |
| 7 | "bridge_cidr": "10.42.0.0/24" | ||
| 7 | } | 8 | } |
internal/server/api/types/types.go
| Old | New | ||
|---|---|---|---|
| @@ -170,6 +170,13 @@ type EnrollRequest struct { | |||
| 170 | OS string `json:"os"` | 170 | OS string `json:"os"` |
| 171 | Arch string `json:"arch"` | 171 | Arch string `json:"arch"` |
| 172 | Provisioner string `json:"provisioner"` | 172 | Provisioner string `json:"provisioner"` |
| 173 | // BridgeCIDR is the subnet this host says its guests are on. A POINTER | ||
| 174 | // because absent and empty differ: absent is "no opinion" — an older agent, | ||
| 175 | // or a host that wants the fleet's suggestion — while an explicit "" is "no | ||
| 176 | // subnet of my own", which is what a host whose OS owns the guest network | ||
| 177 | // says. The first gets an allocation; the second is left alone to report its | ||
| 178 | // own later. RevokeSSHCertRequest.Serial is the same trick. | ||
| 179 | BridgeCIDR *string `json:"bridge_cidr"` | ||
| 173 | } | 180 | } |
| 174 | 181 | ||
| 175 | // CreateVMRequest is the POST /api/v1/vms body. Every field except host_id is | 182 | // CreateVMRequest is the POST /api/v1/vms body. Every field except host_id is |
internal/server/api/wire_golden_test.go
| Old | New | ||
|---|---|---|---|
| @@ -134,7 +134,12 @@ func TestWireGolden(t *testing.T) { | |||
| 134 | Reason: "key compromised", | 134 | Reason: "key compromised", |
| 135 | }}) | 135 | }}) |
| 136 | 136 | ||
| 137 | // bridge_cidr carries a proposal here so the golden pins the field's | ||
| 138 | // presence and its pointer shape; nil and "" are separately meaningful and | ||
| 139 | // covered by the store's own tests. | ||
| 140 | proposal := "10.42.0.0/24" | ||
| 137 | goldenCheck(t, "enroll-request", types.EnrollRequest{ | 141 | goldenCheck(t, "enroll-request", types.EnrollRequest{ |
| 142 | BridgeCIDR: &proposal, | ||
| 138 | Token: "tok-secret-01", | 143 | Token: "tok-secret-01", |
| 139 | Name: "host-nine", | 144 | Name: "host-nine", |
| 140 | OS: "linux", | 145 | OS: "linux", |
internal/server/boot/sshgate_test.go
| Old | New | ||
|---|---|---|---|
| @@ -31,7 +31,7 @@ func makeTenantHost(t *testing.T, s *store.Store, subject, email string) (string | |||
| 31 | require.NoError(t, err) | 31 | require.NoError(t, err) |
| 32 | tok, err := s.CreateEnrollmentToken(tn.ID) | 32 | tok, err := s.CreateEnrollmentToken(tn.ID) |
| 33 | require.NoError(t, err) | 33 | require.NoError(t, err) |
| 34 | h, err := s.RedeemEnrollmentToken(tok, "host-"+tn.ID, "linux", "amd64", "cloudhv", "") | 34 | h, err := s.RedeemEnrollmentToken(tok, store.EnrollFacts{Name: "host-" + tn.ID, OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 35 | require.NoError(t, err) | 35 | require.NoError(t, err) |
| 36 | return tn.ID, h | 36 | return tn.ID, h |
| 37 | } | 37 | } |
internal/server/store/allocation_test.go
| Old | New | ||
|---|---|---|---|
| @@ -49,7 +49,7 @@ func TestAllocatedByHostSeparatesHosts(t *testing.T) { | |||
| 49 | s := newStore(t) | 49 | s := newStore(t) |
| 50 | h1 := enrollHost(t, s) | 50 | h1 := enrollHost(t, s) |
| 51 | tok, _ := s.CreateEnrollmentToken(testTenant) | 51 | tok, _ := s.CreateEnrollmentToken(testTenant) |
| 52 | h2, _ := s.RedeemEnrollmentToken(tok, "h2", "linux", "amd64", "cloudhv", "") | 52 | h2, _ := s.RedeemEnrollmentToken(tok, EnrollFacts{Name: "h2", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 53 | vmWithResources(t, s, h1, "a", 2, 2048, 10) | 53 | vmWithResources(t, s, h1, "a", 2, 2048, 10) |
| 54 | vmWithResources(t, s, h2, "b", 8, 8192, 40) | 54 | vmWithResources(t, s, h2, "b", 8, 8192, 40) |
| 55 | 55 | ||
| @@ -64,3 +64,62 @@ func (s *Store) mustAllocated(t *testing.T) map[string]Alloc { | |||
| 64 | require.NoError(t, err) | 64 | require.NoError(t, err) |
| 65 | return alloc | 65 | return alloc |
| 66 | } | 66 | } |
| 67 | |||
| 68 | // TestEnrollHostProposesTheFleetSuggests pins the inversion at the moment a | ||
| 69 | // host joins. The three states are deliberately distinct, which is why the | ||
| 70 | // field is a pointer: no opinion takes the fleet's suggestion, a stated subnet | ||
| 71 | // is recorded as given, and an explicit none is left alone for a host that will | ||
| 72 | // report its own once it can see it. | ||
| 73 | func TestEnrollHostProposesTheFleetSuggests(t *testing.T) { | ||
| 74 | cidr := func(s string) *string { return &s } | ||
| 75 | |||
| 76 | for _, tc := range []struct { | ||
| 77 | name string | ||
| 78 | proposal *string | ||
| 79 | want string // "" = expect the pool's allocation | ||
| 80 | wantErr string | ||
| 81 | }{ | ||
| 82 | {name: "no opinion takes the suggestion", proposal: nil, want: "10.77.1.0/24"}, | ||
| 83 | {name: "a stated subnet is recorded as given", proposal: cidr("192.168.64.0/24"), want: "192.168.64.0/24"}, | ||
| 84 | {name: "an explicit none is left alone", proposal: cidr(""), want: ""}, | ||
| 85 | {name: "unparseable is refused", proposal: cidr("not-a-network"), wantErr: "guest cidr"}, | ||
| 86 | {name: "a bare address is refused", proposal: cidr("192.168.64.1"), wantErr: "guest cidr"}, | ||
| 87 | {name: "IPv6 is refused", proposal: cidr("fd00::/64"), wantErr: "must be IPv4"}, | ||
| 88 | } { | ||
| 89 | t.Run(tc.name, func(t *testing.T) { | ||
| 90 | s := newStore(t) | ||
| 91 | tok, err := s.CreateEnrollmentToken(testTenant) | ||
| 92 | require.NoError(t, err) | ||
| 93 | h, err := s.RedeemEnrollmentToken(tok, EnrollFacts{ | ||
| 94 | Name: "h", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", | ||
| 95 | BridgeCIDR: tc.proposal, | ||
| 96 | }) | ||
| 97 | if tc.wantErr != "" { | ||
| 98 | require.Error(t, err) | ||
| 99 | assert.Contains(t, err.Error(), tc.wantErr) | ||
| 100 | return | ||
| 101 | } | ||
| 102 | require.NoError(t, err) | ||
| 103 | assert.Equal(t, tc.want, h.BridgeCIDR) | ||
| 104 | }) | ||
| 105 | } | ||
| 106 | } | ||
| 107 | |||
| 108 | // TestEnrollWithAProposalDoesNotSpendAnAllocation pins that a host stating its | ||
| 109 | // own subnet leaves the pool where it was — the fleet allocates for hosts that | ||
| 110 | // want advice, not for every host that joins. | ||
| 111 | func TestEnrollWithAProposalDoesNotSpendAnAllocation(t *testing.T) { | ||
| 112 | s := newStore(t) | ||
| 113 | own := "192.168.64.0/24" | ||
| 114 | tok, _ := s.CreateEnrollmentToken(testTenant) | ||
| 115 | _, err := s.RedeemEnrollmentToken(tok, EnrollFacts{ | ||
| 116 | Name: "mac", OS: "darwin", Arch: "arm64", Provisioner: "vfkit", BridgeCIDR: &own}) | ||
| 117 | require.NoError(t, err) | ||
| 118 | |||
| 119 | tok2, _ := s.CreateEnrollmentToken(testTenant) | ||
| 120 | linux, err := s.RedeemEnrollmentToken(tok2, EnrollFacts{ | ||
| 121 | Name: "linux", OS: "linux", Arch: "amd64", Provisioner: "cloudhv"}) | ||
| 122 | require.NoError(t, err) | ||
| 123 | assert.Equal(t, "10.77.1.0/24", linux.BridgeCIDR, | ||
| 124 | "the first allocation is still available: the Mac consumed none") | ||
| 125 | } | ||
internal/server/store/decommission_test.go
| Old | New | ||
|---|---|---|---|
| @@ -45,29 +45,36 @@ func TestDecommissionHostTombstonesVMsAndSetsStatus(t *testing.T) { | |||
| 45 | assert.Greater(t, epochAfter, epochBefore, "decommission must bump epoch") | 45 | assert.Greater(t, epochAfter, epochBefore, "decommission must bump epoch") |
| 46 | } | 46 | } |
| 47 | 47 | ||
| 48 | func TestRemoveHostFreesCIDRForReuse(t *testing.T) { | 48 | // TestRemoveHostDoesNotReissueItsSubnet pins the reversal of what this test |
| 49 | // used to assert. A departing host's bridge_cidr is NOT returned to the pool, | ||
| 50 | // because post-inversion that column holds what the HOST claimed — so | ||
| 51 | // re-issuing it would hand one host's subnet to the next as a suggestion, and a | ||
| 52 | // Mac's 192.168.64.0/24 is not something to advise a Linux host to build. | ||
| 53 | // | ||
| 54 | // The monotonic index alone is sufficient: a /16 pool is 65k /24s, and leaking | ||
| 55 | // indices across host churn is cheaper than re-issuing a value the fleet never | ||
| 56 | // chose. | ||
| 57 | func TestRemoveHostDoesNotReissueItsSubnet(t *testing.T) { | ||
| 49 | s := newStore(t) | 58 | s := newStore(t) |
| 50 | h1 := enrollHost(t, s) | 59 | h1 := enrollHost(t, s) |
| 51 | assert.Equal(t, "10.77.1.0/24", h1.BridgeCIDR) | 60 | assert.Equal(t, "10.77.1.0/24", h1.BridgeCIDR) |
| 52 | tok2, _ := s.CreateEnrollmentToken(testTenant) | 61 | tok2, _ := s.CreateEnrollmentToken(testTenant) |
| 53 | h2, _ := s.RedeemEnrollmentToken(tok2, "b", "linux", "amd64", "cloudhv", "") | 62 | h2, _ := s.RedeemEnrollmentToken(tok2, EnrollFacts{Name: "b", OS: "linux", Arch: "amd64", Provisioner: "cloudhv"}) |
| 54 | assert.Equal(t, "10.77.2.0/24", h2.BridgeCIDR) | 63 | assert.Equal(t, "10.77.2.0/24", h2.BridgeCIDR) |
| 55 | 64 | ||
| 56 | // Decommission h1 and simulate the agent reaping its VMs (hard-delete). | ||
| 57 | vm := makeVM(t, s, h1, "vm-a") | 65 | vm := makeVM(t, s, h1, "vm-a") |
| 58 | require.NoError(t, s.DecommissionHost(h1.ID)) | 66 | require.NoError(t, s.DecommissionHost(h1.ID)) |
| 59 | require.NoError(t, s.HardDeleteVM(vm.ID)) | 67 | require.NoError(t, s.HardDeleteVM(vm.ID)) |
| 60 | |||
| 61 | require.NoError(t, s.RemoveHost(h1.ID)) | 68 | require.NoError(t, s.RemoveHost(h1.ID)) |
| 62 | 69 | ||
| 63 | _, err := s.GetHost(h1.ID) | 70 | _, err := s.GetHost(h1.ID) |
| 64 | assert.Error(t, err, "removed host should be gone") | 71 | assert.Error(t, err, "removed host should be gone") |
| 65 | 72 | ||
| 66 | // A new enrollment reuses h1's freed CIDR rather than allocating a fresh one. | ||
| 67 | tok3, _ := s.CreateEnrollmentToken(testTenant) | 73 | tok3, _ := s.CreateEnrollmentToken(testTenant) |
| 68 | h3, err := s.RedeemEnrollmentToken(tok3, "c", "linux", "amd64", "cloudhv", "") | 74 | h3, err := s.RedeemEnrollmentToken(tok3, EnrollFacts{Name: "c", OS: "linux", Arch: "amd64", Provisioner: "cloudhv"}) |
| 69 | require.NoError(t, err) | 75 | require.NoError(t, err) |
| 70 | assert.Equal(t, "10.77.1.0/24", h3.BridgeCIDR, "freed CIDR should be reused") | 76 | assert.Equal(t, "10.77.3.0/24", h3.BridgeCIDR, |
| 77 | "the allocator moves on; a departed host's subnet is never re-issued") | ||
| 71 | } | 78 | } |
| 72 | 79 | ||
| 73 | func TestRemoveHostRefusesWhileVMsRemain(t *testing.T) { | 80 | func TestRemoveHostRefusesWhileVMsRemain(t *testing.T) { |
internal/server/store/evolve.go
| Old | New | ||
|---|---|---|---|
| @@ -28,3 +28,17 @@ func ensureColumn(db *sql.DB, table, column, decl string) error { | |||
| 28 | } | 28 | } |
| 29 | return nil | 29 | return nil |
| 30 | } | 30 | } |
| 31 | |||
| 32 | // dropTable removes a table that is no longer part of the schema, so an | ||
| 33 | // existing database stops carrying it. The schema const only creates tables; | ||
| 34 | // removing one has to go through here for the same reason adding a column does. | ||
| 35 | // | ||
| 36 | // table is interpolated verbatim (SQLite cannot bind identifiers): pass trusted | ||
| 37 | // compile-time constants only, never caller- or request-derived strings — the | ||
| 38 | // same rule ensureColumn states, and for the same reason. | ||
| 39 | func dropTable(db *sql.DB, table string) error { | ||
| 40 | if _, err := db.Exec(fmt.Sprintf(`DROP TABLE IF EXISTS %s`, table)); err != nil { | ||
| 41 | return fmt.Errorf("drop %s: %w", table, err) | ||
| 42 | } | ||
| 43 | return nil | ||
| 44 | } | ||
internal/server/store/store.go
| Old | New | ||
|---|---|---|---|
| @@ -166,12 +166,6 @@ CREATE TABLE IF NOT EXISTS vms ( | |||
| 166 | -- vms name uniqueness is per-tenant. | 166 | -- vms name uniqueness is per-tenant. |
| 167 | CREATE UNIQUE INDEX IF NOT EXISTS vms_tenant_name ON vms(tenant, name) WHERE deleted_at IS NULL; | 167 | CREATE UNIQUE INDEX IF NOT EXISTS vms_tenant_name ON vms(tenant, name) WHERE deleted_at IS NULL; |
| 168 | 168 | ||
| 169 | -- bridge CIDRs returned to the pool by host decommission, available for reuse | ||
| 170 | -- before the monotonic next_cidr_index allocator is consulted. | ||
| 171 | CREATE TABLE IF NOT EXISTS freed_cidrs ( | ||
| 172 | bridge_cidr TEXT PRIMARY KEY | ||
| 173 | ); | ||
| 174 | |||
| 175 | -- revoked SSH user certs: a tenant can revoke a specific user cert by its serial | 169 | -- revoked SSH user certs: a tenant can revoke a specific user cert by its serial |
| 176 | -- (crypto-random uint64, set at mint) so it is rejected at the jump gate before | 170 | -- (crypto-random uint64, set at mint) so it is rejected at the jump gate before |
| 177 | -- its short TTL expires. serial is stored as the int64 bit-pattern of the uint64 | 171 | -- its short TTL expires. serial is stored as the int64 bit-pattern of the uint64 |
| @@ -294,6 +288,15 @@ func Open(path, cidrPool string) (*Store, error) { | |||
| 294 | return nil, err | 288 | return nil, err |
| 295 | } | 289 | } |
| 296 | } | 290 | } |
| 291 | // freed_cidrs held bridge CIDRs returned by a departing host, to be re-issued | ||
| 292 | // before the monotonic allocator was consulted. Nothing is recycled now: the | ||
| 293 | // column holds what the HOST claims, so re-issuing it would hand one host's | ||
| 294 | // subnet to another as a suggestion. | ||
| 295 | if err := dropTable(db, "freed_cidrs"); err != nil { | ||
| 296 | db.Close() | ||
| 297 | return nil, err | ||
| 298 | } | ||
| 299 | |||
| 297 | // One identity binds at most one tenant (per issuer). Partial index so | 300 | // One identity binds at most one tenant (per issuer). Partial index so |
| 298 | // unbound rows (empty issuer+subject) don't collide. | 301 | // unbound rows (empty issuer+subject) don't collide. |
| 299 | if _, err := db.Exec(`CREATE UNIQUE INDEX IF NOT EXISTS tenants_identity | 302 | if _, err := db.Exec(`CREATE UNIQUE INDEX IF NOT EXISTS tenants_identity |
| @@ -378,12 +381,30 @@ func (s *Store) CreateEnrollmentToken(tenant string) (string, error) { | |||
| 378 | return tok, nil | 381 | return tok, nil |
| 379 | } | 382 | } |
| 380 | 383 | ||
| 384 | // EnrollFacts is what a joining host says about itself. It is a struct rather | ||
| 385 | // than more positional strings because the enrollment transaction already | ||
| 386 | // carried six of them, and because BridgeCIDR needs three states that a plain | ||
| 387 | // string cannot express — see its field. | ||
| 388 | type EnrollFacts struct { | ||
| 389 | Name, OS, Arch, Provisioner string | ||
| 390 | // Remote is the client address, recorded in the audit row. | ||
| 391 | Remote string | ||
| 392 | // BridgeCIDR is the subnet this host says its guests are on, and it is a | ||
| 393 | // POINTER because absent and empty mean different things. Nil is "this host | ||
| 394 | // has no opinion" — an older agent, or a Linux host started without | ||
| 395 | // --bridge-cidr — and takes the pool's suggestion. A non-nil empty string is | ||
| 396 | // "this host deliberately has none", which is what a Mac says: its OS owns | ||
| 397 | // the guest network and will report the subnet later, so allocating one for | ||
| 398 | // it would invent a fact. A non-nil value is stored as given. | ||
| 399 | BridgeCIDR *string | ||
| 400 | } | ||
| 401 | |||
| 381 | // RedeemEnrollmentToken atomically consumes tok and creates the host row. The | 402 | // RedeemEnrollmentToken atomically consumes tok and creates the host row. The |
| 382 | // created host inherits the tenant bound to the consumed token. remote (the | 403 | // created host inherits the tenant bound to the consumed token. remote (the |
| 383 | // enrolling client's IP) is recorded in a host.enroll audit row written in | 404 | // enrolling client's IP) is recorded in a host.enroll audit row written in |
| 384 | // the SAME transaction, so an enrolled host can never exist without its | 405 | // the SAME transaction, so an enrolled host can never exist without its |
| 385 | // durable audit record. | 406 | // durable audit record. |
| 386 | func (s *Store) RedeemEnrollmentToken(tok, name, osName, arch, provisioner, remote string) (Host, error) { | 407 | func (s *Store) RedeemEnrollmentToken(tok string, f EnrollFacts) (Host, error) { |
| 387 | h := sha256.Sum256([]byte(tok)) | 408 | h := sha256.Sum256([]byte(tok)) |
| 388 | hash := hex.EncodeToString(h[:]) | 409 | hash := hex.EncodeToString(h[:]) |
| 389 | 410 | ||
| @@ -427,14 +448,29 @@ func (s *Store) RedeemEnrollmentToken(tok, name, osName, arch, provisioner, remo | |||
| 427 | return Host{}, fmt.Errorf("parse cidr_pool: %w", err) | 448 | return Host{}, fmt.Errorf("parse cidr_pool: %w", err) |
| 428 | } | 449 | } |
| 429 | 450 | ||
| 430 | // Reuse a CIDR freed by a prior decommission before extending the monotonic | 451 | // The host proposes; the fleet only suggests. A host that states its subnet |
| 431 | // allocator, so the pool doesn't leak across host churn. | 452 | // is recorded as given — validated as a network, never checked against the |
| 453 | // pool, because a host whose OS owns the guest network is on a subnet no | ||
| 454 | // allocation of ours will ever contain. | ||
| 455 | // | ||
| 456 | // Nothing is recycled back into the pool. Post-inversion this column holds | ||
| 457 | // what the HOST claims, so returning it would hand a Mac's 192.168.64.0/24 | ||
| 458 | // to the next Linux host as its suggested bridge. A /16 pool is 65k /24s and | ||
| 459 | // the monotonic index alone is sufficient: leaking indices is cheaper than | ||
| 460 | // re-issuing a value the fleet did not choose. | ||
| 432 | var bridgeCIDR string | 461 | var bridgeCIDR string |
| 433 | if err := tx.QueryRow(`SELECT bridge_cidr FROM freed_cidrs ORDER BY bridge_cidr LIMIT 1`).Scan(&bridgeCIDR); err == nil { | 462 | switch { |
| 434 | if _, err := tx.Exec(`DELETE FROM freed_cidrs WHERE bridge_cidr=?`, bridgeCIDR); err != nil { | 463 | case f.BridgeCIDR != nil && *f.BridgeCIDR != "": |
| 435 | return Host{}, fmt.Errorf("consume freed cidr: %w", err) | 464 | if err := validGuestCIDR(*f.BridgeCIDR); err != nil { |
| 465 | return Host{}, err | ||
| 436 | } | 466 | } |
| 437 | } else if err == sql.ErrNoRows { | 467 | bridgeCIDR = *f.BridgeCIDR |
| 468 | case f.BridgeCIDR != nil: | ||
| 469 | // Deliberately none: this host will report its subnet once it can see | ||
| 470 | // it. Allocating one here would invent a fact and put it on the row. | ||
| 471 | bridgeCIDR = "" | ||
| 472 | default: | ||
| 473 | // No opinion — an older agent, or a host that wants the fleet's advice. | ||
| 438 | bridgeCIDR, err = subnetForIndex(prefix, nextIdx) | 474 | bridgeCIDR, err = subnetForIndex(prefix, nextIdx) |
| 439 | if err != nil { | 475 | if err != nil { |
| 440 | return Host{}, err | 476 | return Host{}, err |
| @@ -442,15 +478,13 @@ func (s *Store) RedeemEnrollmentToken(tok, name, osName, arch, provisioner, remo | |||
| 442 | if _, err := tx.Exec(`UPDATE meta SET value=? WHERE key='next_cidr_index'`, nextIdx+1); err != nil { | 478 | if _, err := tx.Exec(`UPDATE meta SET value=? WHERE key='next_cidr_index'`, nextIdx+1); err != nil { |
| 443 | return Host{}, fmt.Errorf("increment next_cidr_index: %w", err) | 479 | return Host{}, fmt.Errorf("increment next_cidr_index: %w", err) |
| 444 | } | 480 | } |
| 445 | } else { | ||
| 446 | return Host{}, fmt.Errorf("read freed_cidrs: %w", err) | ||
| 447 | } | 481 | } |
| 448 | 482 | ||
| 449 | id := random.Hex(16) | 483 | id := random.Hex(16) |
| 450 | 484 | ||
| 451 | if _, err := tx.Exec( | 485 | if _, err := tx.Exec( |
| 452 | `INSERT INTO hosts(id, name, os, arch, provisioner, bridge_cidr, enrolled_at, tenant) VALUES (?,?,?,?,?,?,?,?)`, | 486 | `INSERT INTO hosts(id, name, os, arch, provisioner, bridge_cidr, enrolled_at, tenant) VALUES (?,?,?,?,?,?,?,?)`, |
| 453 | id, name, osName, arch, provisioner, bridgeCIDR, now.Format(time.RFC3339), tenant, | 487 | id, f.Name, f.OS, f.Arch, f.Provisioner, bridgeCIDR, now.Format(time.RFC3339), tenant, |
| 454 | ); err != nil { | 488 | ); err != nil { |
| 455 | return Host{}, fmt.Errorf("insert host: %w", err) | 489 | return Host{}, fmt.Errorf("insert host: %w", err) |
| 456 | } | 490 | } |
| @@ -459,8 +493,8 @@ func (s *Store) RedeemEnrollmentToken(tok, name, osName, arch, provisioner, remo | |||
| 459 | // once the enrollment itself commits. The token appears only as a hash | 493 | // once the enrollment itself commits. The token appears only as a hash |
| 460 | // prefix (matches the enrollment_tokens.token_hash the mint row logs). | 494 | // prefix (matches the enrollment_tokens.token_hash the mint row logs). |
| 461 | detail, _ := json.Marshal(map[string]string{ | 495 | detail, _ := json.Marshal(map[string]string{ |
| 462 | "host_id": id, "name": name, "os": osName, "arch": arch, | 496 | "host_id": id, "name": f.Name, "os": f.OS, "arch": f.Arch, |
| 463 | "remote": remote, "token_hash_prefix": hash[:8], | 497 | "remote": f.Remote, "token_hash_prefix": hash[:8], |
| 464 | }) | 498 | }) |
| 465 | if _, err := tx.Exec(`INSERT INTO audit_log(at, tenant, action, detail) VALUES (?, ?, ?, ?)`, | 499 | if _, err := tx.Exec(`INSERT INTO audit_log(at, tenant, action, detail) VALUES (?, ?, ?, ?)`, |
| 466 | now.Format(time.RFC3339), tenant, "host.enroll", string(detail)); err != nil { | 500 | now.Format(time.RFC3339), tenant, "host.enroll", string(detail)); err != nil { |
| @@ -473,10 +507,10 @@ func (s *Store) RedeemEnrollmentToken(tok, name, osName, arch, provisioner, remo | |||
| 473 | 507 | ||
| 474 | return Host{ | 508 | return Host{ |
| 475 | ID: id, | 509 | ID: id, |
| 476 | Name: name, | 510 | Name: f.Name, |
| 477 | OS: osName, | 511 | OS: f.OS, |
| 478 | Arch: arch, | 512 | Arch: f.Arch, |
| 479 | Provisioner: provisioner, | 513 | Provisioner: f.Provisioner, |
| 480 | BridgeCIDR: bridgeCIDR, | 514 | BridgeCIDR: bridgeCIDR, |
| 481 | Status: "enrolled", | 515 | Status: "enrolled", |
| 482 | CredGeneration: 1, | 516 | CredGeneration: 1, |
| @@ -498,6 +532,26 @@ func (s *Store) GetHost(id string) (Host, error) { | |||
| 498 | return h, nil | 532 | return h, nil |
| 499 | } | 533 | } |
| 500 | 534 | ||
| 535 | // validGuestCIDR is the sanity check applied to every subnet a host claims, | ||
| 536 | // wherever it arrives — proposed at enrollment, or reported in a tick. It is | ||
| 537 | // the same check netenv makes before handing the value to the kernel, applied | ||
| 538 | // on the receiving side so a garbage string cannot reach the console, an | ||
| 539 | // operator's eyes, or anything that reads the row. | ||
| 540 | // | ||
| 541 | // It asks whether the value is a network. It never asks WHICH network: a host's | ||
| 542 | // guest subnet is that host's business, and applying sanity to one of a row's | ||
| 543 | // two network columns and not the other would be sanity by coincidence. | ||
| 544 | func validGuestCIDR(cidr string) error { | ||
| 545 | p, err := netip.ParsePrefix(cidr) | ||
| 546 | if err != nil { | ||
| 547 | return fmt.Errorf("guest cidr %q: %w", cidr, err) | ||
| 548 | } | ||
| 549 | if !p.Addr().Is4() { | ||
| 550 | return fmt.Errorf("guest cidr %q must be IPv4", cidr) | ||
| 551 | } | ||
| 552 | return nil | ||
| 553 | } | ||
| 554 | |||
| 501 | // RecordHostNetwork stores the subnet a host says its guests are on. The value | 555 | // RecordHostNetwork stores the subnet a host says its guests are on. The value |
| 502 | // is free text supplied by whoever claims to be that host, so it is validated | 556 | // is free text supplied by whoever claims to be that host, so it is validated |
| 503 | // the same way netenv validates it before handing it to the kernel — parses as | 557 | // the same way netenv validates it before handing it to the kernel — parses as |
| @@ -511,14 +565,10 @@ func (s *Store) GetHost(id string) (Host, error) { | |||
| 511 | // the storage layer; the caller still guards the round trip, because reports | 565 | // the storage layer; the caller still guards the round trip, because reports |
| 512 | // arrive every tick forever and the store runs on one connection. | 566 | // arrive every tick forever and the store runs on one connection. |
| 513 | func (s *Store) RecordHostNetwork(id, cidr string) error { | 567 | func (s *Store) RecordHostNetwork(id, cidr string) error { |
| 514 | p, err := netip.ParsePrefix(cidr) | 568 | if err := validGuestCIDR(cidr); err != nil { |
| 515 | if err != nil { | 569 | return err |
| 516 | return fmt.Errorf("guest cidr %q: %w", cidr, err) | ||
| 517 | } | ||
| 518 | if !p.Addr().Is4() { | ||
| 519 | return fmt.Errorf("guest cidr %q must be IPv4", cidr) | ||
| 520 | } | 570 | } |
| 521 | _, err = s.db.Exec(`UPDATE hosts SET bridge_cidr=? WHERE id=? AND bridge_cidr<>?`, cidr, id, cidr) | 571 | _, err := s.db.Exec(`UPDATE hosts SET bridge_cidr=? WHERE id=? AND bridge_cidr<>?`, cidr, id, cidr) |
| 522 | return err | 572 | return err |
| 523 | } | 573 | } |
| 524 | 574 | ||
| @@ -1014,13 +1064,6 @@ func (s *Store) RemoveHost(id string) error { | |||
| 1014 | return fmt.Errorf("host %s still has %d VM(s); not drained", id, n) | 1064 | return fmt.Errorf("host %s still has %d VM(s); not drained", id, n) |
| 1015 | } | 1065 | } |
| 1016 | 1066 | ||
| 1017 | var bridgeCIDR string | ||
| 1018 | if err := tx.QueryRow(`SELECT bridge_cidr FROM hosts WHERE id=?`, id).Scan(&bridgeCIDR); err != nil { | ||
| 1019 | return fmt.Errorf("lookup host cidr: %w", err) | ||
| 1020 | } | ||
| 1021 | if _, err := tx.Exec(`INSERT INTO freed_cidrs(bridge_cidr) VALUES(?) ON CONFLICT DO NOTHING`, bridgeCIDR); err != nil { | ||
| 1022 | return fmt.Errorf("free cidr: %w", err) | ||
| 1023 | } | ||
| 1024 | res, err := tx.Exec(`DELETE FROM hosts WHERE id=?`, id) | 1067 | res, err := tx.Exec(`DELETE FROM hosts WHERE id=?`, id) |
| 1025 | if err != nil { | 1068 | if err != nil { |
| 1026 | return fmt.Errorf("delete host: %w", err) | 1069 | return fmt.Errorf("delete host: %w", err) |
| @@ -1047,12 +1090,14 @@ func (s *Store) ForceRemoveHost(id string) (int, error) { | |||
| 1047 | } | 1090 | } |
| 1048 | defer tx.Rollback() | 1091 | defer tx.Rollback() |
| 1049 | 1092 | ||
| 1050 | var bridgeCIDR string | 1093 | // Existence check only: nothing is reclaimed from a departing host, so the |
| 1051 | switch err := tx.QueryRow(`SELECT bridge_cidr FROM hosts WHERE id=?`, id).Scan(&bridgeCIDR); { | 1094 | // row's contents no longer matter here. |
| 1095 | var exists int | ||
| 1096 | switch err := tx.QueryRow(`SELECT 1 FROM hosts WHERE id=?`, id).Scan(&exists); { | ||
| 1052 | case errors.Is(err, sql.ErrNoRows): | 1097 | case errors.Is(err, sql.ErrNoRows): |
| 1053 | return 0, sql.ErrNoRows | 1098 | return 0, sql.ErrNoRows |
| 1054 | case err != nil: | 1099 | case err != nil: |
| 1055 | return 0, fmt.Errorf("lookup host cidr: %w", err) | 1100 | return 0, fmt.Errorf("lookup host: %w", err) |
| 1056 | } | 1101 | } |
| 1057 | 1102 | ||
| 1058 | res, err := tx.Exec(`DELETE FROM vms WHERE host_id=?`, id) | 1103 | res, err := tx.Exec(`DELETE FROM vms WHERE host_id=?`, id) |
| @@ -1061,9 +1106,6 @@ func (s *Store) ForceRemoveHost(id string) (int, error) { | |||
| 1061 | } | 1106 | } |
| 1062 | purged, _ := res.RowsAffected() | 1107 | purged, _ := res.RowsAffected() |
| 1063 | 1108 | ||
| 1064 | if _, err := tx.Exec(`INSERT INTO freed_cidrs(bridge_cidr) VALUES(?) ON CONFLICT DO NOTHING`, bridgeCIDR); err != nil { | ||
| 1065 | return 0, fmt.Errorf("free cidr: %w", err) | ||
| 1066 | } | ||
| 1067 | if _, err := tx.Exec(`DELETE FROM hosts WHERE id=?`, id); err != nil { | 1109 | if _, err := tx.Exec(`DELETE FROM hosts WHERE id=?`, id); err != nil { |
| 1068 | return 0, fmt.Errorf("delete host: %w", err) | 1110 | return 0, fmt.Errorf("delete host: %w", err) |
| 1069 | } | 1111 | } |
internal/server/store/store_test.go
| Old | New | ||
|---|---|---|---|
| @@ -39,7 +39,7 @@ func enrollHost(t *testing.T, s *Store) Host { | |||
| 39 | t.Helper() | 39 | t.Helper() |
| 40 | tok, err := s.CreateEnrollmentToken(testTenant) | 40 | tok, err := s.CreateEnrollmentToken(testTenant) |
| 41 | require.NoError(t, err) | 41 | require.NoError(t, err) |
| 42 | h, err := s.RedeemEnrollmentToken(tok, "host-a", "linux", "amd64", "cloudhv", "") | 42 | h, err := s.RedeemEnrollmentToken(tok, EnrollFacts{Name: "host-a", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 43 | require.NoError(t, err) | 43 | require.NoError(t, err) |
| 44 | return h | 44 | return h |
| 45 | } | 45 | } |
| @@ -244,15 +244,15 @@ func TestTenantUserCAs(t *testing.T) { | |||
| 244 | func TestEnrollmentAssignsSequentialCIDRsAndIsOneTimeUse(t *testing.T) { | 244 | func TestEnrollmentAssignsSequentialCIDRsAndIsOneTimeUse(t *testing.T) { |
| 245 | s := newStore(t) | 245 | s := newStore(t) |
| 246 | tok1, _ := s.CreateEnrollmentToken(testTenant) | 246 | tok1, _ := s.CreateEnrollmentToken(testTenant) |
| 247 | h1, err := s.RedeemEnrollmentToken(tok1, "a", "linux", "amd64", "cloudhv", "") | 247 | h1, err := s.RedeemEnrollmentToken(tok1, EnrollFacts{Name: "a", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 248 | require.NoError(t, err) | 248 | require.NoError(t, err) |
| 249 | assert.Equal(t, "10.77.1.0/24", h1.BridgeCIDR) | 249 | assert.Equal(t, "10.77.1.0/24", h1.BridgeCIDR) |
| 250 | 250 | ||
| 251 | _, err = s.RedeemEnrollmentToken(tok1, "b", "linux", "amd64", "cloudhv", "") | 251 | _, err = s.RedeemEnrollmentToken(tok1, EnrollFacts{Name: "b", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 252 | assert.Error(t, err, "token must be one-time use") | 252 | assert.Error(t, err, "token must be one-time use") |
| 253 | 253 | ||
| 254 | tok2, _ := s.CreateEnrollmentToken(testTenant) | 254 | tok2, _ := s.CreateEnrollmentToken(testTenant) |
| 255 | h2, _ := s.RedeemEnrollmentToken(tok2, "b", "linux", "amd64", "cloudhv", "") | 255 | h2, _ := s.RedeemEnrollmentToken(tok2, EnrollFacts{Name: "b", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 256 | assert.Equal(t, "10.77.2.0/24", h2.BridgeCIDR) | 256 | assert.Equal(t, "10.77.2.0/24", h2.BridgeCIDR) |
| 257 | } | 257 | } |
| 258 | 258 | ||
| @@ -368,7 +368,7 @@ func TestEnrollmentCIDRWorksForNonSlash16Pools(t *testing.T) { | |||
| 368 | _, err = s.CreateTenantForIdentity("https://test-issuer", "test-subject", testTenant+"@test.local") | 368 | _, err = s.CreateTenantForIdentity("https://test-issuer", "test-subject", testTenant+"@test.local") |
| 369 | require.NoError(t, err) | 369 | require.NoError(t, err) |
| 370 | tok, _ := s.CreateEnrollmentToken(testTenant) | 370 | tok, _ := s.CreateEnrollmentToken(testTenant) |
| 371 | h, err := s.RedeemEnrollmentToken(tok, "a", "linux", "amd64", "cloudhv", "") | 371 | h, err := s.RedeemEnrollmentToken(tok, EnrollFacts{Name: "a", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 372 | require.NoError(t, err) | 372 | require.NoError(t, err) |
| 373 | assert.Equal(t, "192.168.5.0/24", h.BridgeCIDR, "1st /24 within the pool, 0th reserved") | 373 | assert.Equal(t, "192.168.5.0/24", h.BridgeCIDR, "1st /24 within the pool, 0th reserved") |
| 374 | } | 374 | } |
| @@ -380,11 +380,11 @@ func TestEnrollmentFailsWhenPoolExhausted(t *testing.T) { | |||
| 380 | _, err = s.CreateTenantForIdentity("https://test-issuer", "test-subject", testTenant+"@test.local") | 380 | _, err = s.CreateTenantForIdentity("https://test-issuer", "test-subject", testTenant+"@test.local") |
| 381 | require.NoError(t, err) | 381 | require.NoError(t, err) |
| 382 | tok1, _ := s.CreateEnrollmentToken(testTenant) | 382 | tok1, _ := s.CreateEnrollmentToken(testTenant) |
| 383 | h, err := s.RedeemEnrollmentToken(tok1, "a", "linux", "amd64", "cloudhv", "") | 383 | h, err := s.RedeemEnrollmentToken(tok1, EnrollFacts{Name: "a", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 384 | require.NoError(t, err) | 384 | require.NoError(t, err) |
| 385 | assert.Equal(t, "10.9.9.0/24", h.BridgeCIDR) | 385 | assert.Equal(t, "10.9.9.0/24", h.BridgeCIDR) |
| 386 | tok2, _ := s.CreateEnrollmentToken(testTenant) | 386 | tok2, _ := s.CreateEnrollmentToken(testTenant) |
| 387 | _, err = s.RedeemEnrollmentToken(tok2, "b", "linux", "amd64", "cloudhv", "") | 387 | _, err = s.RedeemEnrollmentToken(tok2, EnrollFacts{Name: "b", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 388 | assert.ErrorContains(t, err, "exhausted") | 388 | assert.ErrorContains(t, err, "exhausted") |
| 389 | } | 389 | } |
| 390 | 390 | ||
| @@ -394,7 +394,7 @@ func TestRecordVMStatusUnknownVMErrors(t *testing.T) { | |||
| 394 | assert.Error(t, err) | 394 | assert.Error(t, err) |
| 395 | } | 395 | } |
| 396 | 396 | ||
| 397 | func TestForceRemoveHostPurgesVMsAndFreesCIDR(t *testing.T) { | 397 | func TestForceRemoveHostPurgesVMs(t *testing.T) { |
| 398 | s := newStore(t) | 398 | s := newStore(t) |
| 399 | h := enrollHost(t, s) | 399 | h := enrollHost(t, s) |
| 400 | makeVM(t, s, h, "vm-a") | 400 | makeVM(t, s, h, "vm-a") |
| @@ -419,13 +419,14 @@ func TestForceRemoveHostPurgesVMsAndFreesCIDR(t *testing.T) { | |||
| 419 | assert.NotEqual(t, h.ID, vm.HostID, "no VM rows should remain for the force-removed host") | 419 | assert.NotEqual(t, h.ID, vm.HostID, "no VM rows should remain for the force-removed host") |
| 420 | } | 420 | } |
| 421 | 421 | ||
| 422 | // The freed CIDR is consulted before the monotonic allocator, so a fresh | 422 | // Its subnet is not returned to the pool: the column holds what the HOST |
| 423 | // enrollment reuses it. | 423 | // claimed, and re-issuing that to the next host would advise it to build a |
| 424 | // network somebody else chose. | ||
| 424 | tok, err := s.CreateEnrollmentToken(testTenant) | 425 | tok, err := s.CreateEnrollmentToken(testTenant) |
| 425 | require.NoError(t, err) | 426 | require.NoError(t, err) |
| 426 | h2, err := s.RedeemEnrollmentToken(tok, "host-b", "linux", "amd64", "cloudhv", "") | 427 | h2, err := s.RedeemEnrollmentToken(tok, EnrollFacts{Name: "host-b", OS: "linux", Arch: "amd64", Provisioner: "cloudhv"}) |
| 427 | require.NoError(t, err) | 428 | require.NoError(t, err) |
| 428 | assert.Equal(t, h.BridgeCIDR, h2.BridgeCIDR, "force-removed host's CIDR must return to the pool") | 429 | assert.NotEqual(t, h.BridgeCIDR, h2.BridgeCIDR, "a force-removed host's subnet is never re-issued") |
| 429 | } | 430 | } |
| 430 | 431 | ||
| 431 | func TestForceRemoveHostUnknownIsNoRows(t *testing.T) { | 432 | func TestForceRemoveHostUnknownIsNoRows(t *testing.T) { |
| @@ -595,7 +596,7 @@ func TestRedeemWritesAuditRowAtomically(t *testing.T) { | |||
| 595 | s := newStore(t) | 596 | s := newStore(t) |
| 596 | tok, err := s.CreateEnrollmentToken(testTenant) | 597 | tok, err := s.CreateEnrollmentToken(testTenant) |
| 597 | require.NoError(t, err) | 598 | require.NoError(t, err) |
| 598 | h, err := s.RedeemEnrollmentToken(tok, "host-a", "linux", "amd64", "cloudhv", "192.0.2.9") | 599 | h, err := s.RedeemEnrollmentToken(tok, EnrollFacts{Name: "host-a", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: "192.0.2.9"}) |
| 599 | require.NoError(t, err) | 600 | require.NoError(t, err) |
| 600 | 601 | ||
| 601 | rows, err := s.ListAudit(testTenant, 5) | 602 | rows, err := s.ListAudit(testTenant, 5) |
| @@ -756,7 +757,7 @@ func TestEnrollmentCarriesTenantToHost(t *testing.T) { | |||
| 756 | 757 | ||
| 757 | tok, err := s.CreateEnrollmentToken("t2") | 758 | tok, err := s.CreateEnrollmentToken("t2") |
| 758 | require.NoError(t, err) | 759 | require.NoError(t, err) |
| 759 | h, err := s.RedeemEnrollmentToken(tok, "box", "linux", "amd64", "cloudhv", "203.0.113.9") | 760 | h, err := s.RedeemEnrollmentToken(tok, EnrollFacts{Name: "box", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: "203.0.113.9"}) |
| 760 | require.NoError(t, err) | 761 | require.NoError(t, err) |
| 761 | assert.Equal(t, "t2", h.Tenant, "host must inherit the token's tenant") | 762 | assert.Equal(t, "t2", h.Tenant, "host must inherit the token's tenant") |
| 762 | got, err := s.GetHost(h.ID) | 763 | got, err := s.GetHost(h.ID) |
internal/server/syncsvc/syncsvc_test.go
| Old | New | ||
|---|---|---|---|
| @@ -56,7 +56,7 @@ func setupWithWriteTimeout(t *testing.T, writeTimeout time.Duration) *fixture { | |||
| 56 | t.Cleanup(func() { st.Close() }) | 56 | t.Cleanup(func() { st.Close() }) |
| 57 | seedTestTenant(t, st) | 57 | seedTestTenant(t, st) |
| 58 | tok, _ := st.CreateEnrollmentToken(testTenant) | 58 | tok, _ := st.CreateEnrollmentToken(testTenant) |
| 59 | host, err := st.RedeemEnrollmentToken(tok, "h", "linux", "amd64", "cloudhv", "") | 59 | host, err := st.RedeemEnrollmentToken(tok, store.EnrollFacts{Name: "h", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 60 | require.NoError(t, err) | 60 | require.NoError(t, err) |
| 61 | 61 | ||
| 62 | reg := registry.New(time.Now) | 62 | reg := registry.New(time.Now) |
| @@ -543,7 +543,7 @@ func TestExpiredCredentialRejectedWhenMaxAgeSet(t *testing.T) { | |||
| 543 | t.Cleanup(func() { st.Close() }) | 543 | t.Cleanup(func() { st.Close() }) |
| 544 | seedTestTenant(t, st) | 544 | seedTestTenant(t, st) |
| 545 | tok, _ := st.CreateEnrollmentToken(testTenant) | 545 | tok, _ := st.CreateEnrollmentToken(testTenant) |
| 546 | host, err := st.RedeemEnrollmentToken(tok, "h", "linux", "amd64", "cloudhv", "") | 546 | host, err := st.RedeemEnrollmentToken(tok, store.EnrollFacts{Name: "h", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 547 | require.NoError(t, err) | 547 | require.NoError(t, err) |
| 548 | 548 | ||
| 549 | reg := registry.New(time.Now) | 549 | reg := registry.New(time.Now) |
| @@ -586,7 +586,7 @@ func TestMaxAgeEnforcedMidSession(t *testing.T) { | |||
| 586 | t.Cleanup(func() { st.Close() }) | 586 | t.Cleanup(func() { st.Close() }) |
| 587 | seedTestTenant(t, st) | 587 | seedTestTenant(t, st) |
| 588 | tok, _ := st.CreateEnrollmentToken(testTenant) | 588 | tok, _ := st.CreateEnrollmentToken(testTenant) |
| 589 | host, err := st.RedeemEnrollmentToken(tok, "h", "linux", "amd64", "cloudhv", "") | 589 | host, err := st.RedeemEnrollmentToken(tok, store.EnrollFacts{Name: "h", OS: "linux", Arch: "amd64", Provisioner: "cloudhv", Remote: ""}) |
| 590 | require.NoError(t, err) | 590 | require.NoError(t, err) |
| 591 | 591 | ||
| 592 | reg := registry.New(time.Now) | 592 | reg := registry.New(time.Now) |
web/src/lib/api-types.ts
| Old | New | ||
|---|---|---|---|
| @@ -1271,6 +1271,7 @@ export interface components { | |||
| 1271 | }; | 1271 | }; |
| 1272 | EnrollRequest: { | 1272 | EnrollRequest: { |
| 1273 | arch?: string; | 1273 | arch?: string; |
| 1274 | bridge_cidr?: string | null; | ||
| 1274 | name?: string; | 1275 | name?: string; |
| 1275 | os?: string; | 1276 | os?: string; |
| 1276 | provisioner?: string; | 1277 | provisioner?: string; |
web/src/routes/+page.svelte
| Old | New | ||
|---|---|---|---|
| @@ -212,7 +212,7 @@ | |||
| 212 | {#if fleet.hosts.length > 0 && shownHosts.length > 0} | 212 | {#if fleet.hosts.length > 0 && shownHosts.length > 0} |
| 213 | <table> | 213 | <table> |
| 214 | <thead> | 214 | <thead> |
| 215 | <tr><th>Name</th><th>Status</th><th>OS</th><th>Version</th><th>Load</th><th>VMs</th><th>CIDR</th><th>Used / total (vCPU · mem · disk)</th><th></th></tr> | 215 | <tr><th>Name</th><th>Status</th><th>OS</th><th>Version</th><th>Load</th><th>VMs</th><th>Guest network</th><th>Used / total (vCPU · mem · disk)</th><th></th></tr> |
| 216 | </thead> | 216 | </thead> |
| 217 | <tbody> | 217 | <tbody> |
| 218 | {#each shownHosts as h (h.id)} | 218 | {#each shownHosts as h (h.id)} |
| @@ -246,7 +246,7 @@ | |||
| 246 | {/if} | 246 | {/if} |
| 247 | </td> | 247 | </td> |
| 248 | <td>{vmCountByHost.get(h.id) ?? 0}</td> | 248 | <td>{vmCountByHost.get(h.id) ?? 0}</td> |
| 249 | <td>{h.bridge_cidr}</td> | 249 | <td>{h.bridge_cidr || '—'}</td> |
| 250 | <td> | 250 | <td> |
| 251 | {h.allocated.vcpus}/{h.capacity.vcpus || '?'}c · | 251 | {h.allocated.vcpus}/{h.capacity.vcpus || '?'}c · |
| 252 | {h.allocated.mem_mb}/{h.capacity.mem_mb || '?'}MB · | 252 | {h.allocated.mem_mb}/{h.capacity.mem_mb || '?'}MB · |
web/src/routes/hosts/[id]/+page.svelte
| Old | New | ||
|---|---|---|---|
| @@ -63,7 +63,16 @@ | |||
| 63 | <tr><th>CPU</th><td>{host.cpu_model || '—'}</td></tr> | 63 | <tr><th>CPU</th><td>{host.cpu_model || '—'}</td></tr> |
| 64 | <tr><th>Virtualization</th><td>{host.virt || '—'}</td></tr> | 64 | <tr><th>Virtualization</th><td>{host.virt || '—'}</td></tr> |
| 65 | <tr><th>Provisioner</th><td>{host.provisioner}</td></tr> | 65 | <tr><th>Provisioner</th><td>{host.provisioner}</td></tr> |
| 66 | <tr><th>Bridge CIDR</th><td>{host.bridge_cidr}</td></tr> | 66 | <tr> |
| 67 | <th>Guest network</th> | ||
| 68 | <td> | ||
| 69 | {#if host.bridge_cidr} | ||
| 70 | {host.bridge_cidr} | ||
| 71 | {:else} | ||
| 72 | <span class="hint">not yet reported</span> | ||
| 73 | {/if} | ||
| 74 | </td> | ||
| 75 | </tr> | ||
| 67 | <tr><th>Capacity</th><td>{host.capacity.vcpus}c / {host.capacity.mem_mb}MB / {host.capacity.disk_gb}GB</td></tr> | 76 | <tr><th>Capacity</th><td>{host.capacity.vcpus}c / {host.capacity.mem_mb}MB / {host.capacity.disk_gb}GB</td></tr> |
| 68 | <tr><th>Enrolled</th><td>{host.enrolled_at}</td></tr> | 77 | <tr><th>Enrolled</th><td>{host.enrolled_at}</td></tr> |
| 69 | </tbody> | 78 | </tbody> |