7dd7ae3c
feat(exposures): a published port can carry datagrams
a73x 2026-08-09 18:21
Commit message
docs/assumptions.md
| Old | New | ||
|---|---|---|---|
| @@ -358,13 +358,24 @@ error and retried on every converge. | |||
| 358 | 358 | ||
| 359 | ### A published port may be bound on every interface | 359 | ### A published port may be bound on every interface |
| 360 | 360 | ||
| 361 | The listener binds 0.0.0.0—every interface, IPv4 only. An exposure is | 361 | The socket binds 0.0.0.0—every interface, IPv4 only. An exposure is |
| 362 | reachable on any IPv4 address the host has, and on none of its IPv6 ones. | 362 | reachable on any IPv4 address the host has, and on none of its IPv6 ones. |
| 363 | Underpins there being one exposure rather than one per network. | 363 | Underpins there being one exposure rather than one per network. |
| 364 | **Proven** in code; the posture is deliberate—see the auth answer in | 364 | **Proven** in code; the posture is deliberate—see the auth answer in |
| 365 | [faq.md](faq.md). A host whose networks are not equally trusted gets more | 365 | [faq.md](faq.md). A host whose networks are not equally trusted gets more |
| 366 | reach than it asked for. | 366 | reach than it asked for. |
| 367 | 367 | ||
| 368 | ### A published UDP port may be used as a reflector | ||
| 369 | |||
| 370 | UDP has no handshake, so a datagram's source address is a claim and nothing | ||
| 371 | more. The host sends the guest's reply to whatever address that claim names, | ||
| 372 | which means a published UDP port can be aimed at a third party. Underpins | ||
| 373 | publishing UDP at all, on the same terms as TCP: reaching the host is reaching | ||
| 374 | the service, and a port is published onto a network the tenant is content to | ||
| 375 | serve. **True by construction**, not mitigated—there is no rate limit and no | ||
| 376 | source check in the proxy. A guest whose service amplifies (answers larger than | ||
| 377 | it is asked) makes the host a better reflector than a bare port would. | ||
| 378 | |||
| 368 | ### eitri holds no signing key for anyone | 379 | ### eitri holds no signing key for anyone |
| 369 | 380 | ||
| 370 | A caller lends eitri a credential instead: eitri offers an ephemeral public key, | 381 | A caller lends eitri a credential instead: eitri offers an ephemeral public key, |
docs/decisions.md
| Old | New | ||
|---|---|---|---|
| @@ -83,7 +83,9 @@ not choose is never recycled back into the pool. | |||
| 83 | ### Networking stays minimal: just enough to be usable | 83 | ### Networking stays minimal: just enough to be usable |
| 84 | 84 | ||
| 85 | A guest gets an address, a way out, and the ports its tenant chose to publish— | 85 | A guest gets an address, a way out, and the ports its tenant chose to publish— |
| 86 | embedded DHCP, the host as NAT gateway, and a userspace port-forward table—any | 86 | embedded DHCP, the host as NAT gateway, and a userspace port-forward table for |
| 87 | TCP and UDP alike (a spliced connection per caller, a session per client | ||
| 88 | address; both capped, both deletable, neither routing anything else)—any | ||
| 87 | of which could be deleted with the VM still booting, syncing, and taking SSH. | 89 | of which could be deleted with the VM still booting, syncing, and taking SSH. |
| 88 | Instead of growing the agent toward a virtual router (guest DNS, ACLs, | 90 | Instead of growing the agent toward a virtual router (guest DNS, ACLs, |
| 89 | east-west, in-path policy). Network machinery that basic lifecycle would come | 91 | east-west, in-path policy). Network machinery that basic lifecycle would come |
docs/faq.md
| Old | New | ||
|---|---|---|---|
| @@ -19,11 +19,18 @@ your tenant's CA. A published port has nothing in front of it: whoever can | |||
| 19 | reach the host on that port reaches the service, exactly as if the service were | 19 | reach the host on that port reaches the service, exactly as if the service were |
| 20 | running on the host itself. | 20 | running on the host itself. |
| 21 | 21 | ||
| 22 | That is the LAN trust posture, deliberately. The listener binds every interface | 22 | That is the LAN trust posture, deliberately. The socket binds every interface |
| 23 | on the host, IPv4 only. Publish what you are content to serve to everything | 23 | on the host, IPv4 only. Publish what you are content to serve to everything |
| 24 | that can reach it; put anything else behind the gate, or behind the service's | 24 | that can reach it; put anything else behind the gate, or behind the service's |
| 25 | own authentication. | 25 | own authentication. |
| 26 | 26 | ||
| 27 | A published UDP port is the same posture carried one step further. Nothing in | ||
| 28 | UDP proves where a datagram came from, so the host answers the address the | ||
| 29 | datagram claimed—someone else's, if that is what it said. A UDP exposure can | ||
| 30 | therefore be pointed at a third party, and a guest service that answers larger | ||
| 31 | than it is asked makes that worse. Publish UDP the way you would publish | ||
| 32 | anything else here: onto a network you are content to serve. | ||
| 33 | |||
| 27 | ## VMs boot and SSH works, but have no outbound network—why? | 34 | ## VMs boot and SSH works, but have no outbound network—why? |
| 28 | 35 | ||
| 29 | Docker. Installing (or starting) Docker on a host sets the kernel's iptables | 36 | Docker. Installing (or starting) Docker on a host sets the kernel's iptables |
docs/mcp.md
| Old | New | ||
|---|---|---|---|
| @@ -22,9 +22,9 @@ Both serve the same tools. | |||
| 22 | | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. | | 22 | | `vm_exec` | Run a shell command in a VM over SSH; returns stdout, stderr, exit code. | |
| 23 | | `vm_write_file` | Write content to a path in a VM over SFTP (parent dirs created). | | 23 | | `vm_write_file` | Write content to a path in a VM over SFTP (parent dirs created). | |
| 24 | | `vm_read_file` | Read a file from a VM over SFTP (capped at 1 MiB, truncation flagged). | | 24 | | `vm_read_file` | Read a file from a VM over SFTP (capped at 1 MiB, truncation flagged). | |
| 25 | | `vm_expose` | Publish a guest TCP port on the VM's host; returns the exposure id, both ports, the address to dial and the listener's state. Omit `host_port` to allocate one from 30000–32767. | | 25 | | `vm_expose` | Publish a guest port on the VM's host, `protocol` `tcp` (the default) or `udp`; returns the exposure id, both ports, the protocol, the address to dial and the socket's state. Omit `host_port` to allocate one from 30000–32767. | |
| 26 | | `vm_exposures` | List a VM's published ports, same shape. | | 26 | | `vm_exposures` | List a VM's published ports, same shape. | |
| 27 | | `vm_unexpose` | Stop publishing a guest port (`host_port` disambiguates when one guest port is published twice). | | 27 | | `vm_unexpose` | Stop publishing a guest port (`host_port` or `protocol` disambiguates when one guest port is published twice). | |
| 28 | | `vm_destroy` | Destroy a VM by id or exact name. Explicit-only—never called automatically. | | 28 | | `vm_destroy` | Destroy a VM by id or exact name. Explicit-only—never called automatically. | |
| 29 | | `ca_upload` | Register your SSH user CA's public key with your tenant, with an optional label. Only the public half is sent. A guest trusts the CA set it was created with, so VMs that already exist will not accept certificates from a CA uploaded now. | | 29 | | `ca_upload` | Register your SSH user CA's public key with your tenant, with an optional label. Only the public half is sent. A guest trusts the CA set it was created with, so VMs that already exist will not accept certificates from a CA uploaded now. | |
| 30 | | `tenant_info` | Show this tenant's setup: registered CAs (fingerprint and label), whether a delegation is live and when it expires, and the gate address. Read-only. | | 30 | | `tenant_info` | Show this tenant's setup: registered CAs (fingerprint and label), whether a delegation is live and when it expires, and the gate address. Read-only. | |
| @@ -174,9 +174,11 @@ result or error. | |||
| 174 | table above. | 174 | table above. |
| 175 | - A published port is **unauthenticated**. `vm_expose` binds a host port and | 175 | - A published port is **unauthenticated**. `vm_expose` binds a host port and |
| 176 | pipes it to the guest; whoever can reach the host on that port reaches the | 176 | pipes it to the guest; whoever can reach the host on that port reaches the |
| 177 | service. The tool descriptions say so, so the model treats publishing as a | 177 | service. A UDP one goes further: with no handshake to prove where a datagram |
| 178 | deliberate act. DNS, TLS certs and routing remain out of scope—the tools | 178 | came from, the host answers the address the datagram claimed, so the port can |
| 179 | hand back a host address and a port. | 179 | be aimed at a third party. The tool descriptions say so, so the model treats |
| 180 | publishing as a deliberate act. DNS, TLS certs and routing remain out of | ||
| 181 | scope—the tools hand back a host address and a port. | ||
| 180 | - The remote endpoint authenticates with a bearer PAT. Browser connectors | 182 | - The remote endpoint authenticates with a bearer PAT. Browser connectors |
| 181 | (claude.ai) need OAuth, which the endpoint does not speak. | 183 | (claude.ai) need OAuth, which the endpoint does not speak. |
| 182 | 184 | ||
| @@ -198,4 +200,5 @@ cover the tools and the HTTP transport against fake API and SSH seams. The | |||
| 198 | deploy boot-gate (`make deploy`) drives a whole VM life through the remote | 200 | deploy boot-gate (`make deploy`) drives a whole VM life through the remote |
| 199 | endpoint with a bearer PAT: register a CA, `delegate_begin`, sign, refuse a | 201 | endpoint with a bearer PAT: register a CA, `delegate_begin`, sign, refuse a |
| 200 | certificate from an unregistered CA, `delegate_complete`, `vm_create`, | 202 | certificate from an unregistered CA, `delegate_complete`, `vm_create`, |
| 201 | `vm_exec`, `vm_expose`, dial the published port, `vm_destroy`. | 203 | `vm_exec`, `vm_expose`, dial the published port, publish a UDP port and echo a |
| 204 | datagram through it, `vm_destroy`. | ||
docs/openapi.json
| Old | New | ||
|---|---|---|---|
| @@ -110,6 +110,9 @@ | |||
| 110 | }, | 110 | }, |
| 111 | "host_port": { | 111 | "host_port": { |
| 112 | "type": "integer" | 112 | "type": "integer" |
| 113 | }, | ||
| 114 | "protocol": { | ||
| 115 | "type": "string" | ||
| 113 | } | 116 | } |
| 114 | }, | 117 | }, |
| 115 | "type": "object" | 118 | "type": "object" |
| @@ -2061,7 +2064,7 @@ | |||
| 2061 | "patToken": [] | 2064 | "patToken": [] |
| 2062 | } | 2065 | } |
| 2063 | ], | 2066 | ], |
| 2064 | "summary": "Publish a guest TCP port on the VM's host. Omit host_port to allocate one from the reserved range 30000-32767; a named port must be \u003e= 1024 and is honored or refused." | 2067 | "summary": "Publish a guest port on the VM's host, protocol \"tcp\" (the default) or \"udp\". Omit host_port to allocate one from the reserved range 30000-32767; a named port must be \u003e= 1024 and is honored or refused, and is taken only by another exposure of the same protocol." |
| 2065 | } | 2068 | } |
| 2066 | }, | 2069 | }, |
| 2067 | "/api/v1/vms/{id}/restore": { | 2070 | "/api/v1/vms/{id}/restore": { |
docs/quickstart.md
| Old | New | ||
|---|---|---|---|
| @@ -149,20 +149,31 @@ The gate reaches a guest over SSH. Anything else a guest serves needs a | |||
| 149 | published port: the fleet binds one on the VM's host and pipes it to the guest. | 149 | published port: the fleet binds one on the VM's host and pipes it to the guest. |
| 150 | 150 | ||
| 151 | Open the VM's page in the console, find **Exposures**, and enter the port your | 151 | Open the VM's page in the console, find **Exposures**, and enter the port your |
| 152 | service listens on inside the guest—8080, say. Leave the host port blank and | 152 | service listens on inside the guest—8080, say. Pick `tcp` or `udp`. Leave the |
| 153 | eitri allocates one from 30000–32767, the range it reserves on every host. The | 153 | host port blank and eitri allocates one from 30000–32767, the range it reserves |
| 154 | row then reads: | 154 | on every host. The row then reads: |
| 155 | 155 | ||
| 156 | guest :8080 → 192.168.0.190:30080 ● active | 156 | guest :8080/tcp → 192.168.0.190:30080 ● active |
| 157 | 157 | ||
| 158 | That address is the host's. Anything that can reach the host on that port | 158 | That address is the host's. Anything that can reach the host on that port |
| 159 | reaches the service—there is no authentication in front of a published port, so | 159 | reaches the service—there is no authentication in front of a published port, so |
| 160 | publish what you are content to serve to everything on that network, and leave | 160 | publish what you are content to serve to everything on that network, and leave |
| 161 | the rest to the gate. | 161 | the rest to the gate. |
| 162 | 162 | ||
| 163 | `active` means the host's listener is bound. Whether anything answers on the | 163 | A host port is claimed per protocol, so `30080/tcp` and `30080/udp` can be two |
| 164 | different services. | ||
| 165 | |||
| 166 | `active` means the host's socket is bound. Whether anything answers on the | ||
| 164 | guest's 8080 is the guest's business. | 167 | guest's 8080 is the guest's business. |
| 165 | 168 | ||
| 169 | A UDP exposure asks for one thing more of you than a TCP one. UDP has no | ||
| 170 | handshake, so nothing proves a datagram came from the address it claims: the | ||
| 171 | host sends the guest's reply wherever that source header says, which makes a | ||
| 172 | published UDP port something an outsider can aim at a third party. That is the | ||
| 173 | same bargain the rest of a published port already is—reaching the host is | ||
| 174 | reaching the service—so it is accepted rather than papered over, and it is one | ||
| 175 | more reason to publish only onto a network you are content to serve. | ||
| 176 | |||
| 166 | Remove the row to take the port down. Deleting the VM takes its exposures with | 177 | Remove the row to take the port down. Deleting the VM takes its exposures with |
| 167 | it. | 178 | it. |
| 168 | 179 | ||
internal/agent/exposeproxy/exposeproxy.go
| Old | New | ||
|---|---|---|---|
| @@ -1,10 +1,15 @@ | |||
| 1 | // Package exposeproxy publishes guest ports on their host. A Manager owns one | 1 | // Package exposeproxy publishes guest ports on their host. A Manager owns one |
| 2 | // listener and one goroutine per exposure id — the same manager shape as the | 2 | // socket and one goroutine per exposure id — the same manager shape as the |
| 3 | // serial pumps — and Converge drives that set toward the exposures the fleet | 3 | // serial pumps — and Converge drives that set toward the exposures the fleet |
| 4 | // says this host should be serving: a new id binds, a vanished id closes, a | 4 | // says this host should be serving: a new id binds, a vanished id closes, a |
| 5 | // changed spec closes and rebinds. | 5 | // changed spec closes and rebinds. |
| 6 | // | 6 | // |
| 7 | // The listener binds 0.0.0.0. All interfaces is the point: the LAN address is | 7 | // A TCP exposure is a listener and a spliced connection per caller; a UDP one |
| 8 | // is a single bound socket and a session per client address (udp.go). The | ||
| 9 | // protocol is part of an exposure's spec, so changing it is a rebind like any | ||
| 10 | // other change. | ||
| 11 | // | ||
| 12 | // The socket binds 0.0.0.0. All interfaces is the point: the LAN address is | ||
| 8 | // the feature, the tailnet address is how an operator reaches it from | 13 | // the feature, the tailnet address is how an operator reaches it from |
| 9 | // elsewhere, and the guest-side bridge seeing it is the same trust domain the | 14 | // elsewhere, and the guest-side bridge seeing it is the same trust domain the |
| 10 | // LAN posture already accepts. There is no authentication in front of a | 15 | // LAN posture already accepts. There is no authentication in front of a |
| @@ -12,10 +17,10 @@ | |||
| 12 | // | 17 | // |
| 13 | // Each accepted connection asks for the VM's address AT THAT MOMENT and dials | 18 | // Each accepted connection asks for the VM's address AT THAT MOMENT and dials |
| 14 | // the guest fresh. That per-connection lookup is what makes an exposure created | 19 | // the guest fresh. That per-connection lookup is what makes an exposure created |
| 15 | // before its guest has booted simply work: the listener binds now, and | 20 | // before its guest has booted simply work: the socket binds now, and |
| 16 | // connections start succeeding when the guest does. | 21 | // connections start succeeding when the guest does. |
| 17 | // | 22 | // |
| 18 | // Nothing here is persisted. Listeners are rebuilt from the first snapshot | 23 | // Nothing here is persisted. Sockets are rebuilt from the first snapshot |
| 19 | // after the agent starts, the same way the consoles are; connections in flight | 24 | // after the agent starts, the same way the consoles are; connections in flight |
| 20 | // across an agent restart drop, and reconnecting works. | 25 | // across an agent restart drop, and reconnecting works. |
| 21 | // | 26 | // |
| @@ -57,7 +62,7 @@ const ( | |||
| 57 | acceptRetryCeiling = time.Second | 62 | acceptRetryCeiling = time.Second |
| 58 | ) | 63 | ) |
| 59 | 64 | ||
| 60 | // Manager runs one listener per active exposure. | 65 | // Manager runs one socket per active exposure. |
| 61 | type Manager struct { | 66 | type Manager struct { |
| 62 | // addr answers a VM's current guest address, or "" when this host does not | 67 | // addr answers a VM's current guest address, or "" when this host does not |
| 63 | // know one. Injected so the package stays a leaf: it never learns what a VM | 68 | // know one. Injected so the package stays a leaf: it never learns what a VM |
| @@ -65,10 +70,13 @@ type Manager struct { | |||
| 65 | // goroutines; it must be safe for concurrent use. | 70 | // goroutines; it must be safe for concurrent use. |
| 66 | addr func(vmID string) string | 71 | addr func(vmID string) string |
| 67 | 72 | ||
| 68 | // maxConns is maxConnsPerExposure, held per-Manager so a test can serve the | 73 | // maxConns, maxSessions and the two idle windows are the package constants, |
| 69 | // same behaviour with a handful of connections instead of hundreds. Written | 74 | // held per-Manager so a test can serve the same behaviour with a handful of |
| 70 | // once at construction and only read after, so it needs no lock. | 75 | // callers instead of hundreds, and with windows it can outlast. Written |
| 71 | maxConns int64 | 76 | // once at construction and only read after, so they need no lock. |
| 77 | maxConns int64 | ||
| 78 | maxSessions int | ||
| 79 | unrepliedIdle, repliedIdle time.Duration | ||
| 72 | 80 | ||
| 73 | mu sync.Mutex | 81 | mu sync.Mutex |
| 74 | live map[string]*exposure | 82 | live map[string]*exposure |
| @@ -77,47 +85,77 @@ type Manager struct { | |||
| 77 | // NewManager returns a Manager that resolves each connection's destination | 85 | // NewManager returns a Manager that resolves each connection's destination |
| 78 | // through addr. | 86 | // through addr. |
| 79 | func NewManager(addr func(vmID string) string) *Manager { | 87 | func NewManager(addr func(vmID string) string) *Manager { |
| 80 | return &Manager{addr: addr, maxConns: maxConnsPerExposure, live: map[string]*exposure{}} | 88 | return &Manager{ |
| 89 | addr: addr, | ||
| 90 | maxConns: maxConnsPerExposure, | ||
| 91 | maxSessions: maxSessionsPerExposure, | ||
| 92 | unrepliedIdle: udpUnrepliedIdle, | ||
| 93 | repliedIdle: udpRepliedIdle, | ||
| 94 | live: map[string]*exposure{}, | ||
| 95 | } | ||
| 81 | } | 96 | } |
| 82 | 97 | ||
| 83 | // exposure is one published port's running state: the spec key it was bound | 98 | // exposure is one published port's running state: the spec key it is bound for, |
| 84 | // for, its listener (nil when the bind failed), what the OS said if it did, and | 99 | // its socket — a listener for TCP, a packet socket for UDP, both nil when the |
| 85 | // how many connections it is holding open right now. | 100 | // bind failed — what the OS said if it did, and what it is holding open right |
| 101 | // now: connections for TCP, sessions for UDP. | ||
| 86 | // | 102 | // |
| 87 | // The count lives here rather than with the listener so that an exposure whose | 103 | // One entry lives per exposure id for as long as the fleet wants that id, and a |
| 88 | // accept loop died and was rebound keeps counting what is still draining | 104 | // changed spec closes and rebinds INSIDE it rather than replacing it. That is |
| 89 | // through it: the descriptors those connections hold are still spent. | 105 | // what keeps the count honest: connections spliced under the old spec drain |
| 106 | // through this same struct, and the descriptors they hold are still spent. | ||
| 90 | type exposure struct { | 107 | type exposure struct { |
| 91 | key string | 108 | key string |
| 92 | ln net.Listener | 109 | ln net.Listener |
| 110 | pc *net.UDPConn | ||
| 93 | reason string | 111 | reason string |
| 94 | conns atomic.Int64 | 112 | conns atomic.Int64 |
| 113 | |||
| 114 | // sessions is the UDP session table, guarded by its own mutex because the | ||
| 115 | // packet loop touches it on every datagram while the manager lock is held | ||
| 116 | // across whole converges. nil until the first UDP bind. | ||
| 117 | smu sync.Mutex | ||
| 118 | sessions map[string]*udpSession | ||
| 95 | } | 119 | } |
| 96 | 120 | ||
| 97 | // close tears down the listener, which is what ends its accept goroutine. | 121 | // bound reports whether this exposure currently holds a socket. |
| 122 | func (e *exposure) bound() bool { return e.ln != nil || e.pc != nil } | ||
| 123 | |||
| 124 | // close tears down the socket, which is what ends the goroutine serving it, and | ||
| 125 | // drops every UDP session with it: the sessions belong to the socket they were | ||
| 126 | // created on, and there is nothing left to answer them through. | ||
| 98 | func (e *exposure) close() { | 127 | func (e *exposure) close() { |
| 99 | if e.ln != nil { | 128 | if e.ln != nil { |
| 100 | e.ln.Close() | 129 | e.ln.Close() |
| 101 | e.ln = nil | 130 | e.ln = nil |
| 102 | } | 131 | } |
| 132 | if e.pc != nil { | ||
| 133 | e.pc.Close() | ||
| 134 | e.pc = nil | ||
| 135 | } | ||
| 136 | e.closeSessions() | ||
| 103 | } | 137 | } |
| 104 | 138 | ||
| 105 | // specKey renders everything about a desired exposure that a bound listener | 139 | // specKey renders everything about a desired exposure that a bound socket |
| 106 | // depends on. A change to any of it is a close and a rebind rather than an | 140 | // depends on. A change to any of it is a close and a rebind rather than an |
| 107 | // edit — the listener's own address is part of what changed. | 141 | // edit — the socket's own address, or the protocol it speaks, is part of what |
| 142 | // changed. | ||
| 108 | func specKey(d *pb.ExposureDesired) string { | 143 | func specKey(d *pb.ExposureDesired) string { |
| 109 | return fmt.Sprintf("%s/%d/%d/%s", d.GetVmId(), d.GetGuestPort(), d.GetHostPort(), d.GetProtocol()) | 144 | return fmt.Sprintf("%s/%d/%d/%s", d.GetVmId(), d.GetGuestPort(), d.GetHostPort(), d.GetProtocol()) |
| 110 | } | 145 | } |
| 111 | 146 | ||
| 112 | // Converge drives the running listeners toward desired and reports what each | 147 | // Converge drives the running sockets toward desired and reports what each |
| 113 | // exposure is doing, in desired order. It is level-triggered: every call | 148 | // exposure is doing, in desired order. It is level-triggered: every call |
| 114 | // re-examines the whole set, so an exposure whose listener is gone — a failed | 149 | // re-examines the whole set, so an exposure whose socket is gone — a failed |
| 115 | // bind, or an accept loop that died — is retried here and heals the moment the | 150 | // bind, or a serving loop that died — is retried here and heals the moment the |
| 116 | // port frees. | 151 | // port frees. |
| 117 | // | 152 | // |
| 118 | // "active" means the listener is bound. Eitri owns the host half of the pipe; | 153 | // "active" means the socket is bound. Eitri owns the host half of the pipe; |
| 119 | // whether anything answers inside the guest is the guest's half, and this does | 154 | // whether anything answers inside the guest is the guest's half, and this does |
| 120 | // not pretend otherwise. | 155 | // not pretend otherwise. A bound exposure still carries a reason when something |
| 156 | // about serving it is going wrong — a descriptor shortage the accept loop is | ||
| 157 | // riding out — because a port that is bound and struggling is not the same | ||
| 158 | // thing as a port that is bound and fine. | ||
| 121 | func (m *Manager) Converge(desired []*pb.ExposureDesired) []*pb.ExposureActual { | 159 | func (m *Manager) Converge(desired []*pb.ExposureDesired) []*pb.ExposureActual { |
| 122 | m.mu.Lock() | 160 | m.mu.Lock() |
| 123 | defer m.mu.Unlock() | 161 | defer m.mu.Unlock() |
| @@ -127,7 +165,7 @@ func (m *Manager) Converge(desired []*pb.ExposureDesired) []*pb.ExposureActual { | |||
| 127 | want[d.GetId()] = d | 165 | want[d.GetId()] = d |
| 128 | } | 166 | } |
| 129 | for id, ex := range m.live { | 167 | for id, ex := range m.live { |
| 130 | if d, ok := want[id]; ok && ex.key == specKey(d) { | 168 | if _, ok := want[id]; ok { |
| 131 | continue | 169 | continue |
| 132 | } | 170 | } |
| 133 | ex.close() | 171 | ex.close() |
| @@ -137,26 +175,65 @@ func (m *Manager) Converge(desired []*pb.ExposureDesired) []*pb.ExposureActual { | |||
| 137 | out := make([]*pb.ExposureActual, 0, len(desired)) | 175 | out := make([]*pb.ExposureActual, 0, len(desired)) |
| 138 | for _, d := range desired { | 176 | for _, d := range desired { |
| 139 | ex, ok := m.live[d.GetId()] | 177 | ex, ok := m.live[d.GetId()] |
| 140 | if !ok { | 178 | switch { |
| 179 | case !ok: | ||
| 141 | ex = &exposure{key: specKey(d)} | 180 | ex = &exposure{key: specKey(d)} |
| 142 | m.live[d.GetId()] = ex | 181 | m.live[d.GetId()] = ex |
| 182 | case ex.key != specKey(d): | ||
| 183 | // A rebind in place, not a fresh entry: the id is what the fleet | ||
| 184 | // named and what callers are still draining through. | ||
| 185 | ex.close() | ||
| 186 | ex.key, ex.reason = specKey(d), "" | ||
| 143 | } | 187 | } |
| 144 | if ex.ln == nil { | 188 | if !ex.bound() { |
| 145 | ln, err := net.Listen("tcp", net.JoinHostPort("0.0.0.0", strconv.Itoa(int(d.GetHostPort())))) | 189 | if err := m.bind(d, ex); err != nil { |
| 146 | if err != nil { | ||
| 147 | ex.reason = err.Error() | 190 | ex.reason = err.Error() |
| 148 | out = append(out, &pb.ExposureActual{Id: d.GetId(), State: "failed", Reason: ex.reason}) | 191 | out = append(out, &pb.ExposureActual{Id: d.GetId(), State: "failed", Reason: ex.reason}) |
| 149 | continue | 192 | continue |
| 150 | } | 193 | } |
| 151 | ex.ln, ex.reason = ln, "" | ||
| 152 | go m.accept(d.GetId(), ex, ln, d.GetVmId(), d.GetGuestPort()) | ||
| 153 | } | 194 | } |
| 154 | out = append(out, &pb.ExposureActual{Id: d.GetId(), State: "active"}) | 195 | // Where the guest is can change under a live session — a VM re-imaged, |
| 196 | // re-addressed, or gone. A session pins the address it was created for, | ||
| 197 | // so this is where a pin that no longer matches is dropped: the next | ||
| 198 | // datagram from that client starts a session pointing at wherever the | ||
| 199 | // guest is now. Only UDP exposures ask, because only they hold anything | ||
| 200 | // pinned, and asking is a read of the VM's record on disk. | ||
| 201 | if ex.pc != nil { | ||
| 202 | ex.evictMovedSessions(m.addr(d.GetVmId())) | ||
| 203 | } | ||
| 204 | out = append(out, &pb.ExposureActual{Id: d.GetId(), State: "active", Reason: ex.reason}) | ||
| 155 | } | 205 | } |
| 156 | return out | 206 | return out |
| 157 | } | 207 | } |
| 158 | 208 | ||
| 159 | // StopAll closes every listener (agent shutdown, tests). | 209 | // bind opens the socket one desired exposure calls for and starts the loop that |
| 210 | // serves it. UDP is a bound packet socket; everything else is a TCP listener — | ||
| 211 | // a server that names no protocol at all means the one this proxy started with. | ||
| 212 | func (m *Manager) bind(d *pb.ExposureDesired, ex *exposure) error { | ||
| 213 | hostAddr := net.JoinHostPort("0.0.0.0", strconv.Itoa(int(d.GetHostPort()))) | ||
| 214 | if d.GetProtocol() == "udp" { | ||
| 215 | ua, err := net.ResolveUDPAddr("udp4", hostAddr) | ||
| 216 | if err != nil { | ||
| 217 | return err | ||
| 218 | } | ||
| 219 | pc, err := net.ListenUDP("udp4", ua) | ||
| 220 | if err != nil { | ||
| 221 | return err | ||
| 222 | } | ||
| 223 | ex.pc, ex.reason = pc, "" | ||
| 224 | go m.serve(d.GetId(), ex, pc, d.GetVmId(), d.GetGuestPort()) | ||
| 225 | return nil | ||
| 226 | } | ||
| 227 | ln, err := net.Listen("tcp", hostAddr) | ||
| 228 | if err != nil { | ||
| 229 | return err | ||
| 230 | } | ||
| 231 | ex.ln, ex.reason = ln, "" | ||
| 232 | go m.accept(d.GetId(), ex, ln, d.GetVmId(), d.GetGuestPort()) | ||
| 233 | return nil | ||
| 234 | } | ||
| 235 | |||
| 236 | // StopAll closes every socket (agent shutdown, tests). | ||
| 160 | func (m *Manager) StopAll() { | 237 | func (m *Manager) StopAll() { |
| 161 | m.mu.Lock() | 238 | m.mu.Lock() |
| 162 | defer m.mu.Unlock() | 239 | defer m.mu.Unlock() |
| @@ -181,8 +258,11 @@ func (m *Manager) StopAll() { | |||
| 181 | // the listener back would unbind a working port and ask the next converge to | 258 | // the listener back would unbind a working port and ask the next converge to |
| 182 | // find a descriptor for a fresh one — exactly what the process has none of — | 259 | // find a descriptor for a fresh one — exactly what the process has none of — |
| 183 | // so the loop pauses and keeps the port, and the backlog keeps callers waiting | 260 | // so the loop pauses and keeps the port, and the backlog keeps callers waiting |
| 184 | // rather than refusing them. Everything else is a broken listener and heals the | 261 | // rather than refusing them. It says so in the exposure's reason for as long as |
| 185 | // only way a broken listener can, by being rebound. | 262 | // the streak lasts: the port is bound, which is what "active" means, and an |
| 263 | // operator looking at a published port nothing is getting through deserves the | ||
| 264 | // one sentence that explains it. Everything else is a broken listener and heals | ||
| 265 | // the only way a broken listener can, by being rebound. | ||
| 186 | func (m *Manager) accept(id string, ex *exposure, ln net.Listener, vmID string, guestPort uint32) { | 266 | func (m *Manager) accept(id string, ex *exposure, ln net.Listener, vmID string, guestPort uint32) { |
| 187 | var pause time.Duration | 267 | var pause time.Duration |
| 188 | capped := false | 268 | capped := false |
| @@ -193,6 +273,7 @@ func (m *Manager) accept(id string, ex *exposure, ln net.Listener, vmID string, | |||
| 193 | if pause == 0 { | 273 | if pause == 0 { |
| 194 | pause = acceptRetryFloor | 274 | pause = acceptRetryFloor |
| 195 | slog.Warn("exposure accept out of descriptors, retrying", "exposure", id, "err", err) | 275 | slog.Warn("exposure accept out of descriptors, retrying", "exposure", id, "err", err) |
| 276 | m.setReason(id, ln, "accept: out of descriptors, retrying: "+err.Error()) | ||
| 196 | } else { | 277 | } else { |
| 197 | pause = min(pause*2, acceptRetryCeiling) | 278 | pause = min(pause*2, acceptRetryCeiling) |
| 198 | } | 279 | } |
| @@ -211,7 +292,11 @@ func (m *Manager) accept(id string, ex *exposure, ln net.Listener, vmID string, | |||
| 211 | m.mu.Unlock() | 292 | m.mu.Unlock() |
| 212 | return | 293 | return |
| 213 | } | 294 | } |
| 214 | pause = 0 | 295 | if pause != 0 { |
| 296 | pause = 0 | ||
| 297 | slog.Info("exposure accept has descriptors again", "exposure", id) | ||
| 298 | m.setReason(id, ln, "") | ||
| 299 | } | ||
| 215 | 300 | ||
| 216 | // Increment first and give it back if the cap says no: counting only | 301 | // Increment first and give it back if the cap says no: counting only |
| 217 | // after the decision would let a burst of simultaneous accepts each read | 302 | // after the decision would let a burst of simultaneous accepts each read |
| @@ -233,6 +318,20 @@ func (m *Manager) accept(id string, ex *exposure, ln net.Listener, vmID string, | |||
| 233 | } | 318 | } |
| 234 | } | 319 | } |
| 235 | 320 | ||
| 321 | // setReason records what an accept loop wants the next report to say about a | ||
| 322 | // still-bound exposure. It writes under the manager lock, which is what makes | ||
| 323 | // the reason a converge reads a whole one rather than a torn one, and only when | ||
| 324 | // the entry still holds THIS loop's listener — the same identity guard the | ||
| 325 | // loop's exit takes, because a loop whose exposure was rebound underneath it | ||
| 326 | // has nothing to say about the listener that replaced its own. | ||
| 327 | func (m *Manager) setReason(id string, ln net.Listener, reason string) { | ||
| 328 | m.mu.Lock() | ||
| 329 | defer m.mu.Unlock() | ||
| 330 | if cur, ok := m.live[id]; ok && cur.ln == ln { | ||
| 331 | cur.reason = reason | ||
| 332 | } | ||
| 333 | } | ||
| 334 | |||
| 236 | // outOfDescriptors reports whether an accept failed because the process (EMFILE) | 335 | // outOfDescriptors reports whether an accept failed because the process (EMFILE) |
| 237 | // or the machine (ENFILE) has no descriptor to give it. The runtime already | 336 | // or the machine (ENFILE) has no descriptor to give it. The runtime already |
| 238 | // retries the other transient accept errors — an interrupted call, a caller that | 337 | // retries the other transient accept errors — an interrupted call, a caller that |
internal/agent/exposeproxy/exposeproxy_test.go
| Old | New | ||
|---|---|---|---|
| @@ -4,6 +4,7 @@ import ( | |||
| 4 | "io" | 4 | "io" |
| 5 | "net" | 5 | "net" |
| 6 | "strconv" | 6 | "strconv" |
| 7 | "strings" | ||
| 7 | "syscall" | 8 | "syscall" |
| 8 | "testing" | 9 | "testing" |
| 9 | "time" | 10 | "time" |
| @@ -82,6 +83,10 @@ func desired(id, vmID string, guestPort, hostPort uint32) *pb.ExposureDesired { | |||
| 82 | return &pb.ExposureDesired{Id: id, VmId: vmID, GuestPort: guestPort, HostPort: hostPort, Protocol: "tcp"} | 83 | return &pb.ExposureDesired{Id: id, VmId: vmID, GuestPort: guestPort, HostPort: hostPort, Protocol: "tcp"} |
| 83 | } | 84 | } |
| 84 | 85 | ||
| 86 | func desiredUDP(id, vmID string, guestPort, hostPort uint32) *pb.ExposureDesired { | ||
| 87 | return &pb.ExposureDesired{Id: id, VmId: vmID, GuestPort: guestPort, HostPort: hostPort, Protocol: "udp"} | ||
| 88 | } | ||
| 89 | |||
| 85 | // speak dials addr, sends msg, and returns what came back. | 90 | // speak dials addr, sends msg, and returns what came back. |
| 86 | func speak(t *testing.T, addr, msg string) string { | 91 | func speak(t *testing.T, addr, msg string) string { |
| 87 | t.Helper() | 92 | t.Helper() |
| @@ -422,6 +427,92 @@ func TestTheCapReleasesWhenAConnectionEnds(t *testing.T) { | |||
| 422 | assert.Equal(t, "after", exchange(t, hold(t, addr), "after")) | 427 | assert.Equal(t, "after", exchange(t, hold(t, addr), "after")) |
| 423 | } | 428 | } |
| 424 | 429 | ||
| 430 | func TestTheCapCountsConnectionsDrainingThroughARebind(t *testing.T) { | ||
| 431 | first, second := newHoldingGuest(t), newHoldingGuest(t) | ||
| 432 | m := newTestManager(t, map[string]string{"vm1": first.addr}) | ||
| 433 | m.maxConns = 1 | ||
| 434 | m.Converge([]*pb.ExposureDesired{desired("e1", "vm1", first.port, 0)}) | ||
| 435 | |||
| 436 | held := hold(t, boundPort(t, m, "e1")) | ||
| 437 | |||
| 438 | // The spec changes and the exposure rebinds. The connection under the old | ||
| 439 | // listener is still open and still spending its two descriptors, so it is | ||
| 440 | // still what the cap is counting — an exposure that forgot it on the rebind | ||
| 441 | // would let this port hold twice what it is allowed. | ||
| 442 | m.Converge([]*pb.ExposureDesired{desired("e1", "vm1", second.port, 0)}) | ||
| 443 | waitConns(t, m, "e1", 1) | ||
| 444 | |||
| 445 | over, err := net.Dial("tcp", boundPort(t, m, "e1")) | ||
| 446 | require.NoError(t, err) | ||
| 447 | defer over.Close() | ||
| 448 | require.NoError(t, over.SetDeadline(time.Now().Add(5*time.Second))) | ||
| 449 | out, err := io.ReadAll(over) | ||
| 450 | require.NoError(t, err) | ||
| 451 | assert.Empty(t, out, "the rebound listener is at its cap, because the draining connection still counts") | ||
| 452 | |||
| 453 | // The old connection drains rather than being cut, and the slot it holds is | ||
| 454 | // the one the next caller gets when it ends. | ||
| 455 | assert.Equal(t, "drain", exchange(t, held, "drain")) | ||
| 456 | require.NoError(t, held.Close()) | ||
| 457 | waitConns(t, m, "e1", 0) | ||
| 458 | assert.Equal(t, "after", exchange(t, hold(t, boundPort(t, m, "e1")), "after")) | ||
| 459 | } | ||
| 460 | |||
| 461 | // waitReason waits for an exposure's reported reason to satisfy ok, reading it | ||
| 462 | // under the lock the accept loop writes it under. | ||
| 463 | func waitReason(t *testing.T, m *Manager, id string, ok func(string) bool, what string) string { | ||
| 464 | t.Helper() | ||
| 465 | var got string | ||
| 466 | for i := 0; i < 200; i++ { | ||
| 467 | m.mu.Lock() | ||
| 468 | ex, live := m.live[id] | ||
| 469 | if live { | ||
| 470 | got = ex.reason | ||
| 471 | } | ||
| 472 | m.mu.Unlock() | ||
| 473 | if live && ok(got) { | ||
| 474 | return got | ||
| 475 | } | ||
| 476 | time.Sleep(10 * time.Millisecond) | ||
| 477 | } | ||
| 478 | t.Fatalf("exposure %q reason %q: %s", id, got, what) | ||
| 479 | return "" | ||
| 480 | } | ||
| 481 | |||
| 482 | func TestADescriptorShortageIsReportedWhileThePortStaysBound(t *testing.T) { | ||
| 483 | g := newFakeGuest(t) | ||
| 484 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 485 | |||
| 486 | d := desired("e1", "vm1", g.port, 0) | ||
| 487 | ln := &scriptedListener{next: make(chan acceptResult), addr: g.ln.Addr()} | ||
| 488 | ex := &exposure{key: specKey(d), ln: ln} | ||
| 489 | m.mu.Lock() | ||
| 490 | m.live["e1"] = ex | ||
| 491 | m.mu.Unlock() | ||
| 492 | go m.accept("e1", ex, ln, "vm1", g.port) | ||
| 493 | |||
| 494 | ln.next <- acceptResult{err: syscall.EMFILE} | ||
| 495 | waitReason(t, m, "e1", func(r string) bool { return strings.Contains(r, "out of descriptors") }, | ||
| 496 | "a port nothing can get through must say why") | ||
| 497 | |||
| 498 | // The port is still bound, so it is still active — with the streak beside | ||
| 499 | // it, which is the whole point: "active" alone would read as healthy. | ||
| 500 | got := m.Converge([]*pb.ExposureDesired{d}) | ||
| 501 | require.Len(t, got, 1) | ||
| 502 | assert.Equal(t, "active", got[0].GetState()) | ||
| 503 | assert.Contains(t, got[0].GetReason(), "out of descriptors") | ||
| 504 | |||
| 505 | // Descriptors come back. The next accept clears the streak, and the report | ||
| 506 | // stops saying something is wrong. | ||
| 507 | client, accepted := net.Pipe() | ||
| 508 | defer client.Close() | ||
| 509 | ln.next <- acceptResult{conn: accepted} | ||
| 510 | waitReason(t, m, "e1", func(r string) bool { return r == "" }, "a recovered port must stop reporting a shortage") | ||
| 511 | got = m.Converge([]*pb.ExposureDesired{d}) | ||
| 512 | assert.Equal(t, "active", got[0].GetState()) | ||
| 513 | assert.Empty(t, got[0].GetReason()) | ||
| 514 | } | ||
| 515 | |||
| 425 | // scriptedListener hands an accept loop exactly the sequence a test writes to | 516 | // scriptedListener hands an accept loop exactly the sequence a test writes to |
| 426 | // it — the errors a real listener only produces under a load a test cannot | 517 | // it — the errors a real listener only produces under a load a test cannot |
| 427 | // stage. | 518 | // stage. |
internal/agent/exposeproxy/udp.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,283 @@ | |||
| 1 | package exposeproxy | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "errors" | ||
| 5 | "log/slog" | ||
| 6 | "net" | ||
| 7 | "os" | ||
| 8 | "strconv" | ||
| 9 | "sync" | ||
| 10 | "sync/atomic" | ||
| 11 | "time" | ||
| 12 | ) | ||
| 13 | |||
| 14 | // A UDP exposure is one bound socket on the host and one session per client | ||
| 15 | // address behind it. The session is the whole design: UDP has no connection to | ||
| 16 | // follow, so the proxy invents the smallest thing that lets a reply find its | ||
| 17 | // way home — a connected socket toward the guest, whose local port is what the | ||
| 18 | // guest answers to, held for as long as the conversation looks live. | ||
| 19 | // | ||
| 20 | // The guest's address is resolved ONCE, when the session is created, and pinned | ||
| 21 | // there. A datagram that arrives before this host knows where the guest is is | ||
| 22 | // dropped and starts nothing: a session pinned to nowhere would have to | ||
| 23 | // re-resolve on some later packet, and a proxy that re-resolves mid-conversation | ||
| 24 | // is one that can quietly start sending a guest's traffic to whatever took its | ||
| 25 | // address over. Converge is where a stale pin is dropped instead. | ||
| 26 | const ( | ||
| 27 | // maxSessionsPerExposure is how many client conversations one published UDP | ||
| 28 | // port may hold at once. Each is one descriptor — the socket toward the | ||
| 29 | // guest; the socket callers reach is shared by all of them — so the sizing | ||
| 30 | // rationale is the TCP cap's, one descriptor per session instead of two, | ||
| 31 | // against the same 65536 the shipped unit grants the agent. A table at its | ||
| 32 | // cap refuses new clients and keeps the ones it has: eviction would trade a | ||
| 33 | // conversation that is working for one that might not be, and a UDP client | ||
| 34 | // has no close to notice. | ||
| 35 | maxSessionsPerExposure = 256 | ||
| 36 | |||
| 37 | // udpUnrepliedIdle and udpRepliedIdle are how long a session outlives its | ||
| 38 | // last datagram. A session the guest has never answered is a guess — a | ||
| 39 | // scanner's single packet, or a service that was not listening — and thirty | ||
| 40 | // seconds is long enough for a slow first reply and short enough that a | ||
| 41 | // sweep of the port space does not fill the table. Once traffic has flowed | ||
| 42 | // back the session is a real conversation, and two minutes carries the | ||
| 43 | // quiet stretches those have (a DNS client between queries, a game tick | ||
| 44 | // paused, an NTP poll interval). | ||
| 45 | udpUnrepliedIdle = 30 * time.Second | ||
| 46 | udpRepliedIdle = 120 * time.Second | ||
| 47 | |||
| 48 | // udpDatagramMax is the read buffer on both halves — larger than the | ||
| 49 | // largest datagram either side can send, so what goes in one end comes out | ||
| 50 | // the other whole. One datagram in is one datagram out: nothing here | ||
| 51 | // batches, coalesces, or reassembles. | ||
| 52 | udpDatagramMax = 64 * 1024 | ||
| 53 | ) | ||
| 54 | |||
| 55 | // udpSession is one client address's conversation with one guest port: the | ||
| 56 | // connected socket the guest sees, the client to send its answers back to, and | ||
| 57 | // the two facts that decide when the session ends. | ||
| 58 | type udpSession struct { | ||
| 59 | // key is the client address as the session table indexes it. | ||
| 60 | key string | ||
| 61 | client *net.UDPAddr | ||
| 62 | |||
| 63 | // guest is CONNECTED to the pinned address, which is what makes a reply | ||
| 64 | // readable here rather than on the shared socket — and what surfaces an | ||
| 65 | // ICMP port-unreachable as a read error instead of silence. | ||
| 66 | guest *net.UDPConn | ||
| 67 | |||
| 68 | // pinned is the guest address this session was created for. Converge | ||
| 69 | // compares it against where the VM is now. | ||
| 70 | pinned string | ||
| 71 | |||
| 72 | // unrepliedIdle and repliedIdle are the manager's two windows, copied here | ||
| 73 | // so a session decides its own expiry without reaching back for them. | ||
| 74 | unrepliedIdle, repliedIdle time.Duration | ||
| 75 | |||
| 76 | // last is the unix-nano time of the last datagram in either direction, and | ||
| 77 | // replied records that the guest has answered at least once. Both are | ||
| 78 | // touched by the packet loop and the relay goroutine, so both are atomic. | ||
| 79 | last atomic.Int64 | ||
| 80 | replied atomic.Bool | ||
| 81 | |||
| 82 | closeOnce sync.Once | ||
| 83 | } | ||
| 84 | |||
| 85 | func newUDPSession(client *net.UDPAddr, guest *net.UDPConn, pinned string, unreplied, replied time.Duration) *udpSession { | ||
| 86 | s := &udpSession{ | ||
| 87 | key: client.String(), client: client, guest: guest, pinned: pinned, | ||
| 88 | unrepliedIdle: unreplied, repliedIdle: replied, | ||
| 89 | } | ||
| 90 | s.touch() | ||
| 91 | return s | ||
| 92 | } | ||
| 93 | |||
| 94 | // touch records that a datagram just moved. | ||
| 95 | func (s *udpSession) touch() { s.last.Store(time.Now().UnixNano()) } | ||
| 96 | |||
| 97 | // expiry is when the session ends if nothing else moves: its last datagram plus | ||
| 98 | // the window its promotion earns it. | ||
| 99 | func (s *udpSession) expiry() time.Time { | ||
| 100 | idle := s.unrepliedIdle | ||
| 101 | if s.replied.Load() { | ||
| 102 | idle = s.repliedIdle | ||
| 103 | } | ||
| 104 | return time.Unix(0, s.last.Load()).Add(idle) | ||
| 105 | } | ||
| 106 | |||
| 107 | // close releases the session's descriptor, which is also what ends its relay. | ||
| 108 | func (s *udpSession) close() { | ||
| 109 | s.closeOnce.Do(func() { s.guest.Close() }) | ||
| 110 | } | ||
| 111 | |||
| 112 | // session returns the session serving client, or nil when there is none. | ||
| 113 | func (e *exposure) session(key string) *udpSession { | ||
| 114 | e.smu.Lock() | ||
| 115 | defer e.smu.Unlock() | ||
| 116 | return e.sessions[key] | ||
| 117 | } | ||
| 118 | |||
| 119 | // addSession puts s in the table unless it is full, in which case the caller | ||
| 120 | // closes s and the datagram that would have started it is dropped. | ||
| 121 | func (e *exposure) addSession(s *udpSession, max int) bool { | ||
| 122 | e.smu.Lock() | ||
| 123 | defer e.smu.Unlock() | ||
| 124 | if len(e.sessions) >= max { | ||
| 125 | return false | ||
| 126 | } | ||
| 127 | if e.sessions == nil { | ||
| 128 | e.sessions = map[string]*udpSession{} | ||
| 129 | } | ||
| 130 | e.sessions[s.key] = s | ||
| 131 | return true | ||
| 132 | } | ||
| 133 | |||
| 134 | // dropSession removes s and closes it. Idempotent, and identity-checked: a | ||
| 135 | // session that has already been replaced under its key is not the one to | ||
| 136 | // remove. | ||
| 137 | func (e *exposure) dropSession(s *udpSession) { | ||
| 138 | e.smu.Lock() | ||
| 139 | if cur, ok := e.sessions[s.key]; ok && cur == s { | ||
| 140 | delete(e.sessions, s.key) | ||
| 141 | } | ||
| 142 | e.smu.Unlock() | ||
| 143 | s.close() | ||
| 144 | } | ||
| 145 | |||
| 146 | // closeSessions empties the table, ending every relay with it. | ||
| 147 | func (e *exposure) closeSessions() { | ||
| 148 | e.smu.Lock() | ||
| 149 | held := e.sessions | ||
| 150 | e.sessions = nil | ||
| 151 | e.smu.Unlock() | ||
| 152 | for _, s := range held { | ||
| 153 | s.close() | ||
| 154 | } | ||
| 155 | } | ||
| 156 | |||
| 157 | // evictMovedSessions drops every session whose pinned guest address is not | ||
| 158 | // where the VM is now — including when the host no longer knows where that is, | ||
| 159 | // which is not an address to keep sending to either. | ||
| 160 | func (e *exposure) evictMovedSessions(guestIP string) { | ||
| 161 | e.smu.Lock() | ||
| 162 | var moved []*udpSession | ||
| 163 | for key, s := range e.sessions { | ||
| 164 | if s.pinned != guestIP { | ||
| 165 | delete(e.sessions, key) | ||
| 166 | moved = append(moved, s) | ||
| 167 | } | ||
| 168 | } | ||
| 169 | e.smu.Unlock() | ||
| 170 | for _, s := range moved { | ||
| 171 | s.close() | ||
| 172 | } | ||
| 173 | } | ||
| 174 | |||
| 175 | // serve reads one exposure's published UDP socket until it ends: a converge | ||
| 176 | // dropped it, StopAll closed it, or the socket itself broke. Like the accept | ||
| 177 | // loop, a broken socket is handed back so the next converge rebinds it — a | ||
| 178 | // bound port nothing is reading is worse than no port at all, because it | ||
| 179 | // swallows datagrams instead of refusing them. | ||
| 180 | // | ||
| 181 | // Every datagram either continues a session or starts one. Nothing here waits: | ||
| 182 | // a client with nowhere to send, a guest that will not take a connected socket, | ||
| 183 | // and a table at its cap all drop the datagram, because a datagram held is a | ||
| 184 | // datagram late, and UDP callers retry. | ||
| 185 | func (m *Manager) serve(id string, ex *exposure, pc *net.UDPConn, vmID string, guestPort uint32) { | ||
| 186 | buf := make([]byte, udpDatagramMax) | ||
| 187 | full := false | ||
| 188 | for { | ||
| 189 | n, client, err := pc.ReadFromUDP(buf) | ||
| 190 | if err != nil { | ||
| 191 | m.mu.Lock() | ||
| 192 | if cur, ok := m.live[id]; ok && cur.pc == pc { | ||
| 193 | cur.close() | ||
| 194 | cur.reason = "read: " + err.Error() | ||
| 195 | // The reason travels in the next report, which nobody watching | ||
| 196 | // the host sees; a broken socket belongs in the agent's log too. | ||
| 197 | slog.Warn("exposure packet loop died", "exposure", id, "err", err) | ||
| 198 | } | ||
| 199 | m.mu.Unlock() | ||
| 200 | return | ||
| 201 | } | ||
| 202 | |||
| 203 | s := ex.session(client.String()) | ||
| 204 | if s == nil { | ||
| 205 | s = m.openSession(ex, pc, client, vmID, guestPort, &full) | ||
| 206 | if s == nil { | ||
| 207 | continue | ||
| 208 | } | ||
| 209 | } | ||
| 210 | if _, err := s.guest.Write(buf[:n]); err != nil { | ||
| 211 | ex.dropSession(s) | ||
| 212 | continue | ||
| 213 | } | ||
| 214 | s.touch() | ||
| 215 | } | ||
| 216 | } | ||
| 217 | |||
| 218 | // openSession resolves where the guest is right now, pins a connected socket | ||
| 219 | // there, and files it under the client's address. It returns nil when the | ||
| 220 | // datagram that asked for it is to be dropped — no guest address, a socket the | ||
| 221 | // OS would not give, or a table with no room — with full carrying the | ||
| 222 | // at-capacity streak so a saturated port logs once rather than per packet. | ||
| 223 | func (m *Manager) openSession(ex *exposure, pc *net.UDPConn, client *net.UDPAddr, vmID string, guestPort uint32, full *bool) *udpSession { | ||
| 224 | ip := m.addr(vmID) | ||
| 225 | if ip == "" { | ||
| 226 | return nil | ||
| 227 | } | ||
| 228 | guestAddr, err := net.ResolveUDPAddr("udp4", net.JoinHostPort(ip, strconv.Itoa(int(guestPort)))) | ||
| 229 | if err != nil { | ||
| 230 | return nil | ||
| 231 | } | ||
| 232 | guest, err := net.DialUDP("udp4", nil, guestAddr) | ||
| 233 | if err != nil { | ||
| 234 | return nil | ||
| 235 | } | ||
| 236 | s := newUDPSession(client, guest, ip, m.unrepliedIdle, m.repliedIdle) | ||
| 237 | if !ex.addSession(s, m.maxSessions) { | ||
| 238 | s.close() | ||
| 239 | if !*full { | ||
| 240 | *full = true | ||
| 241 | slog.Warn("exposure at its UDP session cap, dropping", "cap", m.maxSessions, "client", client.String()) | ||
| 242 | } | ||
| 243 | return nil | ||
| 244 | } | ||
| 245 | *full = false | ||
| 246 | go m.relay(ex, pc, s) | ||
| 247 | return s | ||
| 248 | } | ||
| 249 | |||
| 250 | // relay carries one session's replies back to its client until the session | ||
| 251 | // ends. It ends on the guest socket going quiet for longer than the session's | ||
| 252 | // window, on any error reading it — which is where an ICMP port-unreachable | ||
| 253 | // from a guest with nothing listening arrives — and on a failure to answer the | ||
| 254 | // client. | ||
| 255 | // | ||
| 256 | // The read deadline IS the idle timer: there is no sweeper, because the only | ||
| 257 | // thing that needs to notice an expired session is the goroutine already | ||
| 258 | // waiting on it. A deadline that fires early because the forward direction | ||
| 259 | // moved the session on is simply re-armed against the newer expiry. | ||
| 260 | func (m *Manager) relay(ex *exposure, pc *net.UDPConn, s *udpSession) { | ||
| 261 | defer ex.dropSession(s) | ||
| 262 | buf := make([]byte, udpDatagramMax) | ||
| 263 | for { | ||
| 264 | if err := s.guest.SetReadDeadline(s.expiry()); err != nil { | ||
| 265 | return | ||
| 266 | } | ||
| 267 | n, err := s.guest.Read(buf) | ||
| 268 | if err != nil { | ||
| 269 | if errors.Is(err, os.ErrDeadlineExceeded) && time.Now().Before(s.expiry()) { | ||
| 270 | continue | ||
| 271 | } | ||
| 272 | return | ||
| 273 | } | ||
| 274 | if _, err := pc.WriteToUDP(buf[:n], s.client); err != nil { | ||
| 275 | return | ||
| 276 | } | ||
| 277 | s.touch() | ||
| 278 | // Promotion, and it only happens here: traffic that has come BACK from | ||
| 279 | // the guest is what tells a real conversation apart from a stray packet | ||
| 280 | // at a port. | ||
| 281 | s.replied.Store(true) | ||
| 282 | } | ||
| 283 | } | ||
internal/agent/exposeproxy/udp_test.go
| Old | New | ||
|---|---|---|---|
| @@ -0,0 +1,386 @@ | |||
| 1 | package exposeproxy | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "bytes" | ||
| 5 | "net" | ||
| 6 | "testing" | ||
| 7 | "time" | ||
| 8 | |||
| 9 | "github.com/a73x/eitri/internal/pb" | ||
| 10 | "github.com/stretchr/testify/assert" | ||
| 11 | "github.com/stretchr/testify/require" | ||
| 12 | ) | ||
| 13 | |||
| 14 | // fakeUDPGuest is a bound UDP socket standing in for a service inside a guest. | ||
| 15 | // It echoes every datagram back to whoever sent it, prefixed, so a test can | ||
| 16 | // prove a datagram went both ways through the proxy and came back whole. | ||
| 17 | type fakeUDPGuest struct { | ||
| 18 | pc *net.UDPConn | ||
| 19 | addr string | ||
| 20 | port uint32 | ||
| 21 | } | ||
| 22 | |||
| 23 | func newFakeUDPGuest(t *testing.T) *fakeUDPGuest { | ||
| 24 | t.Helper() | ||
| 25 | pc, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1)}) | ||
| 26 | require.NoError(t, err) | ||
| 27 | t.Cleanup(func() { pc.Close() }) | ||
| 28 | g := &fakeUDPGuest{pc: pc, addr: "127.0.0.1", port: uint32(pc.LocalAddr().(*net.UDPAddr).Port)} | ||
| 29 | go func() { | ||
| 30 | buf := make([]byte, 64*1024) | ||
| 31 | for { | ||
| 32 | n, from, err := pc.ReadFromUDP(buf) | ||
| 33 | if err != nil { | ||
| 34 | return | ||
| 35 | } | ||
| 36 | pc.WriteToUDP(append([]byte("echo:"), buf[:n]...), from) //nolint:errcheck | ||
| 37 | } | ||
| 38 | }() | ||
| 39 | return g | ||
| 40 | } | ||
| 41 | |||
| 42 | // boundUDPPort reads the address the manager's packet socket for id bound. | ||
| 43 | func boundUDPPort(t *testing.T, m *Manager, id string) string { | ||
| 44 | t.Helper() | ||
| 45 | m.mu.Lock() | ||
| 46 | defer m.mu.Unlock() | ||
| 47 | ex, ok := m.live[id] | ||
| 48 | require.True(t, ok, "no exposure %q", id) | ||
| 49 | require.NotNil(t, ex.pc, "exposure %q has no packet socket", id) | ||
| 50 | return ex.pc.LocalAddr().String() | ||
| 51 | } | ||
| 52 | |||
| 53 | // udpClient is one caller of a published UDP port, held open across datagrams | ||
| 54 | // so the proxy sees the same source address each time — which is what makes it | ||
| 55 | // one session rather than several. | ||
| 56 | func udpClient(t *testing.T, addr string) *net.UDPConn { | ||
| 57 | t.Helper() | ||
| 58 | ua, err := net.ResolveUDPAddr("udp4", addr) | ||
| 59 | require.NoError(t, err) | ||
| 60 | c, err := net.DialUDP("udp4", nil, ua) | ||
| 61 | require.NoError(t, err) | ||
| 62 | t.Cleanup(func() { c.Close() }) | ||
| 63 | require.NoError(t, c.SetDeadline(time.Now().Add(5*time.Second))) | ||
| 64 | return c | ||
| 65 | } | ||
| 66 | |||
| 67 | // say sends one datagram and returns the one that comes back. | ||
| 68 | func say(t *testing.T, c *net.UDPConn, msg string) string { | ||
| 69 | t.Helper() | ||
| 70 | _, err := c.Write([]byte(msg)) | ||
| 71 | require.NoError(t, err) | ||
| 72 | buf := make([]byte, 64*1024) | ||
| 73 | n, err := c.Read(buf) | ||
| 74 | require.NoError(t, err) | ||
| 75 | return string(buf[:n]) | ||
| 76 | } | ||
| 77 | |||
| 78 | // sessions is how many the exposure is holding right now. | ||
| 79 | func sessions(t *testing.T, m *Manager, id string) int { | ||
| 80 | t.Helper() | ||
| 81 | m.mu.Lock() | ||
| 82 | ex, ok := m.live[id] | ||
| 83 | m.mu.Unlock() | ||
| 84 | require.True(t, ok, "no exposure %q", id) | ||
| 85 | ex.smu.Lock() | ||
| 86 | defer ex.smu.Unlock() | ||
| 87 | return len(ex.sessions) | ||
| 88 | } | ||
| 89 | |||
| 90 | // waitSessions waits for the session count to reach want. A session is filed | ||
| 91 | // synchronously with the datagram that starts it, but it is torn down by a | ||
| 92 | // goroutine finishing on its own time. | ||
| 93 | func waitSessions(t *testing.T, m *Manager, id string, want int) { | ||
| 94 | t.Helper() | ||
| 95 | var got int | ||
| 96 | for i := 0; i < 200; i++ { | ||
| 97 | if got = sessions(t, m, id); got == want { | ||
| 98 | return | ||
| 99 | } | ||
| 100 | time.Sleep(10 * time.Millisecond) | ||
| 101 | } | ||
| 102 | t.Fatalf("exposure %q holds %d sessions, want %d", id, got, want) | ||
| 103 | } | ||
| 104 | |||
| 105 | // oneSession returns the exposure's single session, for tests that reach past | ||
| 106 | // the wire to what the proxy is holding. | ||
| 107 | func oneSession(t *testing.T, m *Manager, id string) *udpSession { | ||
| 108 | t.Helper() | ||
| 109 | m.mu.Lock() | ||
| 110 | ex, ok := m.live[id] | ||
| 111 | m.mu.Unlock() | ||
| 112 | require.True(t, ok, "no exposure %q", id) | ||
| 113 | ex.smu.Lock() | ||
| 114 | defer ex.smu.Unlock() | ||
| 115 | require.Len(t, ex.sessions, 1, "want exactly one session") | ||
| 116 | for _, s := range ex.sessions { | ||
| 117 | return s | ||
| 118 | } | ||
| 119 | return nil | ||
| 120 | } | ||
| 121 | |||
| 122 | func TestUDPConvergeBindsAndForwardsBothWays(t *testing.T) { | ||
| 123 | g := newFakeUDPGuest(t) | ||
| 124 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 125 | |||
| 126 | got := m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, 0)}) | ||
| 127 | require.Len(t, got, 1) | ||
| 128 | assert.Equal(t, "active", got[0].GetState(), "active means the packet socket is bound") | ||
| 129 | |||
| 130 | c := udpClient(t, boundUDPPort(t, m, "e1")) | ||
| 131 | assert.Equal(t, "echo:hello", say(t, c, "hello")) | ||
| 132 | assert.Equal(t, 1, sessions(t, m, "e1"), "one client address is one session") | ||
| 133 | } | ||
| 134 | |||
| 135 | func TestUDPKeepsDatagramBoundaries(t *testing.T) { | ||
| 136 | g := newFakeUDPGuest(t) | ||
| 137 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 138 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, 0)}) | ||
| 139 | c := udpClient(t, boundUDPPort(t, m, "e1")) | ||
| 140 | |||
| 141 | // Two datagrams in, two out, in order and whole — nothing here coalesces a | ||
| 142 | // stream out of them. | ||
| 143 | assert.Equal(t, "echo:one", say(t, c, "one")) | ||
| 144 | assert.Equal(t, "echo:two", say(t, c, "two")) | ||
| 145 | |||
| 146 | // And a big one survives intact: the read buffers are sized past the | ||
| 147 | // largest datagram either side can send. | ||
| 148 | big := string(bytes.Repeat([]byte("x"), 8000)) | ||
| 149 | assert.Equal(t, "echo:"+big, say(t, c, big)) | ||
| 150 | } | ||
| 151 | |||
| 152 | func TestUDPSessionIsPerClientAddress(t *testing.T) { | ||
| 153 | g := newFakeUDPGuest(t) | ||
| 154 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 155 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, 0)}) | ||
| 156 | addr := boundUDPPort(t, m, "e1") | ||
| 157 | |||
| 158 | first, second := udpClient(t, addr), udpClient(t, addr) | ||
| 159 | assert.Equal(t, "echo:a", say(t, first, "a")) | ||
| 160 | assert.Equal(t, "echo:b", say(t, second, "b")) | ||
| 161 | assert.Equal(t, 2, sessions(t, m, "e1"), "two callers are two conversations") | ||
| 162 | |||
| 163 | // The same caller again is the same session, not a third. | ||
| 164 | assert.Equal(t, "echo:again", say(t, first, "again")) | ||
| 165 | assert.Equal(t, 2, sessions(t, m, "e1")) | ||
| 166 | } | ||
| 167 | |||
| 168 | func TestUDPSessionIsPromotedOnlyByAGuestReply(t *testing.T) { | ||
| 169 | // A guest that takes datagrams and never answers: the session stays on the | ||
| 170 | // short window, however much the client says. | ||
| 171 | silent, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1)}) | ||
| 172 | require.NoError(t, err) | ||
| 173 | t.Cleanup(func() { silent.Close() }) | ||
| 174 | quietPort := uint32(silent.LocalAddr().(*net.UDPAddr).Port) | ||
| 175 | |||
| 176 | m := newTestManager(t, map[string]string{"vm1": "127.0.0.1"}) | ||
| 177 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", quietPort, 0)}) | ||
| 178 | c := udpClient(t, boundUDPPort(t, m, "e1")) | ||
| 179 | _, err = c.Write([]byte("anyone there")) | ||
| 180 | require.NoError(t, err) | ||
| 181 | waitSessions(t, m, "e1", 1) | ||
| 182 | |||
| 183 | s := oneSession(t, m, "e1") | ||
| 184 | assert.False(t, s.replied.Load(), "nothing has come back; the session is still a guess") | ||
| 185 | assert.WithinDuration(t, time.Now().Add(m.unrepliedIdle), s.expiry(), time.Second) | ||
| 186 | |||
| 187 | // Now a guest that answers. The reply is the promotion. | ||
| 188 | g := newFakeUDPGuest(t) | ||
| 189 | m.Converge([]*pb.ExposureDesired{desiredUDP("e2", "vm1", g.port, 0)}) | ||
| 190 | talking := udpClient(t, boundUDPPort(t, m, "e2")) | ||
| 191 | require.Equal(t, "echo:hi", say(t, talking, "hi")) | ||
| 192 | |||
| 193 | promoted := oneSession(t, m, "e2") | ||
| 194 | assert.True(t, promoted.replied.Load()) | ||
| 195 | assert.WithinDuration(t, time.Now().Add(m.repliedIdle), promoted.expiry(), time.Second) | ||
| 196 | } | ||
| 197 | |||
| 198 | func TestUDPSessionExpiresOnBothWindows(t *testing.T) { | ||
| 199 | g := newFakeUDPGuest(t) | ||
| 200 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 201 | // Windows a test can outlast. Everything about them is what the shipped | ||
| 202 | // constants do, at a scale that fits in a test. | ||
| 203 | m.unrepliedIdle, m.repliedIdle = 250*time.Millisecond, 10*time.Second | ||
| 204 | |||
| 205 | // Never answered: gone at the short window. | ||
| 206 | silent, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4(127, 0, 0, 1)}) | ||
| 207 | require.NoError(t, err) | ||
| 208 | t.Cleanup(func() { silent.Close() }) | ||
| 209 | m.Converge([]*pb.ExposureDesired{ | ||
| 210 | desiredUDP("quiet", "vm1", uint32(silent.LocalAddr().(*net.UDPAddr).Port), 0), | ||
| 211 | desiredUDP("live", "vm1", g.port, 0), | ||
| 212 | }) | ||
| 213 | |||
| 214 | unanswered := udpClient(t, boundUDPPort(t, m, "quiet")) | ||
| 215 | _, err = unanswered.Write([]byte("hello?")) | ||
| 216 | require.NoError(t, err) | ||
| 217 | waitSessions(t, m, "quiet", 1) | ||
| 218 | waitSessions(t, m, "quiet", 0) | ||
| 219 | |||
| 220 | // Answered once: the same stretch of quiet does not end it, because the | ||
| 221 | // promotion bought it the longer window. | ||
| 222 | c := udpClient(t, boundUDPPort(t, m, "live")) | ||
| 223 | require.Equal(t, "echo:hi", say(t, c, "hi")) | ||
| 224 | waitSessions(t, m, "live", 1) | ||
| 225 | time.Sleep(600 * time.Millisecond) | ||
| 226 | assert.Equal(t, 1, sessions(t, m, "live"), "a conversation that has answered outlives the short window") | ||
| 227 | assert.Equal(t, "echo:still here", say(t, c, "still here"), "and it is the same session, still carrying traffic") | ||
| 228 | } | ||
| 229 | |||
| 230 | func TestUDPRefusesNewSessionsAtItsCapWithoutEvicting(t *testing.T) { | ||
| 231 | g := newFakeUDPGuest(t) | ||
| 232 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 233 | m.maxSessions = 1 | ||
| 234 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, 0)}) | ||
| 235 | addr := boundUDPPort(t, m, "e1") | ||
| 236 | |||
| 237 | held := udpClient(t, addr) | ||
| 238 | require.Equal(t, "echo:mine", say(t, held, "mine"), "the first caller has the only slot, and has been answered") | ||
| 239 | |||
| 240 | // The table is full. The second caller's datagram is dropped where it | ||
| 241 | // arrives: no session, and nothing comes back. | ||
| 242 | over := udpClient(t, addr) | ||
| 243 | _, err := over.Write([]byte("me too")) | ||
| 244 | require.NoError(t, err) | ||
| 245 | require.NoError(t, over.SetReadDeadline(time.Now().Add(300*time.Millisecond))) | ||
| 246 | buf := make([]byte, 64) | ||
| 247 | _, err = over.Read(buf) | ||
| 248 | assert.Error(t, err, "a caller past the cap is answered by nothing") | ||
| 249 | assert.Equal(t, 1, sessions(t, m, "e1")) | ||
| 250 | |||
| 251 | // And the conversation that was already working is untouched — a full table | ||
| 252 | // refuses the new, it does not sacrifice the live. | ||
| 253 | assert.Equal(t, "echo:still mine", say(t, held, "still mine")) | ||
| 254 | } | ||
| 255 | |||
| 256 | func TestUDPConvergeEvictsASessionWhoseGuestMoved(t *testing.T) { | ||
| 257 | first, second := newFakeUDPGuest(t), newFakeUDPGuest(t) | ||
| 258 | where := map[string]string{"vm1": first.addr} | ||
| 259 | m := NewManager(func(vmID string) string { return where[vmID] }) | ||
| 260 | t.Cleanup(m.StopAll) | ||
| 261 | |||
| 262 | // Both fakes answer on 127.0.0.1, so the guest PORT is what tells them | ||
| 263 | // apart; the pin under test is the address, so move the VM to an address | ||
| 264 | // nothing is at and prove the session goes. | ||
| 265 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", first.port, 0)}) | ||
| 266 | c := udpClient(t, boundUDPPort(t, m, "e1")) | ||
| 267 | require.Equal(t, "echo:before", say(t, c, "before")) | ||
| 268 | s := oneSession(t, m, "e1") | ||
| 269 | |||
| 270 | where["vm1"] = "127.0.0.2" | ||
| 271 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", first.port, 0)}) | ||
| 272 | assert.Equal(t, 0, sessions(t, m, "e1"), "a pin that no longer matches is not a session to keep") | ||
| 273 | _, err := s.guest.Write([]byte("orphan")) | ||
| 274 | assert.Error(t, err, "the evicted session's socket is closed, not leaked") | ||
| 275 | |||
| 276 | // A guest the host has lost track of entirely is no different: there is | ||
| 277 | // nowhere to send, so there is no session. | ||
| 278 | where["vm1"] = second.addr | ||
| 279 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", second.port, 0)}) | ||
| 280 | require.Equal(t, "echo:after", say(t, udpClient(t, boundUDPPort(t, m, "e1")), "after")) | ||
| 281 | where["vm1"] = "" | ||
| 282 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", second.port, 0)}) | ||
| 283 | assert.Equal(t, 0, sessions(t, m, "e1"), "no address is not an address to keep sending to") | ||
| 284 | } | ||
| 285 | |||
| 286 | func TestUDPSessionEndsWhenItsGuestSocketFails(t *testing.T) { | ||
| 287 | g := newFakeUDPGuest(t) | ||
| 288 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 289 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, 0)}) | ||
| 290 | c := udpClient(t, boundUDPPort(t, m, "e1")) | ||
| 291 | require.Equal(t, "echo:up", say(t, c, "up")) | ||
| 292 | |||
| 293 | // Break the socket toward the guest the way an ICMP port-unreachable or a | ||
| 294 | // descriptor going bad would: the relay's read fails, and the session that | ||
| 295 | // cannot reach its guest stops being one. | ||
| 296 | s := oneSession(t, m, "e1") | ||
| 297 | require.NoError(t, s.guest.Close()) | ||
| 298 | waitSessions(t, m, "e1", 0) | ||
| 299 | |||
| 300 | // The port is still published; the next datagram simply starts a fresh | ||
| 301 | // session, which is what makes the failure survivable. | ||
| 302 | assert.Equal(t, "echo:again", say(t, c, "again")) | ||
| 303 | } | ||
| 304 | |||
| 305 | func TestUDPDropsADatagramForAGuestWithNoAddress(t *testing.T) { | ||
| 306 | m := newTestManager(t, map[string]string{}) // the guest is still leasing | ||
| 307 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", 8080, 0)}) | ||
| 308 | |||
| 309 | c := udpClient(t, boundUDPPort(t, m, "e1")) | ||
| 310 | _, err := c.Write([]byte("anyone")) | ||
| 311 | require.NoError(t, err) | ||
| 312 | require.NoError(t, c.SetReadDeadline(time.Now().Add(300*time.Millisecond))) | ||
| 313 | buf := make([]byte, 64) | ||
| 314 | _, err = c.Read(buf) | ||
| 315 | assert.Error(t, err, "nowhere to send is a drop, not a wait") | ||
| 316 | assert.Equal(t, 0, sessions(t, m, "e1"), "a session pinned to nowhere is not a session") | ||
| 317 | } | ||
| 318 | |||
| 319 | func TestUDPConvergeReportsABindFailure(t *testing.T) { | ||
| 320 | g := newFakeUDPGuest(t) | ||
| 321 | m := newTestManager(t, map[string]string{"vm1": g.addr}) | ||
| 322 | |||
| 323 | squatter, err := net.ListenUDP("udp4", &net.UDPAddr{IP: net.IPv4zero}) | ||
| 324 | require.NoError(t, err) | ||
| 325 | held := uint32(squatter.LocalAddr().(*net.UDPAddr).Port) | ||
| 326 | |||
| 327 | got := m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, held)}) | ||
| 328 | require.Len(t, got, 1) | ||
| 329 | assert.Equal(t, "failed", got[0].GetState()) | ||
| 330 | assert.NotEmpty(t, got[0].GetReason(), "the report carries what the OS said") | ||
| 331 | |||
| 332 | require.NoError(t, squatter.Close()) | ||
| 333 | var state string | ||
| 334 | for i := 0; i < 20; i++ { | ||
| 335 | got = m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, held)}) | ||
| 336 | state = got[0].GetState() | ||
| 337 | if state == "active" { | ||
| 338 | break | ||
| 339 | } | ||
| 340 | time.Sleep(50 * time.Millisecond) | ||
| 341 | } | ||
| 342 | assert.Equal(t, "active", state, "a bind-time refusal is a level to converge on, not an error path") | ||
| 343 | } | ||
| 344 | |||
| 345 | func TestChangingTheProtocolRebinds(t *testing.T) { | ||
| 346 | tcpGuest := newFakeGuest(t) | ||
| 347 | udpGuest := newFakeUDPGuest(t) | ||
| 348 | m := newTestManager(t, map[string]string{"vm1": tcpGuest.addr}) | ||
| 349 | |||
| 350 | m.Converge([]*pb.ExposureDesired{desired("e1", "vm1", tcpGuest.port, 0)}) | ||
| 351 | assert.Equal(t, "echo:tcp", speak(t, boundPort(t, m, "e1"), "tcp")) | ||
| 352 | |||
| 353 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", udpGuest.port, 0)}) | ||
| 354 | m.mu.Lock() | ||
| 355 | ex := m.live["e1"] | ||
| 356 | m.mu.Unlock() | ||
| 357 | assert.Nil(t, ex.ln, "the listener the old protocol needed is gone") | ||
| 358 | assert.Equal(t, "echo:udp", say(t, udpClient(t, boundUDPPort(t, m, "e1")), "udp")) | ||
| 359 | } | ||
| 360 | |||
| 361 | func TestStopAllClosesPacketSocketsToo(t *testing.T) { | ||
| 362 | g := newFakeUDPGuest(t) | ||
| 363 | m := NewManager(func(string) string { return g.addr }) | ||
| 364 | m.Converge([]*pb.ExposureDesired{desiredUDP("e1", "vm1", g.port, 0)}) | ||
| 365 | addr := boundUDPPort(t, m, "e1") | ||
| 366 | c := udpClient(t, addr) | ||
| 367 | require.Equal(t, "echo:live", say(t, c, "live")) | ||
| 368 | s := oneSession(t, m, "e1") | ||
| 369 | |||
| 370 | m.StopAll() | ||
| 371 | |||
| 372 | // The socket is unbindable-again proof: the port frees, so something else | ||
| 373 | // can take it. The session that hung off it is closed with it. | ||
| 374 | freed, err := net.ListenUDP("udp4", mustUDPAddr(t, addr)) | ||
| 375 | require.NoError(t, err, "the published port survived StopAll") | ||
| 376 | freed.Close() | ||
| 377 | _, err = s.guest.Write([]byte("orphan")) | ||
| 378 | assert.Error(t, err, "a session outliving its socket would hold a descriptor nothing owns") | ||
| 379 | } | ||
| 380 | |||
| 381 | func mustUDPAddr(t *testing.T, addr string) *net.UDPAddr { | ||
| 382 | t.Helper() | ||
| 383 | ua, err := net.ResolveUDPAddr("udp4", addr) | ||
| 384 | require.NoError(t, err) | ||
| 385 | return ua | ||
| 386 | } | ||
internal/mcpserver/server.go
| Old | New | ||
|---|---|---|---|
| @@ -60,9 +60,9 @@ func NewServer(t *Tools, opts Options) *mcp.Server { | |||
| 60 | register(s, "vm_exec", "Run a shell command in a VM over SSH; returns stdout/stderr/exit code.", t.VMExec) | 60 | register(s, "vm_exec", "Run a shell command in a VM over SSH; returns stdout/stderr/exit code.", t.VMExec) |
| 61 | register(s, "vm_write_file", "Write content to a file in a VM (parents created).", t.VMWriteFile) | 61 | register(s, "vm_write_file", "Write content to a file in a VM (parents created).", t.VMWriteFile) |
| 62 | register(s, "vm_read_file", "Read a file from a VM (capped at 1 MiB).", t.VMReadFile) | 62 | register(s, "vm_read_file", "Read a file from a VM (capped at 1 MiB).", t.VMReadFile) |
| 63 | register(s, "vm_expose", "Publish a VM's guest TCP port on its host and return the address to dial. Omit host_port to allocate one from 30000-32767. WARNING: a published port has NO AUTHENTICATION in front of it — whoever can reach the host on that port reaches the service. Publish only what is meant to be reachable.", t.VMExpose) | 63 | register(s, "vm_expose", "Publish a VM's guest port on its host, TCP or UDP, and return the address to dial. Omit host_port to allocate one from 30000-32767. WARNING: a published port has NO AUTHENTICATION in front of it — whoever can reach the host on that port reaches the service, and a UDP one answers whatever address a datagram claims to come from. Publish only what is meant to be reachable.", t.VMExpose) |
| 64 | register(s, "vm_exposures", "List a VM's published ports, with the address to dial and each listener's state. These ports are unauthenticated.", t.VMExposures) | 64 | register(s, "vm_exposures", "List a VM's published ports, with the protocol, the address to dial and each socket's state. These ports are unauthenticated.", t.VMExposures) |
| 65 | register(s, "vm_unexpose", "Stop publishing a VM's guest port; the host closes the listener.", t.VMUnexpose) | 65 | register(s, "vm_unexpose", "Stop publishing a VM's guest port; the host closes the socket.", t.VMUnexpose) |
| 66 | register(s, "vm_destroy", "Destroy a VM by id or EXACT name. Explicit-only; never called automatically.", t.VMDestroy) | 66 | register(s, "vm_destroy", "Destroy a VM by id or EXACT name. Explicit-only; never called automatically.", t.VMDestroy) |
| 67 | register(s, "ca_upload", | 67 | register(s, "ca_upload", |
| 68 | "Register your SSH user CA's PUBLIC key with your tenant, so your guests trust certificates it signs. "+ | 68 | "Register your SSH user CA's PUBLIC key with your tenant, so your guests trust certificates it signs. "+ |
internal/mcpserver/tools.go
| Old | New | ||
|---|---|---|---|
| @@ -25,7 +25,7 @@ type api interface { | |||
| 25 | DeleteVM(ctx context.Context, id string) error | 25 | DeleteVM(ctx context.Context, id string) error |
| 26 | ListHosts(ctx context.Context) ([]client.Host, error) | 26 | ListHosts(ctx context.Context) ([]client.Host, error) |
| 27 | FirstEligibleHost(ctx context.Context) (client.Host, error) | 27 | FirstEligibleHost(ctx context.Context) (client.Host, error) |
| 28 | CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64) (client.Exposure, error) | 28 | CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) |
| 29 | ListExposures(ctx context.Context, vmID string) ([]client.Exposure, error) | 29 | ListExposures(ctx context.Context, vmID string) ([]client.Exposure, error) |
| 30 | DeleteExposure(ctx context.Context, id string) error | 30 | DeleteExposure(ctx context.Context, id string) error |
| 31 | RegisterUserCA(ctx context.Context, caLine, label string) error | 31 | RegisterUserCA(ctx context.Context, caLine, label string) error |
| @@ -608,16 +608,19 @@ type ExposureView struct { | |||
| 608 | ID string `json:"id"` | 608 | ID string `json:"id"` |
| 609 | GuestPort int64 `json:"guest_port"` | 609 | GuestPort int64 `json:"guest_port"` |
| 610 | HostPort int64 `json:"host_port"` | 610 | HostPort int64 `json:"host_port"` |
| 611 | Protocol string `json:"protocol"` | ||
| 611 | Address string `json:"address,omitempty"` | 612 | Address string `json:"address,omitempty"` |
| 612 | State string `json:"state"` | 613 | State string `json:"state"` |
| 613 | Reason string `json:"reason,omitempty"` | 614 | Reason string `json:"reason,omitempty"` |
| 614 | } | 615 | } |
| 615 | 616 | ||
| 616 | // exposureView folds an API exposure into the MCP view, joining the host | 617 | // exposureView folds an API exposure into the MCP view, joining the host |
| 617 | // address and host port into one dialable string. | 618 | // address and host port into one dialable string. The protocol travels beside |
| 619 | // it rather than inside it: an address is a host and a port, and which of the | ||
| 620 | // two protocols to send is the caller's next decision, not part of the name. | ||
| 618 | func exposureView(e client.Exposure) ExposureView { | 621 | func exposureView(e client.Exposure) ExposureView { |
| 619 | v := ExposureView{ | 622 | v := ExposureView{ |
| 620 | ID: e.ID, GuestPort: e.GuestPort, HostPort: e.HostPort, | 623 | ID: e.ID, GuestPort: e.GuestPort, HostPort: e.HostPort, Protocol: e.Protocol, |
| 621 | State: e.State, Reason: e.Reason, | 624 | State: e.State, Reason: e.Reason, |
| 622 | } | 625 | } |
| 623 | if e.HostAddr != "" { | 626 | if e.HostAddr != "" { |
| @@ -628,8 +631,9 @@ func exposureView(e client.Exposure) ExposureView { | |||
| 628 | 631 | ||
| 629 | type VMExposeIn struct { | 632 | type VMExposeIn struct { |
| 630 | VM string `json:"vm" jsonschema:"VM id or exact name"` | 633 | VM string `json:"vm" jsonschema:"VM id or exact name"` |
| 631 | GuestPort int64 `json:"guest_port" jsonschema:"TCP port the service listens on inside the guest"` | 634 | GuestPort int64 `json:"guest_port" jsonschema:"port the service listens on inside the guest"` |
| 632 | HostPort int64 `json:"host_port,omitempty" jsonschema:"port to bind on the host; omit to allocate one from 30000-32767, a named port must be >= 1024"` | 635 | HostPort int64 `json:"host_port,omitempty" jsonschema:"port to bind on the host; omit to allocate one from 30000-32767, a named port must be >= 1024"` |
| 636 | Protocol string `json:"protocol,omitempty" jsonschema:"tcp (the default) or udp; the same host port can carry one of each"` | ||
| 633 | } | 637 | } |
| 634 | 638 | ||
| 635 | type VMExposeOut struct { | 639 | type VMExposeOut struct { |
| @@ -644,7 +648,7 @@ func (t *Tools) VMExpose(ctx context.Context, in VMExposeIn) (VMExposeOut, error | |||
| 644 | if err != nil { | 648 | if err != nil { |
| 645 | return VMExposeOut{}, err | 649 | return VMExposeOut{}, err |
| 646 | } | 650 | } |
| 647 | e, err := t.API.CreateExposure(ctx, vm.ID, in.GuestPort, in.HostPort) | 651 | e, err := t.API.CreateExposure(ctx, vm.ID, in.GuestPort, in.HostPort, in.Protocol) |
| 648 | if err != nil { | 652 | if err != nil { |
| 649 | return VMExposeOut{}, fmt.Errorf("expose vm %s port %d: %w", vm.Name, in.GuestPort, err) | 653 | return VMExposeOut{}, fmt.Errorf("expose vm %s port %d: %w", vm.Name, in.GuestPort, err) |
| 650 | } | 654 | } |
| @@ -679,12 +683,14 @@ type VMUnexposeIn struct { | |||
| 679 | VM string `json:"vm" jsonschema:"VM id or exact name"` | 683 | VM string `json:"vm" jsonschema:"VM id or exact name"` |
| 680 | GuestPort int64 `json:"guest_port" jsonschema:"the guest port to stop publishing"` | 684 | GuestPort int64 `json:"guest_port" jsonschema:"the guest port to stop publishing"` |
| 681 | HostPort int64 `json:"host_port,omitempty" jsonschema:"host port, only needed when the same guest port is published more than once"` | 685 | HostPort int64 `json:"host_port,omitempty" jsonschema:"host port, only needed when the same guest port is published more than once"` |
| 686 | Protocol string `json:"protocol,omitempty" jsonschema:"tcp or udp, only needed when the same guest port is published in both"` | ||
| 682 | } | 687 | } |
| 683 | 688 | ||
| 684 | type VMUnexposeOut struct { | 689 | type VMUnexposeOut struct { |
| 685 | ID string `json:"id"` | 690 | ID string `json:"id"` |
| 686 | GuestPort int64 `json:"guest_port"` | 691 | GuestPort int64 `json:"guest_port"` |
| 687 | HostPort int64 `json:"host_port"` | 692 | HostPort int64 `json:"host_port"` |
| 693 | Protocol string `json:"protocol"` | ||
| 688 | } | 694 | } |
| 689 | 695 | ||
| 690 | // VMUnexpose revokes one of a VM's published ports. The caller names the guest | 696 | // VMUnexpose revokes one of a VM's published ports. The caller names the guest |
| @@ -699,25 +705,26 @@ func (t *Tools) VMUnexpose(ctx context.Context, in VMUnexposeIn) (VMUnexposeOut, | |||
| 699 | if err != nil { | 705 | if err != nil { |
| 700 | return VMUnexposeOut{}, fmt.Errorf("list exposures for vm %s: %w", vm.Name, err) | 706 | return VMUnexposeOut{}, fmt.Errorf("list exposures for vm %s: %w", vm.Name, err) |
| 701 | } | 707 | } |
| 702 | e, err := matchExposure(exps, vm.Name, in.GuestPort, in.HostPort) | 708 | e, err := matchExposure(exps, vm.Name, in.GuestPort, in.HostPort, in.Protocol) |
| 703 | if err != nil { | 709 | if err != nil { |
| 704 | return VMUnexposeOut{}, err | 710 | return VMUnexposeOut{}, err |
| 705 | } | 711 | } |
| 706 | if err := t.API.DeleteExposure(ctx, e.ID); err != nil { | 712 | if err := t.API.DeleteExposure(ctx, e.ID); err != nil { |
| 707 | return VMUnexposeOut{}, fmt.Errorf("unexpose vm %s port %d: %w", vm.Name, in.GuestPort, err) | 713 | return VMUnexposeOut{}, fmt.Errorf("unexpose vm %s port %d: %w", vm.Name, in.GuestPort, err) |
| 708 | } | 714 | } |
| 709 | return VMUnexposeOut{ID: e.ID, GuestPort: e.GuestPort, HostPort: e.HostPort}, nil | 715 | return VMUnexposeOut{ID: e.ID, GuestPort: e.GuestPort, HostPort: e.HostPort, Protocol: e.Protocol}, nil |
| 710 | } | 716 | } |
| 711 | 717 | ||
| 712 | // matchExposure picks the one exposure of vmName publishing guestPort. Nothing | 718 | // matchExposure picks the one exposure of vmName publishing guestPort. Nothing |
| 713 | // stops a guest port being published on two host ports, so hostPort (0 = any) | 719 | // stops a guest port being published on two host ports, or in both protocols, |
| 714 | // disambiguates; an ambiguous match refuses rather than guessing which | 720 | // so hostPort (0 = any) and protocol ("" = any) disambiguate; an ambiguous |
| 715 | // listener to close. Errors name what IS published, so the model can correct | 721 | // match refuses rather than guessing which socket to close. Errors name what IS |
| 716 | // itself without a second listing call. | 722 | // published, so the model can correct itself without a second listing call. |
| 717 | func matchExposure(exps []client.Exposure, vmName string, guestPort, hostPort int64) (client.Exposure, error) { | 723 | func matchExposure(exps []client.Exposure, vmName string, guestPort, hostPort int64, protocol string) (client.Exposure, error) { |
| 718 | var matches []client.Exposure | 724 | var matches []client.Exposure |
| 719 | for _, e := range exps { | 725 | for _, e := range exps { |
| 720 | if e.GuestPort == guestPort && (hostPort == 0 || e.HostPort == hostPort) { | 726 | if e.GuestPort == guestPort && (hostPort == 0 || e.HostPort == hostPort) && |
| 727 | (protocol == "" || e.Protocol == protocol) { | ||
| 721 | matches = append(matches, e) | 728 | matches = append(matches, e) |
| 722 | } | 729 | } |
| 723 | } | 730 | } |
| @@ -727,18 +734,18 @@ func matchExposure(exps []client.Exposure, vmName string, guestPort, hostPort in | |||
| 727 | case 0: | 734 | case 0: |
| 728 | return client.Exposure{}, fmt.Errorf("vm %s publishes no guest port %d (published: %s)", vmName, guestPort, describeExposures(exps)) | 735 | return client.Exposure{}, fmt.Errorf("vm %s publishes no guest port %d (published: %s)", vmName, guestPort, describeExposures(exps)) |
| 729 | default: | 736 | default: |
| 730 | return client.Exposure{}, fmt.Errorf("vm %s publishes guest port %d more than once (%s); name host_port to pick one", vmName, guestPort, describeExposures(matches)) | 737 | return client.Exposure{}, fmt.Errorf("vm %s publishes guest port %d more than once (%s); name host_port or protocol to pick one", vmName, guestPort, describeExposures(matches)) |
| 731 | } | 738 | } |
| 732 | } | 739 | } |
| 733 | 740 | ||
| 734 | // describeExposures renders exposures as "guest:host" pairs for an error. | 741 | // describeExposures renders exposures as "guest:host/protocol" for an error. |
| 735 | func describeExposures(exps []client.Exposure) string { | 742 | func describeExposures(exps []client.Exposure) string { |
| 736 | if len(exps) == 0 { | 743 | if len(exps) == 0 { |
| 737 | return "none" | 744 | return "none" |
| 738 | } | 745 | } |
| 739 | parts := make([]string, 0, len(exps)) | 746 | parts := make([]string, 0, len(exps)) |
| 740 | for _, e := range exps { | 747 | for _, e := range exps { |
| 741 | parts = append(parts, fmt.Sprintf("%d:%d", e.GuestPort, e.HostPort)) | 748 | parts = append(parts, fmt.Sprintf("%d:%d/%s", e.GuestPort, e.HostPort, e.Protocol)) |
| 742 | } | 749 | } |
| 743 | return strings.Join(parts, ", ") | 750 | return strings.Join(parts, ", ") |
| 744 | } | 751 | } |
internal/mcpserver/tools_test.go
| Old | New | ||
|---|---|---|---|
| @@ -94,15 +94,19 @@ func (f *fakeToolsAPI) FirstEligibleHost(ctx context.Context) (client.Host, erro | |||
| 94 | } | 94 | } |
| 95 | 95 | ||
| 96 | // CreateExposure mirrors the control plane: host port 0 is allocated from the | 96 | // CreateExposure mirrors the control plane: host port 0 is allocated from the |
| 97 | // reserved range, and the host address comes back with the grant. | 97 | // reserved range, an unnamed protocol is tcp, and the host address comes back |
| 98 | func (f *fakeToolsAPI) CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64) (client.Exposure, error) { | 98 | // with the grant. |
| 99 | func (f *fakeToolsAPI) CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) { | ||
| 99 | if hostPort == 0 { | 100 | if hostPort == 0 { |
| 100 | hostPort = 30000 + int64(len(f.exposures[vmID])) | 101 | hostPort = 30000 + int64(len(f.exposures[vmID])) |
| 101 | } | 102 | } |
| 103 | if protocol == "" { | ||
| 104 | protocol = "tcp" | ||
| 105 | } | ||
| 102 | e := client.Exposure{ | 106 | e := client.Exposure{ |
| 103 | ID: fmt.Sprintf("x-%d", len(f.exposures[vmID])+1), VMID: vmID, HostID: "h1", | 107 | ID: fmt.Sprintf("x-%d", len(f.exposures[vmID])+1), VMID: vmID, HostID: "h1", |
| 104 | GuestPort: guestPort, HostPort: hostPort, HostAddr: "10.0.0.4", | 108 | GuestPort: guestPort, HostPort: hostPort, HostAddr: "10.0.0.4", |
| 105 | Protocol: "tcp", Scope: "host", State: "pending", | 109 | Protocol: protocol, Scope: "host", State: "pending", |
| 106 | } | 110 | } |
| 107 | if f.exposures == nil { | 111 | if f.exposures == nil { |
| 108 | f.exposures = map[string][]client.Exposure{} | 112 | f.exposures = map[string][]client.Exposure{} |
| @@ -513,6 +517,32 @@ func TestExposePublishesGuestPortWithDialAddress(t *testing.T) { | |||
| 513 | assert.Equal(t, "10.0.0.4:31500", out.Exposure.Address) | 517 | assert.Equal(t, "10.0.0.4:31500", out.Exposure.Address) |
| 514 | } | 518 | } |
| 515 | 519 | ||
| 520 | func TestExposeCarriesTheProtocolAndUnexposePicksByIt(t *testing.T) { | ||
| 521 | api := &fakeToolsAPI{vms: []client.VM{{ID: "abc123", Name: "web-1", Lifecycle: "ready"}}} | ||
| 522 | tl := newTestTools(api, &fakeRunner{}) | ||
| 523 | |||
| 524 | tcp, err := tl.VMExpose(t.Context(), VMExposeIn{VM: "web-1", GuestPort: 53, HostPort: 30053}) | ||
| 525 | require.NoError(t, err) | ||
| 526 | assert.Equal(t, "tcp", tcp.Exposure.Protocol, "an unnamed protocol is tcp, and the view says which it got") | ||
| 527 | |||
| 528 | udp, err := tl.VMExpose(t.Context(), VMExposeIn{VM: "web-1", GuestPort: 53, HostPort: 30053, Protocol: "udp"}) | ||
| 529 | require.NoError(t, err) | ||
| 530 | assert.Equal(t, "udp", udp.Exposure.Protocol) | ||
| 531 | |||
| 532 | // One guest port, both protocols: the same ambiguity two host ports create, | ||
| 533 | // and protocol is the other way to resolve it. | ||
| 534 | _, err = tl.VMUnexpose(t.Context(), VMUnexposeIn{VM: "web-1", GuestPort: 53}) | ||
| 535 | require.Error(t, err) | ||
| 536 | assert.ErrorContains(t, err, "53:30053/udp", "the error names the protocol of what IS published") | ||
| 537 | assert.Empty(t, api.revoked) | ||
| 538 | |||
| 539 | out, err := tl.VMUnexpose(t.Context(), VMUnexposeIn{VM: "web-1", GuestPort: 53, Protocol: "udp"}) | ||
| 540 | require.NoError(t, err) | ||
| 541 | assert.Equal(t, udp.Exposure.ID, out.ID) | ||
| 542 | assert.Equal(t, "udp", out.Protocol) | ||
| 543 | assert.Equal(t, []string{udp.Exposure.ID}, api.revoked) | ||
| 544 | } | ||
| 545 | |||
| 516 | func TestExposeUnknownVM(t *testing.T) { | 546 | func TestExposeUnknownVM(t *testing.T) { |
| 517 | tl := newTestTools(&fakeToolsAPI{}, &fakeRunner{}) | 547 | tl := newTestTools(&fakeToolsAPI{}, &fakeRunner{}) |
| 518 | _, err := tl.VMExpose(t.Context(), VMExposeIn{VM: "nope", GuestPort: 80}) | 548 | _, err := tl.VMExpose(t.Context(), VMExposeIn{VM: "nope", GuestPort: 80}) |
internal/pb/sync.pb.go
| Old | New | ||
|---|---|---|---|
| @@ -1396,18 +1396,18 @@ func (x *TCPOpened) GetError() string { | |||
| 1396 | } | 1396 | } |
| 1397 | 1397 | ||
| 1398 | // ExposureDesired is one published guest port a host should be serving: bind | 1398 | // ExposureDesired is one published guest port a host should be serving: bind |
| 1399 | // host_port on the host, pipe every accepted connection to guest_port inside | 1399 | // host_port on the host, pipe every accepted connection (or every datagram) |
| 1400 | // the guest. It rides the snapshot at TOP LEVEL rather than nested in | 1400 | // to guest_port inside the guest. It rides the snapshot at TOP LEVEL rather |
| 1401 | // VMDesired, because exposures are their own objects converging on their own | 1401 | // than nested in VMDesired, because exposures are their own objects converging |
| 1402 | // cadence — an exposure can be created while its VM is still imaging, and it | 1402 | // on their own cadence — an exposure can be created while its VM is still |
| 1403 | // binds immediately. | 1403 | // imaging, and it binds immediately. |
| 1404 | type ExposureDesired struct { | 1404 | type ExposureDesired struct { |
| 1405 | state protoimpl.MessageState `protogen:"open.v1"` | 1405 | state protoimpl.MessageState `protogen:"open.v1"` |
| 1406 | Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` | 1406 | Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` |
| 1407 | VmId string `protobuf:"bytes,2,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` | 1407 | VmId string `protobuf:"bytes,2,opt,name=vm_id,json=vmId,proto3" json:"vm_id,omitempty"` |
| 1408 | GuestPort uint32 `protobuf:"varint,3,opt,name=guest_port,json=guestPort,proto3" json:"guest_port,omitempty"` | 1408 | GuestPort uint32 `protobuf:"varint,3,opt,name=guest_port,json=guestPort,proto3" json:"guest_port,omitempty"` |
| 1409 | HostPort uint32 `protobuf:"varint,4,opt,name=host_port,json=hostPort,proto3" json:"host_port,omitempty"` | 1409 | HostPort uint32 `protobuf:"varint,4,opt,name=host_port,json=hostPort,proto3" json:"host_port,omitempty"` |
| 1410 | Protocol string `protobuf:"bytes,5,opt,name=protocol,proto3" json:"protocol,omitempty"` // "tcp" | 1410 | Protocol string `protobuf:"bytes,5,opt,name=protocol,proto3" json:"protocol,omitempty"` // "tcp"|"udp" |
| 1411 | unknownFields protoimpl.UnknownFields | 1411 | unknownFields protoimpl.UnknownFields |
| 1412 | sizeCache protoimpl.SizeCache | 1412 | sizeCache protoimpl.SizeCache |
| 1413 | } | 1413 | } |
| @@ -1478,14 +1478,14 @@ func (x *ExposureDesired) GetProtocol() string { | |||
| 1478 | } | 1478 | } |
| 1479 | 1479 | ||
| 1480 | // ExposureActual is one exposure's state as its host observes it: "active" | 1480 | // ExposureActual is one exposure's state as its host observes it: "active" |
| 1481 | // once the host listener is bound, "failed" with the OS error otherwise. | 1481 | // once the host socket is bound, "failed" with the OS error otherwise. |
| 1482 | // "active" means the HOST half of the pipe exists — whether anything answers | 1482 | // "active" means the HOST half of the pipe exists — whether anything answers |
| 1483 | // inside the guest is the guest's half, and this does not pretend otherwise. | 1483 | // inside the guest is the guest's half, and this does not pretend otherwise. |
| 1484 | type ExposureActual struct { | 1484 | type ExposureActual struct { |
| 1485 | state protoimpl.MessageState `protogen:"open.v1"` | 1485 | state protoimpl.MessageState `protogen:"open.v1"` |
| 1486 | Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` | 1486 | Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` |
| 1487 | State string `protobuf:"bytes,2,opt,name=state,proto3" json:"state,omitempty"` // "active"|"failed" | 1487 | State string `protobuf:"bytes,2,opt,name=state,proto3" json:"state,omitempty"` // "active"|"failed" |
| 1488 | Reason string `protobuf:"bytes,3,opt,name=reason,proto3" json:"reason,omitempty"` // the OS error, when failed | 1488 | Reason string `protobuf:"bytes,3,opt,name=reason,proto3" json:"reason,omitempty"` // the OS error, when failed; a bound port's ongoing trouble otherwise |
| 1489 | unknownFields protoimpl.UnknownFields | 1489 | unknownFields protoimpl.UnknownFields |
| 1490 | sizeCache protoimpl.SizeCache | 1490 | sizeCache protoimpl.SizeCache |
| 1491 | } | 1491 | } |
internal/server/api/client/client.go
| Old | New | ||
|---|---|---|---|
| @@ -150,11 +150,12 @@ func (c *Client) PatchVM(ctx context.Context, id, powerState string) error { | |||
| 150 | return c.do(ctx, http.MethodPatch, "/api/v1/vms/"+url.PathEscape(id), PatchVMRequest{PowerState: powerState}, nil) | 150 | return c.do(ctx, http.MethodPatch, "/api/v1/vms/"+url.PathEscape(id), PatchVMRequest{PowerState: powerState}, nil) |
| 151 | } | 151 | } |
| 152 | 152 | ||
| 153 | // CreateExposure publishes guestPort of a VM on its host. hostPort 0 asks the | 153 | // CreateExposure publishes guestPort of a VM on its host, for protocol "tcp" |
| 154 | // control plane to allocate one from the reserved range; the returned exposure | 154 | // or "udp" (empty means tcp). hostPort 0 asks the control plane to allocate one |
| 155 | // carries whichever port it ended up with. | 155 | // from the reserved range; the returned exposure carries whichever port it |
| 156 | func (c *Client) CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64) (Exposure, error) { | 156 | // ended up with. |
| 157 | req := CreateExposureRequest{GuestPort: guestPort, HostPort: hostPort} | 157 | func (c *Client) CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (Exposure, error) { |
| 158 | req := CreateExposureRequest{GuestPort: guestPort, HostPort: hostPort, Protocol: protocol} | ||
| 158 | var out Exposure | 159 | var out Exposure |
| 159 | return out, c.do(ctx, http.MethodPost, "/api/v1/vms/"+url.PathEscape(vmID)+"/exposures", req, &out) | 160 | return out, c.do(ctx, http.MethodPost, "/api/v1/vms/"+url.PathEscape(vmID)+"/exposures", req, &out) |
| 160 | } | 161 | } |
internal/server/api/client/client_test.go
| Old | New | ||
|---|---|---|---|
| @@ -479,7 +479,7 @@ func TestCreateExposure(t *testing.T) { | |||
| 479 | `"created_at":"2026-08-05T12:00:00Z"}`) | 479 | `"created_at":"2026-08-05T12:00:00Z"}`) |
| 480 | c := &client.Client{BaseURL: srv.URL, Token: "eitri_pat_x"} | 480 | c := &client.Client{BaseURL: srv.URL, Token: "eitri_pat_x"} |
| 481 | 481 | ||
| 482 | got, err := c.CreateExposure(context.Background(), "v-1", 8080, 0) | 482 | got, err := c.CreateExposure(context.Background(), "v-1", 8080, 0, "udp") |
| 483 | if err != nil { | 483 | if err != nil { |
| 484 | t.Fatalf("CreateExposure: %v", err) | 484 | t.Fatalf("CreateExposure: %v", err) |
| 485 | } | 485 | } |
| @@ -490,8 +490,8 @@ func TestCreateExposure(t *testing.T) { | |||
| 490 | if err := json.Unmarshal(cap.body, &sent); err != nil { | 490 | if err := json.Unmarshal(cap.body, &sent); err != nil { |
| 491 | t.Fatalf("decode sent body: %v", err) | 491 | t.Fatalf("decode sent body: %v", err) |
| 492 | } | 492 | } |
| 493 | if sent.GuestPort != 8080 || sent.HostPort != 0 { | 493 | if sent.GuestPort != 8080 || sent.HostPort != 0 || sent.Protocol != "udp" { |
| 494 | t.Errorf("sent = %+v, want guest 8080 / host 0", sent) | 494 | t.Errorf("sent = %+v, want guest 8080 / host 0 / udp", sent) |
| 495 | } | 495 | } |
| 496 | if got.ID != "x-1" || got.HostPort != 30080 { | 496 | if got.ID != "x-1" || got.HostPort != 30080 { |
| 497 | t.Errorf("got = %+v, want the allocated exposure back", got) | 497 | t.Errorf("got = %+v, want the allocated exposure back", got) |
internal/server/api/exposures.go
| Old | New | ||
|---|---|---|---|
| @@ -32,15 +32,19 @@ func (a *API) exposureVM(w http.ResponseWriter, r *http.Request) (store.VM, bool | |||
| 32 | return vm, true | 32 | return vm, true |
| 33 | } | 33 | } |
| 34 | 34 | ||
| 35 | // validateExposurePorts checks a create request, returning (msg, status) on | 35 | // validateExposure checks a create request, returning (msg, status) on failure |
| 36 | // failure or ("", 0) when valid. | 36 | // or ("", 0) when valid. |
| 37 | // | 37 | // |
| 38 | // guest_port is unrestricted within the port space: the guest owns the guest, | 38 | // guest_port is unrestricted within the port space: the guest owns the guest, |
| 39 | // including which of its ports are worth publishing. host_port is where | 39 | // including which of its ports are worth publishing. host_port is where |
| 40 | // privilege lives — the 1024 floor is uniformity as much as safety, because a | 40 | // privilege lives — the 1024 floor is uniformity as much as safety, because a |
| 41 | // macOS agent is unprivileged and cannot bind lower, and low ports are the | 41 | // macOS agent is unprivileged and cannot bind lower, and low ports are the |
| 42 | // future gateway's territory. | 42 | // future gateway's territory. |
| 43 | func validateExposurePorts(req types.CreateExposureRequest) (string, int) { | 43 | // |
| 44 | // protocol is closed at two values because the agent has exactly two proxies | ||
| 45 | // to offer, and a third string accepted here would become a grant no host ever | ||
| 46 | // binds — a row that reads published and serves nothing. | ||
| 47 | func validateExposure(req types.CreateExposureRequest) (string, int) { | ||
| 44 | if req.GuestPort < 1 || req.GuestPort > 65535 { | 48 | if req.GuestPort < 1 || req.GuestPort > 65535 { |
| 45 | return "guest_port must be between 1 and 65535", http.StatusBadRequest | 49 | return "guest_port must be between 1 and 65535", http.StatusBadRequest |
| 46 | } | 50 | } |
| @@ -49,9 +53,22 @@ func validateExposurePorts(req types.CreateExposureRequest) (string, int) { | |||
| 49 | strconv.Itoa(store.MinAllocatedHostPort) + "-" + strconv.Itoa(store.MaxAllocatedHostPort), | 53 | strconv.Itoa(store.MinAllocatedHostPort) + "-" + strconv.Itoa(store.MaxAllocatedHostPort), |
| 50 | http.StatusBadRequest | 54 | http.StatusBadRequest |
| 51 | } | 55 | } |
| 56 | if req.Protocol != "" && req.Protocol != "tcp" && req.Protocol != "udp" { | ||
| 57 | return `protocol must be "tcp" or "udp"`, http.StatusBadRequest | ||
| 58 | } | ||
| 52 | return "", 0 | 59 | return "", 0 |
| 53 | } | 60 | } |
| 54 | 61 | ||
| 62 | // exposureProtocol is the protocol a create request asks for. An omitted one is | ||
| 63 | // tcp: the field arrived after the endpoint did, and every caller that predates | ||
| 64 | // it means the same thing by silence. | ||
| 65 | func exposureProtocol(req types.CreateExposureRequest) string { | ||
| 66 | if req.Protocol == "" { | ||
| 67 | return "tcp" | ||
| 68 | } | ||
| 69 | return req.Protocol | ||
| 70 | } | ||
| 71 | |||
| 55 | // exposureState folds a host's live report into one exposure's state. An | 72 | // exposureState folds a host's live report into one exposure's state. An |
| 56 | // exposure the host has not reported on is "pending": the grant exists, and | 73 | // exposure the host has not reported on is "pending": the grant exists, and |
| 57 | // nothing has said what the host made of it yet. | 74 | // nothing has said what the host made of it yet. |
| @@ -101,17 +118,18 @@ func (a *API) handleCreateExposure(w http.ResponseWriter, r *http.Request) { | |||
| 101 | if !decodeJSON(w, r, &req) { | 118 | if !decodeJSON(w, r, &req) { |
| 102 | return | 119 | return |
| 103 | } | 120 | } |
| 104 | if msg, code := validateExposurePorts(req); msg != "" { | 121 | if msg, code := validateExposure(req); msg != "" { |
| 105 | http.Error(w, msg, code) | 122 | http.Error(w, msg, code) |
| 106 | return | 123 | return |
| 107 | } | 124 | } |
| 108 | e, err := a.st.CreateExposure(vm.ID, req.GuestPort, req.HostPort) | 125 | protocol := exposureProtocol(req) |
| 126 | e, err := a.st.CreateExposure(vm.ID, req.GuestPort, req.HostPort, protocol) | ||
| 109 | if err != nil { | 127 | if err != nil { |
| 110 | switch { | 128 | switch { |
| 111 | case errors.Is(err, store.ErrExposureVMNotFound): | 129 | case errors.Is(err, store.ErrExposureVMNotFound): |
| 112 | http.Error(w, "not found", http.StatusNotFound) | 130 | http.Error(w, "not found", http.StatusNotFound) |
| 113 | case errors.Is(err, store.ErrHostPortTaken): | 131 | case errors.Is(err, store.ErrHostPortTaken): |
| 114 | http.Error(w, "host port already in use on that host", http.StatusConflict) | 132 | http.Error(w, "host port already in use on that host for "+protocol, http.StatusConflict) |
| 115 | case errors.Is(err, store.ErrNoFreeHostPort): | 133 | case errors.Is(err, store.ErrNoFreeHostPort): |
| 116 | http.Error(w, "no free host port in the reserved range on that host", http.StatusConflict) | 134 | http.Error(w, "no free host port in the reserved range on that host", http.StatusConflict) |
| 117 | default: | 135 | default: |
| @@ -123,6 +141,7 @@ func (a *API) handleCreateExposure(w http.ResponseWriter, r *http.Request) { | |||
| 123 | "vm_id": vm.ID, "name": vm.Name, "exposure_id": e.ID, | 141 | "vm_id": vm.ID, "name": vm.Name, "exposure_id": e.ID, |
| 124 | "guest_port": strconv.FormatInt(e.GuestPort, 10), | 142 | "guest_port": strconv.FormatInt(e.GuestPort, 10), |
| 125 | "host_port": strconv.FormatInt(e.HostPort, 10), | 143 | "host_port": strconv.FormatInt(e.HostPort, 10), |
| 144 | "protocol": e.Protocol, | ||
| 126 | }) | 145 | }) |
| 127 | a.hub.Poke(vm.HostID) | 146 | a.hub.Poke(vm.HostID) |
| 128 | a.notif.notify() | 147 | a.notif.notify() |
| @@ -178,6 +197,7 @@ func (a *API) handleDeleteExposure(w http.ResponseWriter, r *http.Request) { | |||
| 178 | "vm_id": e.VMID, "exposure_id": e.ID, | 197 | "vm_id": e.VMID, "exposure_id": e.ID, |
| 179 | "guest_port": strconv.FormatInt(e.GuestPort, 10), | 198 | "guest_port": strconv.FormatInt(e.GuestPort, 10), |
| 180 | "host_port": strconv.FormatInt(e.HostPort, 10), | 199 | "host_port": strconv.FormatInt(e.HostPort, 10), |
| 200 | "protocol": e.Protocol, | ||
| 181 | }) | 201 | }) |
| 182 | a.hub.Poke(e.HostID) | 202 | a.hub.Poke(e.HostID) |
| 183 | a.notif.notify() | 203 | a.notif.notify() |
internal/server/api/exposures_test.go
| Old | New | ||
|---|---|---|---|
| @@ -2,6 +2,7 @@ package api | |||
| 2 | 2 | ||
| 3 | import ( | 3 | import ( |
| 4 | "encoding/json" | 4 | "encoding/json" |
| 5 | "io" | ||
| 5 | "net/http" | 6 | "net/http" |
| 6 | "net/http/httptest" | 7 | "net/http/httptest" |
| 7 | "testing" | 8 | "testing" |
| @@ -76,6 +77,48 @@ func TestCreateExposureHonoursAndRefusesHostPorts(t *testing.T) { | |||
| 76 | assert.Equal(t, 409, resp.StatusCode) | 77 | assert.Equal(t, 409, resp.StatusCode) |
| 77 | } | 78 | } |
| 78 | 79 | ||
| 80 | func TestCreateExposureDefaultsToTCPAndTakesUDP(t *testing.T) { | ||
| 81 | ts, _, _ := testServer(t) | ||
| 82 | host := enroll(t, ts) | ||
| 83 | vmID := createTestVM(t, ts, host["host_id"], "web-1") | ||
| 84 | |||
| 85 | resp := do(t, "POST", ts.URL+"/api/v1/vms/"+vmID+"/exposures", testPAT, | ||
| 86 | map[string]any{"guest_port": 53, "host_port": 8443, "protocol": "udp"}) | ||
| 87 | require.Equal(t, 201, resp.StatusCode) | ||
| 88 | var udp map[string]any | ||
| 89 | require.NoError(t, json.NewDecoder(resp.Body).Decode(&udp)) | ||
| 90 | assert.Equal(t, "udp", udp["protocol"]) | ||
| 91 | |||
| 92 | // The same host port carries one of each, and no more than one of either. | ||
| 93 | resp = do(t, "POST", ts.URL+"/api/v1/vms/"+vmID+"/exposures", testPAT, | ||
| 94 | map[string]any{"guest_port": 53, "host_port": 8443}) | ||
| 95 | require.Equal(t, 201, resp.StatusCode) | ||
| 96 | var tcp map[string]any | ||
| 97 | require.NoError(t, json.NewDecoder(resp.Body).Decode(&tcp)) | ||
| 98 | assert.Equal(t, "tcp", tcp["protocol"], "a request that names no protocol means tcp") | ||
| 99 | |||
| 100 | resp = do(t, "POST", ts.URL+"/api/v1/vms/"+vmID+"/exposures", testPAT, | ||
| 101 | map[string]any{"guest_port": 54, "host_port": 8443, "protocol": "udp"}) | ||
| 102 | assert.Equal(t, 409, resp.StatusCode) | ||
| 103 | } | ||
| 104 | |||
| 105 | func TestCreateExposureRefusesAnotherProtocol(t *testing.T) { | ||
| 106 | ts, _, _ := testServer(t) | ||
| 107 | host := enroll(t, ts) | ||
| 108 | vmID := createTestVM(t, ts, host["host_id"], "web-1") | ||
| 109 | |||
| 110 | for _, proto := range []string{"sctp", "TCP", "udp4", "http"} { | ||
| 111 | t.Run(proto, func(t *testing.T) { | ||
| 112 | resp := do(t, "POST", ts.URL+"/api/v1/vms/"+vmID+"/exposures", testPAT, | ||
| 113 | map[string]any{"guest_port": 8080, "protocol": proto}) | ||
| 114 | require.Equal(t, 400, resp.StatusCode) | ||
| 115 | body, err := io.ReadAll(resp.Body) | ||
| 116 | require.NoError(t, err) | ||
| 117 | assert.Contains(t, string(body), `"tcp" or "udp"`, "a refusal names what it would take") | ||
| 118 | }) | ||
| 119 | } | ||
| 120 | } | ||
| 121 | |||
| 79 | func TestCreateExposureValidatesPorts(t *testing.T) { | 122 | func TestCreateExposureValidatesPorts(t *testing.T) { |
| 80 | ts, _, _ := testServer(t) | 123 | ts, _, _ := testServer(t) |
| 81 | host := enroll(t, ts) | 124 | host := enroll(t, ts) |
internal/server/api/routes.go
| Old | New | ||
|---|---|---|---|
| @@ -233,9 +233,9 @@ var routeTable = []Route{ | |||
| 233 | Doc: "One VM's lifecycle timeline (audit rows carrying its vm_id), newest first; survives the VM row being reaped.", | 233 | Doc: "One VM's lifecycle timeline (audit rows carrying its vm_id), newest first; survives the VM row being reaped.", |
| 234 | handler: (*API).handleListVMEvents, | 234 | handler: (*API).handleListVMEvents, |
| 235 | }, | 235 | }, |
| 236 | // Service exposure: a tenant publishes one TCP port of one VM, and the | 236 | // Service exposure: a tenant publishes one port of one VM, TCP or UDP, and |
| 237 | // fleet binds it on that VM's host. There is no update verb — an exposure | 237 | // the fleet binds it on that VM's host. There is no update verb — an |
| 238 | // is a grant, not a document. | 238 | // exposure is a grant, not a document. |
| 239 | { | 239 | { |
| 240 | Method: "POST", | 240 | Method: "POST", |
| 241 | Path: "/api/v1/vms/{id}/exposures", | 241 | Path: "/api/v1/vms/{id}/exposures", |
| @@ -244,7 +244,7 @@ var routeTable = []Route{ | |||
| 244 | Request: (*types.CreateExposureRequest)(nil), | 244 | Request: (*types.CreateExposureRequest)(nil), |
| 245 | Response: (*types.Exposure)(nil), | 245 | Response: (*types.Exposure)(nil), |
| 246 | Success: http.StatusCreated, | 246 | Success: http.StatusCreated, |
| 247 | Doc: "Publish a guest TCP port on the VM's host. Omit host_port to allocate one from the reserved range 30000-32767; a named port must be >= 1024 and is honored or refused.", | 247 | Doc: "Publish a guest port on the VM's host, protocol \"tcp\" (the default) or \"udp\". Omit host_port to allocate one from the reserved range 30000-32767; a named port must be >= 1024 and is honored or refused, and is taken only by another exposure of the same protocol.", |
| 248 | handler: (*API).handleCreateExposure, | 248 | handler: (*API).handleCreateExposure, |
| 249 | }, | 249 | }, |
| 250 | { | 250 | { |
internal/server/api/testdata/create-exposure-request.golden.json
| Old | New | ||
|---|---|---|---|
| @@ -1,4 +1,5 @@ | |||
| 1 | { | 1 | { |
| 2 | "guest_port": 8080, | 2 | "guest_port": 8080, |
| 3 | "host_port": 30080 | 3 | "host_port": 30080, |
| 4 | "protocol": "udp" | ||
| 4 | } | 5 | } |
internal/server/api/types/types.go
| Old | New | ||
|---|---|---|---|
| @@ -229,17 +229,20 @@ type RevokeSSHCertRequest struct { | |||
| 229 | // CreateExposureRequest is the POST /api/v1/vms/{id}/exposures body: publish | 229 | // CreateExposureRequest is the POST /api/v1/vms/{id}/exposures body: publish |
| 230 | // guest_port of that VM. host_port is optional — omitted (or 0) allocates one | 230 | // guest_port of that VM. host_port is optional — omitted (or 0) allocates one |
| 231 | // from the reserved range 30000-32767; a named port must be >= 1024 and is | 231 | // from the reserved range 30000-32767; a named port must be >= 1024 and is |
| 232 | // honored or refused. Request-side only. | 232 | // honored or refused. protocol is "tcp" (the default when omitted) or "udp", |
| 233 | // and a host port is claimed per protocol, so the same number can carry one of | ||
| 234 | // each. Request-side only. | ||
| 233 | type CreateExposureRequest struct { | 235 | type CreateExposureRequest struct { |
| 234 | GuestPort int64 `json:"guest_port"` | 236 | GuestPort int64 `json:"guest_port"` |
| 235 | HostPort int64 `json:"host_port"` // 0 = allocate | 237 | HostPort int64 `json:"host_port"` // 0 = allocate |
| 238 | Protocol string `json:"protocol"` // "" = tcp | ||
| 236 | } | 239 | } |
| 237 | 240 | ||
| 238 | // Exposure is one published guest port, served by | 241 | // Exposure is one published guest port, served by |
| 239 | // POST and GET /api/v1/vms/{id}/exposures. HostAddr and State are | 242 | // POST and GET /api/v1/vms/{id}/exposures. HostAddr and State are |
| 240 | // server-derived: the address comes from the host's own report, and the state | 243 | // server-derived: the address comes from the host's own report, and the state |
| 241 | // from what its agent last said its listener is doing — "pending" until an | 244 | // from what its agent last said its socket is doing — "pending" until an |
| 242 | // agent has reported at all, then "active" (the listener is bound) or "failed" | 245 | // agent has reported at all, then "active" (the socket is bound) or "failed" |
| 243 | // with the OS error in Reason. "active" describes the host half of the pipe; | 246 | // with the OS error in Reason. "active" describes the host half of the pipe; |
| 244 | // whether anything answers inside the guest is the guest's business. | 247 | // whether anything answers inside the guest is the guest's business. |
| 245 | type Exposure struct { | 248 | type Exposure struct { |
internal/server/api/wire_golden_test.go
| Old | New | ||
|---|---|---|---|
| @@ -260,6 +260,7 @@ func TestWireGolden(t *testing.T) { | |||
| 260 | goldenCheck(t, "create-exposure-request", types.CreateExposureRequest{ | 260 | goldenCheck(t, "create-exposure-request", types.CreateExposureRequest{ |
| 261 | GuestPort: 8080, | 261 | GuestPort: 8080, |
| 262 | HostPort: 30080, | 262 | HostPort: 30080, |
| 263 | Protocol: "udp", | ||
| 263 | }) | 264 | }) |
| 264 | 265 | ||
| 265 | goldenCheck(t, "exposure", []types.Exposure{{ | 266 | goldenCheck(t, "exposure", []types.Exposure{{ |
internal/server/store/evolve.go
| Old | New | ||
|---|---|---|---|
| @@ -44,6 +44,21 @@ func dropTable(db *sql.DB, table string) error { | |||
| 44 | return nil | 44 | return nil |
| 45 | } | 45 | } |
| 46 | 46 | ||
| 47 | // dropIndex removes an index the schema no longer declares. An index whose | ||
| 48 | // columns change is a new index under a new name plus this: SQLite's CREATE | ||
| 49 | // UNIQUE INDEX IF NOT EXISTS leaves an existing index of that name exactly as | ||
| 50 | // it was, so a redefinition under the old name would be silently ignored on | ||
| 51 | // every database that already had one. | ||
| 52 | // | ||
| 53 | // index is interpolated verbatim (SQLite cannot bind identifiers): pass trusted | ||
| 54 | // compile-time constants only, the same rule ensureColumn states. | ||
| 55 | func dropIndex(db *sql.DB, index string) error { | ||
| 56 | if _, err := db.Exec(fmt.Sprintf(`DROP INDEX IF EXISTS %s`, index)); err != nil { | ||
| 57 | return fmt.Errorf("drop index %s: %w", index, err) | ||
| 58 | } | ||
| 59 | return nil | ||
| 60 | } | ||
| 61 | |||
| 47 | // dropColumn removes a column that is no longer part of the schema, so an | 62 | // dropColumn removes a column that is no longer part of the schema, so an |
| 48 | // existing database stops carrying it — and stops carrying whatever was in it. | 63 | // existing database stops carrying it — and stops carrying whatever was in it. |
| 49 | // Idempotent: a database that never had the column, or has already dropped it, | 64 | // Idempotent: a database that never had the column, or has already dropped it, |
internal/server/store/exposures.go
| Old | New | ||
|---|---|---|---|
| @@ -11,10 +11,10 @@ import ( | |||
| 11 | ) | 11 | ) |
| 12 | 12 | ||
| 13 | // Exposure is one published guest port: the fleet binds HostPort on the VM's | 13 | // Exposure is one published guest port: the fleet binds HostPort on the VM's |
| 14 | // host and pipes every accepted connection to GuestPort inside the guest. | 14 | // host and pipes every accepted connection — or every datagram, when Protocol |
| 15 | // Tenant and HostID are both derived from the VM at create — never accepted | 15 | // is "udp" — to GuestPort inside the guest. Tenant and HostID are both derived |
| 16 | // from a caller — so an exposure can only ever name the partition and the | 16 | // from the VM at create — never accepted from a caller — so an exposure can |
| 17 | // machine its VM already lives in. | 17 | // only ever name the partition and the machine its VM already lives in. |
| 18 | type Exposure struct { | 18 | type Exposure struct { |
| 19 | ID, Tenant, VMID, HostID string | 19 | ID, Tenant, VMID, HostID string |
| 20 | GuestPort, HostPort int64 | 20 | GuestPort, HostPort int64 |
| @@ -36,7 +36,7 @@ const ( | |||
| 36 | var ErrExposureVMNotFound = errors.New("vm not found") | 36 | var ErrExposureVMNotFound = errors.New("vm not found") |
| 37 | 37 | ||
| 38 | // ErrHostPortTaken reports that another exposure already holds the requested | 38 | // ErrHostPortTaken reports that another exposure already holds the requested |
| 39 | // host port on that host. | 39 | // host port on that host, for the same protocol. |
| 40 | var ErrHostPortTaken = errors.New("host port already exposed on this host") | 40 | var ErrHostPortTaken = errors.New("host port already exposed on this host") |
| 41 | 41 | ||
| 42 | // ErrNoFreeHostPort reports that the reserved range is fully allocated on the | 42 | // ErrNoFreeHostPort reports that the reserved range is fully allocated on the |
| @@ -82,11 +82,15 @@ func queryExposures(q querier, from string, args ...any) ([]Exposure, error) { | |||
| 82 | return out, rows.Err() | 82 | return out, rows.Err() |
| 83 | } | 83 | } |
| 84 | 84 | ||
| 85 | // CreateExposure publishes guestPort of vmID on that VM's host. hostPort 0 asks | 85 | // CreateExposure publishes guestPort of vmID on that VM's host, for protocol |
| 86 | // for one from the reserved range; a named port is honored or refused. The | 86 | // ("tcp" or "udp" — the API is what holds callers to those two). hostPort 0 |
| 87 | // whole decision — which host, which tenant, which port — happens in one | 87 | // asks for one from the reserved range; a named port is honored or refused, |
| 88 | // transaction, so two concurrent creates cannot agree on the same port. | 88 | // and it is only taken when the SAME protocol already holds it: TCP 30000 and |
| 89 | func (s *Store) CreateExposure(vmID string, guestPort, hostPort int64) (Exposure, error) { | 89 | // UDP 30000 are two ports on the host, and refusing the second would be the |
| 90 | // proxy inventing a scarcity the machine does not have. The whole decision — | ||
| 91 | // which host, which tenant, which port — happens in one transaction, so two | ||
| 92 | // concurrent creates cannot agree on the same port. | ||
| 93 | func (s *Store) CreateExposure(vmID string, guestPort, hostPort int64, protocol string) (Exposure, error) { | ||
| 90 | tx, err := s.db.Begin() | 94 | tx, err := s.db.Begin() |
| 91 | if err != nil { | 95 | if err != nil { |
| 92 | return Exposure{}, err | 96 | return Exposure{}, err |
| @@ -116,7 +120,7 @@ func (s *Store) CreateExposure(vmID string, guestPort, hostPort int64) (Exposure | |||
| 116 | e := Exposure{ | 120 | e := Exposure{ |
| 117 | ID: random.Hex(16), Tenant: tenant, VMID: vmID, HostID: hostID, | 121 | ID: random.Hex(16), Tenant: tenant, VMID: vmID, HostID: hostID, |
| 118 | GuestPort: guestPort, HostPort: hostPort, | 122 | GuestPort: guestPort, HostPort: hostPort, |
| 119 | Protocol: "tcp", Scope: "lan", CreatedAt: time.Now().UTC(), | 123 | Protocol: protocol, Scope: "lan", CreatedAt: time.Now().UTC(), |
| 120 | } | 124 | } |
| 121 | if _, err := tx.Exec( | 125 | if _, err := tx.Exec( |
| 122 | `INSERT INTO exposures(id, tenant, vm_id, host_id, guest_port, host_port, protocol, scope, created_at) | 126 | `INSERT INTO exposures(id, tenant, vm_id, host_id, guest_port, host_port, protocol, scope, created_at) |
| @@ -125,7 +129,7 @@ func (s *Store) CreateExposure(vmID string, guestPort, hostPort int64) (Exposure | |||
| 125 | e.CreatedAt.Format(time.RFC3339), | 129 | e.CreatedAt.Format(time.RFC3339), |
| 126 | ); err != nil { | 130 | ); err != nil { |
| 127 | // SQLITE_CONSTRAINT_UNIQUE (2067): the only UNIQUE constraint this | 131 | // SQLITE_CONSTRAINT_UNIQUE (2067): the only UNIQUE constraint this |
| 128 | // insert can trip besides the random-hex PK is exposures_host_port. | 132 | // insert can trip besides the random-hex PK is exposures_host_port_proto. |
| 129 | // Matched by errno, not message text, which embeds the index's column | 133 | // Matched by errno, not message text, which embeds the index's column |
| 130 | // list and breaks on the next index change. | 134 | // list and breaks on the next index change. |
| 131 | if serr, ok := errors.AsType[*sqlite.Error](err); ok && serr.Code() == 2067 { | 135 | if serr, ok := errors.AsType[*sqlite.Error](err); ok && serr.Code() == 2067 { |
| @@ -143,10 +147,17 @@ func (s *Store) CreateExposure(vmID string, guestPort, hostPort int64) (Exposure | |||
| 143 | return e, nil | 147 | return e, nil |
| 144 | } | 148 | } |
| 145 | 149 | ||
| 146 | // allocateHostPort returns the lowest free port of the reserved range on | 150 | // allocateHostPort returns the lowest port of the reserved range that no |
| 147 | // hostID. It reads the range's held ports into memory before choosing, because | 151 | // exposure on hostID holds — in EITHER protocol. A caller that named nothing |
| 148 | // the store runs on a single connection: an open result set would block the | 152 | // gets a number it can hand to anyone without a protocol beside it, and the |
| 149 | // insert that follows in the same transaction. | 153 | // range is 2768 ports deep on a host that runs dozens of guests, so the ports |
| 154 | // this leaves on the table cost nothing that reasoning about half-taken | ||
| 155 | // numbers would not cost more. Naming a port explicitly still gets the full | ||
| 156 | // per-protocol answer. | ||
| 157 | // | ||
| 158 | // It reads the range's held ports into memory before choosing, because the | ||
| 159 | // store runs on a single connection: an open result set would block the insert | ||
| 160 | // that follows in the same transaction. | ||
| 150 | func allocateHostPort(tx *sql.Tx, hostID string) (int64, error) { | 161 | func allocateHostPort(tx *sql.Tx, hostID string) (int64, error) { |
| 151 | rows, err := tx.Query( | 162 | rows, err := tx.Query( |
| 152 | `SELECT host_port FROM exposures WHERE host_id=? AND host_port BETWEEN ? AND ?`, | 163 | `SELECT host_port FROM exposures WHERE host_id=? AND host_port BETWEEN ? AND ?`, |
internal/server/store/exposures_test.go
| Old | New | ||
|---|---|---|---|
| @@ -21,7 +21,7 @@ func TestCreateExposureAllocatesFromTheReservedRange(t *testing.T) { | |||
| 21 | h := enrollHost(t, s) | 21 | h := enrollHost(t, s) |
| 22 | vm := makeExposureVM(t, s, h, "web-1") | 22 | vm := makeExposureVM(t, s, h, "web-1") |
| 23 | 23 | ||
| 24 | e, err := s.CreateExposure(vm.ID, 8080, 0) | 24 | e, err := s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 25 | require.NoError(t, err) | 25 | require.NoError(t, err) |
| 26 | assert.Equal(t, vm.ID, e.VMID) | 26 | assert.Equal(t, vm.ID, e.VMID) |
| 27 | assert.Equal(t, h.ID, e.HostID) | 27 | assert.Equal(t, h.ID, e.HostID) |
| @@ -38,16 +38,16 @@ func TestCreateExposureAllocatesTheLowestFreePort(t *testing.T) { | |||
| 38 | h := enrollHost(t, s) | 38 | h := enrollHost(t, s) |
| 39 | vm := makeExposureVM(t, s, h, "web-1") | 39 | vm := makeExposureVM(t, s, h, "web-1") |
| 40 | 40 | ||
| 41 | first, err := s.CreateExposure(vm.ID, 8080, 0) | 41 | first, err := s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 42 | require.NoError(t, err) | 42 | require.NoError(t, err) |
| 43 | second, err := s.CreateExposure(vm.ID, 8081, 0) | 43 | second, err := s.CreateExposure(vm.ID, 8081, 0, "tcp") |
| 44 | require.NoError(t, err) | 44 | require.NoError(t, err) |
| 45 | assert.Equal(t, int64(MinAllocatedHostPort), first.HostPort) | 45 | assert.Equal(t, int64(MinAllocatedHostPort), first.HostPort) |
| 46 | assert.Equal(t, int64(MinAllocatedHostPort+1), second.HostPort) | 46 | assert.Equal(t, int64(MinAllocatedHostPort+1), second.HostPort) |
| 47 | 47 | ||
| 48 | // A freed port is the lowest gap, so it is handed out again. | 48 | // A freed port is the lowest gap, so it is handed out again. |
| 49 | require.NoError(t, s.DeleteExposure(first.ID)) | 49 | require.NoError(t, s.DeleteExposure(first.ID)) |
| 50 | third, err := s.CreateExposure(vm.ID, 8082, 0) | 50 | third, err := s.CreateExposure(vm.ID, 8082, 0, "tcp") |
| 51 | require.NoError(t, err) | 51 | require.NoError(t, err) |
| 52 | assert.Equal(t, int64(MinAllocatedHostPort), third.HostPort) | 52 | assert.Equal(t, int64(MinAllocatedHostPort), third.HostPort) |
| 53 | } | 53 | } |
| @@ -57,7 +57,7 @@ func TestCreateExposureHonoursARequestedPort(t *testing.T) { | |||
| 57 | h := enrollHost(t, s) | 57 | h := enrollHost(t, s) |
| 58 | vm := makeExposureVM(t, s, h, "web-1") | 58 | vm := makeExposureVM(t, s, h, "web-1") |
| 59 | 59 | ||
| 60 | e, err := s.CreateExposure(vm.ID, 8080, 8443) | 60 | e, err := s.CreateExposure(vm.ID, 8080, 8443, "tcp") |
| 61 | require.NoError(t, err) | 61 | require.NoError(t, err) |
| 62 | assert.Equal(t, int64(8443), e.HostPort) | 62 | assert.Equal(t, int64(8443), e.HostPort) |
| 63 | } | 63 | } |
| @@ -68,13 +68,29 @@ func TestCreateExposureRefusesAPortAnotherExposureHolds(t *testing.T) { | |||
| 68 | vm := makeExposureVM(t, s, h, "web-1") | 68 | vm := makeExposureVM(t, s, h, "web-1") |
| 69 | other := makeExposureVM(t, s, h, "web-2") | 69 | other := makeExposureVM(t, s, h, "web-2") |
| 70 | 70 | ||
| 71 | _, err := s.CreateExposure(vm.ID, 8080, 8443) | 71 | _, err := s.CreateExposure(vm.ID, 8080, 8443, "tcp") |
| 72 | require.NoError(t, err) | 72 | require.NoError(t, err) |
| 73 | 73 | ||
| 74 | _, err = s.CreateExposure(other.ID, 9090, 8443) | 74 | _, err = s.CreateExposure(other.ID, 9090, 8443, "tcp") |
| 75 | assert.ErrorIs(t, err, ErrHostPortTaken, "the unique index IS the collision check") | 75 | assert.ErrorIs(t, err, ErrHostPortTaken, "the unique index IS the collision check") |
| 76 | } | 76 | } |
| 77 | 77 | ||
| 78 | func TestCreateExposureAllowsTheSamePortInTheOtherProtocol(t *testing.T) { | ||
| 79 | s := newStore(t) | ||
| 80 | h := enrollHost(t, s) | ||
| 81 | vm := makeExposureVM(t, s, h, "web-1") | ||
| 82 | |||
| 83 | _, err := s.CreateExposure(vm.ID, 8080, 8443, "tcp") | ||
| 84 | require.NoError(t, err) | ||
| 85 | |||
| 86 | udp, err := s.CreateExposure(vm.ID, 8080, 8443, "udp") | ||
| 87 | require.NoError(t, err, "a host port is claimed per protocol") | ||
| 88 | assert.Equal(t, "udp", udp.Protocol) | ||
| 89 | |||
| 90 | _, err = s.CreateExposure(vm.ID, 9090, 8443, "udp") | ||
| 91 | assert.ErrorIs(t, err, ErrHostPortTaken, "the second UDP claim on that port is a collision") | ||
| 92 | } | ||
| 93 | |||
| 78 | func TestCreateExposureAllowsTheSamePortOnAnotherHost(t *testing.T) { | 94 | func TestCreateExposureAllowsTheSamePortOnAnotherHost(t *testing.T) { |
| 79 | s := newStore(t) | 95 | s := newStore(t) |
| 80 | h1 := enrollHost(t, s) | 96 | h1 := enrollHost(t, s) |
| @@ -86,9 +102,9 @@ func TestCreateExposureAllowsTheSamePortOnAnotherHost(t *testing.T) { | |||
| 86 | vm1 := makeExposureVM(t, s, h1, "web-1") | 102 | vm1 := makeExposureVM(t, s, h1, "web-1") |
| 87 | vm2 := makeExposureVM(t, s, h2, "web-2") | 103 | vm2 := makeExposureVM(t, s, h2, "web-2") |
| 88 | 104 | ||
| 89 | _, err = s.CreateExposure(vm1.ID, 8080, 8443) | 105 | _, err = s.CreateExposure(vm1.ID, 8080, 8443, "tcp") |
| 90 | require.NoError(t, err) | 106 | require.NoError(t, err) |
| 91 | _, err = s.CreateExposure(vm2.ID, 8080, 8443) | 107 | _, err = s.CreateExposure(vm2.ID, 8080, 8443, "tcp") |
| 92 | assert.NoError(t, err, "the port is unique per host, not per fleet") | 108 | assert.NoError(t, err, "the port is unique per host, not per fleet") |
| 93 | } | 109 | } |
| 94 | 110 | ||
| @@ -97,11 +113,11 @@ func TestCreateExposureRefusesAnUnknownOrTombstonedVM(t *testing.T) { | |||
| 97 | h := enrollHost(t, s) | 113 | h := enrollHost(t, s) |
| 98 | vm := makeExposureVM(t, s, h, "web-1") | 114 | vm := makeExposureVM(t, s, h, "web-1") |
| 99 | 115 | ||
| 100 | _, err := s.CreateExposure("no-such-vm", 8080, 0) | 116 | _, err := s.CreateExposure("no-such-vm", 8080, 0, "tcp") |
| 101 | assert.ErrorIs(t, err, ErrExposureVMNotFound) | 117 | assert.ErrorIs(t, err, ErrExposureVMNotFound) |
| 102 | 118 | ||
| 103 | require.NoError(t, s.TombstoneVM(vm.ID)) | 119 | require.NoError(t, s.TombstoneVM(vm.ID)) |
| 104 | _, err = s.CreateExposure(vm.ID, 8080, 0) | 120 | _, err = s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 105 | assert.ErrorIs(t, err, ErrExposureVMNotFound, "a VM being torn down takes no new exposures") | 121 | assert.ErrorIs(t, err, ErrExposureVMNotFound, "a VM being torn down takes no new exposures") |
| 106 | } | 122 | } |
| 107 | 123 | ||
| @@ -120,7 +136,7 @@ func TestCreateExposureReportsRangeExhaustion(t *testing.T) { | |||
| 120 | require.NoError(t, err) | 136 | require.NoError(t, err) |
| 121 | } | 137 | } |
| 122 | 138 | ||
| 123 | _, err := s.CreateExposure(vm.ID, 8080, 0) | 139 | _, err := s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 124 | assert.ErrorIs(t, err, ErrNoFreeHostPort) | 140 | assert.ErrorIs(t, err, ErrNoFreeHostPort) |
| 125 | } | 141 | } |
| 126 | 142 | ||
| @@ -131,7 +147,7 @@ func TestCreateExposureBumpsTheEpoch(t *testing.T) { | |||
| 131 | 147 | ||
| 132 | before, err := s.Epoch() | 148 | before, err := s.Epoch() |
| 133 | require.NoError(t, err) | 149 | require.NoError(t, err) |
| 134 | _, err = s.CreateExposure(vm.ID, 8080, 0) | 150 | _, err = s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 135 | require.NoError(t, err) | 151 | require.NoError(t, err) |
| 136 | after, err := s.Epoch() | 152 | after, err := s.Epoch() |
| 137 | require.NoError(t, err) | 153 | require.NoError(t, err) |
| @@ -143,26 +159,41 @@ func TestCreateExposureCountsANamedPortInTheReservedRange(t *testing.T) { | |||
| 143 | h := enrollHost(t, s) | 159 | h := enrollHost(t, s) |
| 144 | vm := makeExposureVM(t, s, h, "web-1") | 160 | vm := makeExposureVM(t, s, h, "web-1") |
| 145 | 161 | ||
| 146 | named, err := s.CreateExposure(vm.ID, 8080, MinAllocatedHostPort) | 162 | named, err := s.CreateExposure(vm.ID, 8080, MinAllocatedHostPort, "tcp") |
| 147 | require.NoError(t, err) | 163 | require.NoError(t, err) |
| 148 | assert.Equal(t, int64(MinAllocatedHostPort), named.HostPort) | 164 | assert.Equal(t, int64(MinAllocatedHostPort), named.HostPort) |
| 149 | 165 | ||
| 150 | auto, err := s.CreateExposure(vm.ID, 8081, 0) | 166 | auto, err := s.CreateExposure(vm.ID, 8081, 0, "tcp") |
| 151 | require.NoError(t, err) | 167 | require.NoError(t, err) |
| 152 | assert.Equal(t, int64(MinAllocatedHostPort+1), auto.HostPort, "a named port inside the reserved range is held against auto-allocation") | 168 | assert.Equal(t, int64(MinAllocatedHostPort+1), auto.HostPort, "a named port inside the reserved range is held against auto-allocation") |
| 153 | } | 169 | } |
| 154 | 170 | ||
| 171 | func TestAllocationSkipsAPortHeldByEitherProtocol(t *testing.T) { | ||
| 172 | s := newStore(t) | ||
| 173 | h := enrollHost(t, s) | ||
| 174 | vm := makeExposureVM(t, s, h, "web-1") | ||
| 175 | |||
| 176 | _, err := s.CreateExposure(vm.ID, 8080, MinAllocatedHostPort, "udp") | ||
| 177 | require.NoError(t, err) | ||
| 178 | |||
| 179 | // The allocator hands out a number a caller can name without a protocol | ||
| 180 | // beside it, so the UDP claim takes the port out of the running for TCP too. | ||
| 181 | auto, err := s.CreateExposure(vm.ID, 8081, 0, "tcp") | ||
| 182 | require.NoError(t, err) | ||
| 183 | assert.Equal(t, int64(MinAllocatedHostPort+1), auto.HostPort) | ||
| 184 | } | ||
| 185 | |||
| 155 | func TestListExposuresForVMIsPortOrdered(t *testing.T) { | 186 | func TestListExposuresForVMIsPortOrdered(t *testing.T) { |
| 156 | s := newStore(t) | 187 | s := newStore(t) |
| 157 | h := enrollHost(t, s) | 188 | h := enrollHost(t, s) |
| 158 | vm := makeExposureVM(t, s, h, "web-1") | 189 | vm := makeExposureVM(t, s, h, "web-1") |
| 159 | other := makeExposureVM(t, s, h, "web-2") | 190 | other := makeExposureVM(t, s, h, "web-2") |
| 160 | 191 | ||
| 161 | _, err := s.CreateExposure(vm.ID, 8081, 31000) | 192 | _, err := s.CreateExposure(vm.ID, 8081, 31000, "tcp") |
| 162 | require.NoError(t, err) | 193 | require.NoError(t, err) |
| 163 | _, err = s.CreateExposure(vm.ID, 8080, 30500) | 194 | _, err = s.CreateExposure(vm.ID, 8080, 30500, "tcp") |
| 164 | require.NoError(t, err) | 195 | require.NoError(t, err) |
| 165 | _, err = s.CreateExposure(other.ID, 9090, 30001) | 196 | _, err = s.CreateExposure(other.ID, 9090, 30001, "tcp") |
| 166 | require.NoError(t, err) | 197 | require.NoError(t, err) |
| 167 | 198 | ||
| 168 | got, err := s.ListExposuresForVM(vm.ID) | 199 | got, err := s.ListExposuresForVM(vm.ID) |
| @@ -178,9 +209,9 @@ func TestListExposuresForHostSkipsTombstonedVMs(t *testing.T) { | |||
| 178 | live := makeExposureVM(t, s, h, "web-1") | 209 | live := makeExposureVM(t, s, h, "web-1") |
| 179 | dying := makeExposureVM(t, s, h, "web-2") | 210 | dying := makeExposureVM(t, s, h, "web-2") |
| 180 | 211 | ||
| 181 | _, err := s.CreateExposure(live.ID, 8080, 30001) | 212 | _, err := s.CreateExposure(live.ID, 8080, 30001, "tcp") |
| 182 | require.NoError(t, err) | 213 | require.NoError(t, err) |
| 183 | _, err = s.CreateExposure(dying.ID, 8080, 30002) | 214 | _, err = s.CreateExposure(dying.ID, 8080, 30002, "tcp") |
| 184 | require.NoError(t, err) | 215 | require.NoError(t, err) |
| 185 | 216 | ||
| 186 | require.NoError(t, s.TombstoneVM(dying.ID)) | 217 | require.NoError(t, s.TombstoneVM(dying.ID)) |
| @@ -196,7 +227,7 @@ func TestGetExposureAndDelete(t *testing.T) { | |||
| 196 | h := enrollHost(t, s) | 227 | h := enrollHost(t, s) |
| 197 | vm := makeExposureVM(t, s, h, "web-1") | 228 | vm := makeExposureVM(t, s, h, "web-1") |
| 198 | 229 | ||
| 199 | e, err := s.CreateExposure(vm.ID, 8080, 0) | 230 | e, err := s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 200 | require.NoError(t, err) | 231 | require.NoError(t, err) |
| 201 | 232 | ||
| 202 | got, err := s.GetExposure(e.ID) | 233 | got, err := s.GetExposure(e.ID) |
| @@ -216,7 +247,7 @@ func TestDeleteExposureBumpsTheEpoch(t *testing.T) { | |||
| 216 | s := newStore(t) | 247 | s := newStore(t) |
| 217 | h := enrollHost(t, s) | 248 | h := enrollHost(t, s) |
| 218 | vm := makeExposureVM(t, s, h, "web-1") | 249 | vm := makeExposureVM(t, s, h, "web-1") |
| 219 | e, err := s.CreateExposure(vm.ID, 8080, 0) | 250 | e, err := s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 220 | require.NoError(t, err) | 251 | require.NoError(t, err) |
| 221 | 252 | ||
| 222 | before, err := s.Epoch() | 253 | before, err := s.Epoch() |
| @@ -231,7 +262,7 @@ func TestReapingAVMDestroysItsExposures(t *testing.T) { | |||
| 231 | s := newStore(t) | 262 | s := newStore(t) |
| 232 | h := enrollHost(t, s) | 263 | h := enrollHost(t, s) |
| 233 | vm := makeExposureVM(t, s, h, "web-1") | 264 | vm := makeExposureVM(t, s, h, "web-1") |
| 234 | e, err := s.CreateExposure(vm.ID, 8080, 0) | 265 | e, err := s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 235 | require.NoError(t, err) | 266 | require.NoError(t, err) |
| 236 | 267 | ||
| 237 | require.NoError(t, s.TombstoneVM(vm.ID)) | 268 | require.NoError(t, s.TombstoneVM(vm.ID)) |
| @@ -245,7 +276,7 @@ func TestForceRemovingAHostDestroysItsExposures(t *testing.T) { | |||
| 245 | s := newStore(t) | 276 | s := newStore(t) |
| 246 | h := enrollHost(t, s) | 277 | h := enrollHost(t, s) |
| 247 | vm := makeExposureVM(t, s, h, "web-1") | 278 | vm := makeExposureVM(t, s, h, "web-1") |
| 248 | e, err := s.CreateExposure(vm.ID, 8080, 0) | 279 | e, err := s.CreateExposure(vm.ID, 8080, 0, "tcp") |
| 249 | require.NoError(t, err) | 280 | require.NoError(t, err) |
| 250 | 281 | ||
| 251 | purged, err := s.ForceRemoveHost(h.ID) | 282 | purged, err := s.ForceRemoveHost(h.ID) |
internal/server/store/store.go
| Old | New | ||
|---|---|---|---|
| @@ -214,8 +214,9 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_tenant_user_cas_pubkey ON tenant_user_cas( | |||
| 214 | -- whoever inserts first, and the second insert fails at the storage layer | 214 | -- whoever inserts first, and the second insert fails at the storage layer |
| 215 | -- rather than after a read-then-write race. | 215 | -- rather than after a read-then-write race. |
| 216 | -- | 216 | -- |
| 217 | -- protocol is 'tcp' and scope is 'lan', both stored so 'udp' and 'public' are | 217 | -- protocol is 'tcp' or 'udp' — two different ports on the same number, which |
| 218 | -- additive values later rather than a migration. | 218 | -- is why it is part of the unique index. scope is 'lan', stored so 'public' is |
| 219 | -- an additive value later rather than a migration. | ||
| 219 | -- | 220 | -- |
| 220 | -- ON DELETE CASCADE against vms is what keeps the promise that no listener | 221 | -- ON DELETE CASCADE against vms is what keeps the promise that no listener |
| 221 | -- outlives the thing it pointed at: hard-deleting a reaped VM takes its | 222 | -- outlives the thing it pointed at: hard-deleting a reaped VM takes its |
| @@ -231,7 +232,7 @@ CREATE TABLE IF NOT EXISTS exposures ( | |||
| 231 | scope TEXT NOT NULL DEFAULT 'lan', | 232 | scope TEXT NOT NULL DEFAULT 'lan', |
| 232 | created_at DATETIME NOT NULL | 233 | created_at DATETIME NOT NULL |
| 233 | ); | 234 | ); |
| 234 | CREATE UNIQUE INDEX IF NOT EXISTS exposures_host_port ON exposures(host_id, host_port); | 235 | CREATE UNIQUE INDEX IF NOT EXISTS exposures_host_port_proto ON exposures(host_id, host_port, protocol); |
| 235 | 236 | ||
| 236 | -- Console sessions. Server-side so revocation works and restarts keep | 237 | -- Console sessions. Server-side so revocation works and restarts keep |
| 237 | -- users signed in. id is 256-bit random hex; expiry enforced on read. | 238 | -- users signed in. id is 256-bit random hex; expiry enforced on read. |
| @@ -344,6 +345,15 @@ func Open(path, cidrPool string) (*Store, error) { | |||
| 344 | return nil, err | 345 | return nil, err |
| 345 | } | 346 | } |
| 346 | 347 | ||
| 348 | // A host port is claimed per protocol, so TCP 30000 and UDP 30000 are two | ||
| 349 | // grants rather than a collision. The index that spanned only (host, port) | ||
| 350 | // would refuse the second one, so it goes — exposures_host_port_proto above | ||
| 351 | // has already replaced it by the time this runs. | ||
| 352 | if err := dropIndex(db, "exposures_host_port"); err != nil { | ||
| 353 | db.Close() | ||
| 354 | return nil, err | ||
| 355 | } | ||
| 356 | |||
| 347 | // One identity binds at most one tenant (per issuer). Partial index so | 357 | // One identity binds at most one tenant (per issuer). Partial index so |
| 348 | // unbound rows (empty issuer+subject) don't collide. | 358 | // unbound rows (empty issuer+subject) don't collide. |
| 349 | if _, err := db.Exec(`CREATE UNIQUE INDEX IF NOT EXISTS tenants_identity | 359 | if _, err := db.Exec(`CREATE UNIQUE INDEX IF NOT EXISTS tenants_identity |
internal/server/store/store_test.go
| Old | New | ||
|---|---|---|---|
| @@ -165,7 +165,7 @@ func TestOpenDropsTheEscrowedHostKeyColumn(t *testing.T) { | |||
| 165 | require.NoError(t, err) | 165 | require.NoError(t, err) |
| 166 | // An exposure, because vms is referenced ON DELETE CASCADE: a migration that | 166 | // An exposure, because vms is referenced ON DELETE CASCADE: a migration that |
| 167 | // rebuilt the table by dropping it would take this row with it. | 167 | // rebuilt the table by dropping it would take this row with it. |
| 168 | _, err = s.CreateExposure("vm1", 22, 30001) | 168 | _, err = s.CreateExposure("vm1", 22, 30001, "tcp") |
| 169 | require.NoError(t, err) | 169 | require.NoError(t, err) |
| 170 | require.NoError(t, s.Close()) | 170 | require.NoError(t, s.Close()) |
| 171 | 171 | ||
| @@ -198,6 +198,43 @@ func TestOpenDropsTheEscrowedHostKeyColumn(t *testing.T) { | |||
| 198 | assert.Equal(t, "cert-line", vm.SSHHostCert) | 198 | assert.Equal(t, "cert-line", vm.SSHHostCert) |
| 199 | } | 199 | } |
| 200 | 200 | ||
| 201 | func TestOpenReplacesTheProtocolBlindHostPortIndex(t *testing.T) { | ||
| 202 | path := t.TempDir() + "/eitri.db" | ||
| 203 | s, err := Open(path, "10.77.0.0/16") | ||
| 204 | require.NoError(t, err) | ||
| 205 | _, err = s.CreateTenantForIdentity("https://test-issuer", "test-subject", testTenant+"@test.local") | ||
| 206 | require.NoError(t, err) | ||
| 207 | h := enrollHost(t, s) | ||
| 208 | require.NoError(t, s.CreateVM(VM{ | ||
| 209 | ID: "vm1", HostID: h.ID, Name: "legacy", ImageURL: "u", ImageSHA256: "abc", | ||
| 210 | VCPUs: 1, MemMB: 512, DiskGB: 5, PowerState: "running", | ||
| 211 | })) | ||
| 212 | |||
| 213 | // Put the database back into the shape an earlier release left it: one host | ||
| 214 | // port, one exposure, whatever the protocol. | ||
| 215 | _, err = s.db.Exec(`DROP INDEX exposures_host_port_proto`) | ||
| 216 | require.NoError(t, err) | ||
| 217 | _, err = s.db.Exec(`CREATE UNIQUE INDEX exposures_host_port ON exposures(host_id, host_port)`) | ||
| 218 | require.NoError(t, err) | ||
| 219 | _, err = s.CreateExposure("vm1", 22, 30001, "tcp") | ||
| 220 | require.NoError(t, err) | ||
| 221 | require.NoError(t, s.Close()) | ||
| 222 | |||
| 223 | up, err := Open(path, "10.77.0.0/16") | ||
| 224 | require.NoError(t, err) | ||
| 225 | t.Cleanup(func() { up.Close() }) | ||
| 226 | |||
| 227 | var n int | ||
| 228 | require.NoError(t, up.db.QueryRow( | ||
| 229 | `SELECT count(*) FROM sqlite_master WHERE type='index' AND name='exposures_host_port'`).Scan(&n)) | ||
| 230 | assert.Equal(t, 0, n, "the index that spanned only (host, port) must be gone") | ||
| 231 | |||
| 232 | _, err = up.CreateExposure("vm1", 53, 30001, "udp") | ||
| 233 | assert.NoError(t, err, "the same host port in the other protocol is a second grant") | ||
| 234 | _, err = up.CreateExposure("vm1", 54, 30001, "udp") | ||
| 235 | assert.ErrorIs(t, err, ErrHostPortTaken, "the new index still holds the pair unique") | ||
| 236 | } | ||
| 237 | |||
| 201 | func TestSSHCertRevocation(t *testing.T) { | 238 | func TestSSHCertRevocation(t *testing.T) { |
| 202 | s := newStore(t) | 239 | s := newStore(t) |
| 203 | 240 | ||
internal/server/syncsvc/exposures_test.go
| Old | New | ||
|---|---|---|---|
| @@ -24,7 +24,7 @@ func TestSnapshotCarriesExposures(t *testing.T) { | |||
| 24 | f := setup(t) | 24 | f := setup(t) |
| 25 | exposureVM(t, f, "vm1", "web-1") | 25 | exposureVM(t, f, "vm1", "web-1") |
| 26 | 26 | ||
| 27 | e, err := f.st.CreateExposure("vm1", 8080, 30080) | 27 | e, err := f.st.CreateExposure("vm1", 8080, 30080, "tcp") |
| 28 | require.NoError(t, err) | 28 | require.NoError(t, err) |
| 29 | 29 | ||
| 30 | snap, err := f.svc.buildSnapshot(f.host.ID) | 30 | snap, err := f.svc.buildSnapshot(f.host.ID) |
| @@ -44,7 +44,7 @@ func TestSnapshotCarriesExposures(t *testing.T) { | |||
| 44 | func TestSnapshotDropsExposuresOfATombstonedVM(t *testing.T) { | 44 | func TestSnapshotDropsExposuresOfATombstonedVM(t *testing.T) { |
| 45 | f := setup(t) | 45 | f := setup(t) |
| 46 | exposureVM(t, f, "vm1", "web-1") | 46 | exposureVM(t, f, "vm1", "web-1") |
| 47 | _, err := f.st.CreateExposure("vm1", 8080, 30080) | 47 | _, err := f.st.CreateExposure("vm1", 8080, 30080, "tcp") |
| 48 | require.NoError(t, err) | 48 | require.NoError(t, err) |
| 49 | require.NoError(t, f.st.TombstoneVM("vm1")) | 49 | require.NoError(t, f.st.TombstoneVM("vm1")) |
| 50 | 50 | ||
internal/server/syncsvc/syncsvc.go
| Old | New | ||
|---|---|---|---|
| @@ -324,8 +324,9 @@ func (s *Service) buildSnapshot(hostID string) (*pb.DesiredStateSnapshot, error) | |||
| 324 | return nil, fmt.Errorf("list host exposures: %w", err) | 324 | return nil, fmt.Errorf("list host exposures: %w", err) |
| 325 | } | 325 | } |
| 326 | for _, e := range exps { | 326 | for _, e := range exps { |
| 327 | // In-range by construction: the API validates ports at create | 327 | // In-range by construction: the API validates ports and closes the |
| 328 | // (validateExposurePorts), and the store accepts rows only from the API. | 328 | // protocol at the two the agent can bind (validateExposure), and the |
| 329 | // store accepts rows only from the API. | ||
| 329 | snap.Exposures = append(snap.Exposures, &pb.ExposureDesired{ | 330 | snap.Exposures = append(snap.Exposures, &pb.ExposureDesired{ |
| 330 | Id: e.ID, VmId: e.VMID, | 331 | Id: e.ID, VmId: e.VMID, |
| 331 | GuestPort: uint32(e.GuestPort), HostPort: uint32(e.HostPort), | 332 | GuestPort: uint32(e.GuestPort), HostPort: uint32(e.HostPort), |
internal/smoke/exposure.go
| Old | New | ||
|---|---|---|---|
| @@ -21,11 +21,15 @@ const sshBannerPrefix = "SSH-2.0" | |||
| 21 | // bannerFunc reads the first bytes a TCP peer sends after accepting. | 21 | // bannerFunc reads the first bytes a TCP peer sends after accepting. |
| 22 | type bannerFunc func(ctx context.Context, addr string) (string, error) | 22 | type bannerFunc func(ctx context.Context, addr string) (string, error) |
| 23 | 23 | ||
| 24 | // echoFunc sends one datagram to a published UDP port and returns what came | ||
| 25 | // back on the same socket. | ||
| 26 | type echoFunc func(ctx context.Context, addr, payload string) (string, error) | ||
| 27 | |||
| 24 | // proveExposure publishes the smoke VM's ssh port on its host, dials the | 28 | // proveExposure publishes the smoke VM's ssh port on its host, dials the |
| 25 | // address the grant names, and expects an SSH banner. The exposure is revoked | 29 | // address the grant names, and expects an SSH banner. The exposure is revoked |
| 26 | // before the leg returns, whatever the outcome. | 30 | // before the leg returns, whatever the outcome. |
| 27 | func proveExposure(ctx context.Context, c vmAPI, vmID string, now func() time.Time, sleep func(time.Duration), dial bannerFunc) error { | 31 | func proveExposure(ctx context.Context, c vmAPI, vmID string, now func() time.Time, sleep func(time.Duration), dial bannerFunc) error { |
| 28 | exp, err := c.CreateExposure(ctx, vmID, 22, 0) | 32 | exp, err := c.CreateExposure(ctx, vmID, 22, 0, "tcp") |
| 29 | if err != nil { | 33 | if err != nil { |
| 30 | return fmt.Errorf("create exposure: %w", err) | 34 | return fmt.Errorf("create exposure: %w", err) |
| 31 | } | 35 | } |
| @@ -73,6 +77,32 @@ func truncateBanner(s string) string { | |||
| 73 | return s | 77 | return s |
| 74 | } | 78 | } |
| 75 | 79 | ||
| 80 | // readEcho is the real echoFunc: send payload to a published UDP port and read | ||
| 81 | // the answer off a connected socket, so a reply from anywhere but the port the | ||
| 82 | // grant names is not the fleet answering and does not count. The deadline | ||
| 83 | // bounds a datagram that goes nowhere — every hop here may drop one silently, | ||
| 84 | // which is what the caller's retry loop is for. | ||
| 85 | func readEcho(ctx context.Context, addr, payload string) (string, error) { | ||
| 86 | var d net.Dialer | ||
| 87 | conn, err := d.DialContext(ctx, "udp", addr) | ||
| 88 | if err != nil { | ||
| 89 | return "", err | ||
| 90 | } | ||
| 91 | defer conn.Close() | ||
| 92 | if err := conn.SetDeadline(time.Now().Add(5 * time.Second)); err != nil { | ||
| 93 | return "", err | ||
| 94 | } | ||
| 95 | if _, err := conn.Write([]byte(payload)); err != nil { | ||
| 96 | return "", err | ||
| 97 | } | ||
| 98 | buf := make([]byte, 512) | ||
| 99 | n, err := conn.Read(buf) | ||
| 100 | if err != nil { | ||
| 101 | return "", err | ||
| 102 | } | ||
| 103 | return string(buf[:n]), nil | ||
| 104 | } | ||
| 105 | |||
| 76 | // readBanner is the real bannerFunc: dial addr and read the peer's first line, | 106 | // readBanner is the real bannerFunc: dial addr and read the peer's first line, |
| 77 | // bounded by a deadline so a listener that binds but never answers fails the | 107 | // bounded by a deadline so a listener that binds but never answers fails the |
| 78 | // leg instead of hanging it. It reads until a newline (or 256 bytes) rather | 108 | // leg instead of hanging it. It reads until a newline (or 256 bytes) rather |
internal/smoke/exposure_test.go
| Old | New | ||
|---|---|---|---|
| @@ -19,7 +19,7 @@ func TestProveExposureReadsTheBannerBack(t *testing.T) { | |||
| 19 | clock := &fakeClock{} | 19 | clock := &fakeClock{} |
| 20 | deleted := "" | 20 | deleted := "" |
| 21 | api := &testAPI{ | 21 | api := &testAPI{ |
| 22 | createExposureFunc: func(ctx context.Context, vmID string, guest, host int64) (client.Exposure, error) { | 22 | createExposureFunc: func(ctx context.Context, vmID string, guest, host int64, proto string) (client.Exposure, error) { |
| 23 | if vmID != "vm-1" { | 23 | if vmID != "vm-1" { |
| 24 | t.Errorf("CreateExposure vmID = %q, want vm-1", vmID) | 24 | t.Errorf("CreateExposure vmID = %q, want vm-1", vmID) |
| 25 | } | 25 | } |
| @@ -56,7 +56,7 @@ func TestProveExposureReadsTheBannerBack(t *testing.T) { | |||
| 56 | func TestProveExposureDialsAnIPv6Uplink(t *testing.T) { | 56 | func TestProveExposureDialsAnIPv6Uplink(t *testing.T) { |
| 57 | clock := &fakeClock{} | 57 | clock := &fakeClock{} |
| 58 | api := &testAPI{ | 58 | api := &testAPI{ |
| 59 | createExposureFunc: func(context.Context, string, int64, int64) (client.Exposure, error) { | 59 | createExposureFunc: func(context.Context, string, int64, int64, string) (client.Exposure, error) { |
| 60 | return client.Exposure{ID: "x-1", HostPort: 30080, HostAddr: "2001:db8::1"}, nil | 60 | return client.Exposure{ID: "x-1", HostPort: 30080, HostAddr: "2001:db8::1"}, nil |
| 61 | }, | 61 | }, |
| 62 | deleteExposureFunc: func(context.Context, string) error { return nil }, | 62 | deleteExposureFunc: func(context.Context, string) error { return nil }, |
| @@ -78,7 +78,7 @@ func TestProveExposureDialsAnIPv6Uplink(t *testing.T) { | |||
| 78 | func TestProveExposureRetriesUntilTheListenerConverges(t *testing.T) { | 78 | func TestProveExposureRetriesUntilTheListenerConverges(t *testing.T) { |
| 79 | clock := &fakeClock{} | 79 | clock := &fakeClock{} |
| 80 | api := &testAPI{ | 80 | api := &testAPI{ |
| 81 | createExposureFunc: func(context.Context, string, int64, int64) (client.Exposure, error) { | 81 | createExposureFunc: func(context.Context, string, int64, int64, string) (client.Exposure, error) { |
| 82 | return client.Exposure{ID: "x-1", HostPort: 30080, HostAddr: "192.168.0.190"}, nil | 82 | return client.Exposure{ID: "x-1", HostPort: 30080, HostAddr: "192.168.0.190"}, nil |
| 83 | }, | 83 | }, |
| 84 | deleteExposureFunc: func(context.Context, string) error { return nil }, | 84 | deleteExposureFunc: func(context.Context, string) error { return nil }, |
| @@ -103,7 +103,7 @@ func TestProveExposureRetriesUntilTheListenerConverges(t *testing.T) { | |||
| 103 | func TestProveExposureFailsOnTheWrongBanner(t *testing.T) { | 103 | func TestProveExposureFailsOnTheWrongBanner(t *testing.T) { |
| 104 | clock := &fakeClock{} | 104 | clock := &fakeClock{} |
| 105 | api := &testAPI{ | 105 | api := &testAPI{ |
| 106 | createExposureFunc: func(context.Context, string, int64, int64) (client.Exposure, error) { | 106 | createExposureFunc: func(context.Context, string, int64, int64, string) (client.Exposure, error) { |
| 107 | return client.Exposure{ID: "x-1", HostPort: 30080, HostAddr: "192.168.0.190"}, nil | 107 | return client.Exposure{ID: "x-1", HostPort: 30080, HostAddr: "192.168.0.190"}, nil |
| 108 | }, | 108 | }, |
| 109 | deleteExposureFunc: func(context.Context, string) error { return nil }, | 109 | deleteExposureFunc: func(context.Context, string) error { return nil }, |
| @@ -126,7 +126,7 @@ func TestProveExposureFailsWhenTheGrantNamesNoAddress(t *testing.T) { | |||
| 126 | clock := &fakeClock{} | 126 | clock := &fakeClock{} |
| 127 | revoked := "" | 127 | revoked := "" |
| 128 | api := &testAPI{ | 128 | api := &testAPI{ |
| 129 | createExposureFunc: func(context.Context, string, int64, int64) (client.Exposure, error) { | 129 | createExposureFunc: func(context.Context, string, int64, int64, string) (client.Exposure, error) { |
| 130 | return client.Exposure{ID: "x-1", HostPort: 30080}, nil | 130 | return client.Exposure{ID: "x-1", HostPort: 30080}, nil |
| 131 | }, | 131 | }, |
| 132 | deleteExposureFunc: func(_ context.Context, id string) error { revoked = id; return nil }, | 132 | deleteExposureFunc: func(_ context.Context, id string) error { revoked = id; return nil }, |
internal/smoke/mcp.go
| Old | New | ||
|---|---|---|---|
| @@ -165,14 +165,15 @@ func proveRemoteMCPToolset(ctx context.Context, serverURL, pat string) error { | |||
| 165 | 165 | ||
| 166 | // proveMCP drives one full cycle through the remote MCP endpoint and nothing | 166 | // proveMCP drives one full cycle through the remote MCP endpoint and nothing |
| 167 | // else: delegate access to eitri, create a VM, run a command in it, publish a | 167 | // else: delegate access to eitri, create a VM, run a command in it, publish a |
| 168 | // port, read the guest's banner back through that port, and destroy it. | 168 | // TCP port and read the guest's banner back through it, publish a UDP port and |
| 169 | // read a datagram back through that, and destroy the VM. | ||
| 169 | // | 170 | // |
| 170 | // The CA is registered BEFORE the VM is created, because a guest bakes its CA | 171 | // The CA is registered BEFORE the VM is created, because a guest bakes its CA |
| 171 | // set at create and a CA registered afterwards is one it will never trust. The | 172 | // set at create and a CA registered afterwards is one it will never trust. The |
| 172 | // DELEGATION itself may happen on either side of the create — that is the whole | 173 | // DELEGATION itself may happen on either side of the create — that is the whole |
| 173 | // improvement over holding a signing key, and it is worth stating plainly here | 174 | // improvement over holding a signing key, and it is worth stating plainly here |
| 174 | // so nobody reintroduces an ordering constraint that no longer exists. | 175 | // so nobody reintroduces an ordering constraint that no longer exists. |
| 175 | func proveMCP(ctx context.Context, c mcpTools, vmName string, d delegator, now func() time.Time, sleep func(time.Duration), dial bannerFunc) error { | 176 | func proveMCP(ctx context.Context, c mcpTools, vmName string, d delegator, now func() time.Time, sleep func(time.Duration), dial bannerFunc, echo echoFunc) error { |
| 176 | if err := proveToolList(ctx, c); err != nil { | 177 | if err := proveToolList(ctx, c); err != nil { |
| 177 | return err | 178 | return err |
| 178 | } | 179 | } |
| @@ -282,6 +283,88 @@ func proveMCP(ctx context.Context, c mcpTools, vmName string, d delegator, now f | |||
| 282 | if _, err := c.Call(ctx, "vm_unexpose", map[string]any{"vm": vmName, "guest_port": 22}); err != nil { | 283 | if _, err := c.Call(ctx, "vm_unexpose", map[string]any{"vm": vmName, "guest_port": 22}); err != nil { |
| 283 | return fmt.Errorf("vm_unexpose over MCP: %w", err) | 284 | return fmt.Errorf("vm_unexpose over MCP: %w", err) |
| 284 | } | 285 | } |
| 286 | |||
| 287 | return proveUDPExposure(ctx, c, vmName, now, sleep, echo) | ||
| 288 | } | ||
| 289 | |||
| 290 | // udpEchoPort is where the smoke's own echo listens inside the guest. A TCP | ||
| 291 | // exposure can be proven against the sshd that is already there; UDP has no | ||
| 292 | // such standing service, so the leg brings its own — which is the honest test | ||
| 293 | // anyway, since it proves a port nothing else on the host or the guest is | ||
| 294 | // touching. | ||
| 295 | const udpEchoPort = 17007 | ||
| 296 | |||
| 297 | // udpEchoCommand starts that echo and returns. python3 is what a guest that | ||
| 298 | // booted through cloud-init necessarily has, so this needs no package install | ||
| 299 | // and no image of its own; nohup and the redirects are what let the exec's | ||
| 300 | // session close while the echo keeps running. | ||
| 301 | func udpEchoCommand() string { | ||
| 302 | return fmt.Sprintf(`nohup python3 -c ' | ||
| 303 | import socket | ||
| 304 | s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) | ||
| 305 | s.bind(("0.0.0.0", %d)) | ||
| 306 | while True: | ||
| 307 | datagram, sender = s.recvfrom(2048) | ||
| 308 | s.sendto(datagram, sender) | ||
| 309 | ' >/dev/null 2>&1 &`, udpEchoPort) | ||
| 310 | } | ||
| 311 | |||
| 312 | // proveUDPExposure publishes a UDP port of the MCP leg's own VM and proves a | ||
| 313 | // datagram makes the whole round trip: through the host's packet proxy, into | ||
| 314 | // the guest, and back out to the caller through the same published port. The | ||
| 315 | // grant is revoked before the leg returns; the echo dies with the VM. | ||
| 316 | func proveUDPExposure(ctx context.Context, c mcpTools, vmName string, now func() time.Time, sleep func(time.Duration), echo echoFunc) error { | ||
| 317 | if _, err := c.Call(ctx, "vm_exec", map[string]any{"vm": vmName, "command": udpEchoCommand()}); err != nil { | ||
| 318 | return fmt.Errorf("start the guest UDP echo over MCP: %w", err) | ||
| 319 | } | ||
| 320 | |||
| 321 | exposed, err := c.Call(ctx, "vm_expose", map[string]any{ | ||
| 322 | "vm": vmName, "guest_port": udpEchoPort, "protocol": "udp", | ||
| 323 | }) | ||
| 324 | if err != nil { | ||
| 325 | return fmt.Errorf("vm_expose udp over MCP: %w", err) | ||
| 326 | } | ||
| 327 | if got, _ := exposed["exposure"].(map[string]any); got != nil { | ||
| 328 | if proto, _ := got["protocol"].(string); proto != "udp" { | ||
| 329 | return fmt.Errorf("FAIL: vm_expose published %q, want a udp exposure", proto) | ||
| 330 | } | ||
| 331 | } | ||
| 332 | address, err := exposureAddress(exposed) | ||
| 333 | if err != nil { | ||
| 334 | return err | ||
| 335 | } | ||
| 336 | |||
| 337 | nonce, err := randNonce() | ||
| 338 | if err != nil { | ||
| 339 | return err | ||
| 340 | } | ||
| 341 | var lastErr error | ||
| 342 | err = pollLoop(ctx, now, sleep, 60*time.Second, 3*time.Second, func() (bool, error) { | ||
| 343 | // Everything transient lives in this retry: the host binds the socket on | ||
| 344 | // its next converge, the guest's echo takes a moment to come up, and a | ||
| 345 | // datagram is allowed to go missing on any hop between them. | ||
| 346 | back, derr := echo(ctx, address, nonce) | ||
| 347 | if derr != nil { | ||
| 348 | lastErr = derr | ||
| 349 | return false, nil | ||
| 350 | } | ||
| 351 | if back != nonce { | ||
| 352 | return false, fmt.Errorf("FAIL: published UDP port %s echoed %q, want %q", address, truncateBanner(back), nonce) | ||
| 353 | } | ||
| 354 | return true, nil | ||
| 355 | }) | ||
| 356 | if errors.Is(err, errPollTimeout) { | ||
| 357 | return fmt.Errorf("FAIL: no echo from the MCP-published UDP port %s within 60s: %v", address, lastErr) | ||
| 358 | } | ||
| 359 | if err != nil { | ||
| 360 | return err | ||
| 361 | } | ||
| 362 | |||
| 363 | if _, err := c.Call(ctx, "vm_unexpose", map[string]any{ | ||
| 364 | "vm": vmName, "guest_port": udpEchoPort, "protocol": "udp", | ||
| 365 | }); err != nil { | ||
| 366 | return fmt.Errorf("vm_unexpose udp over MCP: %w", err) | ||
| 367 | } | ||
| 285 | return nil | 368 | return nil |
| 286 | } | 369 | } |
| 287 | 370 | ||
internal/smoke/mcp_test.go
| Old | New | ||
|---|---|---|---|
| @@ -7,6 +7,7 @@ import ( | |||
| 7 | "errors" | 7 | "errors" |
| 8 | "net/http" | 8 | "net/http" |
| 9 | "net/http/httptest" | 9 | "net/http/httptest" |
| 10 | "strconv" | ||
| 10 | "strings" | 11 | "strings" |
| 11 | "testing" | 12 | "testing" |
| 12 | "time" | 13 | "time" |
| @@ -28,8 +29,10 @@ type fakeMCP struct { | |||
| 28 | // hold it: a certificate signed by anything else is refused. | 29 | // hold it: a certificate signed by anything else is refused. |
| 29 | trusted []ssh.PublicKey | 30 | trusted []ssh.PublicKey |
| 30 | pubLine string | 31 | pubLine string |
| 31 | // uploaded records the CA lines ca_upload received. | 32 | // uploaded records the CA lines ca_upload received, and exposedAs the |
| 32 | uploaded []string | 33 | // protocol each vm_expose asked for. |
| 34 | uploaded []string | ||
| 35 | exposedAs []string | ||
| 33 | // delegated is what tenant_info reports; set once a certificate lands. | 36 | // delegated is what tenant_info reports; set once a certificate lands. |
| 34 | delegated bool | 37 | delegated bool |
| 35 | 38 | ||
| @@ -53,7 +56,6 @@ func newFakeMCP(t *testing.T, trusted ...ssh.Signer) *fakeMCP { | |||
| 53 | results: map[string]map[string]any{ | 56 | results: map[string]map[string]any{ |
| 54 | "vm_create": {"id": "v-1", "name": "smoke-mcp"}, | 57 | "vm_create": {"id": "v-1", "name": "smoke-mcp"}, |
| 55 | "vm_exec": {"stdout": "", "exit_code": 0.0}, | 58 | "vm_exec": {"stdout": "", "exit_code": 0.0}, |
| 56 | "vm_expose": {"exposure": map[string]any{"id": "x-1", "address": "10.0.0.1:30001"}}, | ||
| 57 | "vm_unexpose": {"id": "x-1"}, | 59 | "vm_unexpose": {"id": "x-1"}, |
| 58 | "vm_destroy": {"id": "v-1"}, | 60 | "vm_destroy": {"id": "v-1"}, |
| 59 | }, | 61 | }, |
| @@ -93,6 +95,22 @@ func (f *fakeMCP) Call(_ context.Context, name string, args map[string]any) (map | |||
| 93 | if name == "delegate_complete" { | 95 | if name == "delegate_complete" { |
| 94 | return f.complete(args) | 96 | return f.complete(args) |
| 95 | } | 97 | } |
| 98 | if name == "vm_expose" { | ||
| 99 | // The control plane answers with the exposure it made, protocol | ||
| 100 | // included, and an unnamed protocol is tcp. A scripted result stands in | ||
| 101 | // for a plane answering something else. | ||
| 102 | proto, _ := args["protocol"].(string) | ||
| 103 | if proto == "" { | ||
| 104 | proto = "tcp" | ||
| 105 | } | ||
| 106 | f.exposedAs = append(f.exposedAs, proto) | ||
| 107 | if scripted, ok := f.results[name]; ok { | ||
| 108 | return scripted, nil | ||
| 109 | } | ||
| 110 | return map[string]any{"exposure": map[string]any{ | ||
| 111 | "id": "x-1", "address": "10.0.0.1:30001", "protocol": proto, | ||
| 112 | }}, nil | ||
| 113 | } | ||
| 96 | out := map[string]any{} | 114 | out := map[string]any{} |
| 97 | for k, v := range f.results[name] { | 115 | for k, v := range f.results[name] { |
| 98 | out[k] = v | 116 | out[k] = v |
| @@ -144,6 +162,10 @@ func (e echoingMCP) Call(ctx context.Context, name string, args map[string]any) | |||
| 144 | 162 | ||
| 145 | func okBannerDial(context.Context, string) (string, error) { return "SSH-2.0-OpenSSH_9.6\r\n", nil } | 163 | func okBannerDial(context.Context, string) (string, error) { return "SSH-2.0-OpenSSH_9.6\r\n", nil } |
| 146 | 164 | ||
| 165 | // okEcho is a published UDP port with the guest's echo behind it: whatever went | ||
| 166 | // in comes back. | ||
| 167 | func okEcho(_ context.Context, _, payload string) (string, error) { return payload, nil } | ||
| 168 | |||
| 147 | // newCA returns a signer standing in for somebody's own SSH user CA. | 169 | // newCA returns a signer standing in for somebody's own SSH user CA. |
| 148 | func newCA(t *testing.T) ssh.Signer { | 170 | func newCA(t *testing.T) ssh.Signer { |
| 149 | t.Helper() | 171 | t.Helper() |
| @@ -175,14 +197,50 @@ func TestProveMCPDrivesTheWholeCycle(t *testing.T) { | |||
| 175 | f := echoingMCP{newFakeMCP(t, ca)} | 197 | f := echoingMCP{newFakeMCP(t, ca)} |
| 176 | clock := &fakeClock{} | 198 | clock := &fakeClock{} |
| 177 | 199 | ||
| 178 | require.NoError(t, proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial)) | 200 | require.NoError(t, proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho)) |
| 179 | 201 | ||
| 180 | assert.Equal(t, []string{authorizedLine(ca.PublicKey())}, f.uploaded, | 202 | assert.Equal(t, []string{authorizedLine(ca.PublicKey())}, f.uploaded, |
| 181 | "the CA is registered through the tool, before anything is created") | 203 | "the CA is registered through the tool, before anything is created") |
| 182 | assert.Equal(t, | 204 | assert.Equal(t, |
| 183 | []string{"ca_upload", "delegate_begin", "delegate_complete", "delegate_complete", | 205 | []string{"ca_upload", "delegate_begin", "delegate_complete", "delegate_complete", |
| 184 | "tenant_info", "vm_create", "vm_exec", "vm_expose", "vm_unexpose", "vm_destroy"}, | 206 | "tenant_info", "vm_create", "vm_exec", "vm_expose", "vm_unexpose", |
| 185 | f.calls) | 207 | "vm_exec", "vm_expose", "vm_unexpose", "vm_destroy"}, |
| 208 | f.calls, "the UDP leg starts its own listener in the guest before publishing a port for it") | ||
| 209 | assert.Equal(t, []string{"tcp", "udp"}, f.exposedAs) | ||
| 210 | } | ||
| 211 | |||
| 212 | // TestProveUDPExposureRequiresTheDatagramBack pins what the UDP leg is for: a | ||
| 213 | // published port that binds and answers nothing passes no gate. | ||
| 214 | func TestProveUDPExposureRequiresTheDatagramBack(t *testing.T) { | ||
| 215 | f := newFakeMCP(t) | ||
| 216 | clock := &fakeClock{} | ||
| 217 | |||
| 218 | // A port that answers something else entirely is a failure on the spot — | ||
| 219 | // something is listening, and it is not the guest's echo. | ||
| 220 | wrong := func(context.Context, string, string) (string, error) { return "not-the-nonce", nil } | ||
| 221 | err := proveUDPExposure(t.Context(), f, "smoke-mcp", clock.now, clock.sleep, wrong) | ||
| 222 | require.Error(t, err) | ||
| 223 | assert.Contains(t, err.Error(), "echoed") | ||
| 224 | |||
| 225 | // A port that answers nothing at all is retried to the deadline and then | ||
| 226 | // fails with what the last attempt said. | ||
| 227 | silent := func(context.Context, string, string) (string, error) { | ||
| 228 | return "", errors.New("i/o timeout") | ||
| 229 | } | ||
| 230 | err = proveUDPExposure(t.Context(), f, "smoke-mcp", clock.now, clock.sleep, silent) | ||
| 231 | require.Error(t, err) | ||
| 232 | assert.Contains(t, err.Error(), "no echo from the MCP-published UDP port") | ||
| 233 | assert.Contains(t, err.Error(), "i/o timeout") | ||
| 234 | } | ||
| 235 | |||
| 236 | // TestUDPEchoCommandDetachesFromTheExecSession pins the two things about the | ||
| 237 | // guest-side echo that a passing leg depends on: it binds the port the leg | ||
| 238 | // publishes, and it outlives the exec that started it. | ||
| 239 | func TestUDPEchoCommandDetachesFromTheExecSession(t *testing.T) { | ||
| 240 | cmd := udpEchoCommand() | ||
| 241 | assert.Contains(t, cmd, strconv.Itoa(udpEchoPort), "the echo binds the port the leg publishes") | ||
| 242 | assert.Contains(t, cmd, "nohup", "the echo must survive the session that started it") | ||
| 243 | assert.Contains(t, cmd, ">/dev/null 2>&1 &", "a background process holding stdout would hang the exec") | ||
| 186 | } | 244 | } |
| 187 | 245 | ||
| 188 | // TestProveMCPRequiresTheUnregisteredCAToBeRefused is the negative leg itself: | 246 | // TestProveMCPRequiresTheUnregisteredCAToBeRefused is the negative leg itself: |
| @@ -193,7 +251,7 @@ func TestProveMCPRequiresTheUnregisteredCAToBeRefused(t *testing.T) { | |||
| 193 | f := echoingMCP{newFakeMCP(t, ca, stranger)} // a plane that trusts everyone | 251 | f := echoingMCP{newFakeMCP(t, ca, stranger)} // a plane that trusts everyone |
| 194 | clock := &fakeClock{} | 252 | clock := &fakeClock{} |
| 195 | 253 | ||
| 196 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 254 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 197 | require.Error(t, err) | 255 | require.Error(t, err) |
| 198 | assert.Contains(t, err.Error(), "accepted a certificate from an unregistered CA") | 256 | assert.Contains(t, err.Error(), "accepted a certificate from an unregistered CA") |
| 199 | } | 257 | } |
| @@ -207,7 +265,7 @@ func TestProveMCPRejectsAWrongPrincipal(t *testing.T) { | |||
| 207 | f.results["delegate_begin"] = map[string]any{"public_key": f.pubLine, "principal": "root"} | 265 | f.results["delegate_begin"] = map[string]any{"public_key": f.pubLine, "principal": "root"} |
| 208 | clock := &fakeClock{} | 266 | clock := &fakeClock{} |
| 209 | 267 | ||
| 210 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 268 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 211 | require.Error(t, err) | 269 | require.Error(t, err) |
| 212 | assert.Contains(t, err.Error(), `asked for principal "root"`) | 270 | assert.Contains(t, err.Error(), `asked for principal "root"`) |
| 213 | } | 271 | } |
| @@ -219,7 +277,7 @@ func TestProveMCPRejectsADelegationWithNoExpiry(t *testing.T) { | |||
| 219 | f := noExpiryMCP{newFakeMCP(t, ca)} | 277 | f := noExpiryMCP{newFakeMCP(t, ca)} |
| 220 | clock := &fakeClock{} | 278 | clock := &fakeClock{} |
| 221 | 279 | ||
| 222 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 280 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 223 | require.Error(t, err) | 281 | require.Error(t, err) |
| 224 | assert.Contains(t, err.Error(), "reports no expiry") | 282 | assert.Contains(t, err.Error(), "reports no expiry") |
| 225 | } | 283 | } |
| @@ -242,7 +300,7 @@ func TestProveMCPDestroysItsVMOnFailure(t *testing.T) { | |||
| 242 | f.failOn = "vm_exec" | 300 | f.failOn = "vm_exec" |
| 243 | clock := &fakeClock{} | 301 | clock := &fakeClock{} |
| 244 | 302 | ||
| 245 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 303 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 246 | require.Error(t, err) | 304 | require.Error(t, err) |
| 247 | assert.Contains(t, err.Error(), "vm_exec over MCP") | 305 | assert.Contains(t, err.Error(), "vm_exec over MCP") |
| 248 | assert.Contains(t, f.calls, "vm_destroy", "the leg must reap its own VM even when it fails") | 306 | assert.Contains(t, f.calls, "vm_destroy", "the leg must reap its own VM even when it fails") |
| @@ -294,7 +352,7 @@ func TestProveMCPRejectsAShortToolset(t *testing.T) { | |||
| 294 | f.tools = f.tools[:5] | 352 | f.tools = f.tools[:5] |
| 295 | clock := &fakeClock{} | 353 | clock := &fakeClock{} |
| 296 | 354 | ||
| 297 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 355 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 298 | require.Error(t, err) | 356 | require.Error(t, err) |
| 299 | assert.Contains(t, err.Error(), "advertises 5 tools") | 357 | assert.Contains(t, err.Error(), "advertises 5 tools") |
| 300 | assert.Empty(t, f.calls, "a wrong toolset must fail before any tool is called") | 358 | assert.Empty(t, f.calls, "a wrong toolset must fail before any tool is called") |
| @@ -312,7 +370,7 @@ func TestProveMCPRejectsAMissingDelegateTool(t *testing.T) { | |||
| 312 | } | 370 | } |
| 313 | clock := &fakeClock{} | 371 | clock := &fakeClock{} |
| 314 | 372 | ||
| 315 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 373 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 316 | require.Error(t, err) | 374 | require.Error(t, err) |
| 317 | assert.Contains(t, err.Error(), "does not advertise delegate_begin") | 375 | assert.Contains(t, err.Error(), "does not advertise delegate_begin") |
| 318 | } | 376 | } |
| @@ -325,7 +383,7 @@ func TestProveMCPRejectsAnExposureWithNoAddress(t *testing.T) { | |||
| 325 | f.results["vm_expose"] = map[string]any{"exposure": map[string]any{"id": "x-1"}} | 383 | f.results["vm_expose"] = map[string]any{"exposure": map[string]any{"id": "x-1"}} |
| 326 | clock := &fakeClock{} | 384 | clock := &fakeClock{} |
| 327 | 385 | ||
| 328 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 386 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 329 | require.Error(t, err) | 387 | require.Error(t, err) |
| 330 | assert.Contains(t, err.Error(), "no address to dial") | 388 | assert.Contains(t, err.Error(), "no address to dial") |
| 331 | } | 389 | } |
| @@ -338,7 +396,7 @@ func TestProveMCPRejectsAWrongBanner(t *testing.T) { | |||
| 338 | clock := &fakeClock{} | 396 | clock := &fakeClock{} |
| 339 | 397 | ||
| 340 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, | 398 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, |
| 341 | func(context.Context, string) (string, error) { return "HTTP/1.1 200 OK", nil }) | 399 | func(context.Context, string) (string, error) { return "HTTP/1.1 200 OK", nil }, okEcho) |
| 342 | require.Error(t, err) | 400 | require.Error(t, err) |
| 343 | assert.Contains(t, err.Error(), "want an SSH-2.0 banner") | 401 | assert.Contains(t, err.Error(), "want an SSH-2.0 banner") |
| 344 | } | 402 | } |
| @@ -350,7 +408,7 @@ func TestProveMCPRejectsAnEchoThatDoesNotComeBack(t *testing.T) { | |||
| 350 | f := newFakeMCP(t, ca) // plain fake: vm_exec returns an empty stdout | 408 | f := newFakeMCP(t, ca) // plain fake: vm_exec returns an empty stdout |
| 351 | clock := &fakeClock{} | 409 | clock := &fakeClock{} |
| 352 | 410 | ||
| 353 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial) | 411 | err := proveMCP(t.Context(), f, "smoke-mcp", testDelegator(ca, stranger), clock.now, clock.sleep, okBannerDial, okEcho) |
| 354 | require.Error(t, err) | 412 | require.Error(t, err) |
| 355 | assert.Contains(t, err.Error(), "want it to contain") | 413 | assert.Contains(t, err.Error(), "want it to contain") |
| 356 | } | 414 | } |
internal/smoke/run.go
| Old | New | ||
|---|---|---|---|
| @@ -168,7 +168,7 @@ func Run() error { | |||
| 168 | stranger: stranger, | 168 | stranger: stranger, |
| 169 | principal: cfg.SmokeVMUser, | 169 | principal: cfg.SmokeVMUser, |
| 170 | now: time.Now, | 170 | now: time.Now, |
| 171 | }, time.Now, time.Sleep, readBanner) | 171 | }, time.Now, time.Sleep, readBanner, readEcho) |
| 172 | } | 172 | } |
| 173 | 173 | ||
| 174 | msg, err := runScenario(ctx, vmName, api, api.DialConsole, gate, mcp, time.Now, time.Sleep, realReadPubKey, readBanner) | 174 | msg, err := runScenario(ctx, vmName, api, api.DialConsole, gate, mcp, time.Now, time.Sleep, realReadPubKey, readBanner) |
internal/smoke/scenario.go
| Old | New | ||
|---|---|---|---|
| @@ -53,7 +53,7 @@ type vmAPI interface { | |||
| 53 | ListVMs(ctx context.Context) ([]client.VM, error) | 53 | ListVMs(ctx context.Context) ([]client.VM, error) |
| 54 | DeleteVM(ctx context.Context, id string) error | 54 | DeleteVM(ctx context.Context, id string) error |
| 55 | PatchVM(ctx context.Context, id, powerState string) error | 55 | PatchVM(ctx context.Context, id, powerState string) error |
| 56 | CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64) (client.Exposure, error) | 56 | CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) |
| 57 | DeleteExposure(ctx context.Context, id string) error | 57 | DeleteExposure(ctx context.Context, id string) error |
| 58 | } | 58 | } |
| 59 | 59 | ||
| @@ -193,7 +193,9 @@ func runScenario(ctx context.Context, vmName string, c vmAPI, dialConsole consol | |||
| 193 | exposureOK = true | 193 | exposureOK = true |
| 194 | 194 | ||
| 195 | // Remote MCP: the same fleet, driven by a bearer PAT over HTTP with no | 195 | // Remote MCP: the same fleet, driven by a bearer PAT over HTTP with no |
| 196 | // local install and no uploaded CA. It creates and destroys its own VM. | 196 | // local install and no uploaded CA. It creates and destroys its own VM, |
| 197 | // and it is where the UDP published port is proven — the proof needs a | ||
| 198 | // listener inside the guest, and this is the leg that can start one. | ||
| 197 | mcpOK := false | 199 | mcpOK := false |
| 198 | if mcp != nil { | 200 | if mcp != nil { |
| 199 | if err := mcp(ctx, vmName+"-mcp"); err != nil { | 201 | if err := mcp(ctx, vmName+"-mcp"); err != nil { |
| @@ -288,7 +290,7 @@ func runScenario(ctx context.Context, vmName string, c vmAPI, dialConsole consol | |||
| 288 | msg += ", exposed port: ok" | 290 | msg += ", exposed port: ok" |
| 289 | } | 291 | } |
| 290 | if mcpOK { | 292 | if mcpOK { |
| 291 | msg += ", remote MCP: ok" | 293 | msg += ", remote MCP: ok, published UDP port: ok" |
| 292 | } | 294 | } |
| 293 | return msg, nil | 295 | return msg, nil |
| 294 | } | 296 | } |
internal/smoke/scenario_test.go
| Old | New | ||
|---|---|---|---|
| @@ -70,7 +70,7 @@ type testAPI struct { | |||
| 70 | listVMsFunc func(ctx context.Context) ([]client.VM, error) | 70 | listVMsFunc func(ctx context.Context) ([]client.VM, error) |
| 71 | deleteVMFunc func(ctx context.Context, id string) error | 71 | deleteVMFunc func(ctx context.Context, id string) error |
| 72 | patchVMFunc func(ctx context.Context, id, powerState string) error | 72 | patchVMFunc func(ctx context.Context, id, powerState string) error |
| 73 | createExposureFunc func(ctx context.Context, vmID string, guestPort, hostPort int64) (client.Exposure, error) | 73 | createExposureFunc func(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) |
| 74 | deleteExposureFunc func(ctx context.Context, id string) error | 74 | deleteExposureFunc func(ctx context.Context, id string) error |
| 75 | } | 75 | } |
| 76 | 76 | ||
| @@ -90,11 +90,11 @@ func (a *testAPI) PatchVM(ctx context.Context, id, powerState string) error { | |||
| 90 | } | 90 | } |
| 91 | return a.patchVMFunc(ctx, id, powerState) | 91 | return a.patchVMFunc(ctx, id, powerState) |
| 92 | } | 92 | } |
| 93 | func (a *testAPI) CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64) (client.Exposure, error) { | 93 | func (a *testAPI) CreateExposure(ctx context.Context, vmID string, guestPort, hostPort int64, protocol string) (client.Exposure, error) { |
| 94 | if a.createExposureFunc == nil { | 94 | if a.createExposureFunc == nil { |
| 95 | return client.Exposure{ID: "x-fake", HostPort: 30080, HostAddr: "192.168.0.190"}, nil | 95 | return client.Exposure{ID: "x-fake", HostPort: 30080, HostAddr: "192.168.0.190", Protocol: protocol}, nil |
| 96 | } | 96 | } |
| 97 | return a.createExposureFunc(ctx, vmID, guestPort, hostPort) | 97 | return a.createExposureFunc(ctx, vmID, guestPort, hostPort, protocol) |
| 98 | } | 98 | } |
| 99 | 99 | ||
| 100 | func (a *testAPI) DeleteExposure(ctx context.Context, id string) error { | 100 | func (a *testAPI) DeleteExposure(ctx context.Context, id string) error { |
proto/eitri/v1/sync.proto
| Old | New | ||
|---|---|---|---|
| @@ -198,25 +198,25 @@ message TCPOpened { | |||
| 198 | } | 198 | } |
| 199 | 199 | ||
| 200 | // ExposureDesired is one published guest port a host should be serving: bind | 200 | // ExposureDesired is one published guest port a host should be serving: bind |
| 201 | // host_port on the host, pipe every accepted connection to guest_port inside | 201 | // host_port on the host, pipe every accepted connection (or every datagram) |
| 202 | // the guest. It rides the snapshot at TOP LEVEL rather than nested in | 202 | // to guest_port inside the guest. It rides the snapshot at TOP LEVEL rather |
| 203 | // VMDesired, because exposures are their own objects converging on their own | 203 | // than nested in VMDesired, because exposures are their own objects converging |
| 204 | // cadence — an exposure can be created while its VM is still imaging, and it | 204 | // on their own cadence — an exposure can be created while its VM is still |
| 205 | // binds immediately. | 205 | // imaging, and it binds immediately. |
| 206 | message ExposureDesired { | 206 | message ExposureDesired { |
| 207 | string id = 1; | 207 | string id = 1; |
| 208 | string vm_id = 2; | 208 | string vm_id = 2; |
| 209 | uint32 guest_port = 3; | 209 | uint32 guest_port = 3; |
| 210 | uint32 host_port = 4; | 210 | uint32 host_port = 4; |
| 211 | string protocol = 5; // "tcp" | 211 | string protocol = 5; // "tcp"|"udp" |
| 212 | } | 212 | } |
| 213 | 213 | ||
| 214 | // ExposureActual is one exposure's state as its host observes it: "active" | 214 | // ExposureActual is one exposure's state as its host observes it: "active" |
| 215 | // once the host listener is bound, "failed" with the OS error otherwise. | 215 | // once the host socket is bound, "failed" with the OS error otherwise. |
| 216 | // "active" means the HOST half of the pipe exists — whether anything answers | 216 | // "active" means the HOST half of the pipe exists — whether anything answers |
| 217 | // inside the guest is the guest's half, and this does not pretend otherwise. | 217 | // inside the guest is the guest's half, and this does not pretend otherwise. |
| 218 | message ExposureActual { | 218 | message ExposureActual { |
| 219 | string id = 1; | 219 | string id = 1; |
| 220 | string state = 2; // "active"|"failed" | 220 | string state = 2; // "active"|"failed" |
| 221 | string reason = 3; // the OS error, when failed | 221 | string reason = 3; // the OS error, when failed; a bound port's ongoing trouble otherwise |
| 222 | } | 222 | } |
web/src/lib/api-types.ts
| Old | New | ||
|---|---|---|---|
| @@ -1393,7 +1393,7 @@ export interface paths { | |||
| 1393 | }; | 1393 | }; |
| 1394 | }; | 1394 | }; |
| 1395 | put?: never; | 1395 | put?: never; |
| 1396 | /** Publish a guest TCP port on the VM's host. Omit host_port to allocate one from the reserved range 30000-32767; a named port must be >= 1024 and is honored or refused. */ | 1396 | /** Publish a guest port on the VM's host, protocol "tcp" (the default) or "udp". Omit host_port to allocate one from the reserved range 30000-32767; a named port must be >= 1024 and is honored or refused, and is taken only by another exposure of the same protocol. */ |
| 1397 | post: { | 1397 | post: { |
| 1398 | parameters: { | 1398 | parameters: { |
| 1399 | query?: never; | 1399 | query?: never; |
| @@ -1516,6 +1516,7 @@ export interface components { | |||
| 1516 | CreateExposureRequest: { | 1516 | CreateExposureRequest: { |
| 1517 | guest_port?: number; | 1517 | guest_port?: number; |
| 1518 | host_port?: number; | 1518 | host_port?: number; |
| 1519 | protocol?: string; | ||
| 1519 | }; | 1520 | }; |
| 1520 | CreateVMRequest: { | 1521 | CreateVMRequest: { |
| 1521 | cloud_init?: string; | 1522 | cloud_init?: string; |
web/src/lib/fleet.svelte.ts
| Old | New | ||
|---|---|---|---|
| @@ -21,8 +21,9 @@ export type VMEvent = components['schemas']['AuditEvent']; | |||
| 21 | export type UserCA = components['schemas']['UserCA']; | 21 | export type UserCA = components['schemas']['UserCA']; |
| 22 | 22 | ||
| 23 | /** Exposure is one published guest port: the fleet binds host_port on the VM's | 23 | /** Exposure is one published guest port: the fleet binds host_port on the VM's |
| 24 | * host and pipes it to guest_port inside the guest. host_addr is the address | 24 | * host and pipes it to guest_port inside the guest, in protocol ('tcp' or |
| 25 | * to dial; state is what the host says its listener is doing. */ | 25 | * 'udp'). host_addr is the address to dial; state is what the host says its |
| 26 | * socket is doing. */ | ||
| 26 | export type Exposure = components['schemas']['Exposure']; | 27 | export type Exposure = components['schemas']['Exposure']; |
| 27 | 28 | ||
| 28 | export type CreateVMRequest = components['schemas']['CreateVMRequest']; | 29 | export type CreateVMRequest = components['schemas']['CreateVMRequest']; |
| @@ -265,17 +266,19 @@ export async function listExposures(id: string): Promise<Exposure[]> { | |||
| 265 | return (await req('GET', `/api/v1/vms/${id}/exposures`)).json(); | 266 | return (await req('GET', `/api/v1/vms/${id}/exposures`)).json(); |
| 266 | } | 267 | } |
| 267 | 268 | ||
| 268 | /** createExposure publishes a guest port. hostPort 0 asks the control plane to | 269 | /** createExposure publishes a guest port over protocol ('tcp' or 'udp'). |
| 269 | * allocate one from the reserved range. */ | 270 | * hostPort 0 asks the control plane to allocate one from the reserved range. */ |
| 270 | export async function createExposure( | 271 | export async function createExposure( |
| 271 | id: string, | 272 | id: string, |
| 272 | guestPort: number, | 273 | guestPort: number, |
| 273 | hostPort: number | 274 | hostPort: number, |
| 275 | protocol: string | ||
| 274 | ): Promise<Exposure> { | 276 | ): Promise<Exposure> { |
| 275 | return ( | 277 | return ( |
| 276 | await req('POST', `/api/v1/vms/${id}/exposures`, { | 278 | await req('POST', `/api/v1/vms/${id}/exposures`, { |
| 277 | guest_port: guestPort, | 279 | guest_port: guestPort, |
| 278 | host_port: hostPort | 280 | host_port: hostPort, |
| 281 | protocol | ||
| 279 | }) | 282 | }) |
| 280 | ).json(); | 283 | ).json(); |
| 281 | } | 284 | } |
| @@ -427,9 +430,9 @@ export function eventLabel(ev: VMEvent): string { | |||
| 427 | case 'vm.reap': | 430 | case 'vm.reap': |
| 428 | return `Destroyed (${detail.reason ?? '?'})`; | 431 | return `Destroyed (${detail.reason ?? '?'})`; |
| 429 | case 'exposure.create': | 432 | case 'exposure.create': |
| 430 | return `Exposed guest :${detail.guest_port ?? '?'} on host :${detail.host_port ?? '?'}`; | 433 | return `Exposed guest :${detail.guest_port ?? '?'}/${detail.protocol ?? 'tcp'} on host :${detail.host_port ?? '?'}`; |
| 431 | case 'exposure.delete': | 434 | case 'exposure.delete': |
| 432 | return `Exposure removed (guest :${detail.guest_port ?? '?'})`; | 435 | return `Exposure removed (guest :${detail.guest_port ?? '?'}/${detail.protocol ?? 'tcp'})`; |
| 433 | default: | 436 | default: |
| 434 | return ev.action; | 437 | return ev.action; |
| 435 | } | 438 | } |
web/src/routes/vms/[id]/+page.svelte
| Old | New | ||
|---|---|---|---|
| @@ -71,6 +71,7 @@ | |||
| 71 | let exposuresFailed = $state(false); | 71 | let exposuresFailed = $state(false); |
| 72 | let guestPortInput = $state(''); | 72 | let guestPortInput = $state(''); |
| 73 | let hostPortInput = $state(''); | 73 | let hostPortInput = $state(''); |
| 74 | let protocolInput = $state('tcp'); | ||
| 74 | 75 | ||
| 75 | async function loadExposures(forId: string | undefined) { | 76 | async function loadExposures(forId: string | undefined) { |
| 76 | if (!forId) return; | 77 | if (!forId) return; |
| @@ -116,10 +117,12 @@ | |||
| 116 | if (!vm) return; | 117 | if (!vm) return; |
| 117 | const guest = Number(guestPortInput); | 118 | const guest = Number(guestPortInput); |
| 118 | const host = hostPortInput === '' ? 0 : Number(hostPortInput); | 119 | const host = hostPortInput === '' ? 0 : Number(hostPortInput); |
| 120 | const proto = protocolInput; | ||
| 119 | const vmId = vm.id; | 121 | const vmId = vm.id; |
| 120 | if (await action(() => createExposure(vmId, guest, host))) { | 122 | if (await action(() => createExposure(vmId, guest, host, proto))) { |
| 121 | guestPortInput = ''; | 123 | guestPortInput = ''; |
| 122 | hostPortInput = ''; | 124 | hostPortInput = ''; |
| 125 | protocolInput = 'tcp'; | ||
| 123 | await loadExposures(vmId); | 126 | await loadExposures(vmId); |
| 124 | } | 127 | } |
| 125 | } | 128 | } |
| @@ -251,7 +254,7 @@ | |||
| 251 | <tbody> | 254 | <tbody> |
| 252 | {#each exposures as e (e.id)} | 255 | {#each exposures as e (e.id)} |
| 253 | <tr> | 256 | <tr> |
| 254 | <th>guest :{e.guest_port}</th> | 257 | <th>guest :{e.guest_port}/{e.protocol}</th> |
| 255 | <td> | 258 | <td> |
| 256 | {#if exposureTarget(e)} | 259 | {#if exposureTarget(e)} |
| 257 | <code>{exposureTarget(e)}</code> | 260 | <code>{exposureTarget(e)}</code> |
| @@ -288,6 +291,10 @@ | |||
| 288 | bind:value={hostPortInput} | 291 | bind:value={hostPortInput} |
| 289 | placeholder="host port (optional)" | 292 | placeholder="host port (optional)" |
| 290 | /> | 293 | /> |
| 294 | <select bind:value={protocolInput} aria-label="protocol"> | ||
| 295 | <option value="tcp">tcp</option> | ||
| 296 | <option value="udp">udp</option> | ||
| 297 | </select> | ||
| 291 | <button type="submit">Expose</button> | 298 | <button type="submit">Expose</button> |
| 292 | </form> | 299 | </form> |
| 293 | {/if} | 300 | {/if} |