feat(marketplace): a node proves it can carry a customer before it takes one - #369
feat(marketplace): a node proves it can carry a customer before it takes one#369v0l wants to merge 1 commit into
Conversation
…kes one
Increment 4c3b, and the end of 4c. Approving a node no longer enables it.
Everything between an admin approving hardware they cannot see and a customer's
VM working — a tunnel that handshakes, a route server that routes, a bridge, a
packet filter, a forwarding knob — is machinery nobody has tested on that
particular machine. The gate tests it, on the customer's own path: an address
from a real customer range in the node's region, sent to the node as a guest,
pinged **from the route server**.
That path is the point. A VM's address is statically routed from the core
network to the node's tunnel address, forwarded across the guest bridge, and
answered back out the node's default route, while the guest routes to the core
router's address that the node answers for by proxy ARP. A probe on any other
address, or from anywhere else, would test a path no customer takes — and the
failure worth catching is the node that believes it is fine.
The gate runs when a node's tunnel is allocated, which is the first moment it
can run at all, and on demand from the admin API for the cases after that: an
operator who has fixed their firewall, a node whose route server was down.
Decisions worth the words:
- **The run is recorded before it starts.** A gate that only wrote a row once it
had taken an address would tell the operator whose node never handshook
precisely nothing — and that is the most common failure.
- **The address is released in the same statement as the verdict.** A gate that
recorded its result and then failed to give the address back would leak one
address per attempt out of a range customers are waiting for.
- **The allocator can see held probe addresses.** They are not
`vm_ip_assignment` rows, so without that a VM could be handed the address a
gate is proving a node with — two machines answering for one address, which is
the failure this increment exists to prevent, not cause.
- **IPv4 ranges first.** A guest's IPv6 address is normally derived from its MAC
and a probe has no guest, so the v4 path is the one a probe stands in for
exactly. A v6-only region is still gated, from the first free address.
- **The node is told to apply the document rather than left to its heartbeat**,
through a new `POST /api/v1/dataplane/refresh` on the node control API.
Nothing about the document is sent: the node fetches it itself, with its own
credential. This only says *when* — an approval that took a minute to conclude
would be a minute of an operator watching nothing happen.
- **`probe_address` echoes its verdict instead of using ping's exit code.** To
the SSH transport a non-zero exit means "the command failed", which is right
everywhere else and wrong here: "no reply" is the answer the gate asked for,
and it has to stay distinguishable from a route server that cannot be reached
at all. Those two results need different people.
- **A failure never retries forever.** A failed gate is a recorded verdict, not
a job to repeat; `Err` is reserved for not being able to *ask*, which is
LNVPS's problem rather than the operator's.
Caught while writing the tests: the node's guest list (`node_guests`) and the
route server's plan (`guest_addresses`) are two separate functions over the same
idea. Adding the probe to one and not the other would have produced a node
holding an address the route server never routed — a gate failing for a reason
that had nothing to do with the node.
A failure leaves the node approved and unusable with the failing step named,
which is the safe direction: a node that never carries a customer is a support
conversation, and one that carries a customer badly is an outage. `GET
/marketplace/nodes/{id}` gains `host_enabled` and `probe`, which together answer
"why is my node approved and empty?".
|
Closing — this does not do what its name claims. It never spawns a VM. It takes an IPv4 address from a customer range, has the node hold it on It also spends an IPv4 address per run, on a platform where IPv4 is the scarce resource, to check the cheapest of the properties worth checking. The right shape is a real probe VM — built by the node through exactly the customer path, IPv6 only, held in LNVPS's memory rather than the database (so an API restart mid-probe means the VM is simply absent from the next document and the node tears it down), chosen at random when a node polls, then measured over SSH: login works, memory actually allocates rather than paging, disk read/write speed, and time from asked to answering. Results stored as a series, not a verdict. That is blocked on the node being able to build a VM at all: its document describes addresses, not machines, and |
Increment 4c3b, and the end of 4c. Approving a node no longer enables it.
Everything between an admin approving hardware they cannot see and a customer's VM working — a tunnel that handshakes, a route server that routes, a bridge, a packet filter, a forwarding knob — is machinery nobody has tested on that particular machine. The gate tests it on the customer's own path: an address from a real customer range in the node's region, sent to the node as a guest, pinged from the route server.
That path is the point, and it is the one you described: a VM's address is statically routed from the core network to the node's
wgln0, forwarded acrossbr-lnvpsto the guest, and answered back out the node's default route, while the guest routes to the core router's address that the node answers for by proxy ARP. A probe on any other address, or from anywhere else, tests a path no customer takes — and the failure worth catching is the node that believes it is fine.Runs when a node's tunnel is allocated (the first moment it can) and on demand via
POST /api/admin/v1/marketplace/nodes/{id}/health_check.Decisions worth reviewing
vm_ip_assignmentrows, so without that a VM could be handed the address a gate is proving a node with — two machines on one address, the failure this increment exists to prevent rather than cause.POST /api/v1/dataplane/refreshon the node control API. Nothing about the document is sent — the node fetches it itself with its own credential; this only says when.probe_addressechoes its verdict instead of using ping's exit code. To the SSH transport a non-zero exit means "the command failed", which is right everywhere else and wrong here: "no reply" is the answer the gate asked for and must stay distinguishable from a route server that cannot be reached at all.Erris reserved for not being able to ask, which is LNVPS's problem rather than the operator's.Found while writing the tests
The node's guest list (
node_guests) and the route server's plan (guest_addresses) are two separate functions over the same idea. Adding the probe to one and not the other produces a node holding an address the route server never routes — a gate that fails for a reason having nothing to do with the node. Both now carry it, and a test asserts both.Failure behaviour
Approved and unusable, with the failing step named — a node that never carries a customer is a support conversation, one that carries a customer badly is an outage.
GET /marketplace/nodes/{id}gainshost_enabledandprobe, which together answer "why is my node approved and empty?".Testing
8 gate tests covering pass, unreachable-from-the-route-server, no handshake, unfiltered, node-not-answering, address held against the allocator, and the probe being routed like a guest. The control client is behind a trait so the gate can be tested against a node that answers in a chosen way — every interesting case is a node behaving badly, and none of those are states a real node can be asked to be in.
Migration applied against a real MariaDB, including the
NULL-collides-with-nothing behaviour the released-address design relies on. Workspace suite green, clippy clean, netns harness green, 100% function coverage on everything added.