Skip to content

feat(marketplace): a node proves it can carry a customer before it takes one - #369

Closed
v0l wants to merge 1 commit into
masterfrom
feat/marketplace-health-gate
Closed

feat(marketplace): a node proves it can carry a customer before it takes one#369
v0l wants to merge 1 commit into
masterfrom
feat/marketplace-health-gate

Conversation

@v0l

@v0l v0l commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Increment 4c3b, and the end of 4c. Approving a node no longer enables it.

Everything between an admin approving hardware they cannot see and a customer's VM working — a tunnel that handshakes, a route server that routes, a bridge, a packet filter, a forwarding knob — is machinery nobody has tested on that particular machine. The gate tests it on the customer's own path: an address from a real customer range in the node's region, sent to the node as a guest, pinged from the route server.

That path is the point, and it is the one you described: a VM's address is statically routed from the core network to the node's wgln0, forwarded across br-lnvps to the guest, and answered back out the node's default route, while the guest routes to the core router's address that the node answers for by proxy ARP. A probe on any other address, or from anywhere else, tests a path no customer takes — and the failure worth catching is the node that believes it is fine.

Runs when a node's tunnel is allocated (the first moment it can) and on demand via POST /api/admin/v1/marketplace/nodes/{id}/health_check.

Decisions worth reviewing

  • The run is recorded before it starts. A gate that only wrote a row once it had taken an address would tell the operator whose node never handshook precisely nothing — the most common failure.
  • The address is released in the same statement as the verdict. Otherwise a gate that failed halfway leaks one address per attempt out of a range customers are waiting for.
  • The allocator can see held probe addresses. They are not vm_ip_assignment rows, so without that a VM could be handed the address a gate is proving a node with — two machines on one address, the failure this increment exists to prevent rather than cause.
  • IPv4 ranges first. A guest's v6 address is normally EUI64 from its MAC and a probe has no guest, so v4 is the path a probe stands in for exactly. A v6-only region is still gated, from its range's first free address.
  • The node is told to apply the document rather than left to its heartbeat, via a new POST /api/v1/dataplane/refresh on the node control API. Nothing about the document is sent — the node fetches it itself with its own credential; this only says when.
  • probe_address echoes its verdict instead of using ping's exit code. To the SSH transport a non-zero exit means "the command failed", which is right everywhere else and wrong here: "no reply" is the answer the gate asked for and must stay distinguishable from a route server that cannot be reached at all.
  • A failure never retries forever. A failed gate is a recorded verdict; Err is reserved for not being able to ask, which is LNVPS's problem rather than the operator's.

Found while writing the tests

The node's guest list (node_guests) and the route server's plan (guest_addresses) are two separate functions over the same idea. Adding the probe to one and not the other produces a node holding an address the route server never routes — a gate that fails for a reason having nothing to do with the node. Both now carry it, and a test asserts both.

Failure behaviour

Approved and unusable, with the failing step named — a node that never carries a customer is a support conversation, one that carries a customer badly is an outage. GET /marketplace/nodes/{id} gains host_enabled and probe, which together answer "why is my node approved and empty?".

Testing

8 gate tests covering pass, unreachable-from-the-route-server, no handshake, unfiltered, node-not-answering, address held against the allocator, and the probe being routed like a guest. The control client is behind a trait so the gate can be tested against a node that answers in a chosen way — every interesting case is a node behaving badly, and none of those are states a real node can be asked to be in.

Migration applied against a real MariaDB, including the NULL-collides-with-nothing behaviour the released-address design relies on. Workspace suite green, clippy clean, netns harness green, 100% function coverage on everything added.

…kes one

Increment 4c3b, and the end of 4c. Approving a node no longer enables it.

Everything between an admin approving hardware they cannot see and a customer's
VM working — a tunnel that handshakes, a route server that routes, a bridge, a
packet filter, a forwarding knob — is machinery nobody has tested on that
particular machine. The gate tests it, on the customer's own path: an address
from a real customer range in the node's region, sent to the node as a guest,
pinged **from the route server**.

That path is the point. A VM's address is statically routed from the core
network to the node's tunnel address, forwarded across the guest bridge, and
answered back out the node's default route, while the guest routes to the core
router's address that the node answers for by proxy ARP. A probe on any other
address, or from anywhere else, would test a path no customer takes — and the
failure worth catching is the node that believes it is fine.

The gate runs when a node's tunnel is allocated, which is the first moment it
can run at all, and on demand from the admin API for the cases after that: an
operator who has fixed their firewall, a node whose route server was down.

Decisions worth the words:

- **The run is recorded before it starts.** A gate that only wrote a row once it
  had taken an address would tell the operator whose node never handshook
  precisely nothing — and that is the most common failure.
- **The address is released in the same statement as the verdict.** A gate that
  recorded its result and then failed to give the address back would leak one
  address per attempt out of a range customers are waiting for.
- **The allocator can see held probe addresses.** They are not
  `vm_ip_assignment` rows, so without that a VM could be handed the address a
  gate is proving a node with — two machines answering for one address, which is
  the failure this increment exists to prevent, not cause.
- **IPv4 ranges first.** A guest's IPv6 address is normally derived from its MAC
  and a probe has no guest, so the v4 path is the one a probe stands in for
  exactly. A v6-only region is still gated, from the first free address.
- **The node is told to apply the document rather than left to its heartbeat**,
  through a new `POST /api/v1/dataplane/refresh` on the node control API.
  Nothing about the document is sent: the node fetches it itself, with its own
  credential. This only says *when* — an approval that took a minute to conclude
  would be a minute of an operator watching nothing happen.
- **`probe_address` echoes its verdict instead of using ping's exit code.** To
  the SSH transport a non-zero exit means "the command failed", which is right
  everywhere else and wrong here: "no reply" is the answer the gate asked for,
  and it has to stay distinguishable from a route server that cannot be reached
  at all. Those two results need different people.
- **A failure never retries forever.** A failed gate is a recorded verdict, not
  a job to repeat; `Err` is reserved for not being able to *ask*, which is
  LNVPS's problem rather than the operator's.

Caught while writing the tests: the node's guest list (`node_guests`) and the
route server's plan (`guest_addresses`) are two separate functions over the same
idea. Adding the probe to one and not the other would have produced a node
holding an address the route server never routed — a gate failing for a reason
that had nothing to do with the node.

A failure leaves the node approved and unusable with the failing step named,
which is the safe direction: a node that never carries a customer is a support
conversation, and one that carries a customer badly is an outage. `GET
/marketplace/nodes/{id}` gains `host_enabled` and `probe`, which together answer
"why is my node approved and empty?".
@v0l

v0l commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Closing — this does not do what its name claims.

It never spawns a VM. It takes an IPv4 address from a customer range, has the node hold it on br-lnvps, pings it from the route server, and enables the host if the ping answers. That proves the tunnel handshook, the route server routes and admits the address, the node's bridge route and nft binding exist, forwarding is on, and proxy ARP answers. It proves nothing at all about the node building and running a VM, which is the first thing a customer touches.

It also spends an IPv4 address per run, on a platform where IPv4 is the scarce resource, to check the cheapest of the properties worth checking.

The right shape is a real probe VM — built by the node through exactly the customer path, IPv6 only, held in LNVPS's memory rather than the database (so an API restart mid-probe means the VM is simply absent from the next document and the node tears it down), chosen at random when a node polls, then measured over SSH: login works, memory actually allocates rather than paging, disk read/write speed, and time from asked to answering. Results stored as a series, not a verdict.

That is blocked on the node being able to build a VM at all: its document describes addresses, not machines, and get_host_client has no arm for VmHostKind::MarketplaceNode. That is the next increment, and doing it first means LNVPS's own probes exercise the provisioning path before a customer is the first VM a node ever builds.

@v0l v0l closed this Aug 7, 2026
@v0l
v0l deleted the feat/marketplace-health-gate branch August 7, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant