Marketplace: drive libvirt over the tunnel, and prove a node with a real VM - #370
Open
v0l wants to merge 23 commits into
Open
Marketplace: drive libvirt over the tunnel, and prove a node with a real VM#370v0l wants to merge 23 commits into
v0l wants to merge 23 commits into
Conversation
The gate in #369 never spawned a VM. It held an IPv4 address from a customer range, had the node hold it on the bridge, and pinged it from the route server — a data-plane reachability check wearing the health gate's name. It proved routing, filtering and forwarding; it proved nothing about the node building and running a VM, which is the first thing a customer touches. It also spent an address from the platform's scarcest pool to check the cheapest property worth checking. Replaced in the plan by probe VMs: a real VM built through the customer path, IPv6 only, held in memory rather than the database so an API restart mid-probe leaves it absent from the next document and the node tears it down, chosen at random when a node polls, measured over SSH for login, memory that actually allocates, disk speed and provisioning time, and recorded as a series. Blocked on the node being able to build a VM at all, which is increment 5.
The node holds desired state and manages libvirt itself; LNVPS reads state over the tunnel through a host client whose creating operations are no-ops, because creation is the document's job. Not libvirtd on the tunnel: libvirt is machine-wide, so a connection to it hands LNVPS every domain on the operator's machine including their own, and it adds a second authentication scheme beside the mutually-pinned channel already built. The node is the only party that needs libvirt and is already the party that configures the machine. Still a host client, because every existing worker flow speaks VmHostClient: a node that implements the reading half slots into all of them with no special cases, and the writing half being a no-op is honest about where the write path actually is. Split into 4d1 (the document), 4d2 (the node's VM manager) and 4d3 (the host client), with probe VMs following once a node can build a VM at all.
Increment 4d1. `GET /api/v1/node/state` returns a node's whole desired state: the data plane it already got, plus every VM it should be running — shape, image, rendered cloud-init, MAC, addresses, and whether LNVPS wants it running. The node makes its machine match: it creates what is missing and destroys what is absent. There is no delete message, because absence is a better one — it survives LNVPS failing. An API that dies mid-provision leaves nothing to clean up, since the half-built VM is simply not in the next document, and a probe VM that only ever existed in LNVPS's memory disappears the same way. Three things the node deliberately does not get: - **Database rows.** The format is node-shaped. A node has no business knowing about users, cost plans or subscriptions, and a wire format that mirrored the schema would make every migration a node-compatibility question. - **The decision about cloud-init.** It arrives rendered. Sending the ingredients would put the decision on the operator's machine, where a node running an older daemon would configure a guest differently from the rest of the fleet. - **The image itself.** A URL and a checksum: the node fetches it over its own connection and verifies it. Streaming images through LNVPS would make every provision wait on our bandwidth for a file the node can get directly. Two distinctions that matter more than they look: - A **deleted** VM is absent, which destroys it. A **disabled** one stays and is marked not-running — removing it would tell the node to destroy the customer's disk, which is a very different thing from "stop billing this". - A node with no tunnel is told its data plane is `null` but is still told about its VMs. Its disks and domains outlive its network, and a tunnel being re-allocated must not read as an instruction to destroy every customer on the machine.
This reverts commit 5b8d13e.
…ment The node stays dumb: tunnel, data plane, firewall, and a libvirtd that LNVPS drives directly with the existing client. get_host_client gains an arm and every worker flow works, because a marketplace node becomes just another libvirt host. Two facts decide the shape. libvirtd's namespace determines where VM taps land, so it has to run inside /run/netns/lnvps to reach br-lnvps — as a dedicated instance, because moving the operator's libvirtd there would take the networking off every VM they already run and hand LNVPS their domains. And guests can reach the node's tunnel address from the bridge, so a listener there needs TLS client certificates: one nft regression must not hand a customer VM control of the node and everyone else on it. Reverts the desired-state VM document, which this replaces.
…or's The first half of driving VMs on a marketplace node over the tunnel: the node runs a libvirtd for LNVPS's guests. Which libvirtd it is turns out to be the whole design, and every part of it is forced by something in the code rather than chosen. It runs in the data plane network namespace, because a domain's tap is created in the namespace of the libvirtd that starts it and the guest bridge only exists there. A libvirtd in the machine's namespace would build VMs with nowhere to plug them in. It is a second instance rather than the machine's. Moving the operator's libvirtd into our namespace would take the networking off every VM they already run, and would put their domains inside the connection LNVPS drives. An operator listing spare capacity on a box they already use is most of the marketplace pitch. It gets a private mount namespace as well as a network one. Two system libvirtds sharing /var/lib/libvirt is not supported and not subtle either: they contend over domain state, storage pool definitions and their sockets. Four BindPaths lines give the second instance a complete separate world. It writes logs directly instead of through virtlogd, which is reached through a socket in /run/libvirt — one of the paths we replace, so the host's socket-activated one is not visible in the sandbox. stdio_handler = "file" removes the dependency rather than running a second helper daemon on hardware we do not own. Access is TLS with a client certificate, and that is load-bearing rather than belt-and-braces. The guest bridge shares a namespace with the tunnel interface, so a customer VM can address this listener; the packet filter drops that today, and the certificate is what makes a filter regression survivable instead of handing a guest control of the node and everyone else on it. libvirtd binds the tunnel address only — the machine's other interfaces are the operator's LAN and their uplink. The server certificate is self-signed and CA-capable so LNVPS can pin it directly as this node's trust anchor; libvirt verifies a chain rather than a hash, and running our own CA would be a key to hold and a rotation to run for no additional guarantee. LNVPS's client CA arrives in the node's already authenticated document instead of being compiled in, where a rotation would strand every deployed node. The identity persists across restarts, and is regenerated only when the tunnel address moves, because a certificate that does not name the address it is served on leaves the node silently unreachable.
The other half: LNVPS dials the node's libvirtd, and a marketplace node becomes just another libvirt host. get_host_client gains an arm, so every existing worker flow — create, start, stop, resize, migrate, terminal, stats — works unchanged rather than being reimplemented against a node-specific protocol. What differs from a libvirt host on our own network is trust, and that is the bulk of this change. Each node is verified against the certificate it registered, not against a fleet CA. A single CA would mean any node's certificate satisfies a connection to any other, which is exactly the substitution the certificate exists to prevent: something else answering on a node's tunnel address and reporting that a customer's VM is fine when it is not. libvirt's client verifies a chain against a CA file rather than pinning a hash, which is why the node's whole certificate is stored where the control API stores only a fingerprint. libvirt reads its credentials from a directory, so each node gets one, with its certificate as the CA and LNVPS's client certificate and key beside it. That directory is a cache rather than a record: the node re-presents its certificate on every poll, so an API host that loses it — a fresh container, a moved volume — repairs itself within one poll interval instead of leaving nodes undialable until somebody notices. The client key is written owner-only, because a key any process on the API host can read is one that drives every node in the fleet. One client certificate for the whole fleet, not one per node. Every node is told to accept exactly one DN; a per-node client certificate would be a per-node key to issue, ship and rotate for a decision that is identical everywhere. The CA that signs it reaches nodes in their data-plane document, so rotating it takes one poll rather than a new node package. When no client identity is configured, nodes are still enrolled and networked but no VM is placed on one, and the node is told to leave libvirt alone. The alternative — an unauthenticated listener — would put hypervisor control on an address the node's own guests can reach. The node applies libvirt after its data plane and never before: libvirtd binds the tunnel address and its unit names the namespace, so both must exist first. A failure there is logged rather than fatal, because a node whose network is up still carries the guests it already has, and refusing to continue would take them down over a hypervisor LNVPS may not yet be placing VMs on.
Tests for the parts that would otherwise fail quietly, plus the changelog entry and the work-file record. The three that matter most are refusals. A node that never registered a certificate is refused rather than dialled, because connecting without one means not checking which machine answered — on hardware LNVPS does not own, at an address that node's own guests share a namespace with. A marketplace host with no node behind it is named as a broken row rather than producing a connection to nowhere. And with no client identity configured, no client is built at all: an unauthenticated hypervisor connection over the tunnel is worse than not having the feature. The rest cover rotation, which is the ordinary case rather than the exotic one — a node restored from backup, or one whose tunnel address moved, regenerates its identity and re-presents it, and both the stored certificate and the on-disk trust anchor follow. A node that could not do that would be undialable for good. MockDb's update dropped the new column, which would have made every test here pass against a database that never stored anything. The netns harness passes unchanged: it proves the network, and a hypervisor started inside it would be testing systemd.
A disposable node stack, in lnvps_e2e::stack, with every name and address derived from one tag and one index. The literals were not a tidiness problem. A harness full of constants can only run one stack at a time, so tests serialise, a run killed part-way leaves debris the next one trips over, and two tests that both want "the node" fight over the same interfaces. Worse, when the address in a test and the address in the fixture are two separate constants, they can disagree — and what that looks like is a failing assertion about production code. This was not hypothetical. Deriving the addresses immediately exposed two places where the harness had been getting away with agreeing with itself by accident: the route server was configured to listen on one port while the node was told to dial another, and the pool block was written as the route server's own address rather than its network. Both had been invisible because every value was 0 or 1. Names now come from the constraint that decides them. Namespaces are named after the tag, because a human reading `ip netns list` during a failure wants to know which test left them. Interfaces are named after the index and asserted against the kernel's 15-byte limit, because `ip` refuses a longer name outright and a descriptive name is worth nothing if the link cannot be created. Teardown sweeps one list of interfaces, the same list the fixture creates from, so a link added in one place cannot be forgotten in the other. It runs on construction as well as on drop: a previous run killed with SIGKILL leaves its namespaces behind, and a harness that fails because of the last failure hides which one was real. The node's own constants — the bridge, the tunnel interface — are read from production rather than restated, and the operator-owned `wg0` is named in the fixture with the reason it exists: an operator using the obvious name for their own tunnel is the case that made production rename its interface to wgln0. Also removes the node's duplicate copy of LNVPS's client DN. The node holds no opinion about it — LNVPS states it in the document — and a constant on that side would only be a second opinion to disagree with. The libvirt instance's private directories are now one list used both to create them and to render the unit that bind-mounts them; two lists would be a mount of a directory nobody made, which systemd reports as a unit that will not start and nothing else.
Stack::bring_up configures the route server and applies the node's data plane from a single document, replacing the copy that lived in the tunnel test. A second copy is a second place for the two ends to be configured differently, and a harness whose ends disagree fails as a production bug — which is exactly how the port and pool-block mismatches showed up in the previous commit. The libvirt instance's namespace is now carried in Paths rather than taken from the constant, so a harness can stand a whole node up beside a real one without either taking the other's guests. Preparation for running a real libvirtd under the harness. The instance is started from the unit production renders, written where systemd reads units, because the unit is the artefact least covered by anything else: two namespaces, four bind mounts and a config file, none of which a unit test can do more than assert the text of.
Running the thing found four defects, three of them in code already merged. **The pid file.** libvirtd's default is /run/libvirtd.pid, which is in /run — a directory this instance does *not* replace. The operator's libvirtd holds that lock, so ours exited with "resource temporarily unavailable" naming a path neither daemon appears to configure. It now writes inside the private /run/libvirt, which is one of the four directories the unit already binds. **The certificate.** libvirt refuses to serve a CA certificate as its own leaf: "basic constraints show a CA, but we need one for a server". The whole self-signed-and-CA-capable design was therefore unstartable. The node now roots its own one-certificate chain — a CA, which is what it registers with LNVPS and what LNVPS verifies against, and a server certificate signed by it, which is what libvirtd presents. LNVPS still runs no CA for node identities. **The trust files.** Both ends validate the certificate they *present* against the same file they verify the far end with. With only the peer's CA in it, the daemon will not start and the client will not connect, and both errors name the complainant's own certificate rather than anything about the peer. Each side's file now carries two CAs. The cost is stated in the code: a client certificate issued by the node's own CA would also pass the node's check — tls_allowed_dn_list still admits only LNVPS's DN, and anything holding that CA key already owns the machine. **The filter.** The node's input chain accepted ICMP and nothing else, so every TCP call LNVPS makes to a node was dropped — the libvirt connection and the control API alike. That is a node which pings, handshakes, reports itself healthy, and answers nothing, and it has been the case since the filter landed. TCP is now accepted on the control and libvirt ports, bound to the tunnel interface and never the guest bridge: the bridge shares this namespace with the tunnel, and libvirtd on that address is root on the machine. Also gives the unit a RestartSec. Without it systemd spent all five attempts inside a second and left "start request repeated too quickly" as the last thing an operator sees, which says nothing about the cause — as it did here.
Five findings, three in already-merged code, none reachable by a test that asserts what code decides rather than what a kernel does. The filter one is the lesson: 4c2 and 4c3a each shipped with tests asserting the right ruleset and the right client, and the combination was inert, because nothing asserted a packet.
The foundation for probe VMs: an address a probe can be reached on, and a series to record what it found. **The address is derived, not allocated.** A probe stores nothing — no VM row, no IP assignment, no subscription — because a probe that outlives the process which made it is our VM left running on hardware we do not own, and a table of them needs a reaper, which is one more thing to fail quietly. But a guest is only reachable if its address is in three places it does not control: the route server's routes, the peer's AllowedIPs, and the node's packet filter, all built from the database. An address that is not in the database is an address the network drops. So the probe's is a pure function of the node's own inner address — both ends can work it out, and an API that dies mid-probe leaves nothing to reclaim. It is IPv6 only. IPv4 is the scarce resource the marketplace exists to stretch, so spending one to check a machine is backwards, and a node that cannot carry a v6 guest cannot carry a dual-stack one. The address is in the node's document **whether or not a probe is running**. A probe lives for a few minutes between two of the node's polls; a document that named it only while it ran would give probes that fail because the node had not fetched the document yet, and nodes that tear the address down while LNVPS is still logged in. It costs one entry in an anti-spoof list, and it buys a check whose result never depends on timing. **The results are a series, not a verdict.** One bad run is a bad afternoon — a backup job, a neighbour compiling something — and suspending a node for it would make the marketplace hostile to the people it needs. A trend is a node to act on, and increment 12's trust tier wants the history rather than the last answer. Failures are rows too: a node that never completes a probe looks identical to one nobody probed unless they are written down. Each row carries the shape and image it measured. Regions sell different templates, so comparing raw seconds across nodes would rank machines by what we happened to ask them for. Denormalised on purpose — a template edited later must not change what an old measurement appears to say.
The runner. LNVPS assembles a VM in memory, builds it on the node through the ordinary client, hands it to a measurement, and destroys it on every path out — success, failure, or panic. **The id is the safety property.** Domains on a node are named from the VM id and delete_vm finds a domain by that name, so a probe sharing an id with a customer's VM would destroy a customer's disk. Probe ids start at 0xFFFF_0000_0000_0000: an AUTO_INCREMENT column would have to issue eighteen quintillion rows to reach it, which makes the collision impossible rather than unlikely. The id is also stable per node, so a probe left behind by a killed process is found and removed by the next one instead of accumulating — and it is removed *before* a probe starts as well as after, because a stale domain would otherwise make every later probe on that node fail for a reason that has nothing to do with the machine. **Nothing is stored about the VM.** It exists in this process's memory and in the node's libvirt. A row pointing at one would need a reaper, and a reaper that fails leaves our VM running on hardware we do not own. A delete that fails is reported on the result rather than logged, because a node whose probes cannot be cleaned up is one LNVPS is accumulating VMs on, and that belongs in the series rather than in a log nobody reads. **It is built from what customers buy** — the region's cheapest enabled template and a real image of the matching architecture, with cloud-init rendered by the same code a customer's VM uses. A dedicated probe shape would be one more thing to keep in step, and a node could pass on it while failing on everything for sale. The synthetic IP assignment carries a bare address because that is what the database stores: the renderer parses it as an address and silently drops anything with a prefix, which boots a guest with no network on a node that is perfectly fine. An unreachable node is a failed probe rather than an error thrown away, because an unreachable node is exactly what this exists to detect.
AdminResource::MarketplaceNode (28) and MarketplaceOperator (29) were added to the enum and used by every admin marketplace endpoint, and never granted to any role. There is no super-admin bypass — has_permission is a set lookup against the tuples a role holds — so the entire admin surface of the marketplace has been answering 403 to every caller since it landed: approve, suspend, drain, reject, revenue share, payouts, node status. Nothing failed when the resources were introduced, because a resource with no grants is a legitimate state: it means nobody may use it. That is the same shape as the node filter that accepted only ICMP — each piece correct, the combination inert. The two resources stay separate, as intended: node actions are operational and operator actions are money, and an admin who can pull a misbehaving node out of service should not thereby be able to change what its owner is paid.
A resource nobody granted is a set of endpoints that answer 403 to every caller, including super_admin, and nothing fails at the point the omission is made — a resource with no grants is a legitimate state as far as the schema is concerned. That is how the marketplace surface shipped unusable. The test reads the migrations as text and compares the grants it finds against the AdminResource enum. Crude, and the only way to check this without a database: the grants live in SQL, the enum lives in Rust, and nothing else compares them. Verified by removing the grant migration, at which point it names exactly the eight missing marketplace tuples. Writing it turned up the three shapes the migrations use — SELECT-based grants, three-column tuples in the original seed with the role as a SQL variable, and four-column tuples carrying an explicit created_at. A parser that understood only one of them would have reported healthy resources as ungranted, which is the failure mode that gets a test deleted rather than fixed.
The measurement half of the probe. Everything LNVPS can see from outside is what the operator's node chooses to report; this is the part that finds out what a customer would actually get, by being one for a few minutes. Three numbers, each chosen for what it catches: - **Time to first login**, from asking for the VM to having a shell. It catches the node that is technically working and unusably slow. - **Memory allocated and touched**, written through /dev/shm. Allocation alone proves nothing where the host overcommits — the pages have to be faulted in and filled before the machine admits it does not have them. A node reselling four times its RAM passes an allocation check and fails this. - **Disk write and read rates**, with conv=fdatasync and the cache dropped. An operator can present an NVMe node backed by a network volume, or a disk that has begun to fail; both are visible as a rate and invisible as a specification. Nothing is installed in the guest. A probe that waited on a package mirror would be timing the internet, and a node with no outbound access would fail a test about the node. Rates are timed here rather than parsed out of dd, whose output format varies with version and locale — a parser that quietly returned zero on an unfamiliar line would report healthy nodes as broken. The login window is deliberately long: a slow node is a finding to record, and a tight limit turns "this node is slow" into "this node did not answer", which is less useful and more alarming. The keypair is generated per probe and dropped with it, and never printed: a long-lived key that opened a shell on every operator's node would be the most valuable secret LNVPS holds. Key generation lives in lnvps_api_common beside the SSH client that has to load it back, and a test proves the pair matches — a mismatch would look exactly like a node refusing logins. Finding the right RNG took four attempts: the dependency graph carries three versions of rand_core, ssh-key's CryptoRng bound resolves to 0.10's, and 0.10 removed OsRng in favour of rand::rng(). The compiler names the trait through x25519_dalek, which is why it is worth writing down.
Ties the two halves together and adds the scheduling. run_probe builds a VM on a node, logs in as the image's own default user — a probe that assumed root would fail on every image that disables it, which is most of them, and look like a broken node — measures, destroys, and records. The result is written down whatever it is: the first thing anybody will ask about a suspended node is what it did before. Selection is a cooldown and an ordering, not a scheduler. A node that has never been probed comes first, because it is the one LNVPS knows nothing about and may already be placing customers on; after that, longest since. A node probed within six hours is left alone — a probe costs the operator a disk clone, a boot and a few hundred megabytes of I/O on hardware they are barely being paid for, and doing it every few minutes would be indistinguishable from abuse. A *failed* probe starts the same cooldown, because retrying a broken node immediately hammers the machine least able to cope, and the failure is already recorded for whoever decides what to do about it. An approved node with no host yet is skipped rather than recorded as a failure: it is a state every node passes through on the way in, and a failure row would make a normal enrolment look like a broken machine. The shell commands are now pure functions with their own tests, which is both better structure and the only way to cover them: the wrappers around them need a live guest, which is the e2e's job. Worth having done — the tests state why the memory test writes 45% of RAM (tmpfs defaults to half, and the guest still has to run), why it frees the memory in the same command (a lost connection must not leave RAM full), and why the disk write syncs (without it a slow disk with plenty of RAM reports a gigabyte a second, which is the exact node this exists to catch).
Wires the probe into the worker and makes a passing probe the thing that opens a node to customers. This is the gate the whole marketplace rests on. An enabled host is one LNVPS will place paying customers on, and until now the only thing standing between an operator's machine and a customer was that the machine answered questions about itself. Everything LNVPS can see from outside is what the node chooses to report; a VM having actually run there is not. One node per sweep, every five minutes. The per-node cooldown decides how often any given machine is touched; this only decides how quickly a newly approved node gets its first VM, which is what enables its host and lets it start earning. A sweep that probed the whole fleet at once would arrive as a thundering herd on the operators least able to absorb it. A failing probe is recorded and not acted on. One bad run is a bad afternoon — a backup job, a noisy neighbour — and taking a node out of service for it would make the marketplace hostile to the people it needs; suspension on a trend is increment 12's job. A node that cannot be probed at all does not have its host enabled, which is asserted directly, because the failure mode here is not a crash but a customer placed on hardware nobody has tested. A build without the linux-ssh feature refuses to probe rather than probing without measuring: creating and destroying a VM proves the node can build one, and reporting that as health would be the overclaim this whole increment exists to avoid.
GET /api/admin/v1/marketplace/nodes/{id}/health — the probe series, newest
first, paged at the database.
A series rather than a verdict, because that is the question an admin actually
has. One bad run is a bad afternoon; a node worth suspending is one whose numbers
have been getting worse. Failures are rows with the reason verbatim: somebody
deciding whether to take an operator's hardware out of service needs what
actually happened rather than a category, and a node that never completes a probe
is indistinguishable from one nobody probed unless the failures are visible.
Every row carries the shape and image it measured. Regions sell different
templates, so comparing raw numbers across nodes would rank machines by what
LNVPS happened to ask them for.
Remembered rather than live, unlike /status: these are measurements that were
made, and re-running one because somebody opened a page would put real load on an
operator's machine.
The permission test is deliberate: this endpoint checks marketplace_node::view,
which is the check that was granted to nobody until this week.
The instance's /etc/libvirt is its own and starts empty, so the pools on the operator's machine are not visible to it — that is the isolation working, not a problem to route around. But it means every VM creation would have failed with "storage pool not found", which nothing so far would have caught: the client asks for a pool by name, and a name that does not exist is a runtime error on the first real provision. Defined by writing the XML libvirtd reads at startup rather than through libvirt's API, because this instance's /etc/libvirt is ours to write. The node therefore still needs no libvirt client: the daemon stays a thing that writes configuration files, which is the whole reason it is small enough to audit. Autostart is a symlink into the definitions directory, which is how libvirt records it itself — a copy would be a second definition free to drift, and a pool that has to be started by hand is a node that works until it reboots. The pool name is fixed rather than configurable. LNVPS's client asks for a pool by name, so a setting here would be a setting an operator could change into a node that silently refuses every VM. Its target is a plain directory an operator can bind-mount or symlink elsewhere, which keeps the arrangement one libvirt understands without LNVPS having to describe every storage layout. The e2e now asserts the pool exists and is running on the real daemon.
Holding an address on an interface makes the kernel route it, and that route is not the node's to remove. With IPv6 the attempt fails outright — "no such process" — which aborts the whole apply, so a node with an IPv6 gateway could not bring its data plane up at all. Found by the first probe VM, which is the first guest anywhere with an IPv6 gateway: every customer so far has had a v4 one, where the kernel keeps the equivalent route in the local table and the reconcile never saw it. The unit test fails without the fix. Also copies libvirt's stock packet-filter definitions into the LNVPS instance. A guest's interface references `no-mac-spoofing`, and libvirt refuses to start a domain whose filter is missing; the operator's copies are not visible in this instance, which is the isolation working. Copied rather than reimplemented — writing our own would filter a marketplace guest differently from every other libvirt guest on earth, invisibly until it mattered. The stack fixture gains apply_pool, which configures the route server from the database through plan_pool exactly as the worker does. Building that side by hand let the harness configure it more helpfully than the API would, which is how a probe that fails in production passes in a test.
A probe VM now runs end to end against a real node: LNVPS builds it through the customer path, logs in over the tunnel, measures it and destroys it. The e2e takes 75 seconds and is in scripts/tunnel-e2e.sh with the other three. Getting there found four more bugs, three of them in code every customer VM already goes through. **An all-digit MAC is a number to YAML.** `52:54:01:00:00:01` unquoted is sexagesimal — 41135256001 — so netplan rejects it, cloud-init aborts its network stage, and the guest boots with no network on a host that is working. A MAC with any hex letter is a string and works, so this breaks roughly one VM in sixteen and looks like bad hardware. Values are now quoted. The test asserts the rendered text rather than round-tripping: the parser in this workspace is YAML 1.2, which dropped sexagesimal and reads it correctly either way, while the parser that consumes the file is netplan's 1.1. **A gateway outside the guest's prefix needs on-link.** Without it netplan silently produces a configuration in which the guest cannot reach its router at all. **A probe's gateway must not be the route server's address.** The node holds its guests' gateway on the bridge, and the route server holds its own address on the same pool: sharing it means the guest's replies are delivered to the node and the route server never sees them. Confirmed by tcpdump — echo requests arriving, replies vanishing into the node. **A guest whose prefix covers the pool treats the route server as on-link** and resolves it on a bridge where nothing answers. The probe is now addressed with a host prefix and an on-link gateway, so everything it sends goes to the node, which is the only thing on that link that can route. The last two are the same mistake from opposite ends, and both were invisible in unit tests: every assertion about addressing passed while no packet could complete a round trip. The login window now starts when the guest does rather than when the VM was requested. Building it includes fetching an OS image the node has never seen and cloning a disk — minutes on a cold node — and a deadline measured from the request spends its whole budget there and then condemns the node for being new.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LNVPS manages VMs on a marketplace node by driving a libvirtd on it over the tunnel, and a node is only opened to customers once a VM has actually run there.
What lands
A libvirtd LNVPS can drive, and nothing of the operator's. Which libvirtd it is turns out to be the whole design, and every part is forced by something in the code:
/var/lib/libvirtcontend over domain state, storage pools and sockets./etc/libvirtstarts empty.Access is TLS with a client certificate, and that is load-bearing rather than belt-and-braces. The guest bridge shares a namespace with the tunnel interface, so a customer VM can address the listener; the certificate is what makes a packet-filter regression survivable instead of handing a guest root on the node and everyone else on it. Each node is verified against the certificate it registered, not a fleet CA — one CA would mean any node's certificate satisfies a connection to any other, which is exactly the substitution being prevented.
A host is enabled only by a passing probe. LNVPS builds a real VM on the node through the ordinary customer path, logs in, measures time-to-login, memory allocated and touched (allocation alone proves nothing where the host overcommits), and disk rates with the cache bypassed — then destroys it. Results are a series (
GET /api/admin/v1/marketplace/nodes/{id}/health) including failures, because one bad run is a bad afternoon and a trend is a node to act on. Nothing about the probe VM is stored, its address is derived rather than allocated, and its id sits in a range no customer VM can reach — domains are named from the VM id, so a probe sharing one would delete a customer's disk.Bugs this found in already-merged code
Running the thing found six defects that no unit test could have caught. Three were in code that shipped weeks ago:
AdminResource::MarketplaceNodeandMarketplaceOperatorwere granted to nobody, and there is no super-admin bypass — so the entire admin marketplace surface has been answering 403 to every caller since increment 1: approve, suspend, drain, reject, revenue share, payouts. A test now compares the migrations against the enum so the next resource cannot repeat it./run/libvirtd.pid— in/run, which the instance does not replace — so the operator's daemon holds the lock and ours refused to start.The common shape: every component correct in isolation, the assembly dead. That is what the netns harness exists for, and it is now three harnesses — the tunnel, the node's libvirtd started from production's own systemd unit, and (still failing, see below) a full probe.
Testing
sudo ./scripts/tunnel-e2e.sh— tunnel and node-libvirt harnesses pass on a real kernel, with a real libvirtd running beside the machine's own.Not proven yet
No probe has completed end to end. The probe e2e gets as far as a running VM — tunnel carries, libvirt authenticates, disk clones, domain starts — and then times out waiting for SSH. That test is deliberately not in this branch: the guest either is not getting its address or is not reachable across the bridge, and I would rather land the parts that are proven than a test that is red. Given this branch's track record I expect that last hop to surface one or two more assembly bugs.
So: the probe is written and unit-tested, not yet demonstrated. Everything else here has been run.