Hold enrolled workloads until Phantom Engine is enforce-ready

docs/planning/boot-hold/FD-REBOOT-1-LAB-PROCEDURE.md

@@ -0,0 +1,112 @@
+# FD-REBOOT-1 lab procedure — reboot hold reproof
+
+**Audience:** INTERNAL_RESTRICTED  
+**Do not run this from the open-core PR.** This file is the procedure for a later authorized Free-plan host reproof.  
+**Row:** `B1-REBOOT-EXPOSURE` / GAP-FIP-014  
+**Until that host reproof:** **NOT_PROVED**. Unknown is never PASS.  
+**Frozen seal (not retro-claimed):** `e0b19d557891b1ee8bbd20e702df11669d175e4083ef5bbe2f7077cf30093b5e`  
+**Ceiling:** `INTERNAL_CLEAN_HOST_PROOF`  
+**Account:** `960577828987` only  
+**Expected out-of-pocket:** `expected_oop_usd=0` (OOP $0). No Cost Explorer. No Paid instance sizes. No publish. No soak, fleet, or k3s.
+
+Measured on the short run `2026-09-30T2225ET-b1-short` before this change: `W_unprotected≈23.05s`, `W_race≈12.84s`. Loader down and pins empty during the window. The enrolled timeout was not a Phantom Engine deny. An unenrolled connect completed. This procedure exists so a later run can record hold time and show zero enrolled allows. It does not convert that earlier run into a pass.
+
+## Guards (same shape as the B1 short)
+
+1. Fresh cost gate at launch: Free/Active, `expected_oop_usd=0`, `abort=false`, account `960577828987`, no Cost Explorer, no Paid.
+2. One lab instance. Prefer `t3.micro`, then `t3.small`. `associate_public_ipv4=false`. `stop_after_minutes` stays inside the short-lab cap. No public IPv4.
+3. Region and tags follow the B1 short plan. Re-establish access with SSM or serial.
+4. Install the open-core boot hold on the guest (`vantio-boot-hold install`) and set `VANTIO_BOOT_HOLD_LIVE=1` only for the root install and boot units. Enroll one workload and record one unenrolled control with `observe-unenrolled`.
+5. Point the loader at `/sys/fs/cgroup/vantio-enrolled.slice` and keep the Phantom Engine container on `--restart=no`, outside that slice.
+6. Export timestamps before any terminate. Do not mark the row PASS in the export. A missing timestamp is NOT_PROVED.
+
+## Clock and metrics
+
+Record `T_boot`, `T_egress`, and `T_ready` with the guest clock source (chrony/UTC). Also record `T_hold`, the guest time when `vantio-boot-hold` state is `HELD` and the scoped rules are present.
+
+- `W_unprotected = max(0, T_ready − T_boot)`
+- `W_race = T_ready − T_egress` when `T_egress < T_ready`, else `0`
+- `W_hold = T_ready − T_hold` when both exist
+
+Report the measured numbers. Do not invent a maximum-seconds marketing threshold.
+
+Target for a future PASS, which this procedure does not award: zero enrolled allows at any point during boot, hold time recorded, and either a deny of the first enrolled egress or a documented fail-closed hold while the loader is absent. An unenrolled connect may complete. That shows the host was not placed on a host-wide default-route hold.
+
+## Matrix
+
+Run these as separate boots. Each boot exports its own `timestamps.json`, `egress-probe.txt`, `hold-status.json`, and a journal excerpt. The enrolled probe and the unenrolled probe race from t=0 (a unit that fires as soon as its gate allows).
+
+### 1. Enrolled and unenrolled from t=0 (both mechanisms on)
+
+Default config: hold on, ordering on. Reboot. From the first userspace moment, attempt enrolled egress and unenrolled egress.
+
+Expect: enrolled unit does not become active before `vantio-pe-enforce-ready.service`. If it is forced to run, its egress does not complete while `state` is `HELD`. Unenrolled egress may complete. SSH session survives. Record `W_hold`.
+
+### 2. Loader is running but enforcement is not attached
+
+Start the loader process without attaching `cgroup_skb_egress_enforce` to `/sys/fs/cgroup/vantio-enrolled.slice` (pins may be present). Reboot or start from a held boot.
+
+Expect: `vantio-boot-hold release --require-enforce-ready` refuses. Status stays `HELD` / `DEGRADED`. The message says the loader is running and enforcement is not attached. Enrolled workloads stay stopped. Do not treat the running process as enforce-ready.
+
+### 3. Loader fails, hold stays, SSH works, break-glass releases the start gate
+
+Leave `/etc/vantio/pe-loader.argv.json` absent or point it at a missing binary. Reboot.
+
+Expect: `vantio-pe-loader.service` fails, `vantio-pe-enforce-ready.service` fails, enrolled workload stays inactive, `vantio-boot-hold status` shows `HELD` and `DEGRADED` with the operator message. Open an SSH session and a console session. From SSH as root, one command, `vantio-boot-hold release --break-glass --i-am-root-operator`, returns `RELEASED`. Health stays `DEGRADED`. `audit.log` contains `BREAK_GLASS` and names what was released: `packet-hold-ipv4`, `packet-hold-ipv6`, `file-hold`, and `start-gate:` for each enrolled unit. After that command, `systemctl start` of the enrolled unit is not blocked by `Requires=vantio-pe-enforce-ready.service`. Do not call that a reboot PASS.
+
+Set `deny_probe` in `/etc/vantio/boot-hold.json` before the enforce-ready boots, for example the lab's public IPv6 destination and port 443. A boot that never runs the deny self-check stays held.
+
+### 4. Docker restart policy
+
+Create a second container with `--restart=always` and try to enroll it. Expect enroll to refuse until `docker update --restart=no`. After enroll, reboot and confirm `docker ps` does not show the enrolled container before enforce-ready. A container left on `always` and not enrolled is unprotected in status; record it that way rather than calling it held.
+
+### 5. Hold alone
+
+As root, `vantio-boot-hold configure --hold on --ordering off`, then reboot.
+
+Expect: the enrolled unit may start (no `Requires=` on enforce-ready) and its egress still fails while the cgroup and subnet rules are installed. Unenrolled egress may complete. Record hold time. This boot is not a PASS by itself.
+
+### 6. Ordering alone
+
+As root, `vantio-boot-hold configure --hold off --ordering on`, then reboot.
+
+Expect: the packet rules are absent, and the enrolled unit stays inactive until enforce-ready. Record the start time relative to `T_ready`. This boot is not a PASS by itself. Ordering alone leaves a hole if something starts the workload by hand before ready; that is why the hold exists.
+
+### 7. Both together
+
+`vantio-boot-hold opt-in` (hold on and ordering on). Reboot.
+
+Expect: the unit does not start early, and the rules are present until the enforce-ready release. Record `W_hold` and the enrolled and unenrolled probe results.
+
+### 8. Five or more reboots
+
+Repeat the both-together boot five times (5+). Each iteration records `W_hold`, `W_unprotected`, and `W_race` in `reboot-N/timestamps.json`. A missing sample is NOT_PROVED for that iteration. Do not average them into a pass.
+
+### 9. Enrolled workload tries break-glass
+
+From a process in `vantio-enrolled.slice` (the enrolled unit, including uid 0 inside that cgroup), run `vantio-boot-hold release --break-glass --i-am-root-operator`.
+
+Expect: exit non-zero, JSON `state` `FAILED_SAFE`, and an audit line with `"result": "REFUSED"`. The hold remains. Repeat as a non-root caller and expect the same refusal.
+
+## Evidence
+
+```text
+ROW-B1-REBOOT-EXPOSURE/
+  timestamps.json
+  probe-spec.txt
+  egress-probe.txt
+  hold-status.json
+  audit-excerpt.log
+  journal-boot.txt
+  matrix/
+    hold-alone/
+    ordering-alone/
+    both/
+    loader-fail/
+    docker-restart/
+    reboot-1/ ... reboot-5/
+```
+
+`hold-status.json` is the `vantio-boot-hold status` object. `reboot_row` in that object stays `NOT_PROVED` even when the lab looks clean. A human pass decision waits on the verifier and a fresh seal if Phantom Engine changed. This open-core change does not seal Phantom Engine.
+
+GAP-BH-002 and GAP-BH-005 through GAP-BH-009 stay open. This procedure does not close them.

docs/planning/boot-hold/PE-COMPANION.md

@@ -0,0 +1,52 @@
+# Phantom Engine companion note — FD-REBOOT-1
+
+**Audience:** INTERNAL_RESTRICTED  
+**Repo this note belongs to:** open-core packaging. It does not patch `vantio-phantom-engine` and it does not invent a seal.
+
+## What open-core ships
+
+`packages/vantio-install` installs these units:
+
+- `vantio-boot-hold.service` — early, `DefaultDependencies=no`, `Before=docker.service containerd.service`, `WantedBy=sysinit.target`
+- `vantio-pe-loader.service` — `After=vantio-boot-hold.service docker.service`, `Before=vantio-pe-enforce-ready.service`, `Restart=no`
+- `vantio-pe-enforce-ready.service` — `Requires=vantio-pe-loader.service`, runs `release --require-enforce-ready`
+- `vantio-enrolled.slice` and `vantio-enrolled-docker@.service`
+
+The loader command is not baked into the unit. A root admin writes `/etc/vantio/pe-loader.argv.json` as a JSON list. Example for a container that was created with `--restart=no` and is not in the enrolled slice:
+
+```json
+["/usr/bin/docker", "start", "-a", "vantio-pe"]
+```
+
+The container's loader arguments need `--enforce`, `--cgroup-skb-enforce`, and:
+
+```text
+--startup-enroll-cgroup /sys/fs/cgroup/vantio-enrolled.slice
+```
+
+Enrolled systemd units set `Slice=vantio-enrolled.slice`, so their cgroup is `/sys/fs/cgroup/vantio-enrolled.slice/<unit>`.
+
+## What the PE tree does today
+
+The Phantom Engine tree on this machine has no systemd unit. Startup enrollment is the loop in `vantio-loader/src/main.rs` that calls `resolve_cgroup_spec` for each `--startup-enroll-cgroup` value and inserts that one cgroup id. `cgroup_skb` is then attached to that cgroup path.
+
+A PE repository change is not required for open-core to install the units. If the parent wants the unit to live in the Phantom Engine repo, add `vantio-pe-loader.service` with the same ordering as `vantio_install/boot_hold/units.py` (`pe_loader_service`) and point `ExecStart` at the loader binary with the slice path above. Do not treat that copy as a seal. A merge that changes loader behavior needs a new seal from the parent. Seal `e0b19d557891b1ee8bbd20e702df11669d175e4083ef5bbe2f7077cf30093b5e` stays frozen and is not retro-claimed.
+
+## What the parent should re-verify on the next seal
+
+The boot units set `VANTIO_BOOT_HOLD_LIVE=1` so apply, release, and loader start actually touch the host. Fixture tests leave that variable unset and pass a fake root, so they record commands instead of changing packet filters. The enforce-ready unit does not set `VANTIO_BOOT_HOLD_ALLOW_FACTS` or `VANTIO_BOOT_HOLD_ALLOW_CALLER_FIXTURE`.
+
+The open-core probe treats enforce-ready as all of the following. A running loader is not enough:
+
+- a live `vantio-loader` command line containing `--enforce`
+- the pinned names `vantio_trace_map`, `vantio_enrolled_cgroups`, `vantio_debug_counters`, `vantio_debug_last_comm`, `vantio_tls_severed_pids`
+- `bpftool map show` containing `vantio_enforce` (policy loaded)
+- `bpftool cgroup show /sys/fs/cgroup/vantio-enrolled.slice` containing `cgroup_skb_egress_enforce` (attached, not only loaded)
+- loader health `OK` (process state R, S, or D)
+- a deny self-check: enrolled connect fails and the same connect outside the slice succeeds, after a one-destination exception in the hold chain that is removed before release
+
+A loader that is running while that cgroup show does not list the program keeps the hold.
+
+Unknown on any of those keeps the hold. The parent should confirm on the sealed loader that attaching `cgroup_skb` to `/sys/fs/cgroup/vantio-enrolled.slice` covers descendant cgroups under that slice. This note does not record that confirmation.
+
+A root attacker who can disable host controls is Battery 4. This hold does not detect that tampering.

packages/vantio-install/README.md

@@ -17,6 +17,8 @@ Run the commands from the host that will keep the node:
 - `vantio-install uninstall` on a customer host is the dual-gated command in `docs/UNINSTALL.md`.
 - `vantio-install verify-removal` checks whether that scope is actually gone, including the known pin names on `/sys/fs/bpf` for a Phantom Engine scope.
 
+`vantio-boot-hold` is the early-boot hold for enrolled workloads. It is on unless a root admin opts out. Read `docs/BOOT-HOLD.md`. The command prints the same JSON envelope, and the reboot exposure row stays `NOT_PROVED` until a later host reproof.
+
 Every command prints one JSON object. `proof_state` stays `NOT_PROVED` in this package. `vantio-verify` reads the evidence directory and the bundle from disk, and it ignores an installer exit code of 0.
 
 A live change needs `VANTIO_INSTALL_ALLOW_LIVE=1` and `--i-accept-live-mutations` together, plus `--plan` and `--plan-sha256`, on `apply`, `rollback`, or `uninstall`. The installer then runs an allowlisted observe-only command list and checks the host again before it treats the step as verified.

packages/vantio-install/bin/vantio-boot-hold

@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+from vantio_install.boot_hold.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())