diff --git a/.agents/skills/add-mission/references/mission-checklist.md b/.agents/skills/add-mission/references/mission-checklist.md index 51701b0..b86ecc0 100644 --- a/.agents/skills/add-mission/references/mission-checklist.md +++ b/.agents/skills/add-mission/references/mission-checklist.md @@ -10,7 +10,7 @@ Every mission supplies: - An effective learning `track` and `environment`; omitted values preserve the legacy `linux` and `simulated` defaults. - Teaching content: `story`, `objective`, one or more unique base names in `suggested_commands`, `hints`, and `explanation`. - Simulated labs: an absolute clean `start_dir` plus declarative `setup` containing `directories`, `files`, `processes`, `environment`, and `archives` as needed. -- Docker labs: `track: docker`, `environment: docker`, and a declarative `docker` setup containing digest-pinned image references and logical `running` or `stopped` container fixtures. They do not define simulated setup or a start directory. +- Docker labs: `track: docker`, `environment: docker`, and a declarative `docker` setup containing digest-pinned image references and logical `running` or `stopped` container fixtures. Fixtures may select fixed Go-implemented behaviors: an optional startup `log`, a `health` probe (`healthy` or `unhealthy`), a stopped one-shot job (`log` + `exit_code`), or a running bounded crash loop (`log` + non-zero `exit_code` + `restart: on-failure`). They do not define simulated setup or a start directory. - Declarative `validation.all`: one or more observable conditions. - `rewards`: positive `xp` and non-negative `hint_penalty`. @@ -26,7 +26,7 @@ Use the smallest combination that proves the outcome without encoding a command - Virtual file state: `file_content_equals`, `file_content_contains`, `file_lines_equal`, `file_mode_equals`, `file_owner_equals`. - Virtual processes: `process_stopped`, `process_running`. - Virtual environment: `env_equals` with `NAME=value`. -- Docker state: `docker_container_running` with a declared logical container alias and `docker_container_count_equals` with a non-negative count. +- Docker state: `docker_container_running`, `docker_container_stopped`, or `docker_container_absent` with a declared logical container alias, and `docker_container_count_equals` with a non-negative count. Running and stopped both require the container to still exist. Output-only validation is appropriate when producing the exact useful output is the lesson. Prefer filesystem, process, or environment state when the operational result should survive the final command. Combine positive and negative conditions when distractors could otherwise produce a false completion. diff --git a/AGENTS.md b/AGENTS.md index 35c47b9..b45d707 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # OpsQuest agent guide -OpsQuest is a Go 1.26 CLI game with 23 in-memory Linux missions and 6 optional Docker Foundations labs. Linux player commands stay inside the teaching shell. Docker input is parsed into a deliberately small command subset that can affect only disposable, OpsQuest-labeled resources. Kubernetes remains future scope unless a task explicitly adds it. +OpsQuest is a Go 1.26 CLI game with 23 in-memory Linux missions and 11 optional Docker labs across two worlds. Linux player commands stay inside the teaching shell. Docker input is parsed into a deliberately small command subset that can affect only disposable, OpsQuest-labeled resources. Kubernetes remains future scope unless a task explicitly adds it. ## Repository map diff --git a/README.md b/README.md index 09fcf08..55c9d90 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ OpsQuest validates the result, not a prescribed command. Equivalent supported so ## What is included - 23 Linux missions across four ordered learning worlds -- 6 optional, disposable Docker Foundations missions +- 11 optional, disposable Docker missions across two worlds: Foundations and Container Triage - An isolated in-memory filesystem, environment, process table, and archives - Quote-aware globs and variables, pipelines, redirection, and command history - A compact virtual `vi` and bounded virtual shell scripts @@ -85,7 +85,10 @@ $ opsquest doctor $ opsquest play 20 ``` -OpsQuest never pulls the image automatically. See +OpsQuest never pulls the image automatically. If an OpsQuest process is +killed before it can clean up, the next Docker mission removes its leftover lab +containers; `opsquest doctor` reports them and `opsquest doctor --cleanup` +removes them on demand. See [Docker Foundations](docs/play/docker-foundations.md) for Docker Desktop, Docker Engine, and OrbStack setup, plus the teaching subset and isolation boundary. @@ -94,7 +97,9 @@ boundary. Player-entered Linux commands—including virtual scripts—are parsed and executed by OpsQuest. They never reach a host shell, host process, or host path. The teaching shell operates only on bounded in-memory state. -Docker input is parsed into a small set of typed teaching actions. OpsQuest constructs Docker arguments itself, uses only exact resources labeled for the current attempt, applies resource restrictions, and verifies ownership again during cleanup. It does not expose arbitrary Docker passthrough, privileged mode, host mounts, host networking, devices, or the Docker socket. +Docker input is parsed into a small set of typed teaching actions. OpsQuest constructs Docker arguments itself, uses only exact resources labeled for the current attempt, applies resource restrictions, and verifies ownership again during cleanup. Fixture processes live at most 24 hours, and leftovers from a crashed process are removed only when every ownership label matches and the recorded owner process is gone. It does not expose arbitrary Docker passthrough, privileged mode, host mounts, host networking, devices, or the Docker socket. + +The optional web companion binds only to an ephemeral `127.0.0.1` port, uses a one-time pairing URL, and exposes a read-only mission projection. It cannot submit commands, mutate the profile or attempt, address Docker resources, or approve completion. The optional web companion binds only to an ephemeral `127.0.0.1` port, uses a one-time pairing URL, and exposes a read-only mission projection. It cannot submit commands, mutate the profile or attempt, address Docker resources, or approve completion. diff --git a/docs/README.md b/docs/README.md index 5bf1fa6..67b2f6c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -31,7 +31,7 @@ The game describes an observable objective, gives you a disposable environment, ## Current scope - 23 Linux missions across four ordered learning worlds -- 6 optional Docker Foundations missions +- 11 optional Docker missions across two worlds - Outcome-based validation rather than one required command transcript - An in-memory filesystem, process table, environment, archives, editor, and shell scripts - Persistent XP, ranks, command practice, hints, completions, and achievements diff --git a/docs/play/diagrams/learning-journey.excalidraw b/docs/play/diagrams/learning-journey.excalidraw index 820a829..cdca9f3 100644 --- a/docs/play/diagrams/learning-journey.excalidraw +++ b/docs/play/diagrams/learning-journey.excalidraw @@ -22,9 +22,9 @@ {"id":"docker-lane","type":"rectangle","x":40,"y":545,"width":1410,"height":215,"angle":0,"strokeColor":"#8b5cf6","backgroundColor":"#f5f3ff","fillStyle":"solid","strokeWidth":2,"strokeStyle":"solid","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":{"type":3},"seed":214,"version":1,"versionNonce":2014,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false}, {"id":"docker-label","type":"text","x":70,"y":565,"width":370,"height":20,"angle":0,"strokeColor":"#6d28d9","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1,"strokeStyle":"solid","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":null,"seed":215,"version":1,"versionNonce":2015,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false,"fontSize":16,"fontFamily":2,"text":"DOCKER TRACK · OPTIONAL · INDEPENDENT WORLD NUMBERING","textAlign":"left","verticalAlign":"top","containerId":null,"originalText":"DOCKER TRACK · OPTIONAL · INDEPENDENT WORLD NUMBERING","autoResize":true,"lineHeight":1.25}, {"id":"docker-world","type":"rectangle","x":85,"y":610,"width":390,"height":115,"angle":0,"strokeColor":"#7c3aed","backgroundColor":"#ffffff","fillStyle":"solid","strokeWidth":2,"strokeStyle":"solid","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":{"type":3},"seed":216,"version":1,"versionNonce":2016,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false}, - {"id":"docker-world-text","type":"text","x":115,"y":630,"width":330,"height":75,"angle":0,"strokeColor":"#4c1d95","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1,"strokeStyle":"solid","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":null,"seed":217,"version":1,"versionNonce":2017,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false,"fontSize":18,"fontFamily":2,"text":"DOCKER WORLD 1\nIt Works on My Machine\nMissions 20–25 · lifecycle + diagnostics","textAlign":"left","verticalAlign":"top","containerId":null,"originalText":"DOCKER WORLD 1\nIt Works on My Machine\nMissions 20–25 · lifecycle + diagnostics","autoResize":true,"lineHeight":1.25}, + {"id":"docker-world-text","type":"text","x":115,"y":630,"width":330,"height":75,"angle":0,"strokeColor":"#4c1d95","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1,"strokeStyle":"solid","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":null,"seed":217,"version":1,"versionNonce":2017,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false,"fontSize":18,"fontFamily":2,"text":"DOCKER WORLDS 1–2\nIt Works on My Machine → Container Triage\nMissions 20–34 · lifecycle, diagnostics, triage","textAlign":"left","verticalAlign":"top","containerId":null,"originalText":"DOCKER WORLDS 1–2\nIt Works on My Machine → Container Triage\nMissions 20–34 · lifecycle, diagnostics, triage","autoResize":true,"lineHeight":1.25}, {"id":"docker-note","type":"rectangle","x":650,"y":610,"width":700,"height":115,"angle":0,"strokeColor":"#a78bfa","backgroundColor":"#ede9fe","fillStyle":"solid","strokeWidth":2,"strokeStyle":"dashed","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":{"type":3},"seed":218,"version":1,"versionNonce":2018,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false}, - {"id":"docker-note-text","type":"text","x":685,"y":630,"width":630,"height":75,"angle":0,"strokeColor":"#4c1d95","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1,"strokeStyle":"solid","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":null,"seed":219,"version":1,"versionNonce":2019,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false,"fontSize":18,"fontFamily":2,"text":"Separate readiness and progress\nPinned local fixture · no automatic pull · Linux never blocked\n6 bounded lifecycle and diagnostic missions","textAlign":"left","verticalAlign":"top","containerId":null,"originalText":"Separate readiness and progress\nPinned local fixture · no automatic pull · Linux never blocked\n6 bounded lifecycle and diagnostic missions","autoResize":true,"lineHeight":1.25}, + {"id":"docker-note-text","type":"text","x":685,"y":630,"width":630,"height":75,"angle":0,"strokeColor":"#4c1d95","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":1,"strokeStyle":"solid","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":null,"seed":219,"version":1,"versionNonce":2019,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false,"fontSize":18,"fontFamily":2,"text":"Separate readiness and progress\nPinned local fixture · no automatic pull · Linux never blocked\n11 bounded lifecycle and diagnostic missions","textAlign":"left","verticalAlign":"top","containerId":null,"originalText":"Separate readiness and progress\nPinned local fixture · no automatic pull · Linux never blocked\n11 bounded lifecycle and diagnostic missions","autoResize":true,"lineHeight":1.25}, {"id":"a-docker","type":"arrow","x":475,"y":668,"width":175,"height":0,"angle":0,"strokeColor":"#7c3aed","backgroundColor":"transparent","fillStyle":"solid","strokeWidth":3,"strokeStyle":"dashed","roughness":1,"opacity":100,"groupIds":[],"frameId":null,"roundness":{"type":2},"seed":220,"version":1,"versionNonce":2020,"isDeleted":false,"boundElements":null,"updated":1784505600000,"link":null,"locked":false,"points":[[0,0],[175,0]],"lastCommittedPoint":null,"startBinding":null,"endBinding":null,"startArrowhead":null,"endArrowhead":"arrow","elbowed":false} ], "appState": {"gridSize": null, "viewBackgroundColor": "#f8fafc"}, diff --git a/docs/play/diagrams/learning-journey.svg b/docs/play/diagrams/learning-journey.svg index 57178cb..114efe1 100644 --- a/docs/play/diagrams/learning-journey.svg +++ b/docs/play/diagrams/learning-journey.svg @@ -1,7 +1,7 @@ OpsQuest learning journey - Two-lane curriculum overview showing four Linux worlds with twenty-three missions and one independent six-mission Docker Foundations world. + Two-lane curriculum overview showing four Linux worlds with twenty-three missions and two independent Docker worlds with eleven missions. @@ -74,16 +74,16 @@ DOCKER TRACK · OPTIONAL · INDEPENDENT WORLD NUMBERING - Docker World 1 - It Works on My Machine - Missions 20–25 · 6 beginner lifecycle and diagnostic stages + Docker Worlds 1–2 + Foundations → Container Triage + Missions 20–34 · 11 stages; Linux 26–29 sit between Separate readiness and progress Pinned local fixture · no automatic pull · Docker never blocks Linux - 6 bounded missions · typed list, logs, inspect, and lifecycle actions only. + 11 bounded missions · typed list, filter, logs, inspect, lifecycle, and rm only. OUTCOME-BASED VALIDATION diff --git a/docs/play/docker-foundations.md b/docs/play/docker-foundations.md index 314a2e2..ba9a520 100644 --- a/docs/play/docker-foundations.md +++ b/docs/play/docker-foundations.md @@ -1,5 +1,5 @@ --- -description: Prepare and play the optional, disposable OpsQuest Docker Foundations missions. +description: Prepare and play the optional, disposable OpsQuest Docker missions. audience: players status: current --- @@ -39,7 +39,7 @@ $ opsquest play --track docker ## Current lessons -The 6 beginner missions cover a deliberately narrow operational loop: +The track has 11 missions in two worlds. **World 1: It Works on My Machine** is 6 beginner missions covering a deliberately narrow lifecycle loop: 1. **Container Census** — list containers and start an existing stopped service. 2. **Last Broadcast** — read bounded logs from an exited one-shot job. @@ -48,12 +48,43 @@ The 6 beginner missions cover a deliberately narrow operational loop: 5. **Recovery Pair** — restore two stopped services without replacing them. 6. **Shift Handoff** — combine `start` and `stop` while preserving supporting metrics. -The delivered teaching subset recognizes container listing plus `start`, `restart`, `stop`, `inspect`, `logs`, and `help` forms. It accepts exact logical aliases without forwarding arbitrary Docker CLI arguments or flags. +**World 2: Container Triage** adds health checks, restart policies, log tails, and removal: + +1. **Janitor Duty** — remove finished job containers while services keep running. +2. **Tail End** — show only the last lines of a long job log. +3. **Running Isn't Healthy** — pull a running but unhealthy replica from service and start its standby. +4. **Crash Loop** — stop a container its restart policy keeps relaunching, then show why it fails. +5. **Postmortem Triage** — combine health, exit codes, stop, and remove, and keep the evidence the postmortem needs. + +## Teaching subset + +| Command | Supported forms | +| --- | --- | +| List | `docker ps` or `docker container ls`, with `-a`/`--all` and up to four `--filter`/`-f` values: `status=created\|running\|restarting\|exited` or `health=starting\|healthy\|unhealthy\|none` | +| Lifecycle | `start`, `restart`, and `stop` with one alias | +| Remove | `docker rm ALIAS` for a stopped container; `--force` is refused, so stop the container first | +| Inspect | `docker inspect ALIAS` shows logical state, exit code, health, restart count, and restart policy | +| Logs | `docker logs ALIAS` or `docker logs --tail N ALIAS` (also `-n N`, `--tail=N`, and `--tail all`) | + +Every command also has a `docker container ...` form. OpsQuest parses these forms itself and accepts exact logical aliases. It never forwards other Docker CLI arguments or flags, and filters run inside OpsQuest rather than being passed to Docker. ## Isolation boundary OpsQuest generates unique names and ownership labels, maps player-visible aliases to exact container IDs, applies resource restrictions, and removes only resources verified as belonging to the current attempt. Labs do not use privileged mode, host bind mounts, host networking, devices, or a mounted Docker socket. +Health probes and restart policies are fixed fixture behaviors chosen by the mission, never player input. A crash-loop fixture's restart policy allows at most 50 retries. + +## Cleaning up after a crash + +OpsQuest removes a lab's containers when the attempt ends. If the process is killed first, its containers stay behind. Their processes exit on their own within 24 hours, and the next Docker mission removes them. You can also check and clean up yourself: + +```console +$ opsquest doctor # reports orphaned lab containers +$ opsquest doctor --cleanup # removes them +``` + +A container counts as orphaned only when it has every OpsQuest ownership label and its generated name, and either its recorded owner process on this machine has exited or it is older than 24 hours. Containers belonging to other running OpsQuest sessions on this machine are left alone for their 24-hour lifetime. + The selected Docker-compatible engine remains a powerful external dependency. OpsQuest constrains the lesson and cleanup scope; it does not present the engine itself as an untrusted-code security boundary. See [Sandbox and safety](../technical/sandbox-and-safety.md#docker-teaching-boundary) diff --git a/docs/play/mission-map.md b/docs/play/mission-map.md index 76b7c3d..28aa62c 100644 --- a/docs/play/mission-map.md +++ b/docs/play/mission-map.md @@ -17,9 +17,9 @@ Editable source: [`learning-journey.excalidraw`](diagrams/learning-journey.excal The curriculum has two independent tracks: - **Linux:** 23 missions across four ordered worlds. Bare `opsquest play` follows this track. -- **Docker:** 6 optional Foundations missions in one beginner world. Docker readiness never blocks Linux play. +- **Docker:** 11 optional missions across two worlds: Foundations, then Container Triage. Docker readiness never blocks Linux play. -Display numbers are global across both tracks: Docker occupies Missions 20–25 and Linux World 4 resumes at Mission 26. World and stage positions are derived separately inside each track. Persisted completions use stable mission IDs rather than display numbers, so profiles created before this expansion remain compatible. +Display numbers are global across both tracks: Docker World 1 occupies Missions 20–25, Linux World 4 resumes at Mission 26, and Docker World 2 continues at Mission 30. World and stage positions are derived separately inside each track. Persisted completions use stable mission IDs rather than display numbers, so profiles created before this expansion remain compatible. The [learning philosophy](../game/learning-philosophy.md) explains the incident loop and feedback model. [Mission authoring](../technical/mission-authoring.md) @@ -30,7 +30,7 @@ describes how validators preserve equivalent solutions. ```console $ opsquest play # resume Linux progress $ opsquest map # see every world and stage -$ opsquest map --track docker # see the optional Docker world +$ opsquest map --track docker # see the optional Docker worlds $ opsquest play --world 3 # stay within Production Friday $ opsquest play 19 # begin at a global mission number $ opsquest play linux-find-logs # begin at a stable mission ID @@ -75,6 +75,11 @@ The “tools” column records suggested commands, not a mandatory solution. Out | 27 | Linux W4.2 | Beginner | Make a focused configuration edit with a modal editor | `cat`, `vi` | Content and mode equal | | 28 | Linux W4.3 | Intermediate | Repair, permit, and run a reusable report script | `cat`, `less`, `vi`, `sed`, `chmod`, `sh` | Report content/lines and script mode | | 29 | Linux W4.4 | Advanced | Reason about child-shell directory and environment scope | `cat`, `less`, `vi`, `sed`, `chmod`, `sh`, `pwd`, `env` | Parent scope preserved; report and script state correct | +| 30 | Docker W2.1 | Beginner | Remove finished job containers while services keep running | `docker` | Jobs absent; services running; count equals | +| 31 | Docker W2.2 | Beginner | Read only the end of a long job log | `docker` | Output equals final log lines | +| 32 | Docker W2.3 | Intermediate | Replace a running but unhealthy replica with its standby | `docker` | Unhealthy target stopped; healthy and standby running; count equals | +| 33 | Docker W2.4 | Intermediate | Stop a restart-policy crash loop and surface its fault | `docker` | Target stopped; service running; count equals; output contains fault | +| 34 | Docker W2.5 | Advanced | Triage health and exit codes while preserving evidence | `docker` | Canary and failed job stopped; successful jobs absent; api running; count equals | ## World progression @@ -98,9 +103,13 @@ The final Linux world moves from executing a supplied script to repairing reusab Docker Foundations progresses through listing, logs, sanitized exit status, targeted stop, multi-service recovery, and a mixed lifecycle handoff. Logical aliases hide real container names and IDs, keeping every lesson focused on observable state for exact attempt-owned resources. +### Docker World 2: Container Triage + +Container Triage separates evidence from clutter. It adds removal of stopped containers, log tails, health probes, and restart policies, then combines them in an advanced boss where the player must decide what to stop, what to remove, and what to keep for the postmortem. + ## Curriculum evolution -The expanded map fills the early file-reading, focused log-preview, simple counting, supplied-script, and beginner Docker lifecycle gaps. Later curriculum work can deepen those skills through new declarative missions without weakening existing outcome validators or widening the Docker boundary casually. +The expanded map fills the early file-reading, focused log-preview, simple counting, supplied-script, beginner Docker lifecycle, and Docker triage gaps. Later curriculum work can deepen those skills through new declarative missions without weakening existing outcome validators or widening the Docker boundary casually. Read [Missions, hints, and progress](missions-and-progress.md) for route, reward, and persistence behavior, and [Mission authoring](../technical/mission-authoring.md) diff --git a/docs/roadmap/README.md b/docs/roadmap/README.md index b5f0c4c..52d1a34 100644 --- a/docs/roadmap/README.md +++ b/docs/roadmap/README.md @@ -11,17 +11,19 @@ behavior. Shipped behavior belongs in the player and technical guides. ## Docker Foundations -The current six-mission campaign intentionally supports only listing, logs, -sanitized inspection, and bounded lifecycle actions for attempt-owned -containers. +The current two Docker worlds intentionally support only listing with fixed +filters, logs, sanitized inspection, bounded lifecycle actions, and removal of +stopped attempt-owned containers. Fixture health probes and crash loops are +fixed behaviors chosen by the mission. Possible later increments include: -- environment-aware container creation; +- environment-aware container creation through a limited `docker run`; - limited port publication; - bounded volumes and networking; -- Dockerfile or Compose concepts; -- more advanced outcome-based troubleshooting. +- Dockerfile or Compose concepts, possibly taught in the simulated shell; +- restart-to-recover health lessons, which need fixture state that survives a + restart. Each expansion would widen the external-engine boundary. It requires its own parser contract, threat model, resource ownership rules, cleanup behavior, and diff --git a/docs/technical/mission-authoring.md b/docs/technical/mission-authoring.md index 7bbc19c..eeab863 100644 --- a/docs/technical/mission-authoring.md +++ b/docs/technical/mission-authoring.md @@ -21,7 +21,7 @@ Good evidence includes: - a target process is stopped while a healthy process remains running; - output contains all relevant paths and excludes a distractor; - a report contains exactly the required logical lines; -- an attempt-owned Docker container is in the required state. +- an attempt-owned Docker container is in the required state, or has been removed. Avoid requiring one command name, argument order, or intermediate state when another supported solution demonstrates the same understanding. @@ -62,14 +62,24 @@ Linux remains the default track and `simulated` remains the default environment - missing narrative, guidance, setup, validation, or reward data; - unsafe, conflicting, or inconsistent virtual paths and state; - validators with missing, extra, or incompatible fields; -- unpinned Docker images, invalid aliases, oversized Docker setup, or incomplete diagnostic fixtures; +- unpinned Docker images, invalid aliases, oversized Docker setup, or inconsistent fixture behaviors; - a campaign that reappears as separate worlds in one track. Catalog access is indexed by ID and number. Returned missions and worlds are deep copies so adapters cannot mutate embedded content. ## Observable validation -Conditions cover output, working directory, path existence, file content and logical lines, modes, owners, process state, environment values, and bounded Docker container state. Docker fixtures may additionally declare a bounded log and exit code together for a stopped one-shot diagnostic job; container conditions can require either running or stopped state. The game layer compares output conditions; the active environment observes state conditions through the shared `Environment` contract. +Conditions cover output, working directory, path existence, file content and logical lines, modes, owners, process state, environment values, and bounded Docker container state. Docker fixtures select fixed behaviors implemented in Go rather than supplying commands: + +| Fixture fields | Behavior | Rules | +| --- | --- | --- | +| none | Long-lived service | `running` or `stopped` | +| `log` | Long-lived service that prints a startup log | Log at most 8 KiB, no NUL | +| `health: healthy` or `unhealthy` | Service with a fixed passing or failing health probe | Not allowed on exiting fixtures; setup waits for a running probe to settle | +| `log` + `exit_code` | One-shot diagnostic job | Must be `stopped`; setup runs it to completion | +| `log` + non-zero `exit_code` + `restart: on-failure` | Bounded crash loop (`on-failure:50`) | Must be `running`; setup waits until at least two restarts are visible | + +Container conditions can require running or stopped state (both require the container to still exist), or `docker_container_absent` after the player removes it. Every container a condition names must be declared in the fixture setup. The game layer compares output conditions; the active environment observes state conditions through the shared `Environment` contract. Suggested commands identify the intended tool family but do not constrain validation. One to five hints should progress from concept, through inspection diff --git a/docs/technical/sandbox-and-safety.md b/docs/technical/sandbox-and-safety.md index d89eca9..096f42e 100644 --- a/docs/technical/sandbox-and-safety.md +++ b/docs/technical/sandbox-and-safety.md @@ -122,14 +122,25 @@ Docker missions are opt-in and follow a narrower path: 1. The mission catalog accepts only pinned image references and bounded logical fixtures. 2. Availability checks the Docker executable, active Docker-compatible engine, and exact local image without creating resources or pulling images. An active `orbstack` context is identified for provider-specific guidance only. -3. The factory generates a random session ID and creates containers with generated names and five ownership labels. -4. Player text is parsed into `list`, `start`, `restart`, `stop`, `inspect`, `logs`, or `help`; flags and aliases must match the small grammar. +3. The factory first sweeps orphaned fixtures (see below), then generates a random session ID and creates containers with generated names, five ownership labels, and two advisory owner labels (process ID and host name). +4. Player text is parsed into `list`, `start`, `restart`, `stop`, `rm`, `inspect`, `logs`, or `help`; flags and aliases must match the small grammar. `ps` filters are limited to fixed `status` and `health` values and are evaluated in Go over sanitized inspection data rather than passed to Docker. `logs --tail` accepts only a whole number up to 10,000 or `all`. `rm` has no `--force` form and refuses a running container. 5. The adapter resolves a validated logical alias to a tracked exact container ID. 6. `exec.CommandContext` receives fixed arguments constructed by the adapter. 7. Observations inspect exact IDs or enumerate by the session label, then verify the complete ownership-label set. -8. `Close` seals the attempt immediately, re-inspects ownership, removes only matching exact IDs, and retains unresolved resources for a retry. +8. `Close` seals the attempt immediately, re-inspects ownership, removes only matching exact IDs, and retains unresolved resources for a retry. Containers the player already removed count as resolved. +9. Engine errors that name a tracked container are rewritten to its logical identity, and a missing container is reported as removed, so real IDs and generated names never reach the player. -Created containers use no network, a read-only root filesystem, a bounded temporary filesystem, an unprivileged numeric user, no Linux capabilities, `no-new-privileges`, and explicit PID, memory, CPU, file-descriptor, restart, and stop-timeout limits. Diagnostic fixtures use a fixed shell program; bounded log text and exit status are passed as data arguments rather than interpolated into that program. Containers do not receive host bind mounts, devices, privileged mode, host networking, or a Docker socket. +Created containers use no network, a read-only root filesystem, a bounded temporary filesystem, an unprivileged numeric user, no Linux capabilities, `no-new-privileges`, and explicit PID, memory, CPU, file-descriptor, restart, and stop-timeout limits. Diagnostic and logging fixtures use fixed shell programs; bounded log text, exit status, and lifetime are passed as data arguments rather than interpolated into those programs. Long-lived fixture processes exit after 24 hours. Health probes are the fixed BusyBox applets `true` or `false`, selected by a validated `health` enum. The restart policy is `no` except for a declared crash-loop fixture, which uses `on-failure:50` and therefore stops relaunching on its own. Containers do not receive host bind mounts, devices, privileged mode, host networking, published ports, or a Docker socket. + +### Orphaned fixtures + +A process that is killed before `Close` leaves its fixtures behind. Before each Docker attempt, and on demand through `opsquest doctor --cleanup`, the factory lists containers labeled `com.opsquest.managed=true` and removes an exact ID only when all of these hold: + +- the managed, schema, session, mission, and alias labels are present and well formed; +- the container name is the generated `opsquest--cNN` form for that same session; +- either the owner-host label matches this machine and the recorded owner process is no longer running, or the container is older than the 24-hour fixture lifetime. + +Owners on other hosts, and fixtures created before owner labels existed, can therefore only age out. A plain `opsquest doctor` counts orphans without removing anything. A failed automatic sweep never blocks mission setup. Docker-specific ceilings include: @@ -157,6 +168,9 @@ The Docker-compatible engine remains a powerful external dependency; OpsQuest re | Docker input becomes arbitrary CLI flags | Typed parser accepts only exact actions and validated logical aliases | `internal/dockerlab/parser.go` and tests | | Cleanup removes another container | Exact ID plus managed/schema/session/mission/alias label verification | `internal/dockerlab/environment.go` and tests | | Partial Docker setup leaks resources silently | Factory may return a partial environment; managed cleanup and retryable `Close` | `internal/dockerlab/factory.go`, environment contract tests | +| A killed process leaks running fixtures | 24-hour fixture lifetime; owner-process and age-based sweep with complete label and generated-name proof | `internal/dockerlab/janitor.go` and tests | +| Orphan sweep removes a live or foreign container | Live same-host owners are never swept; unknown owners only after the fixture lifetime; exact IDs only | `internal/dockerlab/janitor_test.go` | +| Removed containers leak engine identity | Missing-container errors mapped to the alias; tracked IDs and names redacted from engine errors | `internal/dockerlab/environment.go` and tests | | Persisted display text injects terminal controls | Profile names reject non-printable characters and normalize legacy values | `internal/profile/profile.go` and tests | | Another site reaches the loopback companion | One-time capability pairing, exact Host/Origin checks, same-site HTTP-only cookie, no permissive CORS | `internal/webapp/server.go` and tests | | Browser input reaches a mission environment | Companion exposes only read-only state and SSE routes; `game.Session` has no companion command callback | `internal/webapp`, `internal/game/companion.go` | diff --git a/internal/cli/app.go b/internal/cli/app.go index d859de5..53adf97 100644 --- a/internal/cli/app.go +++ b/internal/cli/app.go @@ -739,7 +739,8 @@ func (a *App) runShow(args []string) error { } func (a *App) runDoctor(args []string) error { - flags := a.newFlagSet("doctor", func() { fmt.Fprintln(a.errOut, "Usage: opsquest doctor") }) + flags := a.newFlagSet("doctor", func() { fmt.Fprintln(a.errOut, "Usage: opsquest doctor [--cleanup]") }) + cleanup := flags.Bool("cleanup", false, "remove Docker lab containers left behind by exited OpsQuest processes") if help, err := parseFlags(flags, args); help || err != nil { return err } @@ -756,14 +757,57 @@ func (a *App) runDoctor(args []string) error { fmt.Fprintf(a.out, " %s profile: version %d, %d completed missions\n", check, player.Version, len(player.Completed)) fmt.Fprintf(a.out, " %s profile path: %s\n", check, a.store.Path()) fmt.Fprintf(a.out, " %s Linux labs: in-memory; no host shell or filesystem access\n", check) + dockerReady := false if item, found := a.catalog.FirstInTrack(mission.TrackDocker); found { availability := game.EnvironmentAvailability(a.ctx, a.factory, item) + dockerReady = availability.Available if availability.Available { fmt.Fprintf(a.out, " %s docker labs: ready · %s\n", check, availability.Detail) } else { fmt.Fprintf(a.out, " %s docker labs: unavailable · %s\n", a.style.Warning("!"), availability.Detail) } } + return a.reportOrphanedLabs(*cleanup, dockerReady) +} + +// reportOrphanedLabs checks for, or with cleanup removes, lab resources left +// behind by OpsQuest processes that exited without closing their attempt. +// A plain doctor run stays read-only and only checks a reachable engine. +func (a *App) reportOrphanedLabs(cleanup, dockerReady bool) error { + janitor, supported := a.factory.(game.ResourceJanitor) + check := a.style.Success("✓") + warning := a.style.Warning("!") + if !supported { + if cleanup { + fmt.Fprintf(a.out, " %s docker cleanup: nothing to clean; Docker labs are not configured\n", check) + } + return nil + } + if cleanup { + removed, err := janitor.RemoveOrphanedResources(a.ctx) + if err != nil { + if removed > 0 { + fmt.Fprintf(a.out, " %s docker cleanup: removed %d, then failed · %v\n", warning, removed, err) + } else { + fmt.Fprintf(a.out, " %s docker cleanup: failed · %v\n", warning, err) + } + return fmt.Errorf("docker cleanup failed: %w", err) + } + fmt.Fprintf(a.out, " %s docker cleanup: removed %d orphaned lab %s\n", check, removed, plural(removed, "container", "containers")) + return nil + } + if !dockerReady { + return nil + } + count, err := janitor.OrphanedResources(a.ctx) + switch { + case err != nil: + fmt.Fprintf(a.out, " %s docker cleanup: could not check for orphaned lab containers · %v\n", warning, err) + case count > 0: + fmt.Fprintf(a.out, " %s docker cleanup: %d orphaned lab %s from exited OpsQuest processes; run 'opsquest doctor --cleanup'\n", warning, count, plural(count, "container", "containers")) + default: + fmt.Fprintf(a.out, " %s docker cleanup: no orphaned lab containers\n", check) + } return nil } @@ -825,7 +869,7 @@ func (a *App) runHelp(args []string) error { case "show", "mission": fmt.Fprintln(a.out, "Usage: opsquest show [MISSION]") case "doctor": - fmt.Fprintln(a.out, "Usage: opsquest doctor") + fmt.Fprintln(a.out, "Usage: opsquest doctor [--cleanup]") case "reset": fmt.Fprintln(a.out, "Usage: opsquest reset [--yes]") case "version": @@ -864,7 +908,7 @@ func (a *App) printUsage() { opsquest commands Show commands practiced successfully opsquest achievements Show learning achievements opsquest show [MISSION] Preview a mission without starting it - opsquest doctor Check the catalog, profile, and safety mode + opsquest doctor Check the catalog, profile, safety mode, and lab cleanup opsquest reset [--yes] Reset local progress opsquest version Print the version diff --git a/internal/cli/app_test.go b/internal/cli/app_test.go index 6d5af61..a649829 100644 --- a/internal/cli/app_test.go +++ b/internal/cli/app_test.go @@ -912,7 +912,9 @@ func TestListDockerTrackAndRejectsUnknownTrack(t *testing.T) { "Container Census", "Shift Handoff", "docker-container-census", - "0/6 missions complete", + "WORLD 2/2 · Container Triage", + "Postmortem Triage", + "0/11 missions complete", "Continue: opsquest play --track docker", "Jump: opsquest play --track docker --world N", "IDs: opsquest map --track docker --ids", @@ -1085,7 +1087,7 @@ func TestProfileRenameShowAndDoctor(t *testing.T) { if err := app.Run([]string{"doctor"}); err != nil { t.Fatal(err) } - if !strings.Contains(out.String(), "29 missions (23 Linux, 6 Docker)") || !strings.Contains(out.String(), "Linux labs: in-memory; no host shell or filesystem access") { + if !strings.Contains(out.String(), "34 missions (23 Linux, 11 Docker)") || !strings.Contains(out.String(), "Linux labs: in-memory; no host shell or filesystem access") { t.Fatalf("doctor output = %s", out.String()) } } @@ -1163,7 +1165,7 @@ func TestListCampaignFilterUsesCampaignTotal(t *testing.T) { if !strings.Contains(out.String(), "0/6 missions complete") { t.Fatalf("campaign-filtered total missing:\n%s", out.String()) } - if strings.Contains(out.String(), "0/29 missions complete") || strings.Contains(out.String(), "Production Friday") { + if strings.Contains(out.String(), "0/34 missions complete") || strings.Contains(out.String(), "Production Friday") { t.Fatalf("campaign filter used catalog-wide scope:\n%s", out.String()) } } @@ -1364,3 +1366,125 @@ func (e *cliDockerEnvironment) Close() error { e.closed = true return nil } + +type cliJanitorFactory struct { + cliDockerFactory + orphans int + checkErr error + removeErr error + checks int + removeCalls int +} + +func (f *cliJanitorFactory) OrphanedResources(context.Context) (int, error) { + f.checks++ + return f.orphans, f.checkErr +} + +func (f *cliJanitorFactory) RemoveOrphanedResources(context.Context) (int, error) { + f.removeCalls++ + if f.removeErr != nil { + return 1, f.removeErr + } + removed := f.orphans + f.orphans = 0 + return removed, nil +} + +func TestDoctorReportsAndCleansOrphanedDockerLabs(t *testing.T) { + run := func(t *testing.T, factory game.Factory, args ...string) (string, error) { + t.Helper() + store := profile.NewStore(filepath.Join(t.TempDir(), "profile.json"), "alex") + catalog, err := mission.LoadCatalog() + if err != nil { + t.Fatal(err) + } + out := &bytes.Buffer{} + app := New(Config{Out: out, ErrOut: &bytes.Buffer{}, Catalog: catalog, Store: store, Factory: factory}) + err = app.Run(append([]string{"doctor"}, args...)) + return out.String(), err + } + + t.Run("plain doctor is read-only", func(t *testing.T) { + factory := &cliJanitorFactory{cliDockerFactory: cliDockerFactory{available: true, detail: "Docker is ready for this mission."}, orphans: 2} + output, err := run(t, factory) + if err != nil || !strings.Contains(output, "docker cleanup: 2 orphaned lab containers from exited OpsQuest processes; run 'opsquest doctor --cleanup'") { + t.Fatalf("doctor output = %s, error = %v", output, err) + } + if factory.checks != 1 || factory.removeCalls != 0 { + t.Fatalf("checks = %d, removals = %d", factory.checks, factory.removeCalls) + } + }) + + t.Run("clean engine", func(t *testing.T) { + factory := &cliJanitorFactory{cliDockerFactory: cliDockerFactory{available: true, detail: "ready"}} + output, err := run(t, factory) + if err != nil || !strings.Contains(output, "docker cleanup: no orphaned lab containers") { + t.Fatalf("doctor output = %s, error = %v", output, err) + } + }) + + t.Run("check failure is a warning", func(t *testing.T) { + factory := &cliJanitorFactory{cliDockerFactory: cliDockerFactory{available: true, detail: "ready"}, checkErr: fmt.Errorf("list failed")} + output, err := run(t, factory) + if err != nil || !strings.Contains(output, "could not check for orphaned lab containers · list failed") { + t.Fatalf("doctor output = %s, error = %v", output, err) + } + }) + + t.Run("unavailable engine skips the check", func(t *testing.T) { + factory := &cliJanitorFactory{cliDockerFactory: cliDockerFactory{detail: "Docker labs unavailable"}, orphans: 2} + output, err := run(t, factory) + if err != nil || strings.Contains(output, "docker cleanup") || factory.checks != 0 { + t.Fatalf("doctor output = %s, error = %v, checks = %d", output, err, factory.checks) + } + }) + + t.Run("cleanup removes orphans", func(t *testing.T) { + factory := &cliJanitorFactory{cliDockerFactory: cliDockerFactory{available: true, detail: "ready"}, orphans: 1} + output, err := run(t, factory, "--cleanup") + if err != nil || !strings.Contains(output, "docker cleanup: removed 1 orphaned lab container\n") || factory.removeCalls != 1 { + t.Fatalf("doctor output = %s, error = %v, removals = %d", output, err, factory.removeCalls) + } + }) + + t.Run("cleanup failure fails the command", func(t *testing.T) { + factory := &cliJanitorFactory{cliDockerFactory: cliDockerFactory{available: true, detail: "ready"}, orphans: 2, removeErr: fmt.Errorf("daemon went away")} + output, err := run(t, factory, "--cleanup") + if err == nil || !strings.Contains(err.Error(), "docker cleanup failed") || !strings.Contains(output, "removed 1, then failed · daemon went away") { + t.Fatalf("doctor output = %s, error = %v", output, err) + } + }) + + t.Run("cleanup failure before removal", func(t *testing.T) { + factory := &zeroRemovalJanitor{&cliJanitorFactory{cliDockerFactory: cliDockerFactory{detail: "unavailable"}, removeErr: fmt.Errorf("docker executable not found in PATH")}} + output, err := run(t, factory, "--cleanup") + if err == nil || !strings.Contains(output, "docker cleanup: failed · docker executable not found in PATH") || strings.Contains(output, "removed 0") { + t.Fatalf("doctor output = %s, error = %v", output, err) + } + }) + + t.Run("cleanup without Docker labs", func(t *testing.T) { + output, err := run(t, game.SandboxFactory{}, "--cleanup") + if err != nil || !strings.Contains(output, "nothing to clean; Docker labs are not configured") { + t.Fatalf("doctor output = %s, error = %v", output, err) + } + }) + + t.Run("rejects positional arguments", func(t *testing.T) { + if _, err := run(t, game.SandboxFactory{}, "cleanup"); err == nil { + t.Fatal("doctor accepted a positional argument") + } + }) +} + +// zeroRemovalJanitor fails before removing anything, as when the Docker CLI +// is absent. +type zeroRemovalJanitor struct { + *cliJanitorFactory +} + +func (f *zeroRemovalJanitor) RemoveOrphanedResources(context.Context) (int, error) { + f.removeCalls++ + return 0, f.removeErr +} diff --git a/internal/dockerlab/dockerlab_test.go b/internal/dockerlab/dockerlab_test.go index 1561b31..a0be794 100644 --- a/internal/dockerlab/dockerlab_test.go +++ b/internal/dockerlab/dockerlab_test.go @@ -10,21 +10,33 @@ import ( "strings" "sync" "testing" + "time" "github.com/aleksandergregersen/opsquest/internal/game" "github.com/aleksandergregersen/opsquest/internal/mission" ) -const testImageReference = "docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" +const ( + testImageReference = "docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" + testOwnerPID = 4242 + testOwnerHost = "test-host" +) + +var testNow = time.Date(2026, time.September, 30, 12, 0, 0, 0, time.UTC) type fakeContainer struct { - labels map[string]string - name string - running bool - status string - log string - exitCode int - oneShot bool + labels map[string]string + name string + created time.Time + running bool + restarting bool + status string + log string + exitCode int + oneShot bool + restartPolicy string + restartCount int + healthCmd string } type fakeDockerRunner struct { @@ -44,6 +56,8 @@ type fakeDockerRunner struct { failNextInspect map[string]error hideNextInspect map[string]bool failNextRemove map[string]error + failManagedList bool + stripHealth bool } func newFakeDockerRunner() *fakeDockerRunner { @@ -86,9 +100,12 @@ func (r *fakeDockerRunner) run(ctx context.Context, args ...string) (runResult, if r.failCreateAt == r.createAttempts { return runResult{stderr: "fixture creation failed"}, errors.New("exit status 1") } - id := strings.Repeat(string(rune('b'+len(r.containerOrder))), 64) + // Created IDs use a prefix no seeded test container shares. + id := fmt.Sprintf("c0ffee%058x", r.createAttempts) labels := make(map[string]string) name := "" + restartPolicy := "" + healthCmd := "" for index := 0; index+1 < len(args); index++ { switch args[index] { case "--name": @@ -98,9 +115,16 @@ func (r *fakeDockerRunner) run(ctx context.Context, args ...string) (runResult, if found { labels[labelName] = value } + case "--restart": + restartPolicy = args[index+1] + case "--health-cmd": + healthCmd = args[index+1] } } - container := &fakeContainer{labels: labels, name: name, status: "created"} + container := &fakeContainer{labels: labels, name: name, created: testNow, status: "created", restartPolicy: restartPolicy, healthCmd: healthCmd} + if len(args) >= 3 && args[len(args)-3] == "opsquest-service" { + container.log = args[len(args)-2] + "\n" + } if len(args) >= 3 && args[len(args)-3] == "opsquest-diagnostic" { container.oneShot = true container.log = args[len(args)-2] + "\n" @@ -126,7 +150,15 @@ func (r *fakeDockerRunner) run(ctx context.Context, args ...string) (runResult, if !exists { return missingContainerResult(id) } - if container.oneShot { + if container.oneShot && strings.HasPrefix(container.restartPolicy, "on-failure") { + // Model a crash loop observed between restarts: Docker reports a + // restarting container as running and appends each attempt's log. + container.running = true + container.restarting = true + container.status = "restarting" + container.restartCount += crashLoopReadyRestarts + container.log = strings.Repeat(strings.SplitAfter(container.log, "\n")[0], container.restartCount+1) + } else if container.oneShot { container.running = false container.status = "exited" } else { @@ -142,6 +174,7 @@ func (r *fakeDockerRunner) run(ctx context.Context, args ...string) (runResult, return missingContainerResult(id) } container.running = false + container.restarting = false container.status = "exited" return runResult{stdout: id + "\n"}, nil case hasPrefix(args, "container", "wait"): @@ -157,6 +190,20 @@ func (r *fakeDockerRunner) run(ctx context.Context, args ...string) (runResult, if !exists { return missingContainerResult(id) } + if len(args) == 5 && args[2] == "--tail" { + tail, err := strconv.Atoi(args[3]) + if err != nil { + return runResult{}, err + } + lines := strings.SplitAfter(container.log, "\n") + if lines[len(lines)-1] == "" { + lines = lines[:len(lines)-1] + } + if tail < len(lines) { + lines = lines[len(lines)-tail:] + } + return runResult{stdout: strings.Join(lines, "")}, nil + } return runResult{stdout: container.log}, nil case hasPrefix(args, "container", "inspect", "--format"): reference := args[len(args)-1] @@ -173,16 +220,32 @@ func (r *fakeDockerRunner) run(ctx context.Context, args ...string) (runResult, if !exists { return missingContainerResult(id) } - inspection := containerInspection{ID: id} + inspection := containerInspection{ID: id, Name: "/" + container.name, Created: container.created.Format(time.RFC3339Nano), RestartCount: container.restartCount} inspection.Config.Labels = cloneStrings(container.labels) inspection.State.Running = container.running + inspection.State.Restarting = container.restarting inspection.State.Status = container.status inspection.State.ExitCode = container.exitCode + if policy, retries, found := strings.Cut(container.restartPolicy, ":"); found { + inspection.HostConfig.RestartPolicy.Name = policy + inspection.HostConfig.RestartPolicy.MaximumRetryCount, _ = strconv.Atoi(retries) + } else { + inspection.HostConfig.RestartPolicy.Name = container.restartPolicy + } + if container.healthCmd != "" && container.running && !r.stripHealth { + health := "unhealthy" + if container.healthCmd == "true" { + health = "healthy" + } + inspection.State.Health = &healthState{Status: health} + } encoded, err := json.Marshal(inspection) if err != nil { return runResult{}, err } return runResult{stdout: string(encoded) + "\n"}, nil + case hasPrefix(args, "container", "ls", "--all", "--quiet", "--no-trunc") && r.failManagedList && args[len(args)-1] == "label="+managedLabel+"=true": + return runResult{stderr: "list failed"}, errors.New("exit status 1") case hasPrefix(args, "container", "ls", "--all", "--quiet", "--no-trunc"): filters := make(map[string]string) for index := 0; index+1 < len(args); index++ { @@ -205,6 +268,17 @@ func (r *fakeDockerRunner) run(ctx context.Context, args ...string) (runResult, output.WriteByte('\n') } return runResult{stdout: output.String()}, nil + case len(args) == 3 && hasPrefix(args, "container", "rm"): + id := args[len(args)-1] + container, exists := r.containers[id] + if !exists { + return missingContainerResult(id) + } + if container.running { + return runResult{stderr: "Error response from daemon: cannot remove container: container is running: stop the container before removing or force remove"}, errors.New("exit status 1") + } + delete(r.containers, id) + return runResult{stdout: id + "\n"}, nil case hasPrefix(args, "container", "rm", "--force"): id := args[len(args)-1] if err := r.failNextRemove[id]; err != nil { @@ -242,6 +316,20 @@ func labelsMatch(labels, filters map[string]string) bool { return true } +func (r *fakeDockerRunner) seedContainer(id string, container *fakeContainer) { + r.mutex.Lock() + defer r.mutex.Unlock() + r.containers[id] = container + r.containerOrder = append(r.containerOrder, id) +} + +func (r *fakeDockerRunner) hasContainer(id string) bool { + r.mutex.Lock() + defer r.mutex.Unlock() + _, exists := r.containers[id] + return exists +} + func (r *fakeDockerRunner) resetCalls() { r.mutex.Lock() defer r.mutex.Unlock() @@ -335,9 +423,29 @@ func testDiagnosticDockerMission() mission.Mission { } func testFactory(commandRunner runner) *Factory { - return newFactory(game.SandboxFactory{}, commandRunner, nil, func() (string, error) { + factory := newFactory(game.SandboxFactory{}, commandRunner, nil, func() (string, error) { return strings.Repeat("a", 24), nil }) + factory.owner = testOwner(func(pid int) bool { return pid == testOwnerPID }) + factory.pollInterval = time.Millisecond + return factory +} + +func testOwner(alive func(int) bool) processOwner { + return processOwner{pid: testOwnerPID, host: testOwnerHost, alive: alive, now: func() time.Time { return testNow }} +} + +// firstCall returns the first recorded Docker invocation with prefix and its +// position, so setup assertions do not depend on unrelated preflight calls. +func firstCall(t *testing.T, calls [][]string, prefix ...string) (int, []string) { + t.Helper() + for index, call := range calls { + if hasPrefix(call, prefix...) { + return index, call + } + } + t.Fatalf("no Docker call with prefix %q in %q", prefix, calls) + return -1, nil } func createTestEnvironment(t *testing.T, commandRunner *fakeDockerRunner) *environment { @@ -471,6 +579,8 @@ func TestCreateUsesExactLabelsAndResourceLimits(t *testing.T) { "--label", "com.opsquest.session=aaaaaaaaaaaaaaaaaaaaaaaa", "--label", "com.opsquest.mission=docker-container-census", "--label", "com.opsquest.alias=api", + "--label", "com.opsquest.owner-pid=4242", + "--label", "com.opsquest.owner-host=test-host", "--network", "none", "--read-only", "--tmpfs", "/tmp:rw,noexec,nosuid,nodev,size=8m", @@ -488,12 +598,13 @@ func TestCreateUsesExactLabelsAndResourceLimits(t *testing.T) { testImageReference, "86400", } - if !reflect.DeepEqual(calls[3], wantCreate) { - t.Fatalf("first create args:\n got: %q\nwant: %q", calls[3], wantCreate) + createIndex, create := firstCall(t, calls, "container", "create") + if !reflect.DeepEqual(create, wantCreate) { + t.Fatalf("first create args:\n got: %q\nwant: %q", create, wantCreate) } metricsID := environment.byAlias["metrics"].id - if !reflect.DeepEqual(calls[5], []string{"container", "start", metricsID}) { - t.Fatalf("running fixture start = %q", calls[5]) + if !reflect.DeepEqual(calls[createIndex+2], []string{"container", "start", metricsID}) { + t.Fatalf("running fixture start = %q", calls[createIndex+2]) } for _, call := range calls { joined := strings.Join(call, " ") @@ -519,7 +630,7 @@ func TestDiagnosticFixtureUsesFixedCommandAndExposesSanitizedObservations(t *tes "-c", `printf '%s\n' "$1"; exit "$2"`, "opsquest-diagnostic", logText, "23", } - create := calls[3] + createIndex, create := firstCall(t, calls, "container", "create") if len(create) < len(wantTail) || !reflect.DeepEqual(create[len(create)-len(wantTail):], wantTail) { t.Fatalf("diagnostic create tail:\n got: %q\nwant: %q", create, wantTail) } @@ -527,8 +638,8 @@ func TestDiagnosticFixtureUsesFixedCommandAndExposesSanitizedObservations(t *tes t.Fatalf("diagnostic data was interpolated into shell program: %q", create[len(create)-4]) } jobID := environment.byAlias["job"].id - if !reflect.DeepEqual(calls[4], []string{"container", "start", jobID}) || !reflect.DeepEqual(calls[5], []string{"container", "wait", jobID}) { - t.Fatalf("diagnostic lifecycle calls = %q", calls[4:6]) + if !reflect.DeepEqual(calls[createIndex+1], []string{"container", "start", jobID}) || !reflect.DeepEqual(calls[createIndex+2], []string{"container", "wait", jobID}) { + t.Fatalf("diagnostic lifecycle calls = %q", calls[createIndex+1:createIndex+3]) } logged, err := environment.Execute(context.Background(), "docker container logs job") diff --git a/internal/dockerlab/environment.go b/internal/dockerlab/environment.go index f3d7276..d79380a 100644 --- a/internal/dockerlab/environment.go +++ b/internal/dockerlab/environment.go @@ -9,6 +9,7 @@ import ( "strconv" "strings" "sync" + "time" "github.com/aleksandergregersen/opsquest/internal/game" "github.com/aleksandergregersen/opsquest/internal/mission" @@ -20,6 +21,24 @@ const ( sessionLabel = "com.opsquest.session" missionLabel = "com.opsquest.mission" aliasLabel = "com.opsquest.alias" + // Owner labels let a later process recognize fixtures whose OpsQuest + // process exited without cleanup. They are advisory: removal still + // requires every ownership label above to match. + ownerPIDLabel = "com.opsquest.owner-pid" + ownerHostLabel = "com.opsquest.owner-host" + + // fixtureLifetime bounds every long-lived fixture process, so a fixture + // orphaned by an abnormal exit stops consuming resources within a day. + fixtureLifetime = 24 * time.Hour + // crashLoopMaxRetries bounds a crash-loop fixture's restart policy. With + // Docker's capped exponential backoff it keeps looping for roughly 40 + // minutes, which comfortably covers a mission attempt. + crashLoopMaxRetries = 50 + // crashLoopReadyRestarts is how many restarts setup waits for, so the + // player's first logs and inspection already show a loop. + crashLoopReadyRestarts = 2 + fixtureReadyTimeout = 10 * time.Second + fixtureReadyInterval = 100 * time.Millisecond ) type trackedContainer struct { @@ -31,11 +50,15 @@ type trackedContainer struct { } type environment struct { - runner runner - sessionID string - missionID string - containers []*trackedContainer - byAlias map[string]*trackedContainer + runner runner + sessionID string + missionID string + ownerPID int + ownerHost string + pollInterval time.Duration + readyTimeout time.Duration + containers []*trackedContainer + byAlias map[string]*trackedContainer cleanupMutex sync.Mutex mutex sync.RWMutex @@ -71,7 +94,7 @@ func (e *environment) Execute(ctx context.Context, line string) (game.Execution, result.Output = dockerHelp result.PracticedCommands = []string{"help"} case actionList: - result.Output, err = e.listContainers(ctx, action.all) + result.Output, err = e.listContainers(ctx, action.all, action.filters) result.PracticedCommands = []string{"docker"} case actionStart: err = e.startContainer(ctx, action.alias) @@ -91,11 +114,17 @@ func (e *environment) Execute(ctx context.Context, line string) (game.Execution, result.Output = action.alias + "\n" } result.PracticedCommands = []string{"docker"} + case actionRemove: + err = e.removeContainer(ctx, action.alias) + if err == nil { + result.Output = action.alias + "\n" + } + result.PracticedCommands = []string{"docker"} case actionInspect: result.Output, err = e.logicalInspect(ctx, action.alias) result.PracticedCommands = []string{"docker"} case actionLogs: - result.Output, err = e.containerLogs(ctx, action.alias) + result.Output, err = e.containerLogs(ctx, action.alias, action.tail) result.PracticedCommands = []string{"docker"} default: err = fmt.Errorf("unsupported Docker action") @@ -122,6 +151,13 @@ func (e *environment) Observe(ctx context.Context, condition mission.Condition) } inspection, exists, err := e.inspect(ctx, tracked.id) return exists && !inspection.State.Running, err + case mission.ConditionDockerContainerAbsent: + tracked, exists := e.container(condition.Container) + if !exists { + return false, nil + } + _, exists, err := e.inspect(ctx, tracked.id) + return err == nil && !exists, err case mission.ConditionDockerContainerCountEqual: if condition.Count == nil { return false, fmt.Errorf("docker_container_count_equals requires count") @@ -218,6 +254,18 @@ func (e *environment) createContainer(ctx context.Context, index int, fixture mi "--label", sessionLabel + "=" + e.sessionID, "--label", missionLabel + "=" + e.missionID, "--label", aliasLabel + "=" + fixture.Name, + } + if e.ownerHost != "" && e.ownerPID > 0 { + args = append(args, + "--label", ownerPIDLabel+"="+strconv.Itoa(e.ownerPID), + "--label", ownerHostLabel+"="+e.ownerHost, + ) + } + restartPolicy := "no" + if fixture.Restart == mission.DockerRestartOnFailure { + restartPolicy = fmt.Sprintf("on-failure:%d", crashLoopMaxRetries) + } + args = append(args, "--network", "none", "--read-only", "--tmpfs", "/tmp:rw,noexec,nosuid,nodev,size=8m", @@ -229,21 +277,40 @@ func (e *environment) createContainer(ctx context.Context, index int, fixture mi "--memory-swap", "128m", "--cpus", "0.5", "--ulimit", "nofile=256:256", - "--restart", "no", + "--restart", restartPolicy, "--stop-timeout", "1", + ) + if probe, configured := healthProbes[fixture.Health]; configured { + // The probe is a fixed BusyBox applet selected by a validated enum; + // mission content and player input never supply the probe command. + args = append(args, + "--health-cmd", probe, + "--health-interval", "1s", + "--health-timeout", "1s", + "--health-retries", "1", + ) } - if fixture.ExitCode != nil { + lifetime := strconv.Itoa(int(fixtureLifetime / time.Second)) + switch { + case fixture.ExitCode != nil: args = append(args, "--entrypoint", "/bin/sh", reference, "-c", `printf '%s\n' "$1"; exit "$2"`, "opsquest-diagnostic", fixture.Log, strconv.Itoa(*fixture.ExitCode), ) - } else { + case fixture.Log != "": + args = append(args, + "--entrypoint", "/bin/sh", + reference, + "-c", `printf '%s\n' "$1"; exec sleep "$2"`, + "opsquest-service", fixture.Log, lifetime, + ) + default: args = append(args, "--entrypoint", "/bin/sleep", reference, - "86400", + lifetime, ) } result, err := e.run(ctx, args...) @@ -291,39 +358,70 @@ func (e *environment) reconcileAmbiguousCreate(tracked *trackedContainer, create return tracked, createErr } +// healthProbes maps a fixture health enum to a fixed probe command. +var healthProbes = map[string]string{ + mission.DockerHealthHealthy: "true", + mission.DockerHealthUnhealthy: "false", +} + func (e *environment) startContainer(ctx context.Context, alias string) error { - tracked, exists := e.container(alias) - if !exists { - return fmt.Errorf("docker: container %q is not part of this mission", alias) - } - _, err := e.run(ctx, "container", "start", tracked.id) + _, err := e.runOnAlias(ctx, alias, "container", "start") return err } func (e *environment) restartContainer(ctx context.Context, alias string) error { - tracked, exists := e.container(alias) - if !exists { - return fmt.Errorf("docker: container %q is not part of this mission", alias) - } - _, err := e.run(ctx, "container", "restart", tracked.id) + _, err := e.runOnAlias(ctx, alias, "container", "restart") return err } func (e *environment) stopContainer(ctx context.Context, alias string) error { + _, err := e.runOnAlias(ctx, alias, "container", "stop") + return err +} + +// removeContainer deletes one stopped mission container. Running containers +// are refused rather than forced so the lesson keeps stop and remove distinct. +func (e *environment) removeContainer(ctx context.Context, alias string) error { tracked, exists := e.container(alias) if !exists { return fmt.Errorf("docker: container %q is not part of this mission", alias) } - _, err := e.run(ctx, "container", "stop", tracked.id) + inspection, exists, err := e.inspect(ctx, tracked.id) + if err != nil { + return err + } + if !exists { + return fmt.Errorf("docker: container %q has already been removed", alias) + } + if inspection.State.Running { + return fmt.Errorf("docker: cannot remove running container %q; stop it first", alias) + } + _, err = e.runOnAlias(ctx, alias, "container", "rm") return err } -func (e *environment) waitContainer(ctx context.Context, alias string, expectedExitCode int) error { +// runOnAlias runs one fixed Docker action against the exact container ID +// behind alias. Engine errors about a missing container are translated so +// removed fixtures never reveal their real container ID. +func (e *environment) runOnAlias(ctx context.Context, alias string, args ...string) (runResult, error) { tracked, exists := e.container(alias) if !exists { - return fmt.Errorf("docker: container %q is not part of this mission", alias) + return runResult{}, fmt.Errorf("docker: container %q is not part of this mission", alias) } - result, err := e.run(ctx, "container", "wait", tracked.id) + operationCtx, cancel := context.WithTimeout(ctx, dockerOperationTimeout) + defer cancel() + result, err := e.runner.run(operationCtx, append(args, tracked.id)...) + if err != nil { + if isMissingContainer(result) { + return result, fmt.Errorf("docker: container %q has been removed", alias) + } + return result, fmt.Errorf("Docker command failed%s", redactIdentifiers(dockerFailureDetail(result, err), tracked)) + } + return result, nil +} + +func (e *environment) waitContainer(ctx context.Context, alias string, expectedExitCode int) error { + result, err := e.runOnAlias(ctx, alias, "container", "wait") if err != nil { return err } @@ -334,40 +432,150 @@ func (e *environment) waitContainer(ctx context.Context, alias string, expectedE return nil } -func (e *environment) containerLogs(ctx context.Context, alias string) (string, error) { +// waitForFixture polls sanitized inspection until ready reports true. Setup +// uses it for fixtures whose teaching state (a health verdict or a visible +// crash loop) appears shortly after start. +func (e *environment) waitForFixture(ctx context.Context, alias string, ready func(containerInspection) bool) error { tracked, exists := e.container(alias) if !exists { - return "", fmt.Errorf("docker: container %q is not part of this mission", alias) + return fmt.Errorf("docker: container %q is not part of this mission", alias) } - result, err := e.run(ctx, "container", "logs", tracked.id) + interval := e.pollInterval + if interval <= 0 { + interval = fixtureReadyInterval + } + timeout := e.readyTimeout + if timeout <= 0 { + timeout = fixtureReadyTimeout + } + deadline := time.NewTimer(timeout) + defer deadline.Stop() + for { + inspection, exists, err := e.inspect(ctx, tracked.id) + if err != nil { + return err + } + if !exists { + return fmt.Errorf("Docker fixture %s disappeared during setup", alias) + } + if ready(inspection) { + return nil + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-deadline.C: + return fmt.Errorf("Docker fixture %s did not reach its teaching state within %s", alias, timeout) + case <-time.After(interval): + } + } +} + +func (e *environment) containerLogs(ctx context.Context, alias string, tail int) (string, error) { + args := []string{"container", "logs"} + if tail >= 0 { + args = append(args, "--tail", strconv.Itoa(tail)) + } + result, err := e.runOnAlias(ctx, alias, args...) if err != nil { return "", err } return result.stdout, nil } -func (e *environment) listContainers(ctx context.Context, all bool) (string, error) { - var output strings.Builder - output.WriteString("CONTAINER ID IMAGE STATUS NAMES\n") +func (e *environment) listContainers(ctx context.Context, all bool, filters []listFilter) (string, error) { + type row struct { + logicalID string + image string + status string + alias string + } + rows := make([]row, 0, len(e.containers)) + // Eight columns preserves the original listing layout for plain states. + statusWidth := 8 for _, tracked := range e.snapshotContainers() { inspection, exists, err := e.inspect(ctx, tracked.id) if err != nil { return "", err } - if !exists || !all && !inspection.State.Running { + // A status filter selects stopped containers even without --all, + // matching docker ps. + if !exists || !all && !hasFilterKey(filters, "status") && !inspection.State.Running || !matchesFilters(inspection, filters) { continue } - status := inspection.State.Status - if status == "" { + status := displayStatus(inspection) + statusWidth = max(statusWidth, len(status)) + rows = append(rows, row{logicalID: tracked.logicalID, image: tracked.imageAlias, status: status, alias: tracked.alias}) + } + var output strings.Builder + fmt.Fprintf(&output, "%-13s %-9s %-*s %s\n", "CONTAINER ID", "IMAGE", statusWidth, "STATUS", "NAMES") + for _, item := range rows { + fmt.Fprintf(&output, "%-13s %-9s %-*s %s\n", item.logicalID, item.image, statusWidth, item.status, item.alias) + } + return output.String(), nil +} + +func displayStatus(inspection containerInspection) string { + status := inspection.State.Status + if status == "" { + if inspection.State.Running { + status = "running" + } else { + status = "stopped" + } + } + if health := healthStatus(inspection); inspection.State.Running && health != "none" { + status += " (" + health + ")" + } + return status +} + +// healthStatus reports the probe verdict, or none for a fixture without a +// health probe. Like docker ps, listings only show it for running containers. +func healthStatus(inspection containerInspection) string { + if inspection.State.Health == nil || inspection.State.Health.Status == "" { + return "none" + } + return inspection.State.Health.Status +} + +func hasFilterKey(filters []listFilter, key string) bool { + for _, filter := range filters { + if filter.key == key { + return true + } + } + return false +} + +// matchesFilters follows docker ps semantics: values for one key are +// alternatives, and different keys must all match. +func matchesFilters(inspection containerInspection, filters []listFilter) bool { + matched := make(map[string]bool, len(filters)) + for _, filter := range filters { + if _, seen := matched[filter.key]; !seen { + matched[filter.key] = false + } + var actual string + switch filter.key { + case "status": + actual = inspection.State.Status + case "health": + actual = "none" if inspection.State.Running { - status = "running" - } else { - status = "stopped" + actual = healthStatus(inspection) } } - fmt.Fprintf(&output, "%-13s %-9s %-8s %s\n", tracked.logicalID, tracked.imageAlias, status, tracked.alias) + if actual == filter.value { + matched[filter.key] = true + } } - return output.String(), nil + for _, ok := range matched { + if !ok { + return false + } + } + return true } func (e *environment) logicalInspect(ctx context.Context, alias string) (string, error) { @@ -382,19 +590,36 @@ func (e *environment) logicalInspect(ctx context.Context, alias string) (string, if !exists { return "", fmt.Errorf("docker: container %q no longer exists", alias) } + type logicalHealth struct { + Status string `json:"Status"` + } logical := struct { - ID string `json:"Id"` - Name string `json:"Name"` - Image string `json:"Image"` - State struct { - Running bool `json:"Running"` - Status string `json:"Status"` - ExitCode int `json:"ExitCode"` + ID string `json:"Id"` + Name string `json:"Name"` + Image string `json:"Image"` + RestartCount int `json:"RestartCount"` + State struct { + Running bool `json:"Running"` + Restarting bool `json:"Restarting"` + Status string `json:"Status"` + ExitCode int `json:"ExitCode"` + Health *logicalHealth `json:"Health,omitempty"` } `json:"State"` - }{ID: tracked.logicalID, Name: tracked.alias, Image: tracked.imageAlias} + HostConfig struct { + RestartPolicy restartPolicy `json:"RestartPolicy"` + } `json:"HostConfig"` + }{ID: tracked.logicalID, Name: tracked.alias, Image: tracked.imageAlias, RestartCount: inspection.RestartCount} logical.State.Running = inspection.State.Running + logical.State.Restarting = inspection.State.Restarting logical.State.Status = inspection.State.Status logical.State.ExitCode = inspection.State.ExitCode + if health := healthStatus(inspection); health != "none" { + logical.State.Health = &logicalHealth{Status: health} + } + logical.HostConfig.RestartPolicy = inspection.HostConfig.RestartPolicy + if logical.HostConfig.RestartPolicy.Name == "" { + logical.HostConfig.RestartPolicy.Name = "no" + } encoded, err := json.MarshalIndent(logical, "", " ") if err != nil { return "", err @@ -402,15 +627,32 @@ func (e *environment) logicalInspect(ctx context.Context, alias string) (string, return string(encoded) + "\n", nil } +type restartPolicy struct { + Name string `json:"Name"` + MaximumRetryCount int `json:"MaximumRetryCount"` +} + +type healthState struct { + Status string `json:"Status"` +} + type containerInspection struct { - ID string `json:"Id"` - Config struct { + ID string `json:"Id"` + Name string `json:"Name"` + Created string `json:"Created"` + RestartCount int `json:"RestartCount"` + Config struct { Labels map[string]string `json:"Labels"` } `json:"Config"` + HostConfig struct { + RestartPolicy restartPolicy `json:"RestartPolicy"` + } `json:"HostConfig"` State struct { - Running bool `json:"Running"` - Status string `json:"Status"` - ExitCode int `json:"ExitCode"` + Running bool `json:"Running"` + Restarting bool `json:"Restarting"` + Status string `json:"Status"` + ExitCode int `json:"ExitCode"` + Health *healthState `json:"Health"` } `json:"State"` } @@ -433,7 +675,13 @@ func (e *environment) inspectUnchecked(ctx context.Context, id string) (containe } func (e *environment) inspectReferenceUnchecked(ctx context.Context, reference string) (containerInspection, bool, error) { - result, err := e.run(ctx, "container", "inspect", "--format", "{{json .}}", reference) + return inspectReference(ctx, e.runner, reference) +} + +// inspectReference inspects one exact container ID or generated name. It is +// shared by attempt environments and the orphan janitor. +func inspectReference(ctx context.Context, commandRunner runner, reference string) (containerInspection, bool, error) { + result, err := runDocker(ctx, commandRunner, "container", "inspect", "--format", "{{json .}}", reference) if err != nil { if isMissingContainer(result) { return containerInspection{}, false, nil @@ -477,15 +725,33 @@ func (e *environment) countOwnedContainers(ctx context.Context) (int, error) { } func (e *environment) run(ctx context.Context, args ...string) (runResult, error) { + return runDocker(ctx, e.runner, args...) +} + +// runDocker runs one fixed Docker invocation with the standard operation +// timeout and a bounded error summary. +func runDocker(ctx context.Context, commandRunner runner, args ...string) (runResult, error) { operationCtx, cancel := context.WithTimeout(ctx, dockerOperationTimeout) defer cancel() - result, err := e.runner.run(operationCtx, args...) + result, err := commandRunner.run(operationCtx, args...) if err != nil { return result, fmt.Errorf("Docker command failed%s", dockerFailureDetail(result, err)) } return result, nil } +// redactIdentifiers replaces a tracked container's engine ID and generated +// name in player-visible error text with its logical identity. +func redactIdentifiers(message string, tracked *trackedContainer) string { + if tracked.id != "" { + message = strings.ReplaceAll(message, tracked.id, tracked.logicalID) + if len(tracked.id) >= 12 { + message = strings.ReplaceAll(message, tracked.id[:12], tracked.logicalID) + } + } + return strings.ReplaceAll(message, tracked.actualName, tracked.alias) +} + func (e *environment) ready(ctx context.Context) error { if err := ctx.Err(); err != nil { return err @@ -525,7 +791,7 @@ func (dockerCompletion) CommandNames() []string { } func (c dockerCompletion) PathCandidates(prefix string) []game.CompletionCandidate { - values := []string{"--all", "-a", "container", "inspect", "logs", "ls", "ps", "restart", "start", "stop"} + values := []string{"--all", "--filter", "--tail", "-a", "container", "inspect", "logs", "ls", "ps", "restart", "rm", "start", "stop"} for _, tracked := range c.environment.snapshotContainers() { values = append(values, tracked.alias) } diff --git a/internal/dockerlab/factory.go b/internal/dockerlab/factory.go index 6093ae0..095c684 100644 --- a/internal/dockerlab/factory.go +++ b/internal/dockerlab/factory.go @@ -6,6 +6,7 @@ import ( "encoding/hex" "errors" "fmt" + "os" "os/exec" "regexp" "strings" @@ -32,11 +33,34 @@ type Factory struct { runner runner lookupErr error newSession func() (string, error) + owner processOwner + // pollInterval and readyTimeout override fixture readiness polling in + // tests. + pollInterval time.Duration + readyTimeout time.Duration +} + +// processOwner identifies the current OpsQuest process for orphan detection +// and decides whether another recorded owner process is still alive. +type processOwner struct { + pid int + host string + alive func(int) bool + now func() time.Time +} + +func currentProcessOwner() processOwner { + host, err := os.Hostname() + if err != nil { + host = "" + } + return processOwner{pid: os.Getpid(), host: host, alive: processAlive, now: time.Now} } var ( _ game.Factory = (*Factory)(nil) _ game.AvailabilityChecker = (*Factory)(nil) + _ game.ResourceJanitor = (*Factory)(nil) ) // NewFactory creates a combined environment factory. Docker remains optional: @@ -56,6 +80,7 @@ func NewFactory(fallback game.Factory) *Factory { runner: commandRunner, lookupErr: err, newSession: randomSessionID, + owner: currentProcessOwner(), } } @@ -66,7 +91,7 @@ func newFactory(fallback game.Factory, commandRunner runner, lookupErr error, ne if newSession == nil { newSession = randomSessionID } - return &Factory{fallback: fallback, runner: commandRunner, lookupErr: lookupErr, newSession: newSession} + return &Factory{fallback: fallback, runner: commandRunner, lookupErr: lookupErr, newSession: newSession, owner: currentProcessOwner()} } func (f *Factory) Create(ctx context.Context, item mission.Mission) (game.Environment, error) { @@ -94,11 +119,20 @@ func (f *Factory) Create(ctx context.Context, item mission.Mission) (game.Enviro return nil, fmt.Errorf("create Docker session ID: generated value is invalid") } + // Reclaim fixtures abandoned by earlier crashed processes before adding + // new ones. This is best effort: a failed sweep must not block play, and + // the same cleanup is available explicitly through opsquest doctor. + _, _ = f.removeOrphans(ctx) + environment := &environment{ - runner: f.runner, - sessionID: sessionID, - missionID: item.ID, - byAlias: make(map[string]*trackedContainer), + runner: f.runner, + sessionID: sessionID, + missionID: item.ID, + ownerPID: f.owner.pid, + ownerHost: f.owner.host, + pollInterval: f.pollInterval, + readyTimeout: f.readyTimeout, + byAlias: make(map[string]*trackedContainer), } images := make(map[string]string, len(item.Docker.Images)) for _, image := range item.Docker.Images { @@ -115,6 +149,18 @@ func (f *Factory) Create(ctx context.Context, item mission.Mission) (game.Enviro } } for _, fixture := range item.Docker.Containers { + if fixture.Restart == mission.DockerRestartOnFailure { + if err := environment.startContainer(ctx, fixture.Name); err != nil { + return environment, joinSetupCleanupError(fmt.Errorf("start crash-loop Docker fixture %s: %w", fixture.Name, err), environment.Close()) + } + looping := func(inspection containerInspection) bool { + return inspection.RestartCount >= crashLoopReadyRestarts + } + if err := environment.waitForFixture(ctx, fixture.Name, looping); err != nil { + return environment, joinSetupCleanupError(fmt.Errorf("wait for crash-loop Docker fixture %s: %w", fixture.Name, err), environment.Close()) + } + continue + } if fixture.ExitCode != nil { if err := environment.startContainer(ctx, fixture.Name); err != nil { return environment, joinSetupCleanupError(fmt.Errorf("start diagnostic Docker fixture %s: %w", fixture.Name, err), environment.Close()) @@ -131,6 +177,20 @@ func (f *Factory) Create(ctx context.Context, item mission.Mission) (game.Enviro return environment, joinSetupCleanupError(fmt.Errorf("start Docker fixture %s: %w", fixture.Name, err), environment.Close()) } } + // Probes run concurrently once every fixture has started, so waiting + // after all starts costs one probe interval rather than one per fixture. + for _, fixture := range item.Docker.Containers { + if fixture.Health == "" || fixture.State != mission.DockerStateRunning { + continue + } + expected := fixture.Health + settled := func(inspection containerInspection) bool { + return healthStatus(inspection) == expected + } + if err := environment.waitForFixture(ctx, fixture.Name, settled); err != nil { + return environment, joinSetupCleanupError(fmt.Errorf("wait for Docker fixture %s health: %w", fixture.Name, err), environment.Close()) + } + } return environment, nil } diff --git a/internal/dockerlab/integration_test.go b/internal/dockerlab/integration_test.go index ba4692d..57ab6f6 100644 --- a/internal/dockerlab/integration_test.go +++ b/internal/dockerlab/integration_test.go @@ -5,6 +5,7 @@ import ( "os" "strings" "testing" + "time" "github.com/aleksandergregersen/opsquest/internal/game" "github.com/aleksandergregersen/opsquest/internal/mission" @@ -137,3 +138,130 @@ func createRealDockerEnvironment(t *testing.T, factory *Factory, item mission.Mi }) return created } + +func TestIntegrationRealDockerContainerTriage(t *testing.T) { + if os.Getenv("OPSQUEST_DOCKER_TEST") != "1" { + t.Skip("set OPSQUEST_DOCKER_TEST=1 to run the disposable Docker integration test") + } + catalog, err := mission.LoadCatalog() + if err != nil { + t.Fatal(err) + } + factory := NewFactory(game.SandboxFactory{}) + run := func(t *testing.T, created game.Environment, line string) string { + t.Helper() + result, err := created.Execute(context.Background(), line) + if err != nil { + t.Fatalf("Execute(%q) error = %v", line, err) + } + return result.Output + } + assertOutcomes := func(t *testing.T, created game.Environment, item mission.Mission, lastOutput string) { + t.Helper() + for _, outcome := range item.Validation.All { + var satisfied bool + var err error + switch outcome.Type { + case mission.ConditionOutputEquals: + satisfied = strings.TrimSpace(lastOutput) == strings.TrimSpace(outcome.Value) + case mission.ConditionOutputContains: + satisfied = strings.Contains(lastOutput, outcome.Value) + default: + satisfied, err = created.Observe(context.Background(), outcome) + } + if err != nil || !satisfied { + t.Fatalf("outcome %#v = %v, %v; last output %q", outcome, satisfied, err, lastOutput) + } + } + } + + t.Run("health, exit codes, and rm", func(t *testing.T) { + item, _ := catalog.Find("docker-postmortem-triage") + created := createRealDockerEnvironment(t, factory, item) + listed := run(t, created, "docker ps") + if !strings.Contains(listed, "(healthy)") || !strings.Contains(listed, "(unhealthy)") { + t.Fatalf("health did not settle before play:\n%s", listed) + } + unhealthy := run(t, created, "docker ps --filter health=unhealthy") + if !strings.Contains(unhealthy, "api-canary") || strings.Contains(unhealthy, "api \n") { + t.Fatalf("health filter:\n%s", unhealthy) + } + if _, err := created.Execute(context.Background(), "docker rm api-canary"); err == nil || !strings.Contains(err.Error(), "stop it first") { + t.Fatalf("rm running error = %v", err) + } + run(t, created, "docker stop api-canary") + run(t, created, "docker rm sync-orders") + last := run(t, created, "docker rm sync-users") + if _, err := created.Execute(context.Background(), "docker logs sync-users"); err == nil || !strings.Contains(err.Error(), "has been removed") { + t.Fatalf("logs after rm error = %v", err) + } + assertOutcomes(t, created, item, last) + }) + + t.Run("crash loop", func(t *testing.T) { + item, _ := catalog.Find("docker-crash-loop") + created := createRealDockerEnvironment(t, factory, item) + inspected := run(t, created, "docker inspect payments") + if !strings.Contains(inspected, `"Name": "on-failure"`) || strings.Contains(inspected, `"RestartCount": 0`) || strings.Contains(inspected, `"RestartCount": 1,`) { + t.Fatalf("crash-loop inspection:\n%s", inspected) + } + run(t, created, "docker stop payments") + time.Sleep(2 * time.Second) + stopped := mission.Condition{Type: mission.ConditionDockerContainerStopped, Container: "payments"} + if satisfied, err := created.Observe(context.Background(), stopped); err != nil || !satisfied { + t.Fatalf("manually stopped crash loop restarted: %v, %v", satisfied, err) + } + logs := run(t, created, "docker logs payments") + if strings.Count(logs, "FATAL cannot read") < 2 { + t.Fatalf("crash-loop logs = %q", logs) + } + assertOutcomes(t, created, item, logs) + }) + + t.Run("tail", func(t *testing.T) { + item, _ := catalog.Find("docker-tail-end") + created := createRealDockerEnvironment(t, factory, item) + assertOutcomes(t, created, item, run(t, created, "docker logs --tail 3 ledger")) + }) +} + +func TestIntegrationRealDockerOrphanSweep(t *testing.T) { + if os.Getenv("OPSQUEST_DOCKER_TEST") != "1" { + t.Skip("set OPSQUEST_DOCKER_TEST=1 to run the disposable Docker integration test") + } + catalog, err := mission.LoadCatalog() + if err != nil { + t.Fatal(err) + } + item, _ := catalog.Find("docker-quiet-worker") + deadPID := 0 + for candidate := 4_000_000; candidate > 3_000_000; candidate-- { + if !processAlive(candidate) { + deadPID = candidate + break + } + } + if deadPID == 0 { + t.Skip("no unused PID found to simulate an exited owner") + } + // Simulate an OpsQuest process that exited without closing its attempt. + crashed := NewFactory(game.SandboxFactory{}) + crashed.owner.pid = deadPID + abandoned := createRealDockerEnvironment(t, crashed, item).(*environment) + + janitor := NewFactory(game.SandboxFactory{}) + live := createRealDockerEnvironment(t, janitor, item).(*environment) + for _, tracked := range abandoned.snapshotContainers() { + if _, exists, err := abandoned.inspectReferenceUnchecked(context.Background(), tracked.id); err != nil || exists { + t.Fatalf("setup did not sweep abandoned fixture %s: exists = %v, error = %v", tracked.alias, exists, err) + } + } + if count, err := janitor.OrphanedResources(context.Background()); err != nil || count != 0 { + t.Fatalf("OrphanedResources() after sweep = %d, %v", count, err) + } + for _, tracked := range live.snapshotContainers() { + if _, exists, err := live.inspectReferenceUnchecked(context.Background(), tracked.id); err != nil || !exists { + t.Fatalf("sweep removed live fixture %s: exists = %v, error = %v", tracked.alias, exists, err) + } + } +} diff --git a/internal/dockerlab/janitor.go b/internal/dockerlab/janitor.go new file mode 100644 index 0000000..b4be904 --- /dev/null +++ b/internal/dockerlab/janitor.go @@ -0,0 +1,120 @@ +package dockerlab + +import ( + "context" + "errors" + "fmt" + "regexp" + "strconv" + "strings" + "time" + + "github.com/aleksandergregersen/opsquest/internal/mission" +) + +var ( + sessionIDPattern = regexp.MustCompile(`^[a-f0-9]{24}$`) + generatedNamePattern = regexp.MustCompile(`^/?opsquest-([a-f0-9]{24})-c[0-9]{2}$`) +) + +// orphanMinimumAge is when a fixture without a verifiable owner process is +// considered abandoned. Every fixture process exits by then, so removing it +// cannot interrupt a playable attempt. +const orphanMinimumAge = fixtureLifetime + +// OrphanedResources counts OpsQuest fixtures left behind by processes that +// exited without cleanup. It never modifies Docker state. +func (f *Factory) OrphanedResources(ctx context.Context) (int, error) { + orphans, err := f.findOrphans(ctx) + return len(orphans), err +} + +// RemoveOrphanedResources removes OpsQuest fixtures left behind by processes +// that exited without cleanup and reports how many were removed. +func (f *Factory) RemoveOrphanedResources(ctx context.Context) (int, error) { + return f.removeOrphans(ctx) +} + +func (f *Factory) removeOrphans(ctx context.Context) (int, error) { + orphans, err := f.findOrphans(ctx) + if err != nil { + return 0, err + } + removed := 0 + var removeErrors []error + for _, id := range orphans { + result, err := runDocker(ctx, f.runner, "container", "rm", "--force", id) + if err != nil && !isMissingContainer(result) { + removeErrors = append(removeErrors, fmt.Errorf("remove orphaned Docker fixture %s: %w", id[:12], err)) + continue + } + if err == nil { + removed++ + } + } + return removed, errors.Join(removeErrors...) +} + +// findOrphans lists managed containers and keeps only exact IDs whose +// complete ownership labels and generated name prove OpsQuest created them +// and whose owner is provably gone. +func (f *Factory) findOrphans(ctx context.Context) ([]string, error) { + if f.lookupErr != nil || f.runner == nil { + return nil, fmt.Errorf("docker executable not found in PATH") + } + if ctx == nil { + ctx = context.Background() + } + result, err := runDocker(ctx, f.runner, + "container", "ls", "--all", "--quiet", "--no-trunc", + "--filter", "label="+managedLabel+"=true", + ) + if err != nil { + return nil, err + } + var orphans []string + for _, id := range strings.Fields(result.stdout) { + if !containerIDPattern.MatchString(id) { + return nil, fmt.Errorf("Docker container listing returned an invalid container ID") + } + inspection, exists, err := inspectReference(ctx, f.runner, id) + if err != nil { + return nil, err + } + if !exists { + continue + } + if inspection.ID != id { + return nil, fmt.Errorf("Docker inspection returned an unexpected container ID") + } + if f.isOrphan(inspection) { + orphans = append(orphans, id) + } + } + return orphans, nil +} + +func (f *Factory) isOrphan(inspection containerInspection) bool { + labels := inspection.Config.Labels + session := labels[sessionLabel] + if labels[managedLabel] != "true" || labels[schemaLabel] != "1" || !sessionIDPattern.MatchString(session) || + !mission.ValidDockerLogicalName(labels[missionLabel]) || !mission.ValidDockerLogicalName(labels[aliasLabel]) { + return false + } + match := generatedNamePattern.FindStringSubmatch(inspection.Name) + if match == nil || match[1] != session { + return false + } + // An owner on this host can be checked directly. Owners on another host, + // and fixtures created before owner labels existed, fall back to age. + if host := labels[ownerHostLabel]; host != "" && host == f.owner.host && f.owner.alive != nil { + if pid, err := strconv.Atoi(labels[ownerPIDLabel]); err == nil && pid > 0 && !f.owner.alive(pid) { + return true + } + } + created, err := time.Parse(time.RFC3339Nano, inspection.Created) + if err != nil || f.owner.now == nil { + return false + } + return f.owner.now().Sub(created) > orphanMinimumAge +} diff --git a/internal/dockerlab/janitor_test.go b/internal/dockerlab/janitor_test.go new file mode 100644 index 0000000..cdd0f10 --- /dev/null +++ b/internal/dockerlab/janitor_test.go @@ -0,0 +1,197 @@ +package dockerlab + +import ( + "context" + "os" + "reflect" + "strings" + "testing" + "time" + + "github.com/aleksandergregersen/opsquest/internal/game" +) + +func orphanLabels(session, pid, host string) map[string]string { + labels := map[string]string{ + managedLabel: "true", + schemaLabel: "1", + sessionLabel: session, + missionLabel: "docker-container-census", + aliasLabel: "api", + } + if pid != "" { + labels[ownerPIDLabel] = pid + } + if host != "" { + labels[ownerHostLabel] = host + } + return labels +} + +func seedOrphanCandidates(commandRunner *fakeDockerRunner) (orphans, kept []string) { + session := func(char string) string { return strings.Repeat(char, 24) } + id := func(char string) string { return strings.Repeat(char, 64) } + old := testNow.Add(-orphanMinimumAge - time.Minute) + young := testNow.Add(-time.Hour) + candidates := []struct { + id string + container *fakeContainer + orphan bool + }{ + // The owner process on this host has exited. + {id: id("1"), orphan: true, container: &fakeContainer{labels: orphanLabels(session("1"), "999", testOwnerHost), name: "opsquest-" + session("1") + "-c01", created: young, running: true}}, + // The owner process on this host is alive. + {id: id("2"), container: &fakeContainer{labels: orphanLabels(session("2"), "4242", testOwnerHost), name: "opsquest-" + session("2") + "-c01", created: young, running: true}}, + // Another host cannot be probed; a young fixture may be live. + {id: id("3"), container: &fakeContainer{labels: orphanLabels(session("3"), "999", "other-host"), name: "opsquest-" + session("3") + "-c01", created: young}}, + // Another host's fixture past its lifetime is abandoned. + {id: id("4"), orphan: true, container: &fakeContainer{labels: orphanLabels(session("4"), "999", "other-host"), name: "opsquest-" + session("4") + "-c02", created: old}}, + // Fixtures from before owner labels fall back to age. + {id: id("5"), orphan: true, container: &fakeContainer{labels: orphanLabels(session("5"), "", ""), name: "opsquest-" + session("5") + "-c01", created: old}}, + {id: id("6"), container: &fakeContainer{labels: orphanLabels(session("6"), "", ""), name: "opsquest-" + session("6") + "-c01", created: young}}, + // Ownership proofs that fail keep the container regardless of age. + {id: id("7"), container: &fakeContainer{labels: orphanLabels(session("7"), "999", testOwnerHost), name: "user-database", created: old}}, + {id: id("8"), container: &fakeContainer{labels: orphanLabels(session("8"), "999", testOwnerHost), name: "opsquest-" + session("9") + "-c01", created: old}}, + {id: id("9"), container: &fakeContainer{labels: func() map[string]string { + labels := orphanLabels(session("9"), "999", testOwnerHost) + labels[schemaLabel] = "2" + return labels + }(), name: "opsquest-" + session("9") + "-c01", created: old}}, + {id: id("a"), container: &fakeContainer{labels: func() map[string]string { + labels := orphanLabels(session("a"), "999", testOwnerHost) + delete(labels, aliasLabel) + return labels + }(), name: "opsquest-" + session("a") + "-c01", created: old}}, + {id: id("d"), container: &fakeContainer{labels: orphanLabels("not-a-session", "999", testOwnerHost), name: "opsquest-not-a-session-c01", created: old}}, + // An unparsable owner PID is not proof of death; it falls back to age. + {id: id("e"), container: &fakeContainer{labels: orphanLabels(session("e"), "pid", testOwnerHost), name: "opsquest-" + session("e") + "-c01", created: young}}, + } + for _, candidate := range candidates { + commandRunner.seedContainer(candidate.id, candidate.container) + if candidate.orphan { + orphans = append(orphans, candidate.id) + } else { + kept = append(kept, candidate.id) + } + } + return orphans, kept +} + +func TestJanitorFindsOnlyProvablyAbandonedFixtures(t *testing.T) { + commandRunner := newFakeDockerRunner() + orphans, kept := seedOrphanCandidates(commandRunner) + factory := testFactory(commandRunner) + var _ game.ResourceJanitor = factory + + count, err := factory.OrphanedResources(context.Background()) + if err != nil || count != len(orphans) { + t.Fatalf("OrphanedResources() = %d, %v; want %d", count, err, len(orphans)) + } + for _, call := range commandRunner.snapshotCalls() { + if hasPrefix(call, "container", "rm") { + t.Fatalf("read-only orphan check removed a container: %q", call) + } + } + + commandRunner.resetCalls() + removed, err := factory.RemoveOrphanedResources(context.Background()) + if err != nil || removed != len(orphans) { + t.Fatalf("RemoveOrphanedResources() = %d, %v; want %d", removed, err, len(orphans)) + } + var removals [][]string + for _, call := range commandRunner.snapshotCalls() { + if hasPrefix(call, "container", "rm") { + removals = append(removals, call) + } + } + var wantRemovals [][]string + for _, id := range orphans { + wantRemovals = append(wantRemovals, []string{"container", "rm", "--force", id}) + } + if !reflect.DeepEqual(removals, wantRemovals) { + t.Fatalf("removals = %q, want %q", removals, wantRemovals) + } + for _, id := range orphans { + if commandRunner.hasContainer(id) { + t.Errorf("orphan %s survived cleanup", id[:12]) + } + } + for _, id := range kept { + if !commandRunner.hasContainer(id) { + t.Errorf("non-orphan %s was removed", id[:12]) + } + } + if count, err := factory.OrphanedResources(context.Background()); err != nil || count != 0 { + t.Fatalf("OrphanedResources() after cleanup = %d, %v", count, err) + } +} + +func TestCreateSweepsOrphansButNeverLiveAttempts(t *testing.T) { + commandRunner := newFakeDockerRunner() + orphans, kept := seedOrphanCandidates(commandRunner) + first := createTestEnvironment(t, commandRunner) + for _, id := range orphans { + if commandRunner.hasContainer(id) { + t.Errorf("Create() left orphan %s", id[:12]) + } + } + for _, id := range kept { + if !commandRunner.hasContainer(id) { + t.Errorf("Create() removed non-orphan %s", id[:12]) + } + } + + factory := testFactory(commandRunner) + factory.newSession = func() (string, error) { return strings.Repeat("f", 24), nil } + second, err := factory.Create(context.Background(), testDockerMission()) + if err != nil { + t.Fatal(err) + } + defer second.Close() + for _, tracked := range first.snapshotContainers() { + if !commandRunner.hasContainer(tracked.id) { + t.Fatalf("a second attempt swept live fixture %s", tracked.alias) + } + } +} + +func TestSweepFailureDoesNotBlockSetup(t *testing.T) { + commandRunner := newFakeDockerRunner() + commandRunner.failManagedList = true + environment := createTestEnvironment(t, commandRunner) + if len(environment.snapshotContainers()) != 2 { + t.Fatalf("setup after failed sweep created %d fixtures", len(environment.snapshotContainers())) + } + if _, err := testFactory(commandRunner).OrphanedResources(context.Background()); err == nil || !strings.Contains(err.Error(), "list failed") { + t.Fatalf("OrphanedResources() error = %v", err) + } +} + +func TestJanitorReportsMissingDocker(t *testing.T) { + factory := newFactory(game.SandboxFactory{}, nil, execNotFoundError{}, nil) + if _, err := factory.OrphanedResources(context.Background()); err == nil || !strings.Contains(err.Error(), "not found") { + t.Fatalf("OrphanedResources() error = %v", err) + } + if _, err := factory.RemoveOrphanedResources(context.Background()); err == nil || !strings.Contains(err.Error(), "not found") { + t.Fatalf("RemoveOrphanedResources() error = %v", err) + } +} + +func TestJanitorRejectsMalformedListing(t *testing.T) { + commandRunner := newFakeDockerRunner() + commandRunner.seedContainer("not-a-container-id", &fakeContainer{labels: orphanLabels(strings.Repeat("1", 24), "999", testOwnerHost)}) + if _, err := testFactory(commandRunner).RemoveOrphanedResources(context.Background()); err == nil || !strings.Contains(err.Error(), "invalid container ID") { + t.Fatalf("RemoveOrphanedResources() error = %v", err) + } +} + +func TestProcessAlive(t *testing.T) { + if !processAlive(os.Getpid()) { + t.Fatal("current process reported dead") + } + for _, pid := range []int{0, -1} { + if processAlive(pid) { + t.Errorf("processAlive(%d) = true", pid) + } + } +} diff --git a/internal/dockerlab/parser.go b/internal/dockerlab/parser.go index e839064..3610e2d 100644 --- a/internal/dockerlab/parser.go +++ b/internal/dockerlab/parser.go @@ -1,27 +1,39 @@ package dockerlab import ( + "errors" "fmt" + "sort" + "strconv" "strings" "unicode" "github.com/aleksandergregersen/opsquest/internal/mission" ) -const maxDockerCommandLineBytes = 64 * 1024 +const ( + maxDockerCommandLineBytes = 64 * 1024 + maxLogTailLines = 10000 + maxListFilters = 4 +) const dockerHelp = `Docker lab commands: docker ps [-a|--all] List mission containers - docker container ls [-a|--all] List mission containers + docker ps --filter KEY=VALUE Filter by status=created|running|restarting|exited + or health=starting|healthy|unhealthy|none + docker container ls [OPTIONS] List mission containers docker start ALIAS Start a mission container docker container start ALIAS Start a mission container docker restart ALIAS Restart a mission container docker container restart ALIAS Restart a mission container docker stop ALIAS Stop a mission container docker container stop ALIAS Stop a mission container + docker rm ALIAS Remove a stopped mission container + docker container rm ALIAS Remove a stopped mission container docker inspect ALIAS Inspect logical container state docker container inspect ALIAS Inspect logical container state docker logs ALIAS Read a mission container's logs + docker logs --tail N ALIAS Read only the last N log lines docker container logs ALIAS Read a mission container's logs help Show this help ` @@ -34,14 +46,30 @@ const ( actionStart actionRestart actionStop + actionRemove actionInspect actionLogs ) +// listFilter is one parsed docker ps filter. Filters are evaluated in Go over +// sanitized inspection data and are never forwarded to the Docker CLI. +type listFilter struct { + key string + value string +} + type dockerAction struct { - kind actionKind - alias string - all bool + kind actionKind + alias string + all bool + filters []listFilter + // tail is the number of trailing log lines to show; negative means all. + tail int +} + +var listFilterValues = map[string]map[string]bool{ + "status": {"created": true, "running": true, "restarting": true, "exited": true}, + "health": {"starting": true, "healthy": true, "unhealthy": true, "none": true}, } func parseAction(line string) (dockerAction, error) { @@ -65,25 +93,31 @@ func parseAction(line string) (dockerAction, error) { } fields = fields[1:] if len(fields) == 0 { - return dockerAction{}, fmt.Errorf("usage: docker ps [-a|--all] | docker start ALIAS | docker stop ALIAS | docker inspect ALIAS | docker logs ALIAS") + return dockerAction{}, fmt.Errorf("usage: docker ps [-a|--all] | docker start ALIAS | docker stop ALIAS | docker rm ALIAS | docker inspect ALIAS | docker logs ALIAS") } if fields[0] == "container" { fields = fields[1:] if len(fields) == 0 { - return dockerAction{}, fmt.Errorf("usage: docker container ls [-a|--all] | docker container start ALIAS | docker container stop ALIAS | docker container inspect ALIAS | docker container logs ALIAS") + return dockerAction{}, fmt.Errorf("usage: docker container ls [-a|--all] | docker container start ALIAS | docker container stop ALIAS | docker container rm ALIAS | docker container inspect ALIAS | docker container logs ALIAS") } } switch fields[0] { case "ps", "ls": - if len(fields) == 1 { - return dockerAction{kind: actionList}, nil + return parseList(fields[1:]) + case "logs": + return parseLogs(fields[1:]) + case "rm": + for _, field := range fields[1:] { + if field == "-f" || field == "--force" { + return dockerAction{}, fmt.Errorf("docker rm --force is outside this teaching subset; stop the container first, then remove it") + } } - if len(fields) == 2 && (fields[1] == "-a" || fields[1] == "--all") { - return dockerAction{kind: actionList, all: true}, nil + if len(fields) != 2 || !mission.ValidDockerLogicalName(fields[1]) { + return dockerAction{}, fmt.Errorf("usage: docker rm ALIAS") } - return dockerAction{}, fmt.Errorf("usage: docker ps [-a|--all]") - case "start", "restart", "stop", "inspect", "logs": + return dockerAction{kind: actionRemove, alias: fields[1]}, nil + case "start", "restart", "stop", "inspect": if len(fields) != 2 || !mission.ValidDockerLogicalName(fields[1]) { return dockerAction{}, fmt.Errorf("usage: docker %s ALIAS", fields[0]) } @@ -94,11 +128,116 @@ func parseAction(line string) (dockerAction, error) { kind = actionStop } else if fields[0] == "inspect" { kind = actionInspect - } else if fields[0] == "logs" { - kind = actionLogs } return dockerAction{kind: kind, alias: fields[1]}, nil default: return dockerAction{}, fmt.Errorf("docker %s is outside this mission's teaching subset; type help", fields[0]) } } + +func parseList(fields []string) (dockerAction, error) { + const usage = "usage: docker ps [-a|--all] [--filter status=VALUE|health=VALUE]" + action := dockerAction{kind: actionList} + for index := 0; index < len(fields); index++ { + field := fields[index] + var filter string + switch { + case field == "-a" || field == "--all": + if action.all { + return dockerAction{}, errors.New(usage) + } + action.all = true + continue + case field == "-f" || field == "--filter": + if index+1 >= len(fields) { + return dockerAction{}, fmt.Errorf("docker ps %s requires KEY=VALUE", field) + } + index++ + filter = fields[index] + case strings.HasPrefix(field, "--filter="): + filter = strings.TrimPrefix(field, "--filter=") + default: + return dockerAction{}, errors.New(usage) + } + parsed, err := parseListFilter(filter) + if err != nil { + return dockerAction{}, err + } + if len(action.filters) == maxListFilters { + return dockerAction{}, fmt.Errorf("docker ps accepts at most %d filters in this lab", maxListFilters) + } + action.filters = append(action.filters, parsed) + } + return action, nil +} + +func parseListFilter(value string) (listFilter, error) { + key, filterValue, found := strings.Cut(value, "=") + allowed, knownKey := listFilterValues[key] + if !found || !knownKey { + return listFilter{}, fmt.Errorf("docker ps filter %q is outside this teaching subset; use status=VALUE or health=VALUE", value) + } + if !allowed[filterValue] { + return listFilter{}, fmt.Errorf("docker ps filter %s accepts %s", key, strings.Join(sortedKeys(allowed), ", ")) + } + return listFilter{key: key, value: filterValue}, nil +} + +func parseLogs(fields []string) (dockerAction, error) { + const usage = "usage: docker logs [--tail N] ALIAS" + action := dockerAction{kind: actionLogs, tail: -1} + tailSet := false + for len(fields) > 1 { + var value string + switch field := fields[0]; { + case field == "--tail" || field == "-n": + if len(fields) < 3 { + return dockerAction{}, errors.New(usage) + } + value = fields[1] + fields = fields[2:] + case strings.HasPrefix(field, "--tail="): + value = strings.TrimPrefix(field, "--tail=") + fields = fields[1:] + default: + return dockerAction{}, errors.New(usage) + } + if tailSet { + return dockerAction{}, errors.New(usage) + } + tail, err := parseTail(value) + if err != nil { + return dockerAction{}, err + } + action.tail = tail + tailSet = true + } + if len(fields) != 1 || !mission.ValidDockerLogicalName(fields[0]) { + return dockerAction{}, errors.New(usage) + } + action.alias = fields[0] + return action, nil +} + +func parseTail(value string) (int, error) { + if value == "all" { + return -1, nil + } + if value == "" || strings.TrimLeft(value, "0123456789") != "" { + return 0, fmt.Errorf("docker logs --tail requires a whole number or all") + } + tail, err := strconv.Atoi(value) + if err != nil || tail > maxLogTailLines { + return 0, fmt.Errorf("docker logs --tail accepts at most %d lines in this lab", maxLogTailLines) + } + return tail, nil +} + +func sortedKeys(values map[string]bool) []string { + keys := make([]string, 0, len(values)) + for key := range values { + keys = append(keys, key) + } + sort.Strings(keys) + return keys +} diff --git a/internal/dockerlab/process_unix.go b/internal/dockerlab/process_unix.go new file mode 100644 index 0000000..dac8e31 --- /dev/null +++ b/internal/dockerlab/process_unix.go @@ -0,0 +1,18 @@ +//go:build !windows + +package dockerlab + +import ( + "errors" + "syscall" +) + +// processAlive reports whether pid names a live process. Signal 0 performs +// only the existence and permission check; EPERM still proves existence. +func processAlive(pid int) bool { + if pid <= 0 { + return false + } + err := syscall.Kill(pid, 0) + return err == nil || errors.Is(err, syscall.EPERM) +} diff --git a/internal/dockerlab/process_windows.go b/internal/dockerlab/process_windows.go new file mode 100644 index 0000000..424883a --- /dev/null +++ b/internal/dockerlab/process_windows.go @@ -0,0 +1,31 @@ +//go:build windows + +package dockerlab + +import ( + "errors" + "syscall" +) + +const ( + processQueryLimitedInformation = 0x1000 + processStillActive = 259 +) + +// processAlive reports whether pid names a live process. Uncertain results +// report alive so orphan cleanup stays conservative. +func processAlive(pid int) bool { + if pid <= 0 { + return false + } + handle, err := syscall.OpenProcess(processQueryLimitedInformation, false, uint32(pid)) + if err != nil { + return errors.Is(err, syscall.ERROR_ACCESS_DENIED) + } + defer syscall.CloseHandle(handle) + var exitCode uint32 + if err := syscall.GetExitCodeProcess(handle, &exitCode); err != nil { + return true + } + return exitCode == processStillActive +} diff --git a/internal/dockerlab/triage_test.go b/internal/dockerlab/triage_test.go new file mode 100644 index 0000000..5aa4063 --- /dev/null +++ b/internal/dockerlab/triage_test.go @@ -0,0 +1,331 @@ +package dockerlab + +import ( + "context" + "reflect" + "strings" + "testing" + "time" + + "github.com/aleksandergregersen/opsquest/internal/mission" +) + +func testTriageDockerMission() mission.Mission { + success := 0 + crash := 70 + return mission.Mission{ + ID: "docker-triage", + Number: 30, + Track: mission.TrackDocker, + Environment: mission.EnvironmentDocker, + Docker: &mission.DockerSetup{ + Images: []mission.DockerImageSpec{{Alias: "fixture", Reference: testImageReference}}, + Containers: []mission.DockerContainerSpec{ + {Name: "web-a", Image: "fixture", State: mission.DockerStateRunning, Health: mission.DockerHealthHealthy}, + {Name: "web-b", Image: "fixture", State: mission.DockerStateRunning, Health: mission.DockerHealthUnhealthy, Log: "INFO boot\nWARN pool 80%\nERROR pool exhausted"}, + {Name: "web-c", Image: "fixture", State: mission.DockerStateStopped, Health: mission.DockerHealthHealthy}, + {Name: "report", Image: "fixture", State: mission.DockerStateStopped, Log: "report complete", ExitCode: &success}, + {Name: "payments", Image: "fixture", State: mission.DockerStateRunning, Log: "FATAL key missing", ExitCode: &crash, Restart: mission.DockerRestartOnFailure}, + }, + }, + } +} + +func TestParserAcceptsTriageSubset(t *testing.T) { + tests := []struct { + line string + want dockerAction + }{ + {line: "docker rm report", want: dockerAction{kind: actionRemove, alias: "report"}}, + {line: "docker container rm report", want: dockerAction{kind: actionRemove, alias: "report"}}, + {line: "docker logs web-b", want: dockerAction{kind: actionLogs, alias: "web-b", tail: -1}}, + {line: "docker logs --tail 3 web-b", want: dockerAction{kind: actionLogs, alias: "web-b", tail: 3}}, + {line: "docker logs -n 0 web-b", want: dockerAction{kind: actionLogs, alias: "web-b", tail: 0}}, + {line: "docker container logs --tail=10000 web-b", want: dockerAction{kind: actionLogs, alias: "web-b", tail: 10000}}, + {line: "docker logs --tail all web-b", want: dockerAction{kind: actionLogs, alias: "web-b", tail: -1}}, + {line: "docker ps --filter status=exited", want: dockerAction{kind: actionList, filters: []listFilter{{key: "status", value: "exited"}}}}, + {line: "docker ps -a -f health=unhealthy", want: dockerAction{kind: actionList, all: true, filters: []listFilter{{key: "health", value: "unhealthy"}}}}, + {line: "docker container ls --filter=status=running --all", want: dockerAction{kind: actionList, all: true, filters: []listFilter{{key: "status", value: "running"}}}}, + {line: "docker ps -f status=exited -f status=created -f health=none", want: dockerAction{kind: actionList, filters: []listFilter{{key: "status", value: "exited"}, {key: "status", value: "created"}, {key: "health", value: "none"}}}}, + } + for _, test := range tests { + got, err := parseAction(test.line) + if err != nil { + t.Errorf("parseAction(%q) error = %v", test.line, err) + continue + } + if !reflect.DeepEqual(got, test.want) { + t.Errorf("parseAction(%q) = %#v, want %#v", test.line, got, test.want) + } + } +} + +func TestParserRejectsOutsideTriageSubsetBeforeTransport(t *testing.T) { + commandRunner := newFakeDockerRunner() + environment := createTestEnvironment(t, commandRunner) + commandRunner.resetCalls() + tests := []struct { + line string + wantErr string + }{ + {line: "docker rm -f api", wantErr: "--force is outside"}, + {line: "docker rm api --force", wantErr: "--force is outside"}, + {line: "docker rm api metrics", wantErr: "usage: docker rm ALIAS"}, + {line: "docker rm", wantErr: "usage: docker rm ALIAS"}, + {line: "docker rm $(docker ps -aq)", wantErr: "usage: docker rm ALIAS"}, + {line: "docker logs --tail api", wantErr: "usage: docker logs"}, + {line: "docker logs --tail -1 api", wantErr: "whole number"}, + {line: "docker logs --tail 3x api", wantErr: "whole number"}, + {line: "docker logs --tail 10001 api", wantErr: "at most 10000"}, + {line: "docker logs --tail 99999999999999999999 api", wantErr: "at most 10000"}, + {line: "docker logs --tail 3 --tail 4 api", wantErr: "usage: docker logs"}, + {line: "docker logs --since 1m api", wantErr: "usage: docker logs"}, + {line: "docker logs --follow api", wantErr: "usage: docker logs"}, + {line: "docker logs api --tail 3", wantErr: "usage: docker logs"}, + {line: "docker ps --filter name=api", wantErr: "outside this teaching subset"}, + {line: "docker ps --filter label=com.opsquest.managed=true", wantErr: "outside this teaching subset"}, + {line: "docker ps --filter status=paused", wantErr: "status accepts"}, + {line: "docker ps --filter status", wantErr: "outside this teaching subset"}, + {line: "docker ps --filter", wantErr: "requires KEY=VALUE"}, + {line: "docker ps -a -a", wantErr: "usage: docker ps"}, + {line: "docker ps --format '{{.ID}}'", wantErr: "usage: docker ps"}, + {line: "docker ps -f status=exited -f status=created -f status=running -f status=restarting -f health=none", wantErr: "at most 4 filters"}, + } + for _, test := range tests { + if _, err := environment.Execute(context.Background(), test.line); err == nil || !strings.Contains(err.Error(), test.wantErr) { + t.Errorf("Execute(%q) error = %v, want substring %q", test.line, err, test.wantErr) + } + } + if calls := commandRunner.snapshotCalls(); len(calls) != 0 { + t.Fatalf("rejected player input reached Docker transport: %q", calls) + } +} + +func TestRemoveOnlyStoppedContainersAndHideRemovedIdentity(t *testing.T) { + commandRunner := newFakeDockerRunner() + environment := createTestEnvironment(t, commandRunner) + api := environment.byAlias["api"] + metricsID := environment.byAlias["metrics"].id + commandRunner.resetCalls() + + if _, err := environment.Execute(context.Background(), "docker rm metrics"); err == nil || !strings.Contains(err.Error(), "stop it first") { + t.Fatalf("removing running container error = %v", err) + } + for _, call := range commandRunner.snapshotCalls() { + if hasPrefix(call, "container", "rm") { + t.Fatalf("running container removal reached Docker: %q", call) + } + } + if !commandRunner.hasContainer(metricsID) { + t.Fatal("running container was removed") + } + + absent := mission.Condition{Type: mission.ConditionDockerContainerAbsent, Container: "api"} + if got, err := environment.Observe(context.Background(), absent); err != nil || got { + t.Fatalf("absent before rm = %v, %v", got, err) + } + removed, err := environment.Execute(context.Background(), "docker container rm api") + if err != nil || removed.Output != "api\n" { + t.Fatalf("rm output = %q, error = %v", removed.Output, err) + } + if _, call := firstCall(t, commandRunner.snapshotCalls(), "container", "rm"); !reflect.DeepEqual(call, []string{"container", "rm", api.id}) { + t.Fatalf("rm transport = %q, want exact ID without --force", call) + } + if got, err := environment.Observe(context.Background(), absent); err != nil || !got { + t.Fatalf("absent after rm = %v, %v", got, err) + } + stopped := mission.Condition{Type: mission.ConditionDockerContainerStopped, Container: "api"} + if got, err := environment.Observe(context.Background(), stopped); err != nil || got { + t.Fatalf("removed container reported stopped = %v, %v", got, err) + } + count := 1 + if got, err := environment.Observe(context.Background(), mission.Condition{Type: mission.ConditionDockerContainerCountEqual, Count: &count}); err != nil || !got { + t.Fatalf("count after rm = %v, %v", got, err) + } + listed, err := environment.Execute(context.Background(), "docker ps -a") + if err != nil || strings.Contains(listed.Output, "api") || !strings.Contains(listed.Output, "metrics") { + t.Fatalf("list after rm = %q, error = %v", listed.Output, err) + } + + for _, line := range []string{"docker start api", "docker stop api", "docker restart api", "docker logs api", "docker logs --tail 1 api"} { + _, err := environment.Execute(context.Background(), line) + if err == nil || !strings.Contains(err.Error(), `container "api" has been removed`) { + t.Errorf("Execute(%q) error = %v", line, err) + continue + } + if strings.Contains(err.Error(), api.id) || strings.Contains(err.Error(), api.id[:12]) || strings.Contains(err.Error(), api.actualName) { + t.Errorf("Execute(%q) leaked engine identity: %v", line, err) + } + } + if _, err := environment.Execute(context.Background(), "docker rm api"); err == nil || !strings.Contains(err.Error(), "already been removed") { + t.Fatalf("second rm error = %v", err) + } + if _, err := environment.Execute(context.Background(), "docker inspect api"); err == nil || !strings.Contains(err.Error(), "no longer exists") { + t.Fatalf("inspect removed error = %v", err) + } + if err := environment.Close(); err != nil { + t.Fatalf("Close() after player removal = %v", err) + } + if commandRunner.containerCount() != 0 { + t.Fatalf("containers remaining after Close = %d", commandRunner.containerCount()) + } +} + +func TestEngineErrorsRedactExactIdentity(t *testing.T) { + tracked := &trackedContainer{id: strings.Repeat("c", 64), logicalID: "lab-01", alias: "api", actualName: "opsquest-" + strings.Repeat("a", 24) + "-c01"} + message := "Error: cannot start " + tracked.id + " (" + tracked.id[:12] + ") named " + tracked.actualName + got := redactIdentifiers(message, tracked) + if strings.Contains(got, tracked.id[:12]) || strings.Contains(got, tracked.actualName) || !strings.Contains(got, "lab-01") || !strings.Contains(got, "api") { + t.Fatalf("redactIdentifiers() = %q", got) + } +} + +func TestTriageFixturesUseFixedBehaviors(t *testing.T) { + commandRunner := newFakeDockerRunner() + environment := createEnvironmentForMission(t, commandRunner, testTriageDockerMission()) + calls := commandRunner.snapshotCalls() + creates := make(map[string][]string) + for _, call := range calls { + if hasPrefix(call, "container", "create") { + for index := range call { + if call[index] == "--label" && strings.HasPrefix(call[index+1], aliasLabel+"=") { + creates[strings.TrimPrefix(call[index+1], aliasLabel+"=")] = call + } + } + } + } + assertArgs := func(alias string, want ...string) { + t.Helper() + call := creates[alias] + for index := 0; index+len(want) <= len(call); index++ { + if reflect.DeepEqual(call[index:index+len(want)], want) { + return + } + } + t.Errorf("%s create args %q do not contain %q", alias, call, want) + } + refuteArg := func(alias, flag string) { + t.Helper() + for _, arg := range creates[alias] { + if arg == flag { + t.Errorf("%s create args unexpectedly contain %q: %q", alias, flag, creates[alias]) + } + } + } + assertArgs("web-a", "--health-cmd", "true", "--health-interval", "1s", "--health-timeout", "1s", "--health-retries", "1") + assertArgs("web-b", "--health-cmd", "false") + assertArgs("web-b", "--entrypoint", "/bin/sh", testImageReference, "-c", `printf '%s\n' "$1"; exec sleep "$2"`, "opsquest-service", "INFO boot\nWARN pool 80%\nERROR pool exhausted", "86400") + assertArgs("web-a", "--entrypoint", "/bin/sleep", testImageReference, "86400") + assertArgs("web-a", "--restart", "no") + assertArgs("payments", "--restart", "on-failure:50") + assertArgs("payments", "opsquest-diagnostic", "FATAL key missing", "70") + refuteArg("payments", "--health-cmd") + refuteArg("report", "--health-cmd") + for alias, call := range creates { + for _, forbidden := range []string{"--privileged", "--volume", "--mount", "-p", "--publish", "--network=host"} { + for _, arg := range call { + if arg == forbidden { + t.Errorf("%s create args contain unsafe %q", alias, forbidden) + } + } + } + } + for _, call := range calls { + if hasPrefix(call, "container", "wait", environment.byAlias["payments"].id) { + t.Fatal("setup waited for a crash-loop fixture to stop") + } + } + + listed, err := environment.Execute(context.Background(), "docker ps") + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"running (healthy)", "running (unhealthy)", "restarting", "web-a", "web-b", "payments"} { + if !strings.Contains(listed.Output, want) { + t.Errorf("ps output missing %q:\n%s", want, listed.Output) + } + } + if strings.Contains(listed.Output, "web-c") || strings.Contains(listed.Output, "report") { + t.Errorf("ps without --all listed stopped containers:\n%s", listed.Output) + } + + unhealthy, err := environment.Execute(context.Background(), "docker ps --filter health=unhealthy") + if err != nil || !strings.Contains(unhealthy.Output, "web-b") || strings.Contains(unhealthy.Output, "web-a") || strings.Contains(unhealthy.Output, "payments") { + t.Fatalf("health filter output = %q, error = %v", unhealthy.Output, err) + } + exited, err := environment.Execute(context.Background(), "docker ps --filter status=exited") + if err != nil || !strings.Contains(exited.Output, "report") || strings.Contains(exited.Output, "web-a") || strings.Contains(exited.Output, "web-c") { + t.Fatalf("status filter without --all output = %q, error = %v", exited.Output, err) + } + created, err := environment.Execute(context.Background(), "docker ps -f status=created -f status=exited") + if err != nil || !strings.Contains(created.Output, "report") || !strings.Contains(created.Output, "web-c") || strings.Contains(created.Output, "web-a") { + t.Fatalf("alternative status filters output = %q, error = %v", created.Output, err) + } + none, err := environment.Execute(context.Background(), "docker ps --filter status=running --filter health=none") + if err != nil || strings.Contains(none.Output, "web-") || !strings.Contains(none.Output, "CONTAINER ID") { + t.Fatalf("combined filters output = %q, error = %v", none.Output, err) + } + + inspected, err := environment.Execute(context.Background(), "docker inspect web-b") + if err != nil || !strings.Contains(inspected.Output, `"Health": {`) || !strings.Contains(inspected.Output, `"Status": "unhealthy"`) || !strings.Contains(inspected.Output, `"Name": "no"`) { + t.Fatalf("health inspection = %q, error = %v", inspected.Output, err) + } + inspected, err = environment.Execute(context.Background(), "docker inspect payments") + if err != nil || !strings.Contains(inspected.Output, `"RestartCount": 2`) || !strings.Contains(inspected.Output, `"Name": "on-failure"`) || !strings.Contains(inspected.Output, `"MaximumRetryCount": 50`) || !strings.Contains(inspected.Output, `"Restarting": true`) { + t.Fatalf("crash-loop inspection = %q, error = %v", inspected.Output, err) + } + if strings.Contains(inspected.Output, environment.byAlias["payments"].id) || strings.Contains(inspected.Output, "Labels") || strings.Contains(inspected.Output, "Created") { + t.Fatalf("inspection leaked engine metadata:\n%s", inspected.Output) + } + + logs, err := environment.Execute(context.Background(), "docker logs --tail 2 web-b") + if err != nil || logs.Output != "WARN pool 80%\nERROR pool exhausted\n" { + t.Fatalf("tail output = %q, error = %v", logs.Output, err) + } + if _, call := firstCall(t, commandRunner.snapshotCalls(), "container", "logs", "--tail"); !reflect.DeepEqual(call, []string{"container", "logs", "--tail", "2", environment.byAlias["web-b"].id}) { + t.Fatalf("tail transport = %q", call) + } + loopLogs, err := environment.Execute(context.Background(), "docker logs payments") + if err != nil || strings.Count(loopLogs.Output, "FATAL key missing") < 2 { + t.Fatalf("crash-loop logs = %q, error = %v", loopLogs.Output, err) + } + + stopped := mission.Condition{Type: mission.ConditionDockerContainerStopped, Container: "payments"} + if got, err := environment.Observe(context.Background(), stopped); err != nil || got { + t.Fatalf("restarting container reported stopped = %v, %v", got, err) + } + if _, err := environment.Execute(context.Background(), "docker stop payments"); err != nil { + t.Fatal(err) + } + if got, err := environment.Observe(context.Background(), stopped); err != nil || !got { + t.Fatalf("stopped crash loop = %v, %v", got, err) + } +} + +func TestFixtureReadinessTimesOutAndCleansUp(t *testing.T) { + commandRunner := newFakeDockerRunner() + item := testTriageDockerMission() + // A healthy probe on a fixture that never reports health cannot settle. + item.Docker.Containers = []mission.DockerContainerSpec{{Name: "web-a", Image: "fixture", State: mission.DockerStateRunning, Health: mission.DockerHealthHealthy}} + factory := testFactory(commandRunner) + factory.readyTimeout = 20 * time.Millisecond + commandRunner.stripHealth = true + if _, err := factory.Create(context.Background(), item); err == nil || !strings.Contains(err.Error(), "did not reach its teaching state") { + t.Fatalf("Create() error = %v", err) + } + if commandRunner.containerCount() != 0 { + t.Fatalf("containers remaining after failed readiness = %d", commandRunner.containerCount()) + } +} + +func TestPlainListingKeepsOriginalLayout(t *testing.T) { + environment := createTestEnvironment(t, newFakeDockerRunner()) + listed, err := environment.Execute(context.Background(), "docker ps") + if err != nil { + t.Fatal(err) + } + want := "CONTAINER ID IMAGE STATUS NAMES\nlab-02 fixture running metrics\n" + if listed.Output != want { + t.Fatalf("docker ps output:\n%q\nwant:\n%q", listed.Output, want) + } +} diff --git a/internal/game/environment.go b/internal/game/environment.go index 29a6df1..d008744 100644 --- a/internal/game/environment.go +++ b/internal/game/environment.go @@ -79,6 +79,15 @@ type AvailabilityChecker interface { Availability(context.Context, mission.Mission) Availability } +// ResourceJanitor is implemented by factories whose attempt resources can +// outlive an OpsQuest process that exits without cleanup. Orphans are only +// resources whose owning process is provably gone or whose lifetime has +// expired; live attempts are never reported or removed. +type ResourceJanitor interface { + OrphanedResources(context.Context) (int, error) + RemoveOrphanedResources(context.Context) (int, error) +} + // EnvironmentAvailability reports ready by default so simple test factories // and the simulated environment do not need boilerplate capability checks. func EnvironmentAvailability(ctx context.Context, factory Factory, item mission.Mission) Availability { diff --git a/internal/game/missions_test.go b/internal/game/missions_test.go index 182cfeb..fa3e635 100644 --- a/internal/game/missions_test.go +++ b/internal/game/missions_test.go @@ -3,6 +3,7 @@ package game import ( "context" "fmt" + "strconv" "strings" "testing" @@ -40,13 +41,23 @@ func TestEveryMissionHasAWorkingOutcome(t *testing.T) { "chmod 750 /deploy/app/deploy.sh", "kill 9001", }, - "docker-container-census": {"docker ps -a", "docker start api"}, - "docker-last-broadcast": {"docker logs checkout"}, - "docker-exit-code-detective": {"docker inspect seed", "docker inspect migrate"}, - "docker-quiet-worker": {"docker ps", "docker stop worker"}, - "docker-recovery-pair": {"docker ps -a", "docker start frontend", "docker start backend"}, - "docker-shift-handoff": {"docker ps -a", "docker start standby", "docker stop retiring"}, - "linux-runbook-runner": {"sh publish-health.sh"}, + "docker-container-census": {"docker ps -a", "docker start api"}, + "docker-last-broadcast": {"docker logs checkout"}, + "docker-exit-code-detective": {"docker inspect seed", "docker inspect migrate"}, + "docker-quiet-worker": {"docker ps", "docker stop worker"}, + "docker-recovery-pair": {"docker ps -a", "docker start frontend", "docker start backend"}, + "docker-shift-handoff": {"docker ps -a", "docker start standby", "docker stop retiring"}, + "docker-janitor-duty": {"docker ps --filter status=exited", "docker rm backup-0914", "docker rm backup-0915", "docker rm report-weekly"}, + "docker-tail-end": {"docker logs --tail 3 ledger"}, + "docker-running-isnt-healthy": {"docker ps", "docker stop web-b", "docker start web-c"}, + "docker-crash-loop": {"docker ps", "docker stop payments", "docker logs payments"}, + "docker-postmortem-triage": { + "docker ps -a", + "docker stop api-canary", + "docker rm sync-orders", + "docker rm sync-users", + }, + "linux-runbook-runner": {"sh publish-health.sh"}, "linux-vi-first-aid": { `printf 'SERVICE=checkout\nLOG_LEVEL=info\n' > release.env`, }, @@ -136,18 +147,24 @@ func canonicalMissionEnvironment(item mission.Mission) (Environment, error) { type missionDockerEnvironment struct { aliases []string running map[string]bool + removed map[string]bool logs map[string]string exitCodes map[string]int oneShot map[string]bool + crashLoop map[string]bool + health map[string]string count int } func newMissionDockerEnvironment(item mission.Mission) *missionDockerEnvironment { environment := &missionDockerEnvironment{ running: make(map[string]bool), + removed: make(map[string]bool), logs: make(map[string]string), exitCodes: make(map[string]int), oneShot: make(map[string]bool), + crashLoop: make(map[string]bool), + health: make(map[string]string), } if item.Docker == nil { return environment @@ -156,10 +173,16 @@ func newMissionDockerEnvironment(item mission.Mission) *missionDockerEnvironment environment.aliases = append(environment.aliases, container.Name) environment.running[container.Name] = container.State == mission.DockerStateRunning environment.logs[container.Name] = container.Log + environment.health[container.Name] = container.Health if container.ExitCode != nil { environment.exitCodes[container.Name] = *container.ExitCode environment.oneShot[container.Name] = true } + if container.Restart == mission.DockerRestartOnFailure { + environment.crashLoop[container.Name] = true + // Setup waits for a visible loop before play begins. + environment.logs[container.Name] = strings.Repeat(container.Log+"\n", 3) + } } environment.count = len(item.Docker.Containers) return environment @@ -167,6 +190,19 @@ func newMissionDockerEnvironment(item mission.Mission) *missionDockerEnvironment func (e *missionDockerEnvironment) PromptLabel() string { return "docker" } +func (e *missionDockerEnvironment) status(alias string) string { + switch { + case e.running[alias] && e.crashLoop[alias]: + return "restarting" + case e.running[alias] && e.health[alias] != "": + return "running (" + e.health[alias] + ")" + case e.running[alias]: + return "running" + default: + return "exited" + } +} + func (e *missionDockerEnvironment) Execute(_ context.Context, line string) (Execution, error) { fields := strings.Fields(line) if len(fields) < 2 || fields[0] != "docker" { @@ -178,24 +214,51 @@ func (e *missionDockerEnvironment) Execute(_ context.Context, line string) (Exec } result := Execution{PracticedCommands: []string{"docker"}, PipelineWidth: 1} if len(fields) >= 1 && (fields[0] == "ps" || fields[0] == "ls") { - all := len(fields) == 2 && (fields[1] == "-a" || fields[1] == "--all") - if len(fields) > 2 || len(fields) == 2 && !all { - return Execution{}, fmt.Errorf("unsupported fake Docker command %q", line) + all := false + filters := make(map[string]string) + for index := 1; index < len(fields); index++ { + switch fields[index] { + case "-a", "--all": + all = true + case "-f", "--filter": + if index+1 >= len(fields) { + return Execution{}, fmt.Errorf("unsupported fake Docker command %q", line) + } + index++ + key, value, _ := strings.Cut(fields[index], "=") + filters[key] = value + default: + return Execution{}, fmt.Errorf("unsupported fake Docker command %q", line) + } } var output strings.Builder for _, alias := range e.aliases { - if !all && !e.running[alias] { + if e.removed[alias] { + continue + } + if want, filtered := filters["status"]; filtered && !strings.HasPrefix(e.status(alias), want) { continue } - status := "stopped" - if e.running[alias] { - status = "running" + if want, filtered := filters["health"]; filtered && (!e.running[alias] || e.health[alias] != want) { + continue } - fmt.Fprintf(&output, "%s %s\n", alias, status) + if !all && filters["status"] == "" && !e.running[alias] { + continue + } + fmt.Fprintf(&output, "%s %s\n", alias, e.status(alias)) } result.Output = output.String() return result, nil } + tail := -1 + if len(fields) == 4 && fields[0] == "logs" && (fields[1] == "--tail" || fields[1] == "-n") { + parsed, err := strconv.Atoi(fields[2]) + if err != nil { + return Execution{}, fmt.Errorf("unsupported fake Docker command %q", line) + } + tail = parsed + fields = []string{"logs", fields[3]} + } if len(fields) != 2 { return Execution{}, fmt.Errorf("unsupported fake Docker command %q", line) } @@ -203,21 +266,31 @@ func (e *missionDockerEnvironment) Execute(_ context.Context, line string) (Exec if _, exists := e.running[alias]; !exists { return Execution{}, fmt.Errorf("unknown fake Docker alias %q", alias) } + if e.removed[alias] { + return Execution{}, fmt.Errorf("docker: container %q has been removed", alias) + } switch action { case "start", "restart": - e.running[alias] = !e.oneShot[alias] + e.running[alias] = !e.oneShot[alias] || e.crashLoop[alias] result.Output = alias + "\n" case "stop": e.running[alias] = false result.Output = alias + "\n" - case "logs": - result.Output = e.logs[alias] + "\n" - case "inspect": - status := "exited" + case "rm": if e.running[alias] { - status = "running" + return Execution{}, fmt.Errorf("docker: cannot remove running container %q; stop it first", alias) + } + e.removed[alias] = true + e.count-- + result.Output = alias + "\n" + case "logs": + lines := strings.Split(strings.TrimSuffix(e.logs[alias], "\n"), "\n") + if tail >= 0 && tail < len(lines) { + lines = lines[len(lines)-tail:] } - result.Output = fmt.Sprintf("{\n \"Name\": %q,\n \"State\": {\n \"Running\": %t,\n \"Status\": %q,\n \"ExitCode\": %d\n }\n}\n", alias, e.running[alias], status, e.exitCodes[alias]) + result.Output = strings.Join(lines, "\n") + "\n" + case "inspect": + result.Output = fmt.Sprintf("{\n \"Name\": %q,\n \"State\": {\n \"Running\": %t,\n \"Status\": %q,\n \"ExitCode\": %d\n }\n}\n", alias, e.running[alias], e.status(alias), e.exitCodes[alias]) default: return Execution{}, fmt.Errorf("unsupported fake Docker command %q", line) } @@ -227,10 +300,12 @@ func (e *missionDockerEnvironment) Execute(_ context.Context, line string) (Exec func (e *missionDockerEnvironment) Observe(_ context.Context, condition mission.Condition) (bool, error) { switch condition.Type { case mission.ConditionDockerContainerRunning: - return e.running[condition.Container], nil + return !e.removed[condition.Container] && e.running[condition.Container], nil case mission.ConditionDockerContainerStopped: _, exists := e.running[condition.Container] - return exists && !e.running[condition.Container], nil + return exists && !e.removed[condition.Container] && !e.running[condition.Container], nil + case mission.ConditionDockerContainerAbsent: + return e.removed[condition.Container], nil case mission.ConditionDockerContainerCountEqual: return condition.Count != nil && e.count == *condition.Count, nil default: @@ -302,6 +377,16 @@ func TestNewDockerMissionsAcceptAlternativesAndRejectIncompleteOutcomes(t *testi {id: "docker-quiet-worker", alternative: []string{"docker container stop worker"}, incomplete: []string{"docker stop metrics"}}, {id: "docker-recovery-pair", alternative: []string{"docker container start backend", "docker container start frontend"}, incomplete: []string{"docker start frontend"}}, {id: "docker-shift-handoff", alternative: []string{"docker container stop retiring", "docker container start standby"}, incomplete: []string{"docker start standby"}}, + {id: "docker-janitor-duty", alternative: []string{"docker container rm report-weekly", "docker container rm backup-0915", "docker container rm backup-0914"}, incomplete: []string{"docker rm backup-0914", "docker rm backup-0915"}}, + {id: "docker-janitor-duty", alternative: []string{"docker rm backup-0914", "docker rm backup-0915", "docker rm report-weekly", "docker ps -a"}, incomplete: []string{"docker stop worker", "docker rm worker", "docker rm backup-0914", "docker rm backup-0915", "docker rm report-weekly"}}, + {id: "docker-tail-end", alternative: []string{"docker container logs -n 3 ledger"}, incomplete: []string{"docker logs ledger"}}, + {id: "docker-tail-end", alternative: []string{"docker logs ledger", "docker logs --tail 3 ledger"}, incomplete: []string{"docker logs --tail 4 ledger"}}, + {id: "docker-running-isnt-healthy", alternative: []string{"docker ps --filter health=unhealthy", "docker container start web-c", "docker container stop web-b"}, incomplete: []string{"docker stop web-a", "docker start web-c"}}, + {id: "docker-running-isnt-healthy", alternative: []string{"docker stop web-b", "docker start web-c", "docker ps"}, incomplete: []string{"docker stop web-b", "docker rm web-b", "docker start web-c"}}, + {id: "docker-crash-loop", alternative: []string{"docker inspect payments", "docker container stop payments", "docker logs --tail 1 payments"}, incomplete: []string{"docker logs payments"}}, + {id: "docker-crash-loop", alternative: []string{"docker stop payments", "docker ps -a", "docker container logs payments"}, incomplete: []string{"docker stop payments", "docker logs payments", "docker rm payments"}}, + {id: "docker-postmortem-triage", alternative: []string{"docker ps --filter status=exited", "docker rm sync-users", "docker container rm sync-orders", "docker ps --filter health=unhealthy", "docker container stop api-canary"}, incomplete: []string{"docker stop api-canary", "docker rm sync-orders", "docker rm sync-users", "docker rm sync-invoices"}}, + {id: "docker-postmortem-triage", alternative: []string{"docker stop api-canary", "docker rm sync-orders", "docker rm sync-users", "docker logs sync-invoices"}, incomplete: []string{"docker stop api-canary", "docker rm api-canary", "docker rm sync-orders", "docker rm sync-users"}}, } for _, test := range tests { item, found := catalog.Find(test.id) diff --git a/internal/game/validator.go b/internal/game/validator.go index 391ef84..1b21f29 100644 --- a/internal/game/validator.go +++ b/internal/game/validator.go @@ -110,6 +110,8 @@ func describeCondition(condition mission.Condition) string { return fmt.Sprintf("Exactly %d mission containers exist", *condition.Count) } return "The required number of mission containers exist" + case mission.ConditionDockerContainerAbsent: + return fmt.Sprintf("Container %s has been removed", condition.Container) default: return fmt.Sprintf("Outcome condition %s is satisfied", condition.Type) } diff --git a/internal/mission/catalog.go b/internal/mission/catalog.go index 3ed8553..c44f151 100644 --- a/internal/mission/catalog.go +++ b/internal/mission/catalog.go @@ -212,7 +212,7 @@ func validateMission(item Mission) error { if err := validateCondition(condition, environment); err != nil { return fmt.Errorf("validation condition %d: %w", index+1, err) } - if (condition.Type == ConditionDockerContainerRunning || condition.Type == ConditionDockerContainerStopped) && !dockerSetupHasContainer(item.Docker, condition.Container) { + if condition.Container != "" && !dockerSetupHasContainer(item.Docker, condition.Container) { return fmt.Errorf("validation condition %d: unknown docker container %q", index+1, condition.Container) } } @@ -268,29 +268,64 @@ func ValidateDockerSetup(setup DockerSetup) error { default: return fmt.Errorf("docker container %q has unknown state %q", container.Name, container.State) } - hasDiagnostic := container.Log != "" || container.ExitCode != nil - if hasDiagnostic { - if container.Log == "" || container.ExitCode == nil { - return fmt.Errorf("docker container %q diagnostic fixture requires both log and exit_code", container.Name) - } - if len(container.Log) > maxDockerFixtureLogBytes { - return fmt.Errorf("docker container %q log exceeds the %d-byte limit", container.Name, maxDockerFixtureLogBytes) - } - if strings.ContainsRune(container.Log, 0) { - return fmt.Errorf("docker container %q log cannot contain NUL", container.Name) - } - if *container.ExitCode < 0 || *container.ExitCode > 255 { - return fmt.Errorf("docker container %q exit_code must be between 0 and 255", container.Name) - } - if container.State != DockerStateStopped { - return fmt.Errorf("docker container %q diagnostic fixture must use stopped state", container.Name) - } + if len(container.Log) > maxDockerFixtureLogBytes { + return fmt.Errorf("docker container %q log exceeds the %d-byte limit", container.Name, maxDockerFixtureLogBytes) + } + if strings.ContainsRune(container.Log, 0) { + return fmt.Errorf("docker container %q log cannot contain NUL", container.Name) + } + if err := validateDockerFixtureBehavior(container); err != nil { + return err } containers[container.Name] = true } return nil } +// validateDockerFixtureBehavior accepts only the fixed fixture behaviors the +// Docker adapter implements: a long-lived service with optional startup log +// and health probe, a one-shot diagnostic job, or a bounded crash loop. +func validateDockerFixtureBehavior(container DockerContainerSpec) error { + switch container.Health { + case "", DockerHealthHealthy, DockerHealthUnhealthy: + default: + return fmt.Errorf("docker container %q has unknown health %q", container.Name, container.Health) + } + switch container.Restart { + case "", DockerRestartOnFailure: + default: + return fmt.Errorf("docker container %q has unknown restart behavior %q", container.Name, container.Restart) + } + if container.ExitCode == nil { + if container.Restart != "" { + return fmt.Errorf("docker container %q restart behavior requires log and a non-zero exit_code", container.Name) + } + return nil + } + if container.Log == "" { + return fmt.Errorf("docker container %q exit_code requires a log", container.Name) + } + if *container.ExitCode < 0 || *container.ExitCode > 255 { + return fmt.Errorf("docker container %q exit_code must be between 0 and 255", container.Name) + } + if container.Health != "" { + return fmt.Errorf("docker container %q exiting fixture cannot declare health", container.Name) + } + if container.Restart == DockerRestartOnFailure { + if *container.ExitCode == 0 { + return fmt.Errorf("docker container %q restart behavior requires log and a non-zero exit_code", container.Name) + } + if container.State != DockerStateRunning { + return fmt.Errorf("docker container %q crash-loop fixture must use running state", container.Name) + } + return nil + } + if container.State != DockerStateStopped { + return fmt.Errorf("docker container %q diagnostic fixture must use stopped state", container.Name) + } + return nil +} + func dockerSetupHasContainer(setup *DockerSetup, name string) bool { if setup == nil { return false @@ -431,6 +466,7 @@ var allowedConditionFields = map[ConditionType]conditionFields{ ConditionDockerContainerRunning: conditionContainer, ConditionDockerContainerStopped: conditionContainer, ConditionDockerContainerCountEqual: conditionCount, + ConditionDockerContainerAbsent: conditionContainer, } var conditionFieldFlags = map[string]conditionFields{ @@ -526,7 +562,7 @@ func validateCondition(condition Condition, environment string) error { if !found || !variablePattern.MatchString(name) { return fmt.Errorf("value must be NAME=value") } - case ConditionDockerContainerRunning, ConditionDockerContainerStopped: + case ConditionDockerContainerRunning, ConditionDockerContainerStopped, ConditionDockerContainerAbsent: if environment != EnvironmentDocker { return fmt.Errorf("%s requires a docker environment", condition.Type) } diff --git a/internal/mission/catalog_test.go b/internal/mission/catalog_test.go index e4e539e..dc6615e 100644 --- a/internal/mission/catalog_test.go +++ b/internal/mission/catalog_test.go @@ -12,8 +12,8 @@ func TestEmbeddedCatalog(t *testing.T) { t.Fatalf("LoadCatalog() error = %v", err) } items := catalog.All() - if len(items) != 29 { - t.Fatalf("len(All()) = %d, want 29", len(items)) + if len(items) != 34 { + t.Fatalf("len(All()) = %d, want 34", len(items)) } for index, item := range items { if item.Number != index+1 { @@ -46,7 +46,7 @@ func TestLegacyMissionDefaultsAndCatalogTrackFiltering(t *testing.T) { linux := catalog.InTrack("") docker := catalog.InTrack(TrackDocker) - if len(linux) != 23 || len(docker) != 6 { + if len(linux) != 23 || len(docker) != 11 { t.Fatalf("track sizes = linux %d, docker %d", len(linux), len(docker)) } if docker[0].ID != "docker-container-census" || docker[0].Number != 20 { @@ -220,6 +220,44 @@ func TestExpandedDockerCurriculumMetadata(t *testing.T) { } } +func TestContainerTriageCurriculum(t *testing.T) { + catalog, err := LoadCatalog() + if err != nil { + t.Fatal(err) + } + wants := []struct { + id string + number int + difficulty string + hints int + }{ + {id: "docker-janitor-duty", number: 30, difficulty: DifficultyBeginner, hints: 3}, + {id: "docker-tail-end", number: 31, difficulty: DifficultyBeginner, hints: 3}, + {id: "docker-running-isnt-healthy", number: 32, difficulty: DifficultyIntermediate, hints: 3}, + {id: "docker-crash-loop", number: 33, difficulty: DifficultyIntermediate, hints: 3}, + {id: "docker-postmortem-triage", number: 34, difficulty: DifficultyAdvanced, hints: 5}, + } + for _, want := range wants { + item, found := catalog.Find(want.id) + if !found { + t.Errorf("mission %q missing", want.id) + continue + } + if item.Number != want.number || item.Campaign != "Container Triage" || item.EffectiveTrack() != TrackDocker || item.Difficulty != want.difficulty || len(item.Hints) != want.hints { + t.Errorf("mission %q curriculum metadata = number %d, campaign %q, track %q, difficulty %q, hints %d", item.ID, item.Number, item.Campaign, item.EffectiveTrack(), item.Difficulty, len(item.Hints)) + } + } + worlds := catalog.Worlds(TrackDocker) + if len(worlds) != 2 || worlds[1].Name != "Container Triage" || len(worlds[1].Missions) != 5 { + t.Fatalf("Docker worlds = %d, want Foundations plus Container Triage", len(worlds)) + } + crashLoop, _ := catalog.Find("docker-crash-loop") + payments := crashLoop.Docker.Containers[0] + if payments.Restart != DockerRestartOnFailure || payments.State != DockerStateRunning || payments.ExitCode == nil || *payments.ExitCode == 0 { + t.Fatalf("crash-loop fixture = %#v", payments) + } +} + func TestMissionValidationRejectsInvalidDockerDefinitions(t *testing.T) { tests := []struct { name string @@ -276,19 +314,85 @@ func TestMissionValidationRejectsInvalidDockerDefinitions(t *testing.T) { wantErr: "unknown state", }, { - name: "diagnostic log without exit code", + name: "diagnostic exit code without log", mutate: func(item *Mission) { - item.Docker.Containers[0].Log = "diagnostic" + exitCode := 1 + item.Docker.Containers[0].ExitCode = &exitCode }, - wantErr: "requires both log and exit_code", + wantErr: "exit_code requires a log", }, { - name: "diagnostic exit code without log", + name: "service log too large", + mutate: func(item *Mission) { + item.Docker.Containers[1].Log = strings.Repeat("x", maxDockerFixtureLogBytes+1) + }, + wantErr: "log exceeds", + }, + { + name: "unknown health", + mutate: func(item *Mission) { + item.Docker.Containers[1].Health = "starting" + }, + wantErr: "unknown health", + }, + { + name: "health on exiting fixture", + mutate: func(item *Mission) { + exitCode := 0 + item.Docker.Containers[0].Log = "complete" + item.Docker.Containers[0].ExitCode = &exitCode + item.Docker.Containers[0].Health = DockerHealthHealthy + }, + wantErr: "cannot declare health", + }, + { + name: "unknown restart behavior", + mutate: func(item *Mission) { + item.Docker.Containers[1].Restart = "always" + }, + wantErr: "unknown restart behavior", + }, + { + name: "restart without crash", + mutate: func(item *Mission) { + item.Docker.Containers[1].Restart = DockerRestartOnFailure + }, + wantErr: "requires log and a non-zero exit_code", + }, + { + name: "restart with successful exit", + mutate: func(item *Mission) { + exitCode := 0 + item.Docker.Containers[1].Log = "complete" + item.Docker.Containers[1].ExitCode = &exitCode + item.Docker.Containers[1].Restart = DockerRestartOnFailure + }, + wantErr: "requires log and a non-zero exit_code", + }, + { + name: "crash loop declared stopped", mutate: func(item *Mission) { exitCode := 1 + item.Docker.Containers[0].Log = "FATAL" item.Docker.Containers[0].ExitCode = &exitCode + item.Docker.Containers[0].Restart = DockerRestartOnFailure + }, + wantErr: "crash-loop fixture must use running state", + }, + { + name: "unknown absent validation container", + mutate: func(item *Mission) { + item.Validation.All[0] = Condition{Type: ConditionDockerContainerAbsent, Container: "missing"} + }, + wantErr: "unknown docker container", + }, + { + name: "absent condition with count", + mutate: func(item *Mission) { + count := 1 + item.Validation.All[0] = Condition{Type: ConditionDockerContainerAbsent, Container: "api", Count: &count} }, - wantErr: "requires both log and exit_code", + wantErr: "does not support count", }, { name: "diagnostic exit code outside range", @@ -389,15 +493,35 @@ func TestMissionValidationRejectsInvalidDockerDefinitions(t *testing.T) { } } +func TestMissionValidationAcceptsDockerFixtureBehaviors(t *testing.T) { + catalog, err := LoadCatalog() + if err != nil { + t.Fatal(err) + } + item, _ := catalog.Find("docker-container-census") + failure := 3 + item.Docker.Containers = []DockerContainerSpec{ + {Name: "api", Image: "fixture", State: DockerStateStopped, Log: "INFO ready", Health: DockerHealthHealthy}, + {Name: "metrics", Image: "fixture", State: DockerStateRunning, Health: DockerHealthUnhealthy}, + {Name: "payments", Image: "fixture", State: DockerStateRunning, Log: "FATAL key missing", ExitCode: &failure, Restart: DockerRestartOnFailure}, + } + item.Validation.All = append(item.Validation.All, Condition{Type: ConditionDockerContainerAbsent, Container: "payments"}) + if err := validateMission(item); err != nil { + t.Fatalf("validateMission() error = %v", err) + } +} + func TestMissionValidationRejectsDockerConditionOnSimulatedMission(t *testing.T) { catalog, err := LoadCatalog() if err != nil { t.Fatal(err) } item, _ := catalog.Find("linux-orientation") - item.Validation.All = []Condition{{Type: "docker_container_running", Container: "api"}} - if err := validateMission(item); err == nil || !strings.Contains(err.Error(), "requires a docker environment") { - t.Fatalf("validateMission() error = %v", err) + for _, conditionType := range []ConditionType{ConditionDockerContainerRunning, ConditionDockerContainerAbsent} { + item.Validation.All = []Condition{{Type: conditionType, Container: "api"}} + if err := validateMission(item); err == nil || !strings.Contains(err.Error(), "requires a docker environment") { + t.Fatalf("validateMission(%s) error = %v", conditionType, err) + } } } @@ -624,6 +748,14 @@ func TestCatalogTrackBoundariesAndAdjacency(t *testing.T) { if !found || last.Number != 29 { t.Fatalf("LastInTrack(linux) = %#v, %v", last, found) } + lastDocker, found := catalog.LastInTrack(TrackDocker) + if !found || lastDocker.ID != "docker-postmortem-triage" || lastDocker.Number != 34 { + t.Fatalf("LastInTrack(docker) = %#v, %v", lastDocker, found) + } + crossWorld, found := catalog.AdjacentInTrack("docker-shift-handoff", 1) + if !found || crossWorld.ID != "docker-janitor-duty" { + t.Fatalf("AdjacentInTrack(after Docker 25) = %#v, %v", crossWorld, found) + } next, found := catalog.AdjacentInTrack("linux-production-friday", 1) if !found || next.ID != "linux-runbook-runner" { t.Fatalf("AdjacentInTrack(after 19) = %#v, %v", next, found) diff --git a/internal/mission/data/30-janitor-duty.json b/internal/mission/data/30-janitor-duty.json new file mode 100644 index 0000000..84be45e --- /dev/null +++ b/internal/mission/data/30-janitor-duty.json @@ -0,0 +1,93 @@ +{ + "id": "docker-janitor-duty", + "number": 30, + "track": "docker", + "environment": "docker", + "title": "Janitor Duty", + "campaign": "Container Triage", + "difficulty": "beginner", + "story": "Last week's one-off backup and report jobs finished long ago, but their exited containers still clutter every listing. The api and worker services are healthy and must keep running.", + "objective": "Remove the three finished one-off job containers, keep api and worker running, and leave exactly two mission containers.", + "suggested_commands": [ + "docker" + ], + "hints": [ + "Finished jobs are exited containers; docker ps -a shows them beside the running services.", + "docker ps --filter status=exited lists only the exited containers.", + "docker rm removes one stopped container per command, for example docker rm backup-0914." + ], + "explanation": "Stopping a container keeps its filesystem, logs, and metadata. Removing it deletes that record. Clearing finished job containers keeps listings readable, and docker rm refuses running containers, so services stay safe unless you stop them first.", + "docker": { + "images": [ + { + "alias": "fixture", + "reference": "docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" + } + ], + "containers": [ + { + "name": "api", + "image": "fixture", + "state": "running" + }, + { + "name": "worker", + "image": "fixture", + "state": "running" + }, + { + "name": "backup-0914", + "image": "fixture", + "state": "stopped", + "log": "backup 2026-09-14 complete: 412 MB written", + "exit_code": 0 + }, + { + "name": "backup-0915", + "image": "fixture", + "state": "stopped", + "log": "backup 2026-09-15 complete: 415 MB written", + "exit_code": 0 + }, + { + "name": "report-weekly", + "image": "fixture", + "state": "stopped", + "log": "weekly report delivered to ops-leads", + "exit_code": 0 + } + ] + }, + "validation": { + "all": [ + { + "type": "docker_container_absent", + "container": "backup-0914" + }, + { + "type": "docker_container_absent", + "container": "backup-0915" + }, + { + "type": "docker_container_absent", + "container": "report-weekly" + }, + { + "type": "docker_container_running", + "container": "api" + }, + { + "type": "docker_container_running", + "container": "worker" + }, + { + "type": "docker_container_count_equals", + "count": 2 + } + ] + }, + "rewards": { + "xp": 75, + "hint_penalty": 10 + } +} diff --git a/internal/mission/data/31-tail-end.json b/internal/mission/data/31-tail-end.json new file mode 100644 index 0000000..0a05648 --- /dev/null +++ b/internal/mission/data/31-tail-end.json @@ -0,0 +1,49 @@ +{ + "id": "docker-tail-end", + "number": 31, + "track": "docker", + "environment": "docker", + "title": "Tail End", + "campaign": "Container Triage", + "difficulty": "beginner", + "story": "The nightly ledger reconciliation failed after dozens of routine progress lines. The incident channel only wants the lines that explain how it ended.", + "objective": "Display only the final three log lines of the ledger container.", + "suggested_commands": [ + "docker" + ], + "hints": [ + "The reason a job stopped is usually at the end of its log.", + "docker logs accepts an option that limits output to the most recent lines.", + "docker logs --tail 3 ledger prints only the last three lines." + ], + "explanation": "docker logs --tail N reads only the newest N lines, the same idea as tail -n on a file. With long logs, that brings the ending into view without scrolling through the whole history.", + "docker": { + "images": [ + { + "alias": "fixture", + "reference": "docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" + } + ], + "containers": [ + { + "name": "ledger", + "image": "fixture", + "state": "stopped", + "log": "INFO ledger reconciliation starting run=2026-09-30\nINFO batch 001 reconciled\nINFO batch 002 reconciled\nINFO batch 003 reconciled\nINFO batch 004 reconciled\nINFO batch 005 reconciled\nINFO batch 006 reconciled\nINFO batch 007 reconciled\nINFO batch 008 reconciled\nINFO batch 009 reconciled\nINFO batch 010 reconciled\nINFO batch 011 reconciled\nINFO batch 012 reconciled\nINFO batch 013 reconciled\nINFO batch 014 reconciled\nINFO batch 015 reconciled\nINFO batch 016 reconciled\nWARN batch 017 retried after lock timeout\nINFO batch 017 reconciled\nINFO batch 018 reconciled\nINFO batch 019 reconciled\nINFO batch 020 reconciled\nINFO batch 021 reconciled\nINFO batch 022 reconciled\nINFO batch 023 reconciled\nINFO batch 024 reconciled\nINFO batch 025 reconciled\nINFO batch 026 reconciled\nINFO batch 027 reconciled\nINFO batch 028 reconciled\nINFO batch 029 reconciled\nINFO batch 030 reconciled\nERROR batch 031 checksum mismatch account=4471\nERROR reconciliation aborted after 30 clean batches\nINFO ledger exiting with status 1", + "exit_code": 1 + } + ] + }, + "validation": { + "all": [ + { + "type": "output_equals", + "value": "ERROR batch 031 checksum mismatch account=4471\nERROR reconciliation aborted after 30 clean batches\nINFO ledger exiting with status 1" + } + ] + }, + "rewards": { + "xp": 80, + "hint_penalty": 10 + } +} diff --git a/internal/mission/data/32-running-isnt-healthy.json b/internal/mission/data/32-running-isnt-healthy.json new file mode 100644 index 0000000..a694adb --- /dev/null +++ b/internal/mission/data/32-running-isnt-healthy.json @@ -0,0 +1,75 @@ +{ + "id": "docker-running-isnt-healthy", + "number": 32, + "track": "docker", + "environment": "docker", + "title": "Running Isn't Healthy", + "campaign": "Container Triage", + "difficulty": "intermediate", + "story": "All the web replicas report as up, yet one in three requests fails. Each replica has a health probe, and one of them has been failing it quietly. A standby replica is ready to take its place.", + "objective": "Take the unhealthy web replica out of service without deleting it, bring the standby replica online, keep the healthy replica running, and leave exactly three mission containers.", + "suggested_commands": [ + "docker" + ], + "hints": [ + "Running means the process exists; the health probe says whether it can do its job. docker ps shows health beside each running container.", + "docker ps --filter health=unhealthy isolates the failing replica, and docker inspect shows State.Health.", + "Stop the unhealthy replica instead of removing it so its logs stay available, then start web-c." + ], + "explanation": "A health check is a probe Docker runs inside the container. A container can be running while its probe fails, so check health as well as state. Stopping the bad replica keeps its logs and metadata for debugging, and starting the standby restores capacity.", + "docker": { + "images": [ + { + "alias": "fixture", + "reference": "docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" + } + ], + "containers": [ + { + "name": "web-a", + "image": "fixture", + "state": "running", + "health": "healthy", + "log": "INFO web-a listening on :8080" + }, + { + "name": "web-b", + "image": "fixture", + "state": "running", + "health": "unhealthy", + "log": "INFO web-b listening on :8080\nERROR upstream pool exhausted; health probe failing" + }, + { + "name": "web-c", + "image": "fixture", + "state": "stopped", + "health": "healthy", + "log": "INFO web-c listening on :8080" + } + ] + }, + "validation": { + "all": [ + { + "type": "docker_container_stopped", + "container": "web-b" + }, + { + "type": "docker_container_running", + "container": "web-a" + }, + { + "type": "docker_container_running", + "container": "web-c" + }, + { + "type": "docker_container_count_equals", + "count": 3 + } + ] + }, + "rewards": { + "xp": 90, + "hint_penalty": 15 + } +} diff --git a/internal/mission/data/33-crash-loop.json b/internal/mission/data/33-crash-loop.json new file mode 100644 index 0000000..51d625d --- /dev/null +++ b/internal/mission/data/33-crash-loop.json @@ -0,0 +1,67 @@ +{ + "id": "docker-crash-loop", + "number": 33, + "track": "docker", + "environment": "docker", + "title": "Crash Loop", + "campaign": "Container Triage", + "difficulty": "intermediate", + "story": "The payments container keeps crashing, and its restart policy keeps launching it again. Every restart pages the on-call engineer. The api container is unaffected.", + "objective": "Stop the crash-looping payments container without removing it, keep api running, and finish by displaying payments' logs so the fault is on screen for the incident channel.", + "suggested_commands": [ + "docker" + ], + "hints": [ + "A restarting status and a climbing RestartCount in docker inspect mean a restart policy keeps relaunching a failing process.", + "Docker does not restart a container you stopped yourself, even when it has an on-failure restart policy.", + "Run docker stop payments, then docker logs payments." + ], + "explanation": "A restart policy relaunches a container whose process exits with an error. When the error is permanent, such as a missing secret, the policy produces a crash loop. docker stop ends the loop and keeps the container, and its logs show the same fatal line once per attempt.", + "docker": { + "images": [ + { + "alias": "fixture", + "reference": "docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" + } + ], + "containers": [ + { + "name": "payments", + "image": "fixture", + "state": "running", + "log": "FATAL cannot read /run/secrets/payments-key: no such file", + "exit_code": 78, + "restart": "on-failure" + }, + { + "name": "api", + "image": "fixture", + "state": "running" + } + ] + }, + "validation": { + "all": [ + { + "type": "docker_container_stopped", + "container": "payments" + }, + { + "type": "docker_container_running", + "container": "api" + }, + { + "type": "docker_container_count_equals", + "count": 2 + }, + { + "type": "output_contains", + "value": "FATAL cannot read /run/secrets/payments-key" + } + ] + }, + "rewards": { + "xp": 95, + "hint_penalty": 15 + } +} diff --git a/internal/mission/data/34-postmortem-triage.json b/internal/mission/data/34-postmortem-triage.json new file mode 100644 index 0000000..f796bf8 --- /dev/null +++ b/internal/mission/data/34-postmortem-triage.json @@ -0,0 +1,98 @@ +{ + "id": "docker-postmortem-triage", + "number": 34, + "track": "docker", + "environment": "docker", + "title": "Postmortem Triage", + "campaign": "Container Triage", + "difficulty": "advanced", + "story": "The morning after a rough release: a canary replica is failing its health probe, and last night's sync jobs have finished. Some succeeded and one did not. The postmortem needs the evidence, and the listings need cleaning up.", + "objective": "Take the unhealthy canary out of service but keep it for debugging, remove the sync jobs that succeeded, keep the failed sync job for the postmortem, keep api serving, and leave exactly three mission containers.", + "suggested_commands": [ + "docker" + ], + "hints": [ + "Look before changing anything. docker ps -a shows every service and job.", + "Health appears beside running containers; docker ps --filter health=unhealthy isolates the failing replica.", + "docker ps --filter status=exited lists the finished jobs, and docker inspect shows each job's State.ExitCode.", + "docker rm removes only stopped containers, and a removed container's logs are gone. Remove only the jobs that exited with code 0.", + "docker stop api-canary, docker rm sync-orders, and docker rm sync-users; leave sync-invoices in place." + ], + "explanation": "Triage separates evidence from clutter. Health shows which running replica is failing, and exit codes show which finished jobs succeeded. Stopping keeps a container's evidence and removing deletes it, so remove only what you will never need to investigate.", + "docker": { + "images": [ + { + "alias": "fixture", + "reference": "docker.io/library/busybox@sha256:73aaf090f3d85aa34ee199857f03fa3a95c8ede2ffd4cc2cdb5b94e566b11662" + } + ], + "containers": [ + { + "name": "api", + "image": "fixture", + "state": "running", + "health": "healthy" + }, + { + "name": "api-canary", + "image": "fixture", + "state": "running", + "health": "unhealthy", + "log": "INFO canary build 2026.09.30 started\nERROR GET /checkout returned HTTP 500" + }, + { + "name": "sync-orders", + "image": "fixture", + "state": "stopped", + "log": "sync-orders: 1204 rows synced", + "exit_code": 0 + }, + { + "name": "sync-users", + "image": "fixture", + "state": "stopped", + "log": "sync-users: 88 rows synced", + "exit_code": 0 + }, + { + "name": "sync-invoices", + "image": "fixture", + "state": "stopped", + "log": "ERROR invoice sync failed: upstream timeout after 30s", + "exit_code": 3 + } + ] + }, + "validation": { + "all": [ + { + "type": "docker_container_running", + "container": "api" + }, + { + "type": "docker_container_stopped", + "container": "api-canary" + }, + { + "type": "docker_container_absent", + "container": "sync-orders" + }, + { + "type": "docker_container_absent", + "container": "sync-users" + }, + { + "type": "docker_container_stopped", + "container": "sync-invoices" + }, + { + "type": "docker_container_count_equals", + "count": 3 + } + ] + }, + "rewards": { + "xp": 130, + "hint_penalty": 20 + } +} diff --git a/internal/mission/mission.go b/internal/mission/mission.go index 1ee8754..f3caaf9 100644 --- a/internal/mission/mission.go +++ b/internal/mission/mission.go @@ -18,6 +18,11 @@ const ( DockerStateRunning = "running" DockerStateStopped = "stopped" + + DockerHealthHealthy = "healthy" + DockerHealthUnhealthy = "unhealthy" + + DockerRestartOnFailure = "on-failure" ) // ConditionType identifies one observable outcome supported by a mission @@ -45,6 +50,7 @@ const ( ConditionDockerContainerRunning ConditionType = "docker_container_running" ConditionDockerContainerStopped ConditionType = "docker_container_stopped" ConditionDockerContainerCountEqual ConditionType = "docker_container_count_equals" + ConditionDockerContainerAbsent ConditionType = "docker_container_absent" ) // Mission is a declarative OpsQuest exercise. Setup describes the isolated @@ -137,12 +143,17 @@ type DockerImageSpec struct { Reference string `json:"reference"` } +// DockerContainerSpec is one logical fixture. Log, ExitCode, Health, and +// Restart select fixed fixture behaviors implemented by the Docker adapter; +// mission content never supplies an executable command. type DockerContainerSpec struct { Name string `json:"name"` Image string `json:"image"` State string `json:"state"` Log string `json:"log,omitempty"` ExitCode *int `json:"exit_code,omitempty"` + Health string `json:"health,omitempty"` + Restart string `json:"restart,omitempty"` } type Validation struct { diff --git a/internal/mission/world_test.go b/internal/mission/world_test.go index 17556d3..f388237 100644 --- a/internal/mission/world_test.go +++ b/internal/mission/world_test.go @@ -28,8 +28,11 @@ func TestCatalogWorldsAreOrderedWithinEachTrack(t *testing.T) { } docker := catalog.Worlds(TrackDocker) - if len(docker) != 1 || docker[0].Number != 1 || docker[0].Name != "It Works on My Machine" || len(docker[0].Missions) != 6 || docker[0].Missions[0].ID != "docker-container-census" { - t.Fatalf("Worlds(docker) = %#v", docker) + if len(docker) != 2 || docker[0].Number != 1 || docker[0].Name != "It Works on My Machine" || len(docker[0].Missions) != 6 || docker[0].Missions[0].ID != "docker-container-census" { + t.Fatalf("Worlds(docker) first world = %d worlds, %q", len(docker), docker[0].Name) + } + if docker[1].Number != 2 || docker[1].Name != "Container Triage" || len(docker[1].Missions) != 5 || docker[1].Missions[0].ID != "docker-janitor-duty" { + t.Fatalf("Worlds(docker) second world = number %d, name %q, stages %d", docker[1].Number, docker[1].Name, len(docker[1].Missions)) } if worlds := catalog.Worlds("missing"); worlds != nil { t.Fatalf("Worlds(missing) = %#v, want nil", worlds) @@ -81,7 +84,11 @@ func TestCatalogWorldLookupAndPlacement(t *testing.T) { }, { missionID: "docker-container-census", - want: Placement{Track: TrackDocker, WorldNumber: 1, WorldTotal: 1, WorldName: "It Works on My Machine", StageNumber: 1, StageTotal: 6}, + want: Placement{Track: TrackDocker, WorldNumber: 1, WorldTotal: 2, WorldName: "It Works on My Machine", StageNumber: 1, StageTotal: 6}, + }, + { + missionID: "docker-postmortem-triage", + want: Placement{Track: TrackDocker, WorldNumber: 2, WorldTotal: 2, WorldName: "Container Triage", StageNumber: 5, StageTotal: 5}, }, } for _, test := range tests { diff --git a/scripts/smoke-test.sh b/scripts/smoke-test.sh index 311d57b..1d181ee 100755 --- a/scripts/smoke-test.sh +++ b/scripts/smoke-test.sh @@ -126,7 +126,8 @@ docker_list_output="$(run_opsquest list --track docker)" assert_no_ansi "docker list" "${docker_list_output}" assert_contains "docker list" "${docker_list_output}" "DOCKER LABS" assert_contains "docker list" "${docker_list_output}" "Container Census" -assert_contains "docker list" "${docker_list_output}" "0/6 missions complete" +assert_contains "docker list" "${docker_list_output}" "Container Triage" +assert_contains "docker list" "${docker_list_output}" "0/11 missions complete" assert_contains "docker list" "${docker_list_output}" "Continue: opsquest play --track docker" assert_contains "docker list" "${docker_list_output}" "Jump: opsquest play --track docker --world N" @@ -152,11 +153,18 @@ assert_contains "profile initialization" "${profile_output}" "Operator: Smoke Op doctor_output="$(run_opsquest doctor)" assert_no_ansi "doctor" "${doctor_output}" -assert_contains "doctor" "${doctor_output}" "embedded catalog: 29 missions" +assert_contains "doctor" "${doctor_output}" "embedded catalog: 34 missions (23 Linux, 11 Docker)" assert_contains "doctor" "${doctor_output}" "profile path: ${PROFILE_HOME}/profile.json" assert_contains "doctor" "${doctor_output}" "Linux labs: in-memory; no host shell or filesystem access" assert_contains "doctor" "${doctor_output}" "docker labs:" assert_contains "doctor" "${doctor_output}" "docker executable not found in PATH" +assert_not_contains "doctor" "${doctor_output}" "docker cleanup:" + +cleanup_output="$(run_opsquest doctor --cleanup 2>&1 || true)" +assert_contains "doctor cleanup" "${cleanup_output}" "docker cleanup failed: docker executable not found in PATH" + +doctor_help_output="$(run_opsquest help doctor)" +assert_contains "doctor help" "${doctor_help_output}" "Usage: opsquest doctor [--cleanup]" play_output="$(printf 'pwd\nopsquest list --completed\nplay 4\nquit\n' | run_opsquest play)" assert_no_ansi "scripted mission" "${play_output}"