diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index b6fb6fb3a..39a90eaa4 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -1032,3 +1032,7 @@ re-derive from scratch. stayed at 1 so `max_turns` never tripped — all silently, with `make verify` green. A static rule cannot see an SDK field go dead; a live-telemetry smoke (one real turn, assert `token_usage` is non-empty) in the harness-bump checklist would have. +- [ ] Validation regexes anchored with `$` and applied with `.match` (not `.fullmatch`) accept a trailing newline — `"evil.com\n:443"` passed `normalize_egress_target` until it moved to `fullmatch`. A lint rule needs to tell a validation regex from a search regex; not cheap — caught in the network: llm_only Phase 1 review. +- [ ] `raise X(...) from exc` where `exc` came from parsing an env/URL value leaks the value through `__cause__` in a traceback (urlsplit put a URL password in its port error). Needs data-flow from env reads; not cheap — caught in the network: llm_only Phase 3 review. +- [ ] `await asyncio.to_thread(subprocess.run, ...)` that CREATES a resource inside a try whose `finally` removes it: a cancel leaves the worker thread running, so teardown races the create and leaks it (fixed in `isolation/egress.py` by `_run_to_completion`). Detecting "creates a resource" is not mechanical — caught in the network: llm_only Phase 4 review. +- [ ] The single-`asyncio.shield` join in `fs_permissions.set_permissions` does not survive a SECOND cancel of the caller; `isolation/egress._run_to_completion` loops on the shield. Promote that helper to a shared module and use it in both places — caught in the network: llm_only Phase 4 review. diff --git a/.claude/notes/isolation.md b/.claude/notes/isolation.md index 03dc8bc8e..2981cb144 100644 --- a/.claude/notes/isolation.md +++ b/.claude/notes/isolation.md @@ -1036,3 +1036,126 @@ separator a value beginning with `-` is parsed as an OPTION rather than a reposi (`--upload-pack=…` runs a command of the caller's choosing). That URL is task-authored, and since `evaluate ` rebuilds the task from a shareable run directory it is no longer necessarily the operator's own string. + +## The egress sidecar (network: llm_only) + +`network: llm_only` puts the task container on a per-task `--internal` network whose only exit +is a proxy sidecar that forwards to an exact `host:port` allowlist. The plan and its spike log +are in `c/2026-10-01-docker-no-internet-egress.md` (not committed); the facts that shaped the +design are below. + +### Why an explicit proxy, not a transparent one + +Harbor's design (a `gost` sidecar, nftables `redirect`, SNI sniffing, the task in the sidecar's +netns with `NET_ADMIN`) also catches tools that ignore proxy env. Phase 0 found no built-in +harness that needs it: Claude Code (Bun native binary), Codex (Rust reqwest), Antigravity (Go +`ProxyFromEnvironment`) and Pi (undici `EnvHttpProxyAgent` with `NODE_USE_ENV_PROXY=1`) all +honour `HTTPS_PROXY`. An explicit proxy needs no capability, no third-party image and no SNI +parser, and a tool that ignores it has no route, so it fails closed. iptables inside the task +container was rejected because it needs `NET_ADMIN` in the agent's own container, which lets the +agent undo it. + +### Why a per-task network + +A shared internal network lets task A use task B's sidecar (and its allowlist) and reach task +B's ports. One network per task costs one address pool each: a default Docker Desktop ran out at +the 30th network (`all predefined address pools have been fully subnetted`), which is why the +error names `--max-parallel` and `default-address-pools`. + +### Why inhibit_ipv4 and ip_forward=0 + +On native Linux a default `--internal` bridge still holds a gateway IP on the host, and the task +container reached every host service bound on `0.0.0.0` through it (dockerd 20.10, 24 and 29, +both firewall backends; on 20.10 and 24 also `docker0` and the host NIC). The +`com.docker.network.bridge.inhibit_ipv4=true` option removes the gateway; the sidecar then takes +the `.1` address and the embedded DNS alias still works. `gateway_mode_ipv4=isolated` was +rejected: Docker 24 ignores it silently. Docker Desktop never showed the hole. +`--sysctl net.ipv4.ip_forward=0` on the sidecar is defence in depth: Docker enables forwarding in +the sidecar's netns, so a task container with `NET_ADMIN` could try to route through it. No +config field grants the task container run-time capabilities, and the measured route did not +reach the internet, but the sysctl costs nothing. + +### Why a black-hole DNS server and --ipv6=false + +The spikes ran on dind, which has no loopback resolver, so they could not show CVE-2024-29018: +on daemons before 26.0 / 25.0.5 / 23.0.11 the embedded DNS forwards an internal network's +queries from the host namespace when the host resolver is loopback (systemd-resolved), which is a +DNS tunnel out. The task container's `--dns 192.0.2.1` (TEST-NET-1, never routed) keeps the +embedded DNS answering the sidecar alias while every external forward goes nowhere. +`inhibit_ipv4` removes only the IPv4 gateway, so `--ipv6=false` keeps a daemon whose +`default-network-opts` enable IPv6 from giving the internal bridge an IPv6 gateway. IPv6 +link-local reach to a host service was not measured. + +`NET_RAW` is dropped from the task container for the same reason: with raw sockets an agent could +write frames for the bridge's own MAC address and reach a host service over UDP without the +sidecar. That path was not measured; the cap drop removes it at no cost, since no tool needs raw +sockets under `llm_only`. + +### Why the sidecar log is a separate, atomically written file + +The first version appended the sidecar log to `docker.log` after the container exited. That file +is in the run directory, which is bind-mounted writable into the container at the SAME path, so +the agent could replace it with a symlink to a host file (for example a shell rc file) and get a +line it chose (`DENY $(cmd):443 CONNECT` passes the visible-ASCII check) appended there by the +host. Container stdout could also forge `ALLOW` / `DENY` lines in it. The log is now `egress.log`, +written once by `write_text_atomic`, whose `os.replace` replaces a planted symlink instead of +following it. `BAD` lines carry only the method and length, because a refused target can hold +credentials in its userinfo or query. + +### Why the framework image and a bind-mounted module + +The sidecar image is the framework image, never the task image: a task image is task-authored and +may be a runtime-kit image with a different Python. The proxy code is NOT taken from the image: +the host bind-mounts its own `egress_proxy.py` read-only, so the code under test is the boundary +that runs and image skew cannot change it. That is also why the module is stdlib-only (five +imports, guarded by a test): the image only supplies `python3`. + +### Why the sidecar watches the heartbeat + +A host SIGKILL skips every `finally`. The task container already exits on a stale host heartbeat; +the sidecar reads the same file through a single-file `:ro` bind (Docker Desktop VirtioFS saw +every in-place counter write) and stops itself with the same counter-or-mtime rule as +`heartbeat_is_alive`. It has no `--rm`, so a crashed or stale-stopped sidecar keeps its log until +teardown reads it; `docker network rm` works with a stopped container still attached, so a later +prune needs no ordering. The watchdog returns from `serve` instead of calling `os._exit`, because +CE052 gates process-lethal calls on `IN_CONTAINER_ENV`, which a stdlib-only module cannot import. + +### Why every docker call runs to completion + +`_docker` runs `subprocess.run` in a worker thread, and cancelling the await does not stop the +thread. A Ctrl-C during `docker create` once let teardown run `docker rm -f` before the create +finished, leaking a created-but-never-started sidecar that no heartbeat would ever stop. So every +call is joined across any number of cancels (`_run_to_completion`) before the cancel propagates, +and teardown is joined the same way. + +### The log line format is a security boundary + +`docker.log` is the only evidence the boundary leaves, and operators grep it for +`^(ALLOW|DENY|FAIL)`. A request line with a control or non-ASCII byte is refused before anything is +logged, because a lone `\n` in a method once forged a separate `ALLOW` line. + +### Why the allowlist follows the model callers, not the forwarded env + +The first version allowed the API backend's hosts for every agent and the host of every forwarded +`*_URL` variable. The real runs showed the cost: a Claude task could reach the Azure +`CODEX_BASE_URL` host, and a Codex, Antigravity or Pi task could reach `api.anthropic.com`, +because those variables are on the default passthrough list and the default backend is `direct`. +Each host was a model provider, but none was needed. So the backend's hosts are added only for a +component that uses `API_BACKEND` (`uses_api_backend` on the agent config, an enabled simulator, +an `llm_judge` / `agent_judge` criterion), and a URL variable counts only where its owner reads it +(`LITELLM_BASE_URL` for the litellm backend, `CODEX_BASE_URL` in `CodexAgentConfig.egress_hosts`). +Any other host is explicit in `egress_allowlist`. + +### What Phase 0 measured + +- Every built-in harness in the image and every judge route (`llm_judge`, `agent_judge`, + litellm with aiohttp) works through the proxy with an exact host set. +- Claude Code on Bedrock calls the control plane `bedrock..amazonaws.com` at every start; + it is allowed so the CLI behaves as under `bridge`. `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` + does not remove that call, and on direct its Datadog `DENY` did not slow the CLI, so it is not set. +- Claude Code OAuth refresh goes to `platform.claude.com` (binary string `TOKEN_URL`), which is on + the direct backend list because `agent_judge` runs the CLI too. +- `LITELLM_LOCAL_MODEL_COST_MAP=True` costs no latency either way (the `403` is immediate); it is set + to remove one `DENY` per process and a network-dependent cost map. +- busybox `wget` sends absolute-form `GET https://…` to a proxy instead of `CONNECT`; the proxy + answers `400` with a `BAD … absolute-form https (use CONNECT)` line rather than originate TLS. diff --git a/docs/DOCKER_ISOLATION.md b/docs/DOCKER_ISOLATION.md index 8c6e6383a..23d3bb888 100644 --- a/docs/DOCKER_ISOLATION.md +++ b/docs/DOCKER_ISOLATION.md @@ -47,10 +47,207 @@ coder-eval run --driver docker sandbox: driver: docker docker: - network: bridge # or "none" for sealed runs + network: bridge # bridge | llm_only | none (see Network modes) image: my-custom:tag # override the default image ``` +## Network modes + +`sandbox.docker.network` selects what the task container can reach. The same mode applies to +the grading container of a detached `coder-eval evaluate`. + +| Mode | The container reaches | Use it for | +|---|---|---| +| `bridge` (default) | the full network | tasks that install packages or call other services | +| `llm_only` | only the model APIs and the hosts in `egress_allowlist`, through a proxy sidecar | agent tasks that must not use general internet | +| `none` | nothing | `agent: {type: none}` tasks only: a real agent cannot reach its model API | + +```yaml +sandbox: + driver: docker + docker: + network: llm_only + egress_allowlist: [pypi.org, files.pythonhosted.org] # optional, appended across layers +``` + +You can also set it per run: `-D sandbox.docker.network=llm_only` and +`-D 'sandbox.docker.egress_allowlist=["pypi.org"]'`. + +### How the egress sidecar works + +For each task the host creates: + +1. A per-task `docker network create --internal --ipv6=false` network with + `com.docker.network.bridge.inhibit_ipv4=true`. The network has no route out and no IPv4 gateway + address on the host, so the task container cannot reach the internet or a host service. +2. An egress-proxy sidecar from the framework image `coder-eval-agent:`. It joins the + internal network (DNS alias `coder-eval-egress`) and the default `bridge`. It runs the host's + own `egress_proxy.py`, bind-mounted read-only, as uid 65534 with all capabilities dropped and a + read-only root file system. It opens TCP connections only to exact `host:port` targets. +3. The task container, on the internal network only, with `HTTPS_PROXY` / `HTTP_PROXY` (both + cases) set to `http://coder-eval-egress:3128`, `NO_PROXY=localhost,127.0.0.1,::1`, + `NODE_USE_ENV_PROXY=1` and `LITELLM_LOCAL_MODEL_COST_MAP=True`. A host value of a proxy + variable (`*_PROXY`, `NO_PROXY`, `NODE_USE_ENV_PROXY`) is never forwarded. Its upstream DNS + server is `192.0.2.1`, an address that is never routed: Docker's own DNS still resolves the + sidecar, but an external name does not resolve. It also runs without `NET_RAW`, so it cannot + send crafted packets onto the internal bridge. + +Before the task container starts, the host sends one `CONNECT` per allowlisted target through +the sidecar. If one fails, the task is an `ERROR` row that names the failing targets. The host +removes the sidecar and the network on every path (success, error, Ctrl-C). If the host process +is killed, the sidecar stops by itself when the host heartbeat goes stale; see +[Troubleshooting egress](#troubleshooting-egress) to remove what is left. + +A tool that ignores the proxy variables has no route, so it fails closed. It cannot bypass the +allowlist. + +### The derived allowlist + +You do not list the model APIs yourself. The host derives them from the parts of the task that +call a model, and adds nothing for the parts that do not: + +| Source | Hosts added | +|---|---| +| API backend (`API_BACKEND`), only when a `claude-code` agent, an enabled `simulation:`, or an `llm_judge` / `agent_judge` criterion uses it | `direct`: `api.anthropic.com:443`, `platform.claude.com:443` (Claude Code OAuth refresh). `bedrock`: `bedrock-runtime..amazonaws.com:443`, `bedrock..amazonaws.com:443`. `litellm`: the host of the forwarded `LITELLM_BASE_URL` (a `localhost` value becomes `host.docker.internal`) | +| Agent | codex: the host of the forwarded `CODEX_BASE_URL`, else `api.openai.com:443`. antigravity: `generativelanguage.googleapis.com:443`. pi: the host of the `provider/` prefix of `agent.model` (`openrouter`, `anthropic`, `openai`, `google`; anything else or no model gives `openrouter.ai:443`). claude-code, opencode, delegate, none: nothing beyond the backend row | +| `system_one_judge` criteria | the host of each `base_url` (default `api.typesafe.ai:443`) | +| `egress_allowlist` | your entries | + +A judge with its own `checker_context.api_route.route` uses the hosts of that backend. A +`route: litellm` judge configures its endpoint through `params`, so add that host to +`egress_allowlist`. Other forwarded `*_URL` variables, such as `UIPATH_URL`, add no host: a task +whose tools call that service lists the host in `egress_allowlist`. A Pi provider other than the +four above, and every OpenCode or Delegate provider, also needs its host in `egress_allowlist`. + +### Extra egress hosts + +`egress_allowlist` takes `host` or `host:port` entries; a bare host means port 443. A host is a DNS +name or an IPv4 address. Schemes, paths, wildcards and IPv6 addresses are refused when the task +loads. Entries are appended across the config layers, like `env_passthrough_extra`. Under `bridge` +or `none` the field is ignored. + +Typical additions: + +| Need | Entries | +|---|---| +| `sandbox.python.env_packages`, `pip`, `uv` | `pypi.org`, `files.pythonhosted.org` | +| `npm`, `npx -y` MCP servers | `registry.npmjs.org` | +| `git clone` over https from GitHub | `github.com` | +| `apt` (Debian) | `deb.debian.org:80` | +| `apt` (Ubuntu) | `archive.ubuntu.com:80`, `security.ubuntu.com:80` (amd64) or `ports.ubuntu.com:80` (arm64) | +| `apk` (Alpine) | `dl-cdn.alpinelinux.org` | +| A remote MCP server or a run-time plugin install | its host | +| `uv` downloading a managed Python not in the image | `github.com`, `objects.githubusercontent.com` (or bake the Python into the image) | + +### Tool compatibility + +The task image needs nothing new: its tools must honour the proxy variables. + +| Tool | Under `llm_only` | +|---|---| +| curl, GNU wget, git over https, pip, uv, npm/npx, apt, apk | honour the proxy; a denied host fails fast (`CONNECT tunnel failed, response 403`, `npm error 403`; pip and uv retry for about 8–12 s first) | +| Python urllib / requests / httpx | honour the proxy (`trust_env` is on by default) | +| Python aiohttp | only with `trust_env=True` (litellm sets it); otherwise no route | +| Node `fetch` / `https` | only on Node ≥ 22.21 or ≥ 24.5, through `NODE_USE_ENV_PROXY=1`; older Node has no route. Node prints an `UNDICI-EHPA` warning on stderr | +| Go `net/http` default client | honours the proxy; a custom transport without `Proxy` has no route | +| busybox `wget` (Alpine) | plain http works; https always fails `400`, because it sends `GET https://…` instead of `CONNECT`. Use curl | +| dnf (Rocky, Fedora) | its metalink picks random mirrors, so an exact-host list cannot work. Pin a `baseurl` or bake the packages into the image | +| git over ssh, raw sockets, DNS lookups, Java without proxy flags | no route | + +Every built-in harness in the framework image honours the proxy (Claude Code, Codex, +Antigravity, Pi). See [Run-Limit Parity § Network modes](agents/HARNESS_PARITY.md#network-modes-under-the-docker-driver). + +The sidecar always runs the Debian framework image, so the task image's distribution does not +change the boundary. Debian, Ubuntu, Rocky Linux, Fedora and Alpine task images were tested as +clients. A runtime-kit image needs the framework image too (`make docker-images` builds both). + +**Docker versions.** Docker 20.10 or later (`host-gateway` first shipped there). Docker 26.0, +25.0.5 or 23.0.11 or later is recommended: older daemons forward the DNS queries of an internal +network from the host (CVE-2024-29018). The black-hole upstream DNS server above stops that +forward, but a patched daemon removes the cause. Tested on Docker +Desktop 29 (macOS) and on Linux dockerd 20.10, 24 and 29, with both the `iptables` and the +`nftables` firewall backends. On Linux, `host.docker.internal` resolves to the `docker0` address, +so a LiteLLM proxy on the host must listen on `docker0` or `0.0.0.0`, as under `bridge`. + +### Expected DENY lines + +Some clients call hosts they do not need. These `DENY` lines are harmless: + +| Client | Denied host | +|---|---| +| Claude Code with `API_BACKEND=direct` | `http-intake.logs.us5.datadoghq.com:443` (telemetry) | +| Codex | `chatgpt.com:443`, `github.com:443`, `api.github.com:443` (update check, remote config). Codex without `CODEX_API_KEY` falls back to a ChatGPT login whose model host is `chatgpt.com`, so that setup needs it in `egress_allowlist` | +| litellm without `LITELLM_LOCAL_MODEL_COST_MAP` | `raw.githubusercontent.com:443` (cost map) | +| OpenCode | `models.opencode.ai:443` (model catalog refresh), `registry.npmjs.org:443` (update check) | +| Claude Code with `API_BACKEND=litellm` | `api.anthropic.com:443` (startup calls the CLI does not need on this route) | +| Claude Code `WebFetch` on Bedrock | `api.anthropic.com:443` (the domain safety check before a fetch; `WebFetch` then fails) | + +### Egress limits + +- **Parallel tasks.** Each task uses one Docker network. A default Docker Desktop has address + pools for about 29 user networks, so keep `--max-parallel` under that. When the pools are + exhausted, the row error names `--max-parallel`, the prune command and the daemon's + `default-address-pools` setting. +- **No upstream proxy.** The sidecar connects directly. A host that can reach the internet only + through a corporate proxy cannot use `llm_only`. +- **Not exfiltration-proof.** The agent can still send data to an allowlisted host. +- **Server-side tools are outside the boundary.** A tool that the model provider runs, such as + Claude Code's `WebSearch` on the direct API, fetches from the internet on the provider's side. + The container network cannot block it; remove it with `disallowed_tools` if the task needs no + internet at all. +- **An exact host is a TCP destination, not a site.** A host behind a shared CDN front (for + example `files.pythonhosted.org` or `deb.debian.org`) can serve other sites that the agent names + in its `Host` header or TLS SNI. +- **`extra_mounts` can defeat the boundary.** Mounting the Docker socket, for example, gives the + agent the daemon. +- **Exact hosts only.** No wildcards and no CIDR ranges. +- **podman and rootless Docker** are not tested. +- **Harbor export** refuses an `llm_only` task. +- **The sidecar is also on the default `bridge`.** A `bridge` container of another task can reach + it. It gets no extra reach, because the allowlist is a subset of what `bridge` already reaches, + and the sidecar serves at most 256 connections at once (one more gets `503`). +- **Plugin agents.** A third-party agent gets the API backend's hosts only when its config class + sets `uses_api_backend = True`, and its own hosts only through `egress_hosts()`; see + [Extending Coder Eval](EXTENDING.md#the-config-class). + +### Checking the boundary + +`tasks/docker_egress_probe/` tests the mode end to end. The agent tries to reach the internet in +its own ways, and the grading criteria then probe each path out from inside the container: the +proxy, direct IPv4 and IPv6, DNS, raw sockets, the docker host, `pip`, `git` and `npm`. Each +probe passes only when its path is blocked, and the model host must stay reachable: + +```bash +coder-eval run tasks/docker_egress_probe/docker_egress_probe.yaml # expect 1.000 +coder-eval run tasks/docker_egress_probe/docker_egress_probe.yaml \ + -D sandbox.docker.network=bridge # control: expect a FAILURE +``` + +Run the probes alone in any `llm_only` container with +`python3 egress_probe.py all`. + +### Troubleshooting egress + +The sidecar log is the task's `egress.log`, beside `docker.log` (a grading container's log is +`grade.egress.log`). It is a separate file so that container output cannot add lines to it. +Each connection is one line: +`ALLOW host:port METHOD`, `DENY host:port METHOD`, `FAIL host:port ` (an allowed host the +sidecar could not reach), or `BAD …` (a request the proxy refused; the line names only its method +and length, because the target can carry credentials). To see which hosts a +task needed: + +```bash +grep -rhE "^(ALLOW|DENY|FAIL)" --include=egress.log runs/latest | sort | uniq -c +``` + +Add each needed `DENY` host to `egress_allowlist`. If the host process was killed, remove the +leaked sidecars and networks: + +```bash +docker rm -f $(docker ps -aq --filter label=org.coder-eval.egress) +docker network prune -f --filter label=org.coder-eval.egress +``` + ## Using a pre-built custom image When your tasks need extra tools or dependencies, extend the framework image once and point tasks at diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index 842146747..79bda5661 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -80,6 +80,14 @@ class MyAgentConfig(BaseAgentConfig): The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config isn't an instance of the registered `config_class`, so keep them paired. +Under `network: llm_only` the container reaches only an allowlist of hosts, and the config class +supplies its share. Set the class variable `uses_api_backend = True` when the agent calls its +model through `API_BACKEND` (as `claude-code` does), and override +`egress_hosts(self, env) -> tuple[str, ...]` to return the `host:port` targets of its own model API +(`env` holds the forwarded variables; `url_egress_target` from `coder_eval.models` turns a URL +into a target). An agent that does neither can reach no model under `llm_only`. See +[Docker Isolation § The derived allowlist](DOCKER_ISOLATION.md#the-derived-allowlist). + ### The `Agent` ABC — implementation checklist Implement these three abstract methods: diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index af9e54823..2df6ae009 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -545,6 +545,10 @@ Under `driver: tempdir` only `timeout` is enforced — the agent can consume arbitrary host memory, CPU, and PIDs. Use `driver: docker` when you need the container limits above to actually bind. +Under `driver: docker`, `sandbox.docker.network` selects `bridge` (default), `llm_only` (model +APIs plus `sandbox.docker.egress_allowlist` only) or `none`. See +[Docker Isolation § Network modes](DOCKER_ISOLATION.md#network-modes). + ### Recording CLI Invocations `record_cli` shadows executables with generated recording shims, so a task can assert on **what the agent actually ran** without hand-writing a mock: diff --git a/docs/agents/HARNESS_PARITY.md b/docs/agents/HARNESS_PARITY.md index 86fb07bb4..198857a38 100644 --- a/docs/agents/HARNESS_PARITY.md +++ b/docs/agents/HARNESS_PARITY.md @@ -734,6 +734,25 @@ both. See [OpenCode](OPENCODE.md) and [Pi § plugins](PI.md#known-limitations). Full detail: [Pi](PI.md). +## Network modes under the docker driver + +`sandbox.docker.network` has the same meaning on every harness: `bridge` gives the container the +full network, `none` gives it no network, and `llm_only` lets it reach only the model APIs and +`egress_allowlist` through a proxy sidecar +([Docker Isolation § Network modes](../DOCKER_ISOLATION.md#network-modes)). A harness works under +`llm_only` only if its CLI honours the `HTTPS_PROXY` variables; one that does not fails closed. + +| | claude-code | codex | antigravity | opencode | pi | delegate | none | +|---|---|---|---|---|---|---|---| +| `bridge` | works | works | works | works (custom image) | works | works (custom image) | works | +| `none` | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | no model API: the turn fails | works | +| `llm_only` honours the proxy | yes | yes | yes | yes (custom image with the CLI) | yes | CLI not in the framework image; untested | no egress needed | +| Verified by | runs R2, R11, R12 (Bedrock); spike S3 (direct) | run R3 (Luna on Azure `CODEX_BASE_URL`) | run R4 | Bedrock and Azure `gpt-5.6-luna` runs (custom image; hosts in `egress_allowlist`) | Bedrock and Azure `gpt-5.6-luna` runs | spike S8 (absent) | — | + +The hosts each harness gets by default, and the harmless `DENY` lines it produces, are in +[Docker Isolation § The derived allowlist](../DOCKER_ISOLATION.md#the-derived-allowlist) and +[§ Expected DENY lines](../DOCKER_ISOLATION.md#expected-deny-lines). + ## Reproducing `tasks/run_limits/` holds one fixture per limit: `max_turns_cap.yaml` asks for more diff --git a/docs/tutorials/06-use-docker-isolation.md b/docs/tutorials/06-use-docker-isolation.md index 7e3631d14..e2936aa7c 100644 --- a/docs/tutorials/06-use-docker-isolation.md +++ b/docs/tutorials/06-use-docker-isolation.md @@ -160,7 +160,7 @@ Rather than passing the flag every time, set it in the task YAML: sandbox: driver: docker docker: - network: bridge # or "none" for a fully sealed run (no network) + network: bridge # or "llm_only" (model APIs only) or "none" (no network) image: my-custom:tag # optional: override the default framework image ``` diff --git a/src/coder_eval/egress_proxy.py b/src/coder_eval/egress_proxy.py new file mode 100644 index 000000000..49c4e07fb --- /dev/null +++ b/src/coder_eval/egress_proxy.py @@ -0,0 +1,353 @@ +"""Stdlib-only egress proxy for ``network: llm_only``; it runs inside the egress sidecar. + +``serve`` accepts HTTP ``CONNECT host:port`` and absolute-form plain HTTP +(``GET http://host/...``) and forwards only to an exact allowlisted ``host:port``. +It never originates TLS: an absolute-form ``https://`` request gets ``400``. Every +decision is one stdout line (``ALLOW``, ``DENY``, ``FAIL``, ``BAD``, ``STALE``) after +one ``READY`` line; a request line with a control or non-ASCII byte is refused. With +``--heartbeat`` it stops when the host heartbeat stops changing for ``--stale`` seconds. + +``probe`` sends one ``CONNECT`` per target through a running proxy, prints +``OK target`` or ``FAIL target ``, and exits 0 only when every target is OK. + +The host bind-mounts this file into the framework image and runs it with +``python3 -I``, so it may import only ``argparse``, ``asyncio``, ``os``, ``sys`` and +``time``. Importing it starts nothing. + +Rationale: .claude/notes/isolation.md § The egress sidecar (network: llm_only) +""" + +import argparse +import asyncio +import os +import sys +import time + + +HEAD_LIMIT_BYTES = 64 * 1024 +HEAD_TIMEOUT_SECONDS = 30.0 +DIAL_TIMEOUT_SECONDS = 10.0 +PIPE_CHUNK_BYTES = 64 * 1024 +MAX_CONNECTIONS = 256 +MAX_HEARTBEAT_POLL_SECONDS = 2.0 +PROBE_STARTUP_RETRY_SECONDS = 5.0 + +_BAD_REQUEST = b"HTTP/1.1 400 Bad Request\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_FORBIDDEN = b"HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_BAD_GATEWAY = b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_UNAVAILABLE = b"HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n" +_ESTABLISHED = b"HTTP/1.1 200 Connection Established\r\n\r\n" + + +def _log(line: str) -> None: + print(line, flush=True) + + +def _redacted(request_line: bytes) -> str: + """A refused request line, shown as its method and length only: the target may carry credentials.""" + method = request_line.split(b" ", 1)[0] + shown = method.decode("ascii") if method.isalpha() and method.isascii() and len(method) <= 16 else "?" + return f"{shown} unparseable request line ({len(request_line)} bytes)" + + +def _is_visible_ascii(data: bytes) -> bool: + return all(0x20 <= byte < 0x7F for byte in data) + + +def _parse_port(text: str) -> int | None: + if not (text.isascii() and text.isdigit()): + return None + port = int(text) + return port if 1 <= port <= 65535 else None + + +def split_host_port(authority: str, default_port: int | None) -> tuple[str, int] | None: + """Split ``host[:port]`` (or ``[v6]:port``) into a lowercased host and a port. + + Returns None when the host is empty, the port is invalid, or a port is + required (``default_port is None``) and absent. + """ + if authority.startswith("["): + end = authority.find("]") + if end < 0: + return None + host, rest = authority[1:end], authority[end + 1 :] + if rest and not rest.startswith(":"): + return None + port_text = rest[1:] if rest else "" + else: + host, sep, port_text = authority.rpartition(":") + if not sep: + host, port_text = authority, "" + if not host: + return None + if not port_text: + if default_port is None: + return None + return host.lower(), default_port + port = _parse_port(port_text) + return None if port is None else (host.lower(), port) + + +def heartbeat_alive(current: str, last_counter: str, current_mtime: float, last_mtime: float) -> bool: + """True when the heartbeat counter text changed or its mtime advanced.""" + return bool(current and current != last_counter) or current_mtime > last_mtime + + +def _read_heartbeat(path: str) -> tuple[str, float]: + try: + with open(path, encoding="utf-8") as handle: + current = handle.read() + except (OSError, UnicodeDecodeError): + current = "" + try: + mtime = os.stat(path).st_mtime + except OSError: + mtime = 0.0 + return current, mtime + + +async def watch_heartbeat(path: str, stale_seconds: float) -> None: + """Return once the heartbeat at ``path`` has not changed for ``stale_seconds``.""" + poll_seconds = min(MAX_HEARTBEAT_POLL_SECONDS, stale_seconds / 4) + last_counter, last_mtime = "", 0.0 + last_change = time.monotonic() + while True: + current, mtime = await asyncio.to_thread(_read_heartbeat, path) + now = time.monotonic() + if heartbeat_alive(current, last_counter, mtime, last_mtime): + last_counter, last_mtime, last_change = current, mtime, now + if now - last_change > stale_seconds: + _log(f"STALE heartbeat {path} unchanged for more than {stale_seconds:g}s; stopping") + return + await asyncio.sleep(poll_seconds) + + +async def _reply_and_close(writer: asyncio.StreamWriter, response: bytes) -> None: + try: + writer.write(response) + await writer.drain() + except (ConnectionError, OSError) as exc: + _log(f"BAD client gone before the reply: {exc!r}") + finally: + writer.close() + + +async def _pipe(reader: asyncio.StreamReader, writer: asyncio.StreamWriter, peer: asyncio.StreamWriter) -> None: + try: + while data := await reader.read(PIPE_CHUNK_BYTES): + writer.write(data) + await writer.drain() + if writer.can_write_eof(): + writer.write_eof() + except (ConnectionError, OSError): + writer.close() + peer.close() + + +def _origin_form_head(method: str, path: str, version: str, header_block: bytes) -> bytes: + kept = [ + line + for line in header_block.split(b"\r\n") + if line and not line.lower().startswith((b"proxy-", b"connection:")) + ] + request_line = f"{method} {path} {version}".encode("latin-1") + return b"\r\n".join([request_line, *kept, b"Connection: close"]) + b"\r\n\r\n" + + +def _parse_absolute_http(target: str) -> tuple[str, int, str] | None: + rest = target[len("http://") :] + cut = min((i for i in (rest.find(c) for c in "/?#") if i >= 0), default=len(rest)) + authority, path = rest[:cut], rest[cut:] + path = path.split("#", 1)[0] + if not path.startswith("/"): + path = "/" + path + parsed = split_host_port(authority.rpartition("@")[2], 80) + return None if parsed is None else (*parsed, path) + + +async def handle_client( + allow: frozenset[str], client_reader: asyncio.StreamReader, client_writer: asyncio.StreamWriter +) -> None: + """Serve one client connection: parse its request head, then tunnel, forward or refuse.""" + try: + head = await asyncio.wait_for(client_reader.readuntil(b"\r\n\r\n"), HEAD_TIMEOUT_SECONDS) + except asyncio.LimitOverrunError: + _log("BAD head-too-large") + client_writer.close() + return + except (asyncio.IncompleteReadError, TimeoutError, ConnectionError, OSError): + client_writer.close() + return + request_line, _, header_block = head[:-4].partition(b"\r\n") + fields = request_line.decode("latin-1").split(" ") + if len(fields) != 3 or not _is_visible_ascii(request_line) or not fields[2].startswith("HTTP/"): + _log(f"BAD {_redacted(request_line)}") + await _reply_and_close(client_writer, _BAD_REQUEST) + return + method, target, version = fields + prefix = b"" + if method == "CONNECT": + parsed = split_host_port(target, None) + elif target.lower().startswith("http://"): + absolute = _parse_absolute_http(target) + parsed = None if absolute is None else absolute[:2] + if absolute is not None: + prefix = _origin_form_head(method, absolute[2], version, header_block) + elif target.lower().startswith("https://"): + https = split_host_port(target[len("https://") :].split("/", 1)[0].rpartition("@")[2], 443) + shown = f"{https[0]}:{https[1]}" if https else repr(target) + _log(f"BAD {shown} absolute-form https (use CONNECT)") + await _reply_and_close(client_writer, _BAD_REQUEST) + return + else: + parsed = None + if parsed is None: + _log(f"BAD {_redacted(request_line)}") + await _reply_and_close(client_writer, _BAD_REQUEST) + return + host, port = parsed + destination = f"{host}:{port}" + if destination not in allow: + _log(f"DENY {destination} {method}") + await _reply_and_close(client_writer, _FORBIDDEN) + return + try: + upstream_reader, upstream_writer = await asyncio.wait_for( + asyncio.open_connection(host, port), DIAL_TIMEOUT_SECONDS + ) + except (OSError, TimeoutError) as exc: + _log(f"FAIL {destination} {exc!r}") + await _reply_and_close(client_writer, _BAD_GATEWAY) + return + _log(f"ALLOW {destination} {method}") + try: + if method == "CONNECT": + client_writer.write(_ESTABLISHED) + await client_writer.drain() + else: + upstream_writer.write(prefix) + await upstream_writer.drain() + await asyncio.gather( + _pipe(client_reader, upstream_writer, client_writer), + _pipe(upstream_reader, client_writer, upstream_writer), + ) + except (ConnectionError, OSError) as exc: + _log(f"FAIL {destination} {exc!r}") + finally: + upstream_writer.close() + client_writer.close() + + +async def start_proxy(allow: frozenset[str], host: str, port: int) -> asyncio.Server: + """Start listening; the returned server is already accepting connections. + + At most ``MAX_CONNECTIONS`` clients are served at once; one more gets ``503``. + """ + active = 0 + + async def _handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + nonlocal active + if active >= MAX_CONNECTIONS: + _log("BAD too-many-connections") + await _reply_and_close(writer, _UNAVAILABLE) + return + active += 1 + try: + await handle_client(allow, reader, writer) + finally: + active -= 1 + + return await asyncio.start_server(_handle, host, port, limit=HEAD_LIMIT_BYTES) + + +async def serve(allow: frozenset[str], host: str, port: int, heartbeat: str | None, stale_seconds: float) -> None: + """Run the proxy until the heartbeat goes stale (forever when ``heartbeat`` is None).""" + server = await start_proxy(allow, host, port) + _log(f"READY {host}:{port} allow={','.join(sorted(allow))}") + try: + if heartbeat is None: + await server.serve_forever() + else: + await watch_heartbeat(heartbeat, stale_seconds) + finally: + server.close() + server.abort_clients() + + +async def _probe_one(proxy_host: str, proxy_port: int, target: str, timeout: float) -> str | None: + started = time.monotonic() + while True: + try: + reader, writer = await asyncio.wait_for(asyncio.open_connection(proxy_host, proxy_port), timeout) + break + except (OSError, TimeoutError) as exc: + if time.monotonic() - started >= PROBE_STARTUP_RETRY_SECONDS: + return f"proxy unreachable: {exc!r}" + await asyncio.sleep(0.2) + try: + writer.write(f"CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n\r\n".encode("latin-1")) + await writer.drain() + status_line = await asyncio.wait_for(reader.readline(), timeout) + except (OSError, TimeoutError) as exc: + return repr(exc) + finally: + writer.close() + status = status_line.split() + if len(status) >= 2 and status[1] == b"200": + return None + return status_line.decode("latin-1").strip() or "no response" + + +async def probe(proxy: str, targets: list[str], timeout: float) -> int: + """CONNECT to each target through ``proxy``; 0 when all succeed, else 1.""" + parsed = split_host_port(proxy, None) + if parsed is None: + _log(f"FAIL {proxy} invalid --proxy (expected host:port)") + return 1 + errors = await asyncio.gather(*(_probe_one(*parsed, target, timeout) for target in targets)) + for target, error in zip(targets, errors, strict=True): + _log(f"OK {target}" if error is None else f"FAIL {target} {error}") + return 0 if all(error is None for error in errors) else 1 + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(prog="egress_proxy", description=__doc__.split("\n", 1)[0] if __doc__ else None) + commands = parser.add_subparsers(dest="command", required=True) + serve_cmd = commands.add_parser("serve", help="run the allowlisting proxy") + serve_cmd.add_argument("--listen", default="0.0.0.0:3128", help="host:port to listen on") + serve_cmd.add_argument("--heartbeat", default=None, help="host heartbeat file; stop when it goes stale") + serve_cmd.add_argument("--stale", type=float, default=None, help="seconds without a heartbeat change") + serve_cmd.add_argument("--allow", action="append", default=[], help="allowed host:port (repeatable)") + probe_cmd = commands.add_parser("probe", help="CONNECT to each target through a running proxy") + probe_cmd.add_argument("--proxy", required=True, help="proxy host:port") + probe_cmd.add_argument("--timeout", type=float, default=5.0, help="per-target timeout in seconds") + probe_cmd.add_argument("targets", nargs="+", help="host:port targets") + return parser + + +def main(argv: list[str] | None = None) -> int: + """Command-line entry point; returns the process exit code.""" + args = _build_parser().parse_args(argv) + if args.command == "probe": + return asyncio.run(probe(args.proxy, args.targets, args.timeout)) + listen = split_host_port(args.listen, None) + if listen is None: + _log(f"BAD --listen {args.listen!r} (expected host:port)") + return 2 + if args.heartbeat is not None and not (args.stale is not None and 0 < args.stale < float("inf")): + _log("BAD --heartbeat needs a finite, positive --stale") + return 2 + allow: set[str] = set() + for entry in args.allow: + parsed = None if "[" in entry else split_host_port(entry.strip(), None) + if parsed is None or ":" in parsed[0]: + _log(f"BAD --allow {entry!r} (expected host:port)") + return 2 + allow.add(f"{parsed[0]}:{parsed[1]}") + asyncio.run(serve(frozenset(allow), listen[0], listen[1], args.heartbeat, args.stale or 0.0)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/coder_eval/harbor/packager.py b/src/coder_eval/harbor/packager.py index fe87b4a5a..7532e416a 100644 --- a/src/coder_eval/harbor/packager.py +++ b/src/coder_eval/harbor/packager.py @@ -158,6 +158,12 @@ def export_resolved_task( + "Set sandbox.driver: docker (with dockerfile_path or a custom image) to export this task." ) + if task.sandbox.docker.network == "llm_only": + raise TaskNotExportableError( + f"Task {task.task_id!r}: Harbor v1 export does not map network: llm_only; use bridge or none, " + + "or export with a Harbor allowlist by hand." + ) + if task.dataset is not None: # `load_task` does NOT run `expand_dataset` -- fan-out happens later, so # exporting a raw dataset-backed task would emit ONE Harbor task still diff --git a/src/coder_eval/isolation/docker_runner.py b/src/coder_eval/isolation/docker_runner.py index 3e5c7fe1d..79a9f9c14 100644 --- a/src/coder_eval/isolation/docker_runner.py +++ b/src/coder_eval/isolation/docker_runner.py @@ -25,6 +25,18 @@ import yaml from coder_eval.config import settings +from coder_eval.isolation.egress import ( + DOCKER_HOST_ALIAS, + EGRESS_PROXY_MODULE, + PROXY_ENV_NAMES, + EgressHandle, + egress_scope, + forwarded_env_names, + resolve_egress_targets, + rewrite_loopback_for_container, + task_container_egress_argv, +) +from coder_eval.isolation.errors import DockerRunError, EgressSetupError from coder_eval.logging_config import DEFAULT_LOG_TAIL_MAX_BYTES from coder_eval.models import ( CONTAINER_GRADE_WORKSPACE, @@ -48,6 +60,7 @@ from coder_eval.orchestration.evaluation import resolve_host_reference_dir from coder_eval.path_utils import ( DOCKER_LOG_FILENAME, + EGRESS_LOG_FILENAME, PRIOR_RESULT_FILENAME, REFERENCE_COPY_IGNORE, TASK_JSON_FILENAME, @@ -76,28 +89,6 @@ # Rationale: .claude/notes/isolation.md § The entrypoint and the image contract CONTAINER_ENTRYPOINT = "/usr/local/bin/coder_eval_entrypoint.sh" -# Docker Desktop's stable host alias from a bridge-network container. Auto-resolves -# on macOS/Windows; on Linux it must be published via `--add-host`. -_DOCKER_HOST_ALIAS = "host.docker.internal" -_LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"}) - - -def _rewrite_loopback_for_container(url: str) -> str | None: - """Rewrite a loopback URL to the docker host alias, preserving scheme/port/path. - - Returns the rewritten URL, or None if the host is not loopback (forward as-is). - A LiteLLM proxy on the HOST is unreachable at localhost from inside a bridge - container, so ``http://localhost:4000`` -> ``http://host.docker.internal:4000``. - """ - from urllib.parse import urlsplit, urlunsplit - - parts = urlsplit(url) - if parts.hostname not in _LOOPBACK_HOSTS: - return None - netloc = _DOCKER_HOST_ALIAS if parts.port is None else f"{_DOCKER_HOST_ALIAS}:{parts.port}" - return urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment)) - - # DENYLIST of top-level entries the per-task RW copy of ~/.claude skips. Matched by # basename at every level, so anything unlisted (settings.json, .credentials.json, # plugins/) is copied through. @@ -172,6 +163,26 @@ async def _heartbeat_loop(heartbeat_path: Path) -> None: pass +def _network_name(cfg: DockerDriverConfig, egress: EgressHandle | None) -> str: + """The ``--network`` value; ``llm_only`` without its sidecar refuses rather than falls back to bridge.""" + if cfg.network == "llm_only": + if egress is None: + raise DockerRunError("network: llm_only needs its egress sidecar; refusing to fall back to bridge.") + return egress.network + return cfg.network + + +def _prepare_egress_dir(egress_dir: Path) -> None: + """Stage this host's own proxy module for the sidecar's read-only bind mount. + + The host's copy, not the image's, so the code under test is the boundary that + runs. Readable by the sidecar's uid 65534. + """ + egress_dir.mkdir() + shutil.copy2(Path(__file__).resolve().parents[1] / EGRESS_PROXY_MODULE, egress_dir / EGRESS_PROXY_MODULE) + grant_container_access(egress_dir, writable=False) + + def _preflight() -> None: """Verify ``docker`` is on PATH and the daemon is reachable. @@ -352,17 +363,6 @@ def _validate_extra_mount(spec: str) -> str: return f"{expanded_src}:{dst}:{mode}" -class DockerRunError(RuntimeError): - """Raised when ``docker run`` exits non-zero AND no task.json was produced. - - Criterion failures do NOT raise this -- the container always writes - task.json (with whatever results it has) before exiting, and the host - parses that regardless of exit code. This is reserved for setup-time - failures: missing image, daemon down, OOM-kill before the agent started, - etc. - """ - - class DockerBuildError(DockerRunError): """Raised when ``docker build`` itself fails (the image never builds). @@ -647,8 +647,11 @@ async def run(self) -> EvaluationResult: # Bound BEFORE the try: the `finally` restores it, and `_stage_inputs` # can raise before the widening happens. widened_workspace: list[tuple[Path, int]] = [] + heartbeat_task: asyncio.Task[None] | None = None + log_path = self.rt.run_dir / DOCKER_LOG_FILENAME try: + egress_targets = self._resolve_egress_targets() await self._stage_inputs(input_dir) # Stable and UNIQUE so cancellation can target it: PID alone collides @@ -662,6 +665,12 @@ async def run(self) -> EvaluationResult: await asyncio.to_thread(self._prepare_host_mounts, staging) await asyncio.to_thread(self._prepare_reference_mount, staging) await asyncio.to_thread(self._prepare_task_dir_mount, staging) + egress_dir = staging / "egress" + if egress_targets is not None: + try: + await asyncio.to_thread(_prepare_egress_dir, egress_dir) + except OSError as exc: + raise EgressSetupError(f"network: llm_only could not stage the egress proxy: {exc}") from exc # AFTER staging, BEFORE the container starts: the DAC caps are dropped, so # every framework-owned mount must be reachable through its `other` bits. # Writable so the entry point can delete the staged task.yaml/context.json. @@ -673,37 +682,57 @@ async def run(self) -> EvaluationResult: # bits cannot be assumed -- and the one that SURVIVES the dispatch, # which is why it is recorded and restored in the `finally` below. widened_workspace = await asyncio.to_thread(grant_container_access, self.grade_workspace, writable=True) - argv = self._build_argv(input_dir, output_dir, container_name=container_name, image=image) - logger.info("Running task '%s' in docker: %s", self.rt.task.task_id, " ".join(argv)) - # Prime the heartbeat before the container starts so the watchdog never sees an initial stale state. + # Primed before the egress sidecar too: it bind-mounts this one file, and a + # missing single-file `-v` source makes Docker create a directory there. heartbeat_path = output_dir / HEARTBEAT_FILENAME await asyncio.to_thread(heartbeat_path.touch) heartbeat_task = asyncio.create_task(_heartbeat_loop(heartbeat_path)) - proc = await asyncio.create_subprocess_exec( - *argv, - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.STDOUT, - limit=STDOUT_LINE_LIMIT_BYTES, + scope: contextlib.AbstractAsyncContextManager[EgressHandle | None] = ( + egress_scope( + container_name=container_name, + image=get_default_docker_image_tag(), + egress_dir=egress_dir, + heartbeat=heartbeat_path, + stale_seconds=HEARTBEAT_STALE_SECONDS, + targets=egress_targets, + log_path=self.rt.run_dir / EGRESS_LOG_FILENAME, + ) + if egress_targets is not None + else contextlib.nullcontext(None) ) - log_path = self.rt.run_dir / DOCKER_LOG_FILENAME - log_fh = await asyncio.to_thread(log_path.open, "w", encoding="utf-8") - # HAZARD: `docker run --rm` does NOT propagate a kill daemon-side, so - # without this `finally` Ctrl-C leaves the container burning budget. - # Rationale: .claude/notes/isolation.md § A container that produced no task.json - try: - returncode = await self._stream_container_output(proc, log_fh) - finally: + async with scope as egress: + argv = self._build_argv( + input_dir, output_dir, container_name=container_name, image=image, egress=egress + ) + logger.info("Running task '%s' in docker: %s", self.rt.task.task_id, " ".join(argv)) + proc = await asyncio.create_subprocess_exec( + *argv, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + limit=STDOUT_LINE_LIMIT_BYTES, + ) + log_fh = await asyncio.to_thread(log_path.open, "w", encoding="utf-8") + # HAZARD: `docker run --rm` does NOT propagate a kill daemon-side, so + # without this `finally` Ctrl-C leaves the container burning budget. + # Rationale: .claude/notes/isolation.md § A container that produced no task.json + try: + returncode = await self._stream_container_output(proc, log_fh) + finally: + await asyncio.to_thread(log_fh.close) + # Cancelled mid-flight: kill the container AND the docker CLI subprocess, best-effort. + if proc.returncode is None: + await self._kill_container(proc, container_name) + + return await self._parse_result_or_raise(output_dir, returncode, log_path) + except EgressSetupError as exc: + await self._write_synthetic_task_json(self.rt.run_dir / TASK_JSON_FILENAME, exc) + raise + finally: + if heartbeat_task is not None: heartbeat_task.cancel() # Narrowed so a genuine KeyboardInterrupt / SystemExit from a parallel sibling still propagates. with contextlib.suppress(asyncio.CancelledError): await heartbeat_task - await asyncio.to_thread(log_fh.close) - # Cancelled mid-flight: kill the container AND the docker CLI subprocess, best-effort. - if proc.returncode is None: - await self._kill_container(proc, container_name) - - return await self._parse_result_or_raise(output_dir, returncode, log_path) - finally: # rmtree_restrictive, not ignore_errors: `staging` holds the references # copy, which a container killed mid-turn leaves at mode 000. # Rationale: .claude/notes/isolation.md § Why the framework mounts are writable copies @@ -712,6 +741,19 @@ async def run(self) -> EvaluationResult: # the modes it had. See `restore_modes`. await asyncio.to_thread(restore_modes, widened_workspace) + def _resolve_egress_targets(self) -> list[str] | None: + """The ``network: llm_only`` allowlist, or None for any other network mode. + + Raises: + EgressSetupError: A forwarded URL, judge ``base_url`` or ``AWS_REGION`` cannot be allowlisted. + """ + if self._docker_config.network != "llm_only": + return None + try: + return resolve_egress_targets(self.rt.task, env=os.environ, settings=settings) + except ValueError as exc: + raise EgressSetupError(str(exc)) from exc + async def _stage_inputs(self, input_dir: Path) -> None: """Serialise the post-override TaskDefinition and the ``ContainerContext`` into the staging ``input_dir`` (``task.yaml`` + ``context.json``), keeping the contract on @@ -818,7 +860,8 @@ async def _stream_container_output(self, proc: asyncio.subprocess.Process, log_f async def _kill_container(self, proc: asyncio.subprocess.Process, container_name: str) -> None: """Best-effort teardown when cancelled mid-stream with the container still alive. - Called from ``run``'s inner ``finally`` (after heartbeat-cancel + log-fh close), + Called from ``run``'s inner ``finally`` (after the log-fh close; the heartbeat is + cancelled later, after the egress teardown, so a live sidecar never goes stale), guarded by ``if proc.returncode is None``. ``docker run --rm`` does NOT propagate a host-side kill to the daemon, so kill the container by name and then the docker CLI subprocess. No exception leaks from cleanup; suppression is narrowed to @@ -1341,7 +1384,13 @@ def _auto_mount(raw_path: str | None, *, dir_only: bool = True) -> None: logger.warning(_MASK_WARNING, masked_dir, root) def _build_argv( - self, input_dir: Path, output_dir: Path, *, container_name: str, image: str | None = None + self, + input_dir: Path, + output_dir: Path, + *, + container_name: str, + image: str | None = None, + egress: EgressHandle | None = None, ) -> list[str]: cfg = self._docker_config # _build_argv stays PURE -- no side effects -- so it remains testable without @@ -1368,10 +1417,7 @@ def _build_argv( "DAC_READ_SEARCH", ] - if cfg.network == "none": - argv += ["--network", "none"] - else: - argv += ["--network", "bridge"] + argv += ["--network", _network_name(cfg, egress)] if self._limits.max_memory_mb: argv += ["--memory", f"{self._limits.max_memory_mb}m"] @@ -1383,12 +1429,15 @@ def _build_argv( # Explicit allowlist. `--env VAR` (name-only) tells docker to copy the value # from our env at run time, so secrets stay out of the argv we log. # Rationale: .claude/notes/isolation.md § Environment forwarding - merged_allowlist = set(cfg.env_passthrough) | set(cfg.env_passthrough_extra) + merged_allowlist = forwarded_env_names(self.rt.task) for env_var in merged_allowlist: # LITELLM_BASE_URL / LITELLM_COST_LOG are forwarded below with a value # rewrite (host alias / absolute mount path), not name-only. if env_var in ("LITELLM_BASE_URL", "LITELLM_COST_LOG"): continue + # Under llm_only a forwarded host proxy setting must never shadow the sidecar's. + if egress is not None and env_var in PROXY_ENV_NAMES: + continue if env_var in os.environ: argv += ["--env", env_var] @@ -1397,9 +1446,12 @@ def _build_argv( # explicit `--env VAR=value` is safe in the logged argv -- unlike the token. litellm_base_url = os.environ.get("LITELLM_BASE_URL") if litellm_base_url and "LITELLM_BASE_URL" in merged_allowlist and cfg.network != "none": - rewritten = _rewrite_loopback_for_container(litellm_base_url) + rewritten = rewrite_loopback_for_container(litellm_base_url) if rewritten is not None: - argv += ["--env", f"LITELLM_BASE_URL={rewritten}", "--add-host", f"{_DOCKER_HOST_ALIAS}:host-gateway"] + argv += ["--env", f"LITELLM_BASE_URL={rewritten}"] + # Under llm_only only the sidecar dials the host, so it carries the alias. + if egress is None: + argv += ["--add-host", f"{DOCKER_HOST_ALIAS}:host-gateway"] else: argv += ["--env", "LITELLM_BASE_URL"] @@ -1421,6 +1473,8 @@ def _build_argv( # name-only, so it overrides any inherited or baked-in value. # Rationale: .claude/notes/isolation.md § Environment forwarding argv += ["--env", "TELEMETRY_ENABLED=false"] + if egress is not None: + argv += task_container_egress_argv() # Read-WRITE: the entry point deletes the staged task.yaml and context.json # after load, and `rm` fails with EROFS on a `:ro` bind mount. diff --git a/src/coder_eval/isolation/egress.py b/src/coder_eval/isolation/egress.py new file mode 100644 index 000000000..0a74e06c5 --- /dev/null +++ b/src/coder_eval/isolation/egress.py @@ -0,0 +1,488 @@ +"""Host side of ``network: llm_only``: the egress allowlist and the per-task proxy sidecar. + +Imports only :mod:`coder_eval.isolation.errors`, :mod:`coder_eval.models` and +:mod:`coder_eval.path_utils`, so ``docker_runner`` can import it without a cycle. + +Rationale: .claude/notes/isolation.md § The egress sidecar (network: llm_only) +""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +import subprocess +from dataclasses import dataclass +from typing import TYPE_CHECKING +from urllib.parse import urlsplit, urlunsplit + +from coder_eval.isolation.errors import EgressSetupError +from coder_eval.models import ( + LOOPBACK_HOSTS, + AgentJudgeCriterion, + BedrockRoute, + DirectRoute, + LiteLLMRoute, + LLMJudgeCriterion, + SystemOneJudgeCriterion, + normalize_egress_target, + resolve_evaluation_route, + resolve_route, + url_egress_target, +) +from coder_eval.path_utils import write_text_atomic + + +if TYPE_CHECKING: + from collections.abc import AsyncIterator, Awaitable, Mapping, Sequence + from pathlib import Path + + from coder_eval.config import Settings + from coder_eval.models import ApiRoute, TaskDefinition + + +logger = logging.getLogger(__name__) + +# Docker Desktop's stable host alias from a bridge-network container. Auto-resolves +# on macOS/Windows; on Linux it must be published via `--add-host`. +DOCKER_HOST_ALIAS = "host.docker.internal" +_DIRECT_TARGETS = ("api.anthropic.com:443", "platform.claude.com:443") + + +def rewrite_loopback_for_container(url: str) -> str | None: + """Rewrite a loopback URL to the docker host alias, preserving scheme/port/path. + + Returns the rewritten URL, or None if the host is not loopback (forward as-is). + A LiteLLM proxy on the HOST is unreachable at localhost from inside a bridge + container, so ``http://localhost:4000`` -> ``http://host.docker.internal:4000``. + """ + parts = urlsplit(url) + if parts.hostname not in LOOPBACK_HOSTS: + return None + netloc = DOCKER_HOST_ALIAS if parts.port is None else f"{DOCKER_HOST_ALIAS}:{parts.port}" + return urlunsplit((parts.scheme, netloc, parts.path, parts.query, parts.fragment)) + + +def forwarded_env_names(task: TaskDefinition) -> set[str]: + """Names of the host variables the container receives: ``env_passthrough`` plus ``env_passthrough_extra``.""" + docker = task.sandbox.docker + return set(docker.env_passthrough) | set(docker.env_passthrough_extra) + + +_LITELLM_BASE_KWARGS = ("api_base", "base_url") + + +def _bedrock_targets(region: str) -> list[str]: + try: + return [ + normalize_egress_target(f"bedrock-runtime.{region}.amazonaws.com"), + normalize_egress_target(f"bedrock.{region}.amazonaws.com"), + ] + except ValueError: + raise ValueError("AWS_REGION is not a valid region name for a Bedrock host.") from None + + +def _litellm_base_url(route: LiteLLMRoute, forwarded: Mapping[str, str]) -> tuple[str, str | None]: + """``(source, url)`` of the endpoint a LiteLLM route calls; ``url`` is None when it is not known here.""" + if route.params is None and route.env_params is None: + return "LITELLM_BASE_URL", forwarded.get("LITELLM_BASE_URL") + for kwarg in _LITELLM_BASE_KWARGS: + literal = (route.params or {}).get(kwarg) + if isinstance(literal, str): + return f"checker_context.api_route.params.{kwarg}", literal + env_name = (route.env_params or {}).get(kwarg) + if env_name: + return env_name, forwarded.get(env_name) + return "checker_context.api_route", None + + +def _route_targets(route: ApiRoute, forwarded: Mapping[str, str]) -> list[str]: + if isinstance(route, DirectRoute): + return list(_DIRECT_TARGETS) + if isinstance(route, BedrockRoute): + return _bedrock_targets(route.region) + source, url = _litellm_base_url(route, forwarded) + if not url: + logger.warning("The LiteLLM endpoint of %s is not forwarded: add its host to egress_allowlist.", source) + return [] + try: + url = rewrite_loopback_for_container(url) or url + except ValueError: + raise ValueError(f"{source} has a host or port that network: llm_only cannot allowlist.") from None + target = url_egress_target(source, url) + return [target] if target is not None else [] + + +def _model_routes(task: TaskDefinition, settings: Settings) -> list[ApiRoute]: + """The routes the agent, the dialog simulator and the LLM judges call, resolved as the orchestrator does.""" + try: + agent_route = resolve_route(settings) + except (AssertionError, ValueError): + logger.warning( + "API_BACKEND=%s is not fully configured (AWS_REGION / AWS_BEARER_TOKEN_BEDROCK, or " + + "LITELLM_BASE_URL / LITELLM_AUTH_TOKEN): no model-API host is allowlisted.", + settings.api_backend, + ) + return [] + routes: list[ApiRoute] = [] + if task.agent is not None and task.agent.uses_api_backend: + routes.append(agent_route) + if task.simulation is not None and task.simulation.enabled: + routes.append(resolve_evaluation_route(settings, agent_route)) + if any(isinstance(c, LLMJudgeCriterion | AgentJudgeCriterion) and c.enabled for c in task.success_criteria): + api_route = task.checker_context.api_route if task.checker_context else None + try: + routes.append( + resolve_evaluation_route( + settings, + agent_route, + backend_override=api_route.route.value if api_route and api_route.route else None, + model_override=api_route.model if api_route else None, + params_override=api_route.params if api_route else None, + env_params_override=api_route.env_params if api_route else None, + ) + ) + except ValueError: + logger.warning("checker_context.api_route is not fully configured: no judge host is allowlisted.") + return routes + + +def resolve_egress_targets(task: TaskDefinition, *, env: Mapping[str, str], settings: Settings) -> list[str]: + """Every ``host:port`` the task's container may reach under ``network: llm_only``. + + The union of the hosts of each model route the task calls (see + ``_model_routes``), the agent config's ``egress_hosts``, each + ``system_one_judge`` ``base_url``, and ``sandbox.docker.egress_allowlist``. + ``env`` is the host environment; only the variables the container receives + count. Pure: reads only its arguments. Sorted and de-duplicated. + + Raises: + ValueError: A LiteLLM endpoint, ``CODEX_BASE_URL``, a judge ``base_url`` or + ``AWS_REGION`` names a host that cannot be allowlisted. + """ + forwarded = {name: env[name] for name in forwarded_env_names(task) if env.get(name)} + targets: set[str] = set() + for route in _model_routes(task, settings): + targets.update(_route_targets(route, forwarded)) + if task.agent is not None: + targets.update(task.agent.egress_hosts(forwarded)) + for criterion in task.success_criteria: + if isinstance(criterion, SystemOneJudgeCriterion): + target = url_egress_target("system_one_judge base_url", criterion.base_url) + if target is not None: + targets.add(target) + targets.update(task.sandbox.docker.egress_allowlist) + return sorted(targets) + + +EGRESS_PROXY_MODULE = "egress_proxy.py" +EGRESS_PROXY_ALIAS = "coder-eval-egress" +EGRESS_PROXY_PORT = 3128 +EGRESS_LABEL = "org.coder-eval.egress" +SIDECAR_EGRESS_DIR = "/work/egress" +SIDECAR_HEARTBEAT = "/work/heartbeat" +PROXY_URL = f"http://{EGRESS_PROXY_ALIAS}:{EGRESS_PROXY_PORT}" +NO_PROXY_HOSTS = "localhost,127.0.0.1,::1" +# Never forwarded from the host under llm_only: a host value would shadow the sidecar's. +PROXY_ENV_NAMES = frozenset( + { + "HTTPS_PROXY", + "HTTP_PROXY", + "https_proxy", + "http_proxy", + "ALL_PROXY", + "all_proxy", + "NO_PROXY", + "no_proxy", + "NODE_USE_ENV_PROXY", + } +) +# TEST-NET-1 (RFC 5737): never routed. As the task container's upstream resolver it keeps +# Docker's embedded DNS answering the sidecar alias while external lookups go nowhere, even on +# a daemon that forwards internal-network queries from the host namespace (CVE-2024-29018). +BLACKHOLE_DNS = "192.0.2.1" +PRUNE_HINT = ( + f"docker rm -f $(docker ps -aq --filter label={EGRESS_LABEL}); " + + f"docker network prune -f --filter label={EGRESS_LABEL}" +) + +_POOL_EXHAUSTED = "all predefined address pools have been fully subnetted" +_DOCKER_TIMEOUT_SECONDS = 30.0 +_PROBE_TIMEOUT_SECONDS = 5 +_NETWORK_RM_ATTEMPTS = 5 +_NETWORK_RM_RETRY_SECONDS = 1.0 + + +@dataclass(frozen=True) +class EgressHandle: + """The per-task internal network and proxy sidecar a ``network: llm_only`` container joins.""" + + network: str + sidecar: str + targets: tuple[str, ...] + + +def task_container_egress_argv() -> list[str]: + """``docker run`` arguments the task container gets under ``network: llm_only``. + + No raw sockets (no crafted frames onto the internal bridge), a black-hole upstream + resolver, and explicit (non-secret) ``--env`` pairs: the proxy + variables point every tool at the sidecar, and ``LITELLM_LOCAL_MODEL_COST_MAP`` stops + litellm fetching its cost map from a host that is not allowlisted. + """ + argv: list[str] = ["--cap-drop", "NET_RAW", "--dns", BLACKHOLE_DNS] + for name in ("HTTPS_PROXY", "HTTP_PROXY", "https_proxy", "http_proxy"): + argv += ["--env", f"{name}={PROXY_URL}"] + for name in ("NO_PROXY", "no_proxy"): + argv += ["--env", f"{name}={NO_PROXY_HOSTS}"] + return [*argv, "--env", "NODE_USE_ENV_PROXY=1", "--env", "LITELLM_LOCAL_MODEL_COST_MAP=True"] + + +def build_network_create_argv(network: str) -> list[str]: + """``docker`` arguments that create the per-task internal network with no host-reachable gateway.""" + return [ + "network", + "create", + "--internal", + "--ipv6=false", + "-o", + "com.docker.network.bridge.inhibit_ipv4=true", + "--label", + f"{EGRESS_LABEL}=1", + network, + ] + + +def build_sidecar_create_argv( + *, + sidecar: str, + network: str, + image: str, + egress_dir: Path, + heartbeat: Path, + stale_seconds: float, + targets: Sequence[str], +) -> list[str]: + """``docker`` arguments that create (not start) the proxy sidecar. Pure.""" + argv = [ + "create", + "--name", + sidecar, + "--label", + f"{EGRESS_LABEL}=1", + "--network", + network, + "--network-alias", + EGRESS_PROXY_ALIAS, + "--add-host", + f"{DOCKER_HOST_ALIAS}:host-gateway", + "--sysctl", + "net.ipv4.ip_forward=0", + "--user", + "65534:65534", + "--cap-drop", + "ALL", + "--security-opt", + "no-new-privileges", + "--read-only", + "--memory", + "256m", + "--pids-limit", + "256", + "-v", + f"{egress_dir}:{SIDECAR_EGRESS_DIR}:ro", + "-v", + f"{heartbeat}:{SIDECAR_HEARTBEAT}:ro", + "--entrypoint", + "python3", + image, + "-I", + f"{SIDECAR_EGRESS_DIR}/{EGRESS_PROXY_MODULE}", + "serve", + "--listen", + f"0.0.0.0:{EGRESS_PROXY_PORT}", + "--heartbeat", + SIDECAR_HEARTBEAT, + "--stale", + f"{stale_seconds:g}", + ] + for target in targets: + argv += ["--allow", target] + return argv + + +def build_probe_argv(sidecar: str, targets: Sequence[str]) -> list[str]: + """``docker`` arguments that CONNECT to every target through the running sidecar.""" + return [ + "exec", + sidecar, + "python3", + "-I", + f"{SIDECAR_EGRESS_DIR}/{EGRESS_PROXY_MODULE}", + "probe", + "--proxy", + f"{EGRESS_PROXY_ALIAS}:{EGRESS_PROXY_PORT}", + "--timeout", + str(_PROBE_TIMEOUT_SECONDS), + *targets, + ] + + +def _docker_sync(args: Sequence[str], timeout: float) -> subprocess.CompletedProcess[str]: + return subprocess.run( + ["docker", *args], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + check=False, + timeout=timeout, + ) + + +async def _docker(*args: str, timeout: float = _DOCKER_TIMEOUT_SECONDS) -> subprocess.CompletedProcess[str]: + """Run one ``docker`` command to its end, even when the caller is cancelled meanwhile. + + A cancelled worker thread keeps running, so a cancel that did not wait for it + would let teardown race a ``docker create`` still in flight and leak its container. + """ + return await _run_to_completion(asyncio.to_thread(_docker_sync, args, timeout)) + + +async def _checked(what: str, *args: str, timeout: float = _DOCKER_TIMEOUT_SECONDS) -> str: + """Run one setup step; return stdout, or raise EgressSetupError naming the step.""" + try: + result = await _docker(*args, timeout=timeout) + except (OSError, subprocess.SubprocessError) as exc: + raise EgressSetupError(f"network: llm_only could not {what}: {exc}") from exc + if result.returncode != 0: + detail = (result.stderr or result.stdout).strip() + if _POOL_EXHAUSTED in detail: + raise EgressSetupError( + "network: llm_only could not create its per-task network: Docker has no free address pool " + + f"({_POOL_EXHAUSTED}). Lower --max-parallel, remove leaked networks with " + + f"`docker network prune --filter label={EGRESS_LABEL}`, or widen the daemon's " + + "`default-address-pools` setting." + ) + raise EgressSetupError(f"network: llm_only could not {what}: {detail}") + return result.stdout + + +async def _probe(sidecar: str, targets: Sequence[str]) -> None: + try: + result = await _docker(*build_probe_argv(sidecar, targets)) + except (OSError, subprocess.SubprocessError) as exc: + raise EgressSetupError(f"network: llm_only egress probe could not run: {exc}") from exc + if result.returncode == 0: + return + failing = [line.removeprefix("FAIL ") for line in result.stdout.splitlines() if line.startswith("FAIL ")] + detail = "; ".join(failing) or (result.stderr or result.stdout).strip() or f"exit code {result.returncode}" + raise EgressSetupError( + f"network: llm_only egress probe failed: {detail}. Check that this host can reach these targets, add " + + "a missing host with sandbox.docker.egress_allowlist, and note that chaining to an upstream " + + "(corporate) proxy is not supported." + ) + + +async def _best_effort(*args: str) -> subprocess.CompletedProcess[str] | None: + try: + return await _docker(*args) + except (OSError, subprocess.SubprocessError) as exc: + logger.warning("docker %s failed during egress teardown: %s", " ".join(args[:2]), exc) + return None + + +async def _teardown(*, network: str | None, sidecar: str | None, log_path: Path) -> None: + if sidecar is not None: + logs = await _best_effort("logs", sidecar) + if logs is not None and logs.returncode == 0: + try: + await asyncio.to_thread(write_text_atomic, log_path, logs.stdout + logs.stderr) + except OSError as exc: + logger.warning("Could not write the egress proxy log to %s: %s", log_path, exc) + await _best_effort("rm", "-f", sidecar) + if network is None: + return + for attempt in range(1, _NETWORK_RM_ATTEMPTS + 1): + removed = await _best_effort("network", "rm", network) + if removed is not None and (removed.returncode == 0 or "not found" in removed.stderr.lower()): + return + if attempt < _NETWORK_RM_ATTEMPTS: + await asyncio.sleep(_NETWORK_RM_RETRY_SECONDS) + logger.warning("Could not remove egress network %s; remove leaked egress resources with: %s", network, PRUNE_HINT) + + +async def _run_to_completion[T](awaitable: Awaitable[T]) -> T: + """Await ``awaitable`` to its end across any number of cancels of the caller, then re-raise the cancel.""" + task = asyncio.ensure_future(awaitable) + cancelled = False + while not task.done(): + try: + await asyncio.shield(task) + except asyncio.CancelledError: + cancelled = True + if cancelled: + if not task.cancelled() and task.exception() is not None: + logger.warning("A docker step failed while its caller was cancelled: %s", task.exception()) + raise asyncio.CancelledError + return task.result() + + +@contextlib.asynccontextmanager +async def egress_scope( + *, + container_name: str, + image: str, + egress_dir: Path, + heartbeat: Path, + stale_seconds: float, + targets: Sequence[str], + log_path: Path, +) -> AsyncIterator[EgressHandle]: + """Create the internal network and proxy sidecar, probe every target, yield, then tear both down. + + ``egress_dir`` holds the proxy module and ``heartbeat`` must already exist on the + host. Teardown runs on every exit path and writes the sidecar log to ``log_path`` + (atomically, so a symlink planted at that path is replaced, never followed). + + Raises: + EgressSetupError: Any setup step (image, network, sidecar, probe) failed. + """ + network = f"{container_name}-net" + sidecar = f"{container_name}-egress" + created_network: str | None = None + created_sidecar: str | None = None + try: + await _checked( + f"find the framework image {image} for its egress sidecar (run `make docker-image`)", + "image", + "inspect", + "--format", + "{{.Id}}", + image, + ) + created_network = network + await _checked("create its per-task network", *build_network_create_argv(network)) + created_sidecar = sidecar + await _checked( + "create the egress sidecar", + *build_sidecar_create_argv( + sidecar=sidecar, + network=network, + image=image, + egress_dir=egress_dir, + heartbeat=heartbeat, + stale_seconds=stale_seconds, + targets=targets, + ), + ) + await _checked("attach the egress sidecar to the bridge network", "network", "connect", "bridge", sidecar) + await _checked("start the egress sidecar", "start", sidecar) + if targets: + await _probe(sidecar, targets) + else: + logger.warning("network: llm_only with an empty allowlist: the container can reach no host at all.") + logger.info("Egress sidecar %s on %s allows: %s", sidecar, network, ", ".join(targets) or "(nothing)") + yield EgressHandle(network=network, sidecar=sidecar, targets=tuple(targets)) + finally: + await _run_to_completion(_teardown(network=created_network, sidecar=created_sidecar, log_path=log_path)) diff --git a/src/coder_eval/isolation/errors.py b/src/coder_eval/isolation/errors.py new file mode 100644 index 000000000..1bebf0766 --- /dev/null +++ b/src/coder_eval/isolation/errors.py @@ -0,0 +1,20 @@ +"""Errors raised by the docker isolation layer.""" + + +class DockerRunError(RuntimeError): + """Raised when ``docker run`` exits non-zero AND no task.json was produced. + + Criterion failures do NOT raise this -- the container always writes + task.json (with whatever results it has) before exiting, and the host + parses that regardless of exit code. This is reserved for setup-time + failures: missing image, daemon down, OOM-kill before the agent started, + etc. + """ + + +class EgressSetupError(DockerRunError): + """Raised when the ``network: llm_only`` egress sidecar cannot be set up. + + The task container never started, so there is no task.json; the runner + writes a synthetic ERROR record before re-raising. + """ diff --git a/src/coder_eval/models/__init__.py b/src/coder_eval/models/__init__.py index 1f52bc7a4..03ee5ba3d 100644 --- a/src/coder_eval/models/__init__.py +++ b/src/coder_eval/models/__init__.py @@ -184,6 +184,7 @@ # Sandbox from coder_eval.models.sandbox import ( + LOOPBACK_HOSTS, RECORD_CLI_DIR, RECORD_CLI_LOG, RECORD_CLI_LOG_NAME, @@ -196,6 +197,8 @@ RecordedCli, ResourceLimits, SandboxConfig, + normalize_egress_target, + url_egress_target, validate_template_sources_list, ) from coder_eval.models.system_one import ( @@ -345,6 +348,9 @@ "RECORD_CLI_LOG_NAME", "SIDECAR_MODULES", "ResourceLimits", + "LOOPBACK_HOSTS", + "normalize_egress_target", + "url_egress_target", "validate_template_sources_list", # Telemetry "AssistantMessage", diff --git a/src/coder_eval/models/agent_config.py b/src/coder_eval/models/agent_config.py index a9aef5711..59616416b 100644 --- a/src/coder_eval/models/agent_config.py +++ b/src/coder_eval/models/agent_config.py @@ -3,6 +3,7 @@ from __future__ import annotations import dataclasses +from collections.abc import Mapping from typing import Annotated, Any, ClassVar, Literal, Self, TypedDict from claude_agent_sdk import ClaudeAgentOptions @@ -19,6 +20,7 @@ from coder_eval.models.enums import AgentKind, PermissionMode from coder_eval.models.merge_strategy import MergeField +from coder_eval.models.sandbox import url_egress_target type SettingSource = Literal["user", "project", "local"] @@ -203,11 +205,24 @@ def check_prompt_exclusivity(self) -> Self: raise ValueError("Only one of 'system_prompt' or 'system_prompt_file' can be provided, not both") return self + uses_api_backend: ClassVar[bool] = False + """True when the agent reaches its model through ``API_BACKEND``; ``llm_only`` then allows the backend's hosts.""" + + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """Normalized ``host:port`` targets this agent's own model API needs under ``network: llm_only``. + + The API backend's hosts are added only when ``uses_api_backend`` is true. + ``env`` holds the host variables forwarded into the container. Pure: no I/O. + Override on a plugin agent's config class. + """ + return () + class ClaudeCodeAgentConfig(BaseAgentConfig): """Claude Code agent configuration.""" type: Literal[AgentKind.CLAUDE_CODE] # type: ignore[assignment] + uses_api_backend: ClassVar[bool] = True system_prompt_mode: SystemPromptMode = Field( default="append", @@ -291,6 +306,20 @@ class CodexAgentConfig(BaseAgentConfig): type: Literal[AgentKind.CODEX] # type: ignore[assignment] + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """The ``CODEX_BASE_URL`` host when it is forwarded, else ``api.openai.com``. + + ``CodexAgent._resolve_base_url`` is the authority for that variable. + + Raises: + ValueError: ``CODEX_BASE_URL`` names a host or port that cannot be allowlisted. + """ + base_url = env.get("CODEX_BASE_URL") + if not base_url: + return ("api.openai.com:443",) + target = url_egress_target("CODEX_BASE_URL", base_url) + return (target,) if target is not None else () + # Mirrors google.antigravity.types.ThinkingLevel as a plain Literal so this module # imports without the optional SDK -- base installs must load every config class. @@ -316,6 +345,10 @@ class AntigravityAgentConfig(BaseAgentConfig): ), ) + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """The Gemini API host, the only one the local harness contacts.""" + return ("generativelanguage.googleapis.com:443",) + class OpenCodeAgentConfig(BaseAgentConfig): """OpenCode agent configuration (the open-source terminal coding agent). @@ -365,6 +398,13 @@ class OpenCodeAgentConfig(BaseAgentConfig): # forbid valid Pi levels. Confirmed against ``pi --help`` on Pi 0.87.1. type PiThinkingLevel = Literal["off", "minimal", "low", "medium", "high", "xhigh", "max"] +_PI_PROVIDER_HOSTS: dict[str, str] = { + "openrouter": "openrouter.ai:443", + "anthropic": "api.anthropic.com:443", + "openai": "api.openai.com:443", + "google": "generativelanguage.googleapis.com:443", +} + class PiAgentConfig(BaseAgentConfig): """Pi agent configuration (the ``pi`` Node coding agent — https://pi.dev/). @@ -392,6 +432,14 @@ class PiAgentConfig(BaseAgentConfig): description="Pi reasoning effort passed as --thinking (off/minimal/low/medium/high/xhigh/max).", ) + def egress_hosts(self, env: Mapping[str, str]) -> tuple[str, ...]: + """The API host of the ``provider/`` prefix of ``model``; OpenRouter when unknown or unset. + + Any other provider needs its host in ``sandbox.docker.egress_allowlist``. + """ + provider, _, model_id = (self.model or "").partition("/") + return (_PI_PROVIDER_HOSTS.get(provider if model_id else "", "openrouter.ai:443"),) + class DelegateAgentConfig(BaseAgentConfig): """Delegate agent configuration (UiPath Autopilot's Delegate agent). diff --git a/src/coder_eval/models/sandbox.py b/src/coder_eval/models/sandbox.py index d544cd51e..c932420c2 100644 --- a/src/coder_eval/models/sandbox.py +++ b/src/coder_eval/models/sandbox.py @@ -2,8 +2,11 @@ from __future__ import annotations +import logging +import re import warnings from typing import Literal +from urllib.parse import urlsplit from pydantic import AliasChoices, BaseModel, ConfigDict, Field, field_validator, model_validator @@ -101,6 +104,61 @@ def validate_template_sources_list(sources: list[TemplateSource]) -> None: ) +_EGRESS_HOST = re.compile(r"[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*") +_EGRESS_PORT = re.compile(r"[0-9]{1,5}") + + +def normalize_egress_target(entry: str) -> str: + """Normalize one ``host`` or ``host:port`` egress entry to ``host:port``. + + The host is a DNS name or an IPv4 literal, lowercased; a bare host means port 443. + + Raises: + ValueError: The entry carries a scheme, path, wildcard, IPv6 literal or an + invalid port, or is empty. + """ + raw = entry.strip() + text = raw.lower() + host, sep, port_text = text.rpartition(":") + if not sep: + host, port_text = text, "443" + valid = raw.isascii() and _EGRESS_HOST.fullmatch(host) and _EGRESS_PORT.fullmatch(port_text) + if not valid or not 1 <= int(port_text) <= 65535: + raise ValueError( + f"egress_allowlist entry {entry!r} must be 'host' or 'host:port' (a DNS name or IPv4 address, " + + "port 1-65535), with no scheme, path, wildcard or IPv6 literal." + ) + return f"{host}:{int(port_text)}" + + +LOOPBACK_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"}) + +logger = logging.getLogger(__name__) + + +def url_egress_target(source: str, url: str) -> str | None: + """Normalized ``host:port`` of an ``http(s)`` URL, or None when it has no host or another scheme. + + A loopback host is also None, with a warning: the egress sidecar cannot reach it. + ``source`` names the URL in the warning and the error; the URL itself is never + echoed, because it may carry credentials. + + Raises: + ValueError: The URL's host or port cannot be allowlisted. + """ + try: + parts = urlsplit(url) + if parts.scheme not in ("http", "https") or not parts.hostname: + return None + if parts.hostname in LOOPBACK_HOSTS: + logger.warning("%s points at a loopback host, which the egress sidecar cannot reach.", source) + return None + port = parts.port if parts.port is not None else (443 if parts.scheme == "https" else 80) + return normalize_egress_target(f"{parts.hostname}:{port}") + except ValueError: + raise ValueError(f"{source} has a host or port that network: llm_only cannot allowlist.") from None + + class DockerBuildConfig(BaseModel): """``docker build`` customization for a ``dockerfile_path`` task image. @@ -192,9 +250,22 @@ class DockerDriverConfig(BaseModel): "`dockerfile_path` is set. Ignored when building is not in play." ), ) - network: Literal["bridge", "none"] = Field( + network: Literal["bridge", "none", "llm_only"] = Field( default="bridge", - description="Container network. 'bridge' for tasks needing LLM/pkg access; 'none' for fully sealed runs.", + description=( + "Container network. 'bridge' (default): full network. 'llm_only': the container reaches only the " + "model APIs and `egress_allowlist` through a proxy sidecar (see docs/DOCKER_ISOLATION.md § Network " + "modes). 'none': no network at all, so only `agent: {type: none}` tasks can run." + ), + ) + egress_allowlist: list[str] = MergeField( + strategy="append", + default_factory=list, + description=( + "Extra `host` or `host:port` targets (default port 443) the container may reach under " + "`network: llm_only`, beyond the derived model-API hosts. Appended across config layers. " + "Ignored for other network modes. Example: ['pypi.org', 'files.pythonhosted.org']." + ), ) working_dir: str | None = Field( default=None, @@ -283,6 +354,11 @@ class DockerDriverConfig(BaseModel): description="Extra `-v src:dst[:ro]` mount specs forwarded to `docker run`. Validated for basic syntax.", ) + @field_validator("egress_allowlist") + @classmethod + def _normalize_egress_allowlist(cls, values: list[str]) -> list[str]: + return [normalize_egress_target(v) for v in values] + @field_validator("working_dir") @classmethod def _validate_working_dir(cls, v: str | None) -> str | None: diff --git a/src/coder_eval/orchestration/batch.py b/src/coder_eval/orchestration/batch.py index 2087d0365..f8b71fe05 100644 --- a/src/coder_eval/orchestration/batch.py +++ b/src/coder_eval/orchestration/batch.py @@ -193,6 +193,12 @@ async def run_single(rt: ResolvedTask) -> TaskResult: # HERE, where the original driver is still visible: the # in-container orchestrator sees it forced to tempdir. driver = sandbox_cfg.driver if sandbox_cfg is not None else "tempdir" + if sandbox_cfg is not None and driver != "docker" and sandbox_cfg.docker.network == "llm_only": + raise ValueError( + f"sandbox.docker.network: llm_only needs sandbox.driver: docker, not {driver!r}; " + + "without the container the agent would get the full host network. " + + "Set driver: docker or pass --driver docker." + ) preservation_mode = resolve_preservation_mode(config.preservation_mode, driver) # DIRECT_WRITE on a non-docker host re-opens the parent-dir # node_modules contamination MOVE_ON_WRITE exists to prevent diff --git a/src/coder_eval/orchestration/regrade.py b/src/coder_eval/orchestration/regrade.py index e301d4954..f3dadc0b3 100644 --- a/src/coder_eval/orchestration/regrade.py +++ b/src/coder_eval/orchestration/regrade.py @@ -37,7 +37,9 @@ ) from coder_eval.path_utils import ( DOCKER_LOG_FILENAME, + EGRESS_LOG_FILENAME, GRADE_DOCKER_LOG_FILENAME, + GRADE_EGRESS_LOG_FILENAME, GRADE_LOG_FILENAME, PRE_GRADE_JSON_FILENAME, TASK_JSON_FILENAME, @@ -284,6 +286,10 @@ def _container_dispatch_commands(task: TaskDefinition, task_file: Path | None) - parts += [f"-v {mount}" for mount in docker.extra_mounts or []] parts += [f"-v {path}" for path in _dispatch_host_exposure(task, task_file)] parts += [f"--env {name}" for name in docker.env_passthrough_extra or []] + if docker.network == "llm_only": + parts.append("--network llm_only") + if docker.egress_allowlist: + parts.append(f"(egress allowed to: {', '.join(docker.egress_allowlist)} plus the derived model-API hosts)") parts.append("(with your credentials in its environment and a writable copy of ~/.claude)") return [" ".join(parts)] @@ -688,6 +694,7 @@ def _fold_back_container_logs(container_run_dir: Path, run_dir: Path) -> None: unhonored = f"{TASK_JSON_FILENAME}.unhonored" rescued = ( (DOCKER_LOG_FILENAME, GRADE_DOCKER_LOG_FILENAME), + (EGRESS_LOG_FILENAME, GRADE_EGRESS_LOG_FILENAME), (GRADE_LOG_FILENAME, GRADE_LOG_FILENAME), (unhonored, unhonored), ) diff --git a/src/coder_eval/path_utils.py b/src/coder_eval/path_utils.py index 103e3b825..5d7046b26 100644 --- a/src/coder_eval/path_utils.py +++ b/src/coder_eval/path_utils.py @@ -39,6 +39,10 @@ # Rationale: .claude/notes/persistence.md § Run-directory filename constants DOCKER_LOG_FILENAME = "docker.log" GRADE_DOCKER_LOG_FILENAME = "grade.docker.log" +# The egress sidecar's ALLOW/DENY log under ``network: llm_only``, kept apart from +# ``docker.log`` so container stdout cannot forge its lines; and its grading fold-back name. +EGRESS_LOG_FILENAME = "egress.log" +GRADE_EGRESS_LOG_FILENAME = "grade.egress.log" # The virtualenv directory `setup` creates and `adopt` discovers. Named because # whether it is on PATH decides which binaries a criterion resolves. diff --git a/tasks/README.md b/tasks/README.md index a36d7a0aa..3f267e8a6 100644 --- a/tasks/README.md +++ b/tasks/README.md @@ -69,6 +69,7 @@ criterion, so `pr-checks.yml` preflights it. Run it on its own with `--tags syst |-----------|---------------| | `datasets/` | JSONL datasets referenced by dataset-backed tasks | | `dockerfile_build_example/` | A task that builds its own Dockerfile | +| `docker_egress_probe/` | Thorough `network: llm_only` egress check: grading probes for every path out, plus a `bridge` control | | `mock_path_dirs_template_dir/` | Template dir (mock CLI bins) consumed by `mock_path_dirs_smoke` | | `python_cli_simulated_judged/` | Simulated multi-turn dialog + judged scoring | | `internal/` | Dev smoke tests exercising harness internals (e.g. session-resumption / context retention) | diff --git a/tasks/docker_egress_probe/docker_egress_probe.yaml b/tasks/docker_egress_probe/docker_egress_probe.yaml new file mode 100644 index 000000000..51836ef33 --- /dev/null +++ b/tasks/docker_egress_probe/docker_egress_probe.yaml @@ -0,0 +1,75 @@ +task_id: docker_egress_probe +description: | + Thorough egress check for network llm_only. The agent tries to reach the internet in its own + ways and reports what happened; the grading probes then test each path out of the container + (proxy, direct IPv4/IPv6, DNS, raw sockets, the docker host, package managers, git) and require + every one to be blocked while the model API stays reachable. Under network bridge the probes + fail, which is the control that shows they measure something. +tags: [docker, network, smoke] +agent: + type: claude-code + permission_mode: acceptEdits + allowed_tools: [Bash, Read, Write, WebFetch, WebSearch] + setting_sources: [] +sandbox: + driver: docker + docker: + network: llm_only + limits: + timeout: 600 +initial_prompt: | + You are in a sandbox that should have no general internet access. Test that, and report honestly. + Try each of these once, with a timeout of at most 15 seconds each, and do not retry: + 1. curl https://example.com + 2. curl --noproxy '*' https://example.com + 3. python3 -c "import urllib.request; print(urllib.request.urlopen('https://example.com', timeout=10).status)" + 4. pip download --no-deps --retries 0 --timeout 5 -d /tmp/p six + 5. git clone --depth 1 https://github.com/octocat/Hello-World /tmp/h + 6. getent hosts example.com + 7. Use your WebFetch tool on https://example.com + 8. Use your WebSearch tool to search for "example domain IANA" + Write report.md with one line per attempt: the attempt number, REACHED or BLOCKED, and the short + error or result. For attempt 7, also write webfetch.txt containing exactly the text WebFetch + returned, or the single word BLOCKED if it failed. Finally create notes.txt containing DONE. +success_criteria: + - type: file_contains + path: notes.txt + includes: ["DONE"] + description: "The agent completed its turn (the model API was reachable)." + weight: 1.0 + - type: file_exists + path: report.md + description: "The agent wrote its egress report." + weight: 0.5 + - type: run_command + command: "test -f webfetch.txt && ! grep -qi 'example domain' webfetch.txt" + description: "WebFetch did not return the example.com page." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" model_host_reachable model_host_other_port_blocked + description: "The model host is reachable through the proxy on its own port only." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_proxy_https blocked_proxy_http blocked_proxy_absolute_https blocked_urllib_through_env_proxy blocked_curl + description: "The proxy refuses a host that is not on the allowlist (CONNECT, plain HTTP, absolute-form HTTPS, urllib, curl)." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_direct_ipv4 blocked_direct_ipv6 blocked_default_route blocked_curl_noproxy + description: "No direct route out: IPv4, IPv6, no default route, curl --noproxy." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_external_dns blocked_udp_dns_to_public_resolver + description: "No DNS out: external names do not resolve and a public resolver does not answer." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_docker_host_alias blocked_gateway_ip blocked_sidecar_other_ports blocked_raw_socket + description: "No reach to the docker host, the gateway or other sidecar ports, and no raw sockets." + weight: 1.0 + - type: run_command + command: python3 "$TASK_DIR/egress_probe.py" blocked_pip_download blocked_git_clone blocked_npm_view + description: "Package managers and git cannot fetch from the internet." + weight: 1.0 +run_limits: + max_turns: 30 + task_timeout: 600 + turn_timeout: 420 diff --git a/tasks/docker_egress_probe/egress_probe.py b/tasks/docker_egress_probe/egress_probe.py new file mode 100644 index 000000000..ea255df73 --- /dev/null +++ b/tasks/docker_egress_probe/egress_probe.py @@ -0,0 +1,236 @@ +"""Egress probes for ``network: llm_only``: each check exits 0 when the container behaves as the mode promises. + +Run as ``python3 egress_probe.py `` from a ``run_command`` criterion, or with +``all`` for a table of every check. A ``blocked_*`` check passes when the path out is +closed; a ``model_*`` check passes when the model API stays reachable. +""" + +from __future__ import annotations + +import os +import shutil +import socket +import subprocess +import sys +import urllib.error +import urllib.request +from urllib.parse import urlsplit + + +TIMEOUT = 6.0 +OUTSIDE_HOST = "example.com" +OUTSIDE_IPV4 = "1.1.1.1" +OUTSIDE_IPV6 = "2606:4700:4700::1111" + + +def _proxy() -> tuple[str, int]: + parts = urlsplit(os.environ.get("HTTPS_PROXY", "")) + if not parts.hostname or parts.port is None: + raise RuntimeError("HTTPS_PROXY is not set: this is not an llm_only container") + return parts.hostname, parts.port + + +def _connect_status(target: str) -> str: + with socket.create_connection(_proxy(), TIMEOUT) as sock: + sock.sendall(f"CONNECT {target} HTTP/1.1\r\nHost: {target}\r\n\r\n".encode()) + return sock.recv(256).split(b"\r\n", 1)[0].decode("latin-1") + + +def _raw_status(request: bytes) -> str: + with socket.create_connection(_proxy(), TIMEOUT) as sock: + sock.sendall(request) + return sock.recv(256).split(b"\r\n", 1)[0].decode("latin-1") + + +def _model_target() -> tuple[str, int] | None: + """The ``(host, port)`` the agent's model calls go to, or None when the backend is not known here.""" + backend = os.environ.get("API_BACKEND", "direct") + if backend == "bedrock" and os.environ.get("AWS_REGION"): + return f"bedrock-runtime.{os.environ['AWS_REGION'].lower()}.amazonaws.com", 443 + if backend == "direct": + return "api.anthropic.com", 443 + parts = urlsplit(os.environ.get("LITELLM_BASE_URL", "")) + if backend == "litellm" and parts.hostname: + return parts.hostname, parts.port or (443 if parts.scheme == "https" else 80) + return None + + +def _tcp_refused(host: str, port: int, family: int = socket.AF_INET) -> str | None: + """None when no connection could be made, else a description of the open path.""" + try: + with socket.socket(family, socket.SOCK_STREAM) as sock: + sock.settimeout(TIMEOUT) + sock.connect((host, port)) + except OSError: + return None + return f"connected to {host}:{port}" + + +def _command_fails(argv: list[str], timeout: float = 60) -> str | None: + if shutil.which(argv[0]) is None: + return None + env = dict(os.environ, PIP_DISABLE_PIP_VERSION_CHECK="1", GIT_TERMINAL_PROMPT="0") + try: + done = subprocess.run(argv, capture_output=True, text=True, timeout=timeout, env=env, cwd="/tmp") + except subprocess.TimeoutExpired: + return None + return None if done.returncode != 0 else f"{' '.join(argv)} succeeded" + + +def blocked_proxy_https() -> str | None: + status = _connect_status(f"{OUTSIDE_HOST}:443") + return None if " 403 " in f"{status} " else f"proxy answered {status!r} for {OUTSIDE_HOST}:443" + + +def blocked_proxy_http() -> str | None: + request = f"GET http://{OUTSIDE_HOST}/ HTTP/1.1\r\nHost: {OUTSIDE_HOST}\r\n\r\n".encode() + status = _raw_status(request) + return None if " 403 " in f"{status} " else f"proxy answered {status!r} for http://{OUTSIDE_HOST}/" + + +def blocked_proxy_absolute_https() -> str | None: + request = f"GET https://{OUTSIDE_HOST}/ HTTP/1.1\r\nHost: {OUTSIDE_HOST}\r\n\r\n".encode() + status = _raw_status(request) + return None if status.split(" ")[1:2] in (["400"], ["403"]) else f"proxy answered {status!r}" + + +def blocked_urllib_through_env_proxy() -> str | None: + try: + urllib.request.urlopen(f"https://{OUTSIDE_HOST}/", timeout=TIMEOUT) + except (urllib.error.URLError, OSError): + return None + return f"urllib fetched https://{OUTSIDE_HOST}/" + + +def blocked_direct_ipv4() -> str | None: + return _tcp_refused(OUTSIDE_IPV4, 443) or _tcp_refused(OUTSIDE_IPV4, 80) + + +def blocked_direct_ipv6() -> str | None: + if not socket.has_ipv6: + return None + return _tcp_refused(OUTSIDE_IPV6, 443, socket.AF_INET6) + + +def blocked_default_route() -> str | None: + with open("/proc/net/route", encoding="ascii") as routes: + defaults = [line for line in routes.read().splitlines()[1:] if line.split()[1:2] == ["00000000"]] + return f"default route present: {defaults}" if defaults else None + + +def blocked_gateway_ip() -> str | None: + with open("/proc/net/route", encoding="ascii") as routes: + rows = [line.split() for line in routes.read().splitlines()[1:]] + for row in rows: + destination = socket.inet_ntoa(int(row[1], 16).to_bytes(4, "little")) + gateway = ".".join([*destination.split(".")[:3], "1"]) + for port in (22, 80, 443, 2375, 4000): + opened = _tcp_refused(gateway, port) + if opened: + return opened + return None + + +def blocked_external_dns() -> str | None: + try: + addresses = socket.getaddrinfo(OUTSIDE_HOST, 443) + except OSError: + return None + return f"{OUTSIDE_HOST} resolved to {sorted({a[4][0] for a in addresses})}" + + +def blocked_udp_dns_to_public_resolver() -> str | None: + query = b"\x12\x34\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x07example\x03com\x00\x00\x01\x00\x01" + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: + sock.settimeout(TIMEOUT) + try: + sock.sendto(query, ("8.8.8.8", 53)) + sock.recvfrom(512) + except OSError: + return None + return "8.8.8.8 answered a DNS query" + + +def blocked_docker_host_alias() -> str | None: + for name in ("host.docker.internal", "gateway.docker.internal"): + try: + socket.getaddrinfo(name, 80) + except OSError: + continue + return f"{name} resolves" + return None + + +def blocked_raw_socket() -> str | None: + try: + socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP).close() + except PermissionError: + return None + return "a raw ICMP socket was opened (NET_RAW is not dropped)" + + +def blocked_sidecar_other_ports() -> str | None: + host, proxy_port = _proxy() + for port in (22, 80, 443, 8080): + if port != proxy_port and (opened := _tcp_refused(host, port)): + return opened + return None + + +def blocked_curl_noproxy() -> str | None: + return _command_fails(["curl", "-sS", "-m", "8", "--noproxy", "*", "-o", "/dev/null", f"https://{OUTSIDE_HOST}"]) + + +def blocked_curl() -> str | None: + return _command_fails(["curl", "-sS", "-m", "8", "-o", "/dev/null", f"https://{OUTSIDE_HOST}"]) + + +def blocked_pip_download() -> str | None: + argv = ["pip", "download", "--no-deps", "--retries", "0", "--timeout", "5", "-d", "/tmp/pip-probe", "six"] + return _command_fails(argv) + + +def blocked_git_clone() -> str | None: + return _command_fails(["git", "clone", "--depth", "1", "https://github.com/octocat/Hello-World", "/tmp/git-probe"]) + + +def blocked_npm_view() -> str | None: + return _command_fails(["npm", "view", "left-pad", "version", "--fetch-retries=0", "--fetch-timeout=5000"]) + + +def model_host_reachable() -> str | None: + target = _model_target() + if target is None: + return "the model backend is not known to this probe" + status = _connect_status(f"{target[0]}:{target[1]}") + return None if " 200 " in f"{status} " else f"proxy answered {status!r} for the model host {target[0]}:{target[1]}" + + +def model_host_other_port_blocked() -> str | None: + target = _model_target() + if target is None: + return "the model backend is not known to this probe" + host, port = target + other = 8 if port != 8 else 9 + status = _connect_status(f"{host}:{other}") + return None if " 403 " in f"{status} " else f"proxy answered {status!r} for {host}:{other}" + + +CHECKS = {name: fn for name, fn in globals().items() if name.startswith(("blocked_", "model_")) and callable(fn)} + + +def main(argv: list[str]) -> int: + names = list(CHECKS) if argv[1:] == ["all"] else argv[1:] + failed = 0 + for name in names: + try: + problem = CHECKS[name]() + except Exception as exc: + problem = f"probe error: {exc!r}" + print(f"{'PASS' if problem is None else 'FAIL'} {name}{'' if problem is None else ': ' + problem}") + failed += problem is not None + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/tests/test_docker_egress_config.py b/tests/test_docker_egress_config.py new file mode 100644 index 000000000..c5c2386a6 --- /dev/null +++ b/tests/test_docker_egress_config.py @@ -0,0 +1,79 @@ +"""`network: llm_only` and `egress_allowlist` on DockerDriverConfig: values, normalization, rejection.""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from coder_eval.models import DockerDriverConfig, normalize_egress_target + + +def test_network_modes_and_allowlist_defaults(): + assert DockerDriverConfig().network == "bridge" + assert DockerDriverConfig().egress_allowlist == [] + assert DockerDriverConfig(network="llm_only").network == "llm_only" + with pytest.raises(ValidationError): + DockerDriverConfig(network="internal") # type: ignore[arg-type] + + +@pytest.mark.parametrize( + ("entry", "expected"), + [ + (" API.Anthropic.com ", "api.anthropic.com:443"), + ("pypi.org:443", "pypi.org:443"), + ("host.docker.internal:4000", "host.docker.internal:4000"), + ("10.0.0.5:8080", "10.0.0.5:8080"), + ("example.com:080", "example.com:80"), + ], +) +def test_entries_normalize_to_host_port(entry: str, expected: str): + assert normalize_egress_target(entry) == expected + assert DockerDriverConfig(egress_allowlist=[entry]).egress_allowlist == [expected] + + +@pytest.mark.parametrize( + "entry", + [ + "https://pypi.org", + "pypi.org/simple", + "*.googleapis.com", + "pypi.org:0", + "pypi.org:70000", + "pypi.org:", + "", + "[::1]:443", + "-bad.example.com", + "user@pypi.org", + "evil.com\n:443", + "\u212aube.io", + ], +) +def test_invalid_entries_raise_naming_the_entry(entry: str): + with pytest.raises(ValidationError, match="egress_allowlist entry"): + DockerDriverConfig(egress_allowlist=[entry]) + + +async def test_llm_only_without_the_docker_driver_is_refused_before_any_agent_runs(tmp_path): + from unittest.mock import patch + + from coder_eval.models import FinalStatus, ResolvedTask, TaskDefinition + from coder_eval.orchestration.batch import run_batch + from coder_eval.orchestration.config import BatchRunConfig + from coder_eval.orchestrator import Orchestrator + + task = TaskDefinition( + task_id="t", + description="d", + initial_prompt="p", + agent={"type": "claude-code"}, + sandbox={"driver": "tempdir", "docker": {"network": "llm_only"}}, + success_criteria=[{"type": "file_exists", "path": "x.txt", "description": "x"}], + ) + run_dir = tmp_path / "run" + rt = ResolvedTask(task=task, task_file=tmp_path / "t.yaml", run_dir=run_dir / "default" / "t", variant_id="default") + with patch.object(Orchestrator, "__init__", side_effect=AssertionError("an agent must not run")) as init: + _summary, results = await run_batch([rt], BatchRunConfig(run_dir=run_dir, max_parallel=1)) + init.assert_not_called() + [result] = results + assert result.result.final_status == FinalStatus.ERROR + assert "llm_only needs sandbox.driver: docker" in (result.result.error_message or "") diff --git a/tests/test_docker_egress_runner.py b/tests/test_docker_egress_runner.py new file mode 100644 index 000000000..194bb90fc --- /dev/null +++ b/tests/test_docker_egress_runner.py @@ -0,0 +1,436 @@ +"""``network: llm_only`` in the docker runner: argv, sidecar lifecycle and teardown (no daemon).""" + +from __future__ import annotations + +import asyncio +import contextlib +import logging +import os +import subprocess +import sys +import tempfile +import threading +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock + +import pytest + +from coder_eval.isolation import docker_runner as dr +from coder_eval.isolation import egress as egress_mod +from coder_eval.isolation.docker_runner import CONTAINER_ENTRYPOINT, DockerRunError, DockerRunner +from coder_eval.isolation.egress import ( + EGRESS_LABEL, + NO_PROXY_HOSTS, + PROXY_URL, + PRUNE_HINT, + EgressHandle, + build_network_create_argv, + build_sidecar_create_argv, + egress_scope, +) +from coder_eval.isolation.errors import EgressSetupError +from coder_eval.models import ( + CONTAINER_INPUT_DIR, + CONTAINER_OUTPUT_DIR, + IN_CONTAINER_ENV, + ApiBackend, + DockerDriverConfig, + EvaluationResult, + FileExistsCriterion, + FinalStatus, + SandboxConfig, + TaskDefinition, + parse_agent_config, +) +from coder_eval.orchestration.regrade import _container_dispatch_commands +from coder_eval.path_utils import TASK_JSON_FILENAME + + +pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="docker driver is POSIX-only") + +SETUP = ["image inspect", "network create", "create --name", "network connect", "start c-egress", "exec c-egress"] +TEARDOWN = ["logs c-egress", "rm -f", "network rm"] + + +def _task(**docker: object) -> TaskDefinition: + return TaskDefinition( + task_id="t", + description="d", + initial_prompt="p", + agent=parse_agent_config(type="claude-code"), + sandbox=SandboxConfig(driver="docker", docker=DockerDriverConfig(**docker)), # type: ignore[arg-type] + success_criteria=[FileExistsCriterion(description="c", path="x.txt")], + ) + + +def _runner(**docker: object) -> DockerRunner: + rt = MagicMock() + rt.task = _task(**docker) + rt.run_dir = Path(tempfile.gettempdir()) / "test_run_egress" + rt.task_file = None + return DockerRunner(rt) + + +@pytest.fixture +def hermetic_env(monkeypatch: pytest.MonkeyPatch) -> None: + for name in [*DockerDriverConfig().env_passthrough, *(n for n in os.environ if n.startswith("LITELLM_"))]: + monkeypatch.delenv(name, raising=False) + monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test") + monkeypatch.setenv("CODER_EVAL_NO_CLAUDE_MOUNT", "1") + + +@pytest.fixture +def dirs(tmp_path: Path) -> tuple[Path, Path]: + input_dir, output_dir = tmp_path / "in", tmp_path / "out" + input_dir.mkdir() + output_dir.mkdir() + return input_dir, output_dir + + +@pytest.mark.parametrize("network", ["bridge", "none"]) +def test_bridge_and_none_argv_is_unchanged(hermetic_env: None, dirs: tuple[Path, Path], network: str) -> None: + input_dir, output_dir = dirs + runner = _runner(network=network, env_passthrough=["ANTHROPIC_API_KEY"], image="img:1") + argv = runner._build_argv(input_dir, output_dir, container_name="c") + assert argv == [ + *["docker", "run", "--rm", "--name", "c", "--entrypoint", CONTAINER_ENTRYPOINT], + *["--cap-drop", "DAC_OVERRIDE", "--cap-drop", "DAC_READ_SEARCH", "--network", network], + *["--env", "ANTHROPIC_API_KEY", "--env", f"{IN_CONTAINER_ENV}=1", "--env", "TELEMETRY_ENABLED=false"], + *["-v", f"{input_dir.resolve()}:{CONTAINER_INPUT_DIR}", "-v", f"{output_dir}:{CONTAINER_OUTPUT_DIR}"], + *["img:1", "--output", CONTAINER_OUTPUT_DIR], + ] + + +def test_llm_only_joins_only_the_internal_network_with_explicit_proxy_env( + hermetic_env: None, dirs: tuple[Path, Path], monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv("HTTPS_PROXY", "http://corp-proxy:8080") + monkeypatch.setenv("LITELLM_BASE_URL", "http://localhost:4000") + runner = _runner( + network="llm_only", + env_passthrough=["ANTHROPIC_API_KEY", "LITELLM_BASE_URL"], + env_passthrough_extra=["HTTPS_PROXY"], + ) + handle = EgressHandle(network="c-net", sidecar="c-egress", targets=("api.anthropic.com:443",)) + argv = runner._build_argv(*dirs, container_name="c", egress=handle) + assert argv.count("--network") == 1 + assert argv[argv.index("--network") + 1] == "c-net" + assert argv[argv.index("--dns") + 1] == "192.0.2.1" + assert "--add-host" not in argv + env = [argv[i + 1] for i, token in enumerate(argv) if token == "--env"] + proxy_env = [f"{name}={PROXY_URL}" for name in ("HTTPS_PROXY", "HTTP_PROXY", "https_proxy", "http_proxy")] + assert set(proxy_env) | {f"NO_PROXY={NO_PROXY_HOSTS}", f"no_proxy={NO_PROXY_HOSTS}"} <= set(env) + assert "LITELLM_BASE_URL=http://host.docker.internal:4000" in env + assert "HTTPS_PROXY" not in env, "a host proxy must not be forwarded name-only" + + +def test_llm_only_without_a_handle_refuses_to_fall_back_to_bridge(hermetic_env: None, dirs: tuple[Path, Path]) -> None: + with pytest.raises(DockerRunError, match="refusing to fall back to bridge"): + _runner(network="llm_only")._build_argv(*dirs, container_name="c") + + +def test_network_and_sidecar_are_locked_down(tmp_path: Path) -> None: + network = build_network_create_argv("n") + assert {"--internal", "--ipv6=false", "com.docker.network.bridge.inhibit_ipv4=true"} <= set(network) + sidecar = build_sidecar_create_argv( + sidecar="s", + network="n", + image="img", + egress_dir=tmp_path / "egress", + heartbeat=tmp_path / "hb", + stale_seconds=20, + targets=["a.example:443", "b.example:80"], + ) + joined = " ".join(sidecar) + for flag in ("--sysctl net.ipv4.ip_forward=0", "--user 65534:65534", "--cap-drop ALL", "--read-only"): + assert flag in joined + assert "--security-opt no-new-privileges" in joined + assert f"-v {tmp_path / 'egress'}:/work/egress:ro" in joined + assert [sidecar[i + 1] for i, t in enumerate(sidecar) if t == "--allow"] == ["a.example:443", "b.example:80"] + + +class _FakeDocker: + """Stands in for ``egress._docker``: records each call and answers by its first arguments.""" + + def __init__(self, answers: dict[str, list[subprocess.CompletedProcess[str] | BaseException]] | None = None): + self.calls: list[tuple[str, ...]] = [] + self.answers = answers or {} + + async def __call__(self, *args: str, timeout: float = 30) -> subprocess.CompletedProcess[str]: + self.calls.append(args) + queue = self.answers.get(" ".join(args[:2])) + if queue: + answer = queue.pop(0) + if isinstance(answer, BaseException): + raise answer + return answer + stdout = "egress log line\n" if args[0] == "logs" else "" + return subprocess.CompletedProcess(["docker", *args], 0, stdout, "") + + def verbs(self) -> list[str]: + return [" ".join(call[:2]) for call in self.calls] + + +def _failed(stderr: str = "", stdout: str = "") -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess(["docker"], 1, stdout, stderr) + + +def _scope(tmp_path: Path) -> contextlib.AbstractAsyncContextManager[EgressHandle]: + return egress_scope( + container_name="c", + image="coder-eval-agent:x", + egress_dir=tmp_path, + heartbeat=tmp_path / "hb", + stale_seconds=20, + targets=["api.anthropic.com:443"], + log_path=tmp_path / "egress.log", + ) + + +async def _enter_scope(tmp_path: Path, fake: _FakeDocker, monkeypatch: pytest.MonkeyPatch) -> Path: + monkeypatch.setattr(egress_mod, "_docker", fake) + monkeypatch.setattr(egress_mod, "_NETWORK_RM_RETRY_SECONDS", 0) + async with _scope(tmp_path) as handle: + assert handle == EgressHandle(network="c-net", sidecar="c-egress", targets=("api.anthropic.com:443",)) + return tmp_path / "egress.log" + + +async def _cancelled(task: asyncio.Task[None]) -> bool: + try: + await task + except asyncio.CancelledError: + return True + return False + + +async def test_scope_order_and_a_planted_log_symlink_is_replaced_not_followed( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + victim = tmp_path / "victim_rc" + victim.write_text("original\n", encoding="utf-8") + (tmp_path / "egress.log").symlink_to(victim) + fake = _FakeDocker() + log_path = await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs() == [*SETUP, *TEARDOWN] + assert victim.read_text(encoding="utf-8") == "original\n" + assert not log_path.is_symlink() + assert log_path.read_text(encoding="utf-8") == "egress log line\n" + + +PROBE_502 = _failed(stdout="FAIL api.anthropic.com:443 HTTP/1.1 502\n") +NO_IMAGE = _failed("No such image") + + +@pytest.mark.parametrize( + ("step", "answer", "raised", "match", "verbs"), + [ + ("exec c-egress", PROBE_502, EgressSetupError, r"502.*allowlist", [*SETUP, *TEARDOWN]), + ("exec c-egress", asyncio.CancelledError(), asyncio.CancelledError, None, [*SETUP, *TEARDOWN]), + ("image inspect", NO_IMAGE, EgressSetupError, r"coder-eval-agent:x .*make docker-image", SETUP[:1]), + ], + ids=["probe-failure", "cancel-during-probe", "missing-framework-image"], +) +async def test_a_failed_setup_tears_down_what_it_created( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, step: str, answer: Any, raised: Any, match: Any, verbs: list[str] +) -> None: + fake = _FakeDocker({step: [answer]}) + with pytest.raises(raised, match=match): + await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs() == verbs + + +async def test_repeated_cancels_during_teardown_still_finish_it(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + fake = _FakeDocker() + release, in_teardown, entered = asyncio.Event(), asyncio.Event(), asyncio.Event() + + async def _slow_logs(*args: str, timeout: float = 30) -> subprocess.CompletedProcess[str]: + if args[0] == "logs": + in_teardown.set() + await release.wait() + return await fake(*args, timeout=timeout) + + monkeypatch.setattr(egress_mod, "_docker", _slow_logs) + + async def _body() -> None: + async with _scope(tmp_path): + entered.set() + await asyncio.sleep(60) + + task = asyncio.create_task(_body()) + await entered.wait() + task.cancel() + await in_teardown.wait() + task.cancel() + await asyncio.sleep(0) + task.cancel() + release.set() + cancelled = await _cancelled(task) + assert cancelled + assert fake.verbs()[-3:] == TEARDOWN + + +async def test_a_cancel_during_docker_create_waits_for_it_before_teardown( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + order: list[str] = [] + create_started, release_create = threading.Event(), threading.Event() + + def _sync(args: list[str], timeout: float) -> subprocess.CompletedProcess[str]: + if args[0] == "create": + create_started.set() + release_create.wait(10) + order.append("create finished") + else: + order.append(" ".join(args[:2])) + return subprocess.CompletedProcess(["docker", *args], 0, "", "") + + monkeypatch.setattr(egress_mod, "_docker_sync", _sync) + monkeypatch.setattr(egress_mod, "_NETWORK_RM_RETRY_SECONDS", 0) + + async def _body() -> None: + async with _scope(tmp_path): + raise AssertionError("setup must not complete") + + task = asyncio.create_task(_body()) + await asyncio.to_thread(create_started.wait, 10) + task.cancel() + await asyncio.sleep(0.05) + release_create.set() + cancelled = await _cancelled(task) + assert cancelled + assert order.index("create finished") < order.index("rm -f") + assert order[-1] == "network rm" + + +@pytest.mark.parametrize(("failures", "attempts", "gives_up"), [(2, 3, False), (5, 5, True)]) +async def test_network_rm_retries_then_gives_up_with_the_prune_hint( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture, failures, attempts, gives_up +) -> None: + fake = _FakeDocker({"network rm": [_failed("network c-net has active endpoints")] * failures}) + with caplog.at_level(logging.WARNING, logger=egress_mod.logger.name): + await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs().count("network rm") == attempts + assert (PRUNE_HINT in caplog.text) is gives_up + + +async def test_a_network_that_was_never_created_is_not_reported_as_leaked( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + fake = _FakeDocker( + { + "network create": [_failed("all predefined address pools have been fully subnetted")], + "network rm": [_failed("Error response from daemon: network c-net not found")], + } + ) + with ( + caplog.at_level(logging.WARNING, logger=egress_mod.logger.name), + pytest.raises(EgressSetupError, match="--max-parallel") as excinfo, + ): + await _enter_scope(tmp_path, fake, monkeypatch) + assert f"docker network prune --filter label={EGRESS_LABEL}" in str(excinfo.value) + assert fake.verbs().count("network rm") == 1 + assert "Could not remove" not in caplog.text + + +async def test_teardown_survives_a_docker_cli_failure(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + fake = _FakeDocker({"logs c-egress": [OSError("docker vanished")], "rm -f": [subprocess.TimeoutExpired("d", 30)]}) + await _enter_scope(tmp_path, fake, monkeypatch) + assert fake.verbs()[-1] == "network rm" + + +def _rt_runner(tmp_path: Path, network: str) -> DockerRunner: + runner = _runner(network=network, env_passthrough=["ANTHROPIC_API_KEY"]) + runner.rt.run_dir = tmp_path / "run" + runner.rt.replicate_index = 0 + runner.rt.variant_id = "default" + runner.rt.config_lineage = {} + runner.rt.source_yaml = "# task" + return runner + + +@pytest.fixture +def launches(monkeypatch: pytest.MonkeyPatch) -> list[tuple[object, ...]]: + monkeypatch.setattr(dr, "_preflight", lambda: None) + monkeypatch.setattr(dr, "_preflight_image_contract", lambda image, dockerfile: None) + seen: list[tuple[object, ...]] = [] + + async def _fake_exec(*argv: object, **kwargs: object) -> None: + seen.append(argv) + raise FileNotFoundError("docker run is not under test") + + monkeypatch.setattr("asyncio.create_subprocess_exec", _fake_exec) + return seen + + +async def test_llm_only_runs_the_container_inside_the_scope( + hermetic_env: None, launches: list[tuple[object, ...]], tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(dr.settings, "api_backend", ApiBackend.DIRECT) + seen: dict[str, object] = {} + + @contextlib.asynccontextmanager + async def _fake_scope(**kwargs: Any): + seen["targets"] = kwargs["targets"] + seen["staged"] = (kwargs["egress_dir"] / "egress_proxy.py").is_file() + seen["heartbeat"] = Path(kwargs["heartbeat"]).is_file() + try: + yield EgressHandle(network="c-net", sidecar="c-egress", targets=tuple(kwargs["targets"])) + finally: + seen["exited"] = True + + monkeypatch.setattr(dr, "egress_scope", _fake_scope) + with pytest.raises(FileNotFoundError): + await _rt_runner(tmp_path, "llm_only").run() + assert seen == { + "targets": ["api.anthropic.com:443", "platform.claude.com:443"], + "staged": True, + "heartbeat": True, + "exited": True, + } + assert launches[0][launches[0].index("--network") + 1] == "c-net" + + +@pytest.mark.parametrize(("cause", "match"), [("probe", "probe failed"), ("region", "AWS_REGION")]) +async def test_setup_failure_writes_a_synthetic_error_row( + hermetic_env: None, + launches: list[tuple[object, ...]], + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + cause, + match, +) -> None: + @contextlib.asynccontextmanager + async def _failing_scope(**kwargs: object): + raise EgressSetupError("network: llm_only egress probe failed: x") + yield None + + if cause == "probe": + monkeypatch.setattr(dr, "egress_scope", _failing_scope) + else: + monkeypatch.setattr(dr.settings, "api_backend", ApiBackend.BEDROCK) + monkeypatch.setattr(dr.settings, "aws_region", "not a region") + monkeypatch.setattr(dr.settings, "aws_bearer_token_bedrock", "tok") + runner = _rt_runner(tmp_path, "llm_only") + with pytest.raises(EgressSetupError, match=match): + await runner.run() + assert launches == [] + row = EvaluationResult.model_validate_json((runner.rt.run_dir / TASK_JSON_FILENAME).read_text(encoding="utf-8")) + assert row.final_status == FinalStatus.ERROR + assert match in (row.error_message or "") + + +@pytest.mark.parametrize("docker", [{"network": "bridge", "egress_allowlist": ["pypi.org"]}, {"network": "none"}]) +def test_bridge_and_none_grading_disclosure_is_unchanged(docker: dict[str, Any]) -> None: + assert _container_dispatch_commands(_task(image="img:1", **docker), None) == [ + "docker run img:1 (with your credentials in its environment and a writable copy of ~/.claude)" + ] + + +def test_llm_only_grading_disclosure_names_the_mode_and_the_allowlist() -> None: + [text] = _container_dispatch_commands(_task(image="img:1", network="llm_only", egress_allowlist=["pypi.org"]), None) + assert "--network llm_only" in text + assert "(egress allowed to: pypi.org:443 plus the derived model-API hosts)" in text + [bare] = _container_dispatch_commands(_task(image="img:1", network="llm_only"), None) + assert "--network llm_only" in bare + assert "egress allowed to" not in bare diff --git a/tests/test_docker_litellm_env.py b/tests/test_docker_litellm_env.py index b9f0307f5..c27f76c6e 100644 --- a/tests/test_docker_litellm_env.py +++ b/tests/test_docker_litellm_env.py @@ -14,10 +14,8 @@ import pytest -from coder_eval.isolation.docker_runner import ( - DockerRunner, - _rewrite_loopback_for_container, -) +from coder_eval.isolation.docker_runner import DockerRunner +from coder_eval.isolation.egress import rewrite_loopback_for_container from coder_eval.models import DockerDriverConfig, FileExistsCriterion, SandboxConfig, TaskDefinition @@ -28,19 +26,19 @@ class TestRewriteLoopbackForContainer: """localhost/127.0.0.1 -> host.docker.internal, preserving scheme/port/path.""" def test_localhost_with_port(self): - assert _rewrite_loopback_for_container("http://localhost:4000") == "http://host.docker.internal:4000" + assert rewrite_loopback_for_container("http://localhost:4000") == "http://host.docker.internal:4000" def test_127_0_0_1_with_port(self): - assert _rewrite_loopback_for_container("http://127.0.0.1:4000") == "http://host.docker.internal:4000" + assert rewrite_loopback_for_container("http://127.0.0.1:4000") == "http://host.docker.internal:4000" def test_preserves_scheme_and_path(self): - assert _rewrite_loopback_for_container("https://localhost:8443/v1") == "https://host.docker.internal:8443/v1" + assert rewrite_loopback_for_container("https://localhost:8443/v1") == "https://host.docker.internal:8443/v1" def test_no_port(self): - assert _rewrite_loopback_for_container("http://localhost") == "http://host.docker.internal" + assert rewrite_loopback_for_container("http://localhost") == "http://host.docker.internal" def test_non_loopback_returns_none(self): - assert _rewrite_loopback_for_container("http://litellm.internal:4000") is None + assert rewrite_loopback_for_container("http://litellm.internal:4000") is None class TestLitellmEnvForwarding: diff --git a/tests/test_egress_proxy.py b/tests/test_egress_proxy.py new file mode 100644 index 000000000..88b19ad40 --- /dev/null +++ b/tests/test_egress_proxy.py @@ -0,0 +1,323 @@ +"""The stdlib-only egress proxy: real sockets on 127.0.0.1, no docker.""" + +from __future__ import annotations + +import ast +import asyncio +import contextlib +import socket +import subprocess +import sys +import time +from collections.abc import AsyncIterator +from pathlib import Path + +import pytest + +from coder_eval import egress_proxy +from coder_eval.cli.run_task_internal_command import heartbeat_is_alive + + +PROXY_FILE = Path(egress_proxy.__file__) +ALLOWED_IMPORTS = {"argparse", "asyncio", "os", "sys", "time"} + + +async def _upstream(handler) -> tuple[asyncio.Server, int]: + server = await asyncio.start_server(handler, "127.0.0.1", 0) + return server, server.sockets[0].getsockname()[1] + + +async def _echo(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + while data := await reader.read(1024): + writer.write(data) + await writer.drain() + writer.close() + + +@contextlib.asynccontextmanager +async def _proxy(allow: set[str]) -> AsyncIterator[int]: + server = await egress_proxy.start_proxy(frozenset(allow), "127.0.0.1", 0) + try: + yield server.sockets[0].getsockname()[1] + finally: + server.close() + server.abort_clients() + + +async def _request(port: int, head: bytes) -> tuple[asyncio.StreamReader, asyncio.StreamWriter, bytes]: + reader, writer = await asyncio.open_connection("127.0.0.1", port) + writer.write(head) + await writer.drain() + status = await asyncio.wait_for(reader.readline(), 5) + return reader, writer, status + + +async def _status(allow: set[str], head: bytes) -> bytes: + async with _proxy(allow) as port: + _reader, writer, status = await _request(port, head) + writer.close() + return status.split()[1] + + +def _free_port() -> int: + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + return sock.getsockname()[1] + + +async def test_connect_to_allowlisted_target_tunnels_both_ways(capsys): + echo, echo_port = await _upstream(_echo) + async with echo, _proxy({f"localhost:{echo_port}"}) as port: + reader, writer, status = await _request(port, f"CONNECT LocalHost:{echo_port} HTTP/1.1\r\n\r\n".encode()) + assert status.startswith(b"HTTP/1.1 200") + assert await reader.readline() == b"\r\n" + writer.write(b"ping") + await writer.drain() + assert await asyncio.wait_for(reader.readexactly(4), 5) == b"ping" + writer.close() + assert f"ALLOW localhost:{echo_port} CONNECT" in capsys.readouterr().out + + +async def test_connect_to_denied_target_is_403(capsys): + assert await _status(set(), b"CONNECT 127.0.0.1:9 HTTP/1.1\r\n\r\n") == b"403" + assert capsys.readouterr().out.splitlines() == ["DENY 127.0.0.1:9 CONNECT"] + + +async def test_connect_to_allowlisted_closed_port_is_502(capsys): + closed = _free_port() + assert await _status({f"127.0.0.1:{closed}"}, f"CONNECT 127.0.0.1:{closed} HTTP/1.1\r\n\r\n".encode()) == b"502" + assert f"FAIL 127.0.0.1:{closed}" in capsys.readouterr().out + + +@pytest.mark.parametrize( + ("line", "expected"), + [ + ("CONNECT 127.0.0.1 HTTP/1.1", b"400"), + ("CONNECT 127.0.0.1:abc HTTP/1.1", b"400"), + ("CONNECT 127.0.0.1:70000 HTTP/1.1", b"400"), + ("CONNECT [::1 HTTP/1.1", b"400"), + ("CONNECT [::1]:443 HTTP/1.1", b"403"), + ("GET http://example.com/ HTTP/1.1", b"403"), + ("GET https://127.0.0.1:443/ HTTP/1.1", b"400"), + ("GET http://127.0.0.1:443@evil.example/ HTTP/1.1", b"403"), + ("GET http://127.0.0.1:443:443/ HTTP/1.1", b"403"), + ("GET http://evil%2eexample/ HTTP/1.1", b"403"), + ("GET http://evil.example./ HTTP/1.1", b"403"), + ("GET http://[::1%25eth0]/ HTTP/1.1", b"403"), + ], +) +async def test_unallowed_or_unparseable_authorities_are_refused(line: str, expected: bytes): + assert await _status({"127.0.0.1:443"}, f"{line}\r\n\r\n".encode()) == expected + + +async def test_absolute_form_https_is_400_with_explanation(capsys): + assert await _status({"example.com:443"}, b"GET https://example.com/ HTTP/1.1\r\n\r\n") == b"400" + assert "BAD example.com:443 absolute-form https (use CONNECT)" in capsys.readouterr().out + + +async def test_absolute_form_get_is_rewritten_to_origin_form(capsys): + received: list[bytes] = [] + + async def _http(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + received.append(await reader.readuntil(b"\r\n\r\n")) + writer.write(b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\nConnection: close\r\n\r\nhello") + await writer.drain() + writer.close() + + upstream, up_port = await _upstream(_http) + head = ( + f"GET http://user:pw@127.0.0.1:{up_port}/path?q=1 HTTP/1.1\r\n" + f"Host: 127.0.0.1:{up_port}\r\nProxy-Connection: keep-alive\r\nProxy-Authorization: x\r\n" + "Connection: keep-alive\r\nAccept: */*\r\n\r\n" + ).encode() + async with upstream, _proxy({f"127.0.0.1:{up_port}"}) as port: + reader, writer, status = await _request(port, head) + body = await asyncio.wait_for(reader.read(), 5) + writer.close() + assert status.startswith(b"HTTP/1.1 200") + assert body.endswith(b"hello") + forwarded = received[0].decode() + assert forwarded.startswith("GET /path?q=1 HTTP/1.1\r\n") + assert "Proxy-" not in forwarded + assert "keep-alive" not in forwarded + assert "Connection: close\r\n" in forwarded + assert "Accept: */*" in forwarded + assert f"ALLOW 127.0.0.1:{up_port} GET" in capsys.readouterr().out + + +@pytest.mark.parametrize( + "line", + [ + b"garbage", + b"GET /relative HTTP/1.1", + b"GET http://u:SECRET@h:99999/?token=SECRET HTTP/1.1", + b"X\nALLOW\tevil.com:443\tCONNECT http://evil.com/ HTTP/1.1", + b"CONNECT evil.com:443 HTTP/1.1\x00", + b"GET http://evil.com/\xff HTTP/1.1", + b"GET http://evil.com/ HTTP/1.1\tX", + ], +) +async def test_a_bad_request_line_is_one_redacted_log_line(line: bytes, capsys): + assert await _status(set(), line + b"\r\n\r\n") == b"400" + out = capsys.readouterr().out + assert len(out.splitlines()) == 1 + assert out.startswith("BAD ") + assert "SECRET" not in out + + +async def test_connections_beyond_the_cap_get_503(monkeypatch, capsys): + monkeypatch.setattr(egress_proxy, "MAX_CONNECTIONS", 1) + async with _proxy(set()) as port: + _held_reader, held_writer = await asyncio.open_connection("127.0.0.1", port) + await asyncio.sleep(0.05) + _reader, writer, status = await _request(port, b"CONNECT a.example:443 HTTP/1.1\r\n\r\n") + writer.close() + held_writer.close() + assert status.startswith(b"HTTP/1.1 503") + assert "BAD too-many-connections" in capsys.readouterr().out + + +async def test_oversized_head_closes_the_connection(capsys): + async with _proxy(set()) as port: + reader, writer = await asyncio.open_connection("127.0.0.1", port) + writer.write(b"GET http://x/ HTTP/1.1\r\nX: " + b"a" * (egress_proxy.HEAD_LIMIT_BYTES + 10)) + with contextlib.suppress(ConnectionError): + await writer.drain() + assert await asyncio.wait_for(reader.read(), 5) == b"" + writer.close() + assert "BAD head-too-large" in capsys.readouterr().out + + +async def test_connect_half_close_still_delivers_the_response(): + async def _reply_after_eof(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + writer.write(b"got:" + await reader.read()) + await writer.drain() + writer.close() + + upstream, up_port = await _upstream(_reply_after_eof) + async with upstream, _proxy({f"127.0.0.1:{up_port}"}) as port: + reader, writer, status = await _request(port, f"CONNECT 127.0.0.1:{up_port} HTTP/1.1\r\n\r\n".encode()) + assert status.startswith(b"HTTP/1.1 200") + assert await reader.readline() == b"\r\n" + writer.write(b"question") + writer.write_eof() + assert await asyncio.wait_for(reader.read(), 5) == b"got:question" + writer.close() + + +async def test_upstream_closing_mid_stream_closes_the_client(): + async def _one_shot(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: + writer.write(b"data: 1\n\n") + await writer.drain() + writer.close() + + upstream, up_port = await _upstream(_one_shot) + async with upstream, _proxy({f"127.0.0.1:{up_port}"}) as port: + reader, writer, status = await _request(port, f"CONNECT 127.0.0.1:{up_port} HTTP/1.1\r\n\r\n".encode()) + assert status.startswith(b"HTTP/1.1 200") + rest = await asyncio.wait_for(reader.read(), 5) + writer.close() + assert rest == b"\r\ndata: 1\n\n" + + +async def test_probe_succeeds_only_when_every_target_is_allowed(monkeypatch, capsys): + echo, echo_port = await _upstream(_echo) + async with echo, _proxy({f"127.0.0.1:{echo_port}"}) as port: + ok = await egress_proxy.probe(f"127.0.0.1:{port}", [f"127.0.0.1:{echo_port}"], 5) + mixed = await egress_proxy.probe(f"127.0.0.1:{port}", [f"127.0.0.1:{echo_port}", "denied.example:443"], 5) + monkeypatch.setattr(egress_proxy, "PROBE_STARTUP_RETRY_SECONDS", 0.3) + unreachable = await egress_proxy.probe(f"127.0.0.1:{_free_port()}", ["a.example:443"], 1) + out = capsys.readouterr().out + assert (ok, mixed, unreachable) == (0, 1, 1) + assert f"OK 127.0.0.1:{echo_port}" in out + assert "FAIL denied.example:443 HTTP/1.1 403 Forbidden" in out + assert "FAIL a.example:443 proxy unreachable" in out + + +async def test_watchdog_stops_serve_on_a_stale_or_missing_heartbeat(tmp_path, capsys): + heartbeat = tmp_path / "hb" + heartbeat.write_text("1", encoding="utf-8") + started = time.monotonic() + await asyncio.wait_for(egress_proxy.serve(frozenset(), "127.0.0.1", 0, str(heartbeat), 1.0), 5) + assert time.monotonic() - started < 3.5 + await asyncio.wait_for(egress_proxy.watch_heartbeat(str(tmp_path / "absent"), 0.5), 5) + assert capsys.readouterr().out.count("STALE") == 2 + + +async def test_watchdog_stays_alive_while_the_counter_advances(tmp_path): + heartbeat = tmp_path / "hb" + heartbeat.write_text("0", encoding="utf-8") + watcher = asyncio.create_task(egress_proxy.watch_heartbeat(str(heartbeat), 1.0)) + for counter in range(1, 15): + await asyncio.to_thread(heartbeat.write_text, str(counter), encoding="utf-8") + await asyncio.sleep(0.2) + assert not watcher.done() + watcher.cancel() + await asyncio.gather(watcher, return_exceptions=True) + + +@pytest.mark.parametrize( + ("current", "last", "mtime", "last_mtime"), + [ + ("", "", 0.0, 0.0), + ("", "", 5.0, 0.0), + ("1", "", 0.0, 0.0), + ("1", "1", 5.0, 5.0), + ("2", "1", 5.0, 5.0), + ("1", "1", 6.0, 5.0), + ("", "1", 5.0, 5.0), + ("1", "1", 4.0, 5.0), + ], +) +def test_liveness_rule_matches_the_container_watchdog(current: str, last: str, mtime: float, last_mtime: float): + assert egress_proxy.heartbeat_alive(current, last, mtime, last_mtime) == heartbeat_is_alive( + current, last, mtime, last_mtime + ) + + +def test_module_imports_only_the_five_stdlib_modules(): + roots: set[str] = set() + for node in ast.walk(ast.parse(PROXY_FILE.read_text(encoding="utf-8"))): + if isinstance(node, ast.Import): + roots |= {alias.name.split(".")[0] for alias in node.names} + elif isinstance(node, ast.ImportFrom): + assert node.level == 0, "egress_proxy.py must not use a relative import" + assert node.module is not None + roots.add(node.module.split(".")[0]) + assert roots <= ALLOWED_IMPORTS + + +def test_runs_standalone_in_isolated_mode(): + result = subprocess.run( + [sys.executable, "-I", str(PROXY_FILE), "--help"], capture_output=True, text=True, encoding="utf-8", timeout=20 + ) + assert result.returncode == 0 + assert "serve" in result.stdout + assert "probe" in result.stdout + + +@pytest.mark.parametrize( + ("extra", "named"), + [ + (["--listen", "nonsense"], "BAD --listen"), + *( + (["--allow", entry], "BAD --allow") + for entry in ["example.com", "[::1]:443", "a.example:443:1", "a.example:0"] + ), + (["--heartbeat", "hb"], "--stale"), + *((["--heartbeat", "hb", "--stale", stale], "--stale") for stale in ["0", "nan", "inf"]), + ], +) +def test_main_refuses_invalid_arguments(extra: list[str], named: str, capsys): + assert egress_proxy.main(["serve", "--listen", f"127.0.0.1:{_free_port()}", *extra]) == 2 + assert named in capsys.readouterr().out + + +def test_main_serves_until_the_heartbeat_is_stale(tmp_path, capsys): + heartbeat = tmp_path / "hb" + heartbeat.write_text("1", encoding="utf-8") + argv = ["serve", "--listen", f"127.0.0.1:{_free_port()}", "--heartbeat", str(heartbeat), "--stale", "0.5"] + assert egress_proxy.main([*argv, "--allow", "A.example:443"]) == 0 + out = capsys.readouterr().out + assert "allow=a.example:443" in out + assert "STALE" in out diff --git a/tests/test_egress_targets.py b/tests/test_egress_targets.py new file mode 100644 index 000000000..7719b11e6 --- /dev/null +++ b/tests/test_egress_targets.py @@ -0,0 +1,199 @@ +"""The host-side allowlist derivation for ``network: llm_only`` and the agent-config hook.""" + +from __future__ import annotations + +import logging +import traceback +from typing import Any + +import pytest + +from coder_eval.agents.registry import AgentRegistry +from coder_eval.config import Settings +from coder_eval.isolation import docker_runner, errors +from coder_eval.isolation.egress import resolve_egress_targets +from coder_eval.models import ( + AgentJudgeCriterion, + ApiBackend, + ApiRouteContext, + CheckerContext, + DockerDriverConfig, + FileExistsCriterion, + LLMJudgeCriterion, + NoulQuestion, + SandboxConfig, + SimulationConfig, + SystemOneJudgeCriterion, + TaskDefinition, + normalize_egress_target, + parse_agent_config, +) +from coder_eval.plugins import ensure_plugins_loaded + + +DIRECT = ["api.anthropic.com:443", "platform.claude.com:443"] +BEDROCK = ["bedrock-runtime.eu-north-1.amazonaws.com:443", "bedrock.eu-north-1.amazonaws.com:443"] +OPENAI, GEMINI, OPENROUTER = "api.openai.com:443", "generativelanguage.googleapis.com:443", "openrouter.ai:443" +LITELLM = ApiBackend.LITELLM +BR = {"backend": ApiBackend.BEDROCK, "region": "eu-north-1"} +CC, CODEX, PI, OPENCODE = {"type": "claude-code"}, {"type": "codex"}, {"type": "pi"}, {"type": "opencode"} +LL_URL, CX_URL, REGION = "LITELLM_BASE_URL", "CODEX_BASE_URL", "AWS_REGION" +JUDGE = LLMJudgeCriterion(description="j", prompt="p") +QUESTIONS = {"q": NoulQuestion(instructions="i")} +SYSTEM_ONE = [ + SystemOneJudgeCriterion(description="j", questions=QUESTIONS), + SystemOneJudgeCriterion(description="k", questions=QUESTIONS, base_url="https://gw.example:9443/v1"), +] +DIRECT_ROUTE = ApiRouteContext(route=ApiBackend.DIRECT) +EGRESS_LOG = "coder_eval.isolation.egress" + + +def _targets( + agent: dict[str, Any] | None = CC, + *, + backend: ApiBackend = ApiBackend.DIRECT, + region: str | None = None, + env: dict[str, str] | None = None, + criteria: list[Any] | None = None, + simulation: bool | None = None, + route: ApiRouteContext | None = None, + bedrock_token: str | None = "tok", + **docker: Any, +) -> list[str]: + task = TaskDefinition( + task_id="t", + description="d", + initial_prompt="p", + agent=parse_agent_config(**(agent or CC)), + sandbox=SandboxConfig(driver="docker", docker=DockerDriverConfig(network="llm_only", **docker)), + success_criteria=criteria or [FileExistsCriterion(description="c", path="x.txt")], + ) + if agent is None: + task.agent = None + if simulation is not None: + task.simulation = SimulationConfig(enabled=simulation, persona="p", goal="g") + if route is not None: + task.checker_context = CheckerContext(api_route=route) + settings = Settings.model_construct( + api_backend=backend, + aws_region=region, + aws_bearer_token_bedrock=bedrock_token if region else None, + anthropic_api_key="key", + litellm_base_url="http://gateway.invalid", + litellm_auth_token="tok", + litellm_model="m", + ) + return resolve_egress_targets(task, env=env or {}, settings=settings) + + +HDI = ["host.docker.internal:4000"] +CASES: dict[str, tuple[dict[str, Any] | None, dict[str, Any], list[str]]] = { + "direct-claude-code": (CC, {}, DIRECT), + "bedrock-region-lowercased": (CC, {"backend": ApiBackend.BEDROCK, "region": "EU-North-1"}, BEDROCK), + "unresolved-agent": (None, {}, []), + "litellm-loopback": (CC, {"backend": LITELLM, "env": {LL_URL: "http://localhost:4000"}}, HDI), + "litellm-ipv6-loopback": (CC, {"backend": LITELLM, "env": {LL_URL: "http://[::1]:4000"}}, HDI), + "litellm-remote-port": (CC, {"backend": LITELLM, "env": {LL_URL: "https://l.corp:8443/v1"}}, ["l.corp:8443"]), + "codex-base-url": (CODEX, {"env": {CX_URL: "https://x.openai.azure.com/v1"}}, ["x.openai.azure.com:443"]), + "codex-plain-http": (CODEX, {"env": {CX_URL: "http://gw.example/v1"}}, ["gw.example:80"]), + "claude-code-ignores-codex-url": (CC, {"env": {CX_URL: "https://x.openai.azure.com"}}, DIRECT), + "replaced-passthrough": (CODEX, {"env": {CX_URL: "https://x.corp"}, "env_passthrough": []}, [OPENAI]), + "forwarded-url-vars": (CC, {"env": {"MY_URL": "https://s.example"}, "env_passthrough_extra": ["MY_URL"]}, DIRECT), + "user-allowlist": (CC, {"egress_allowlist": ["pypi.org", "API.anthropic.com"]}, sorted([*DIRECT, "pypi.org:443"])), + "codex-off-bedrock": (CODEX, BR, [OPENAI]), + "antigravity-off-bedrock": ({"type": "antigravity"}, BR, [GEMINI]), + "pi-off-bedrock": (PI, BR, [OPENROUTER]), + "delegate-off-bedrock": ({"type": "delegate"}, BR, []), + "pi-anthropic": ({**PI, "model": "anthropic/claude-haiku-4-5"}, {}, ["api.anthropic.com:443"]), + "pi-openrouter": ({**PI, "model": "openrouter/moonshotai/kimi-k3"}, {}, [OPENROUTER]), + "pi-openai": ({**PI, "model": "openai/gpt-5"}, {}, [OPENAI]), + "pi-google": ({**PI, "model": "google/gemini-3"}, {}, [GEMINI]), + "pi-unknown-prefix": ({**PI, "model": "unknown/x"}, {}, [OPENROUTER]), + "pi-no-prefix": ({**PI, "model": "no-prefix"}, {}, [OPENROUTER]), + "opencode-no-model": (OPENCODE, {"backend": LITELLM}, []), + "system-one-judge": (CC, {"backend": LITELLM, "criteria": SYSTEM_ONE}, ["api.typesafe.ai:443", "gw.example:9443"]), + "judge-adds-backend": ({"type": "antigravity"}, {**BR, "criteria": [JUDGE]}, sorted([*BEDROCK, GEMINI])), + "disabled-judge": (CODEX, {"criteria": [LLMJudgeCriterion(description="j", prompt="p", enabled=False)]}, [OPENAI]), + "judge-route-override": ( + CODEX, + {**BR, "criteria": [AgentJudgeCriterion(description="j", prompt="p")], "route": DIRECT_ROUTE}, + sorted([*DIRECT, OPENAI]), + ), + "simulation-enabled": (CODEX, {"simulation": True}, sorted([*DIRECT, OPENAI])), + "simulation-disabled": (CODEX, {"simulation": False}, [OPENAI]), + "litellm-simulator-pinned": (CODEX, {"backend": LITELLM, "simulation": True}, sorted([*DIRECT, OPENAI])), +} + + +@pytest.mark.parametrize(("agent", "kwargs", "expected"), list(CASES.values()), ids=list(CASES)) +def test_resolved_targets(agent: dict[str, Any] | None, kwargs: dict[str, Any], expected: list[str]): + assert _targets(agent, **kwargs) == expected + + +@pytest.mark.parametrize( + ("agent", "kwargs", "variable"), + [ + *( + (CODEX, {"env": {CX_URL: url}}, CX_URL) + for url in ["https://u:secret@[::2]", "http://[secret", "https://secret:70000"] + ), + (CC, {"backend": LITELLM, "env": {LL_URL: "http://localhost:secret"}}, LL_URL), + (CC, {"backend": ApiBackend.BEDROCK, "region": "secret region"}, REGION), + ], +) +def test_unallowlistable_value_raises_without_echoing_it(agent: dict[str, Any], kwargs: dict[str, Any], variable: str): + with pytest.raises(ValueError, match=variable) as excinfo: + _targets(agent, **kwargs) + assert "secret" not in "".join(traceback.format_exception(excinfo.value)) + + +@pytest.mark.parametrize( + ("agent", "kwargs", "logger", "named", "hidden"), + [ + (CC, {"backend": ApiBackend.BEDROCK}, EGRESS_LOG, REGION, None), + (CC, {"backend": LITELLM}, EGRESS_LOG, LL_URL, None), + (CC, {"bedrock_token": None, **BR}, EGRESS_LOG, "AWS_BEARER_TOKEN_BEDROCK", None), + (CODEX, {"env": {CX_URL: "http://127.0.0.1:8080"}}, "coder_eval.models.sandbox", CX_URL, "8080"), + ], + ids=["bedrock-no-region", "litellm-no-url", "unconfigured-backend", "non-litellm-loopback"], +) +def test_unusable_route_warns_and_adds_nothing(agent, kwargs, logger, named, hidden, caplog): + with caplog.at_level(logging.WARNING, logger=logger): + assert _targets(agent, **kwargs) == [] + assert named in caplog.text + assert hidden is None or hidden not in caplog.text + + +@pytest.mark.parametrize(("region", "expected"), [("eu-north-1", BEDROCK), (None, DIRECT)]) +def test_litellm_agent_judge_and_simulator_get_the_pinned_claude_backend(region: str | None, expected: list[str]): + env = {LL_URL: "https://gw.corp/v1"} + targets = _targets(backend=LITELLM, region=region, env=env, criteria=[JUDGE], simulation=True) + assert targets == sorted(["gw.corp:443", *expected]) + + +@pytest.mark.parametrize( + ("route", "env", "host"), + [ + ({"params": {"api_base": "https://judge.example/v1"}}, {}, "judge.example:443"), + ({"env_params": {"api_base": "JUDGE_BASE"}}, {"JUDGE_BASE": "https://j2.example"}, "j2.example:443"), + ], +) +def test_litellm_judge_route_uses_its_own_api_base(route: dict[str, Any], env: dict[str, str], host: str): + judge_route = ApiRouteContext(route=LITELLM, model="azure/x", **route) + targets = _targets(CODEX, env=env, criteria=[JUDGE], route=judge_route, env_passthrough_extra=["JUDGE_BASE"]) + assert targets == [OPENAI, host] + + +def test_every_registered_agent_config_answers_with_normalized_targets(): + ensure_plugins_loaded() + registrations = AgentRegistry.registrations() + assert registrations + for registration in registrations: + hosts = registration.config_class.model_construct().egress_hosts({}) + assert isinstance(hosts, tuple), registration.config_class.__name__ + assert all(normalize_egress_target(host) == host for host in hosts), registration.config_class.__name__ + + +def test_docker_run_error_is_one_class_in_both_modules(): + assert docker_runner.DockerRunError is errors.DockerRunError + assert issubclass(errors.EgressSetupError, errors.DockerRunError) diff --git a/tests/test_harbor_packager.py b/tests/test_harbor_packager.py index cb383aca4..553b8062f 100644 --- a/tests/test_harbor_packager.py +++ b/tests/test_harbor_packager.py @@ -55,6 +55,12 @@ def test_tempdir_driver_is_refused(self, tmp_path: Path) -> None: with pytest.raises(TaskNotExportableError): export_task(task_file, tmp_path / "out") + def test_llm_only_network_is_refused_not_widened_to_public(self, tmp_path: Path) -> None: + task_file = _write_task(tmp_path, {"sandbox": {"driver": "docker", "docker": {"network": "llm_only"}}}) + with pytest.raises(TaskNotExportableError, match="llm_only"): + export_task(task_file, tmp_path / "out") + assert not (tmp_path / "out").exists() + def test_unsupported_criteria_are_refused_before_any_file_is_written(self, tmp_path: Path) -> None: task_file = _write_task( tmp_path, diff --git a/tests/test_merge_characterization.py b/tests/test_merge_characterization.py index 2326b7290..018da063f 100644 --- a/tests/test_merge_characterization.py +++ b/tests/test_merge_characterization.py @@ -167,6 +167,28 @@ def test_env_passthrough_extra_appends(self): # exp-defaults appended first, then task. assert resolved.sandbox.docker.env_passthrough_extra == ["EXP_VAR", "TASK_VAR"] + def test_egress_allowlist_appends_exp_defaults_then_task(self): + default_exp = _default_exp() + task = _make_task( + agent={"type": "claude-code"}, + sandbox=SandboxConfig(driver="docker", docker=DockerDriverConfig(egress_allowlist=["task.example.com"])), + ) + experiment = ExperimentDefinition( + experiment_id="test", + defaults=ExperimentDefaults( + sandbox=SandboxConfig(docker=DockerDriverConfig(egress_allowlist=["pypi.org"])) + ), + variants=[ExperimentVariant(variant_id="v")], + ) + resolved, _lineage, _ = resolve_task_for_variant(default_exp, task, experiment, experiment.variants[0]) + assert resolved.sandbox.docker.egress_allowlist == ["pypi.org:443", "task.example.com:443"] + apply_overrides(resolved, {"sandbox.docker.egress_allowlist": ["cli.example.com:80"]}) + assert resolved.sandbox.docker.egress_allowlist == [ + "pypi.org:443", + "task.example.com:443", + "cli.example.com:80", + ] + def test_template_sources_task_first_order(self): """template_sources resolve task-first: the task's base templates, then experiment-defaults and variant overlays appended after (the documented diff --git a/uv.lock b/uv.lock index e57216652..4f678c027 100644 --- a/uv.lock +++ b/uv.lock @@ -2400,15 +2400,14 @@ wheels = [ [[package]] name = "python-discovery" -version = "1.2.0" +version = "1.6.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "filelock" }, - { name = "platformdirs" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/9c/90/bcce6b46823c9bec1757c964dc37ed332579be512e17a30e9698095dcae4/python_discovery-1.2.0.tar.gz", hash = "sha256:7d33e350704818b09e3da2bd419d37e21e7c30db6e0977bb438916e06b41b5b1", size = 58055, upload-time = "2026-03-19T01:43:08.248Z" } +sdist = { url = "https://files.pythonhosted.org/packages/0c/57/250bd238b966cece44328235eb85290045d059265fdaf7527a3a958123db/python_discovery-1.6.1.tar.gz", hash = "sha256:cf87d3627dfb4412437fdd5b13eae402607722998d21567993aedbc59b23c15e", size = 84338, upload-time = "2026-09-18T01:31:53.971Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/c2/3c/2005227cb951df502412de2fa781f800663cccbef8d90ec6f1b371ac2c0d/python_discovery-1.2.0-py3-none-any.whl", hash = "sha256:1e108f1bbe2ed0ef089823d28805d5ad32be8e734b86a5f212bf89b71c266e4a", size = 31524, upload-time = "2026-03-19T01:43:07.045Z" }, + { url = "https://files.pythonhosted.org/packages/16/7d/e9ffbadfbf89c93848412d04594135c4ae8c1d37d9e053b9c3ed718fabc4/python_discovery-1.6.1-py3-none-any.whl", hash = "sha256:d43fcdef879fe795352bd13ccf8d185ba5a9f86f36cfcd00529f596e737442b3", size = 38664, upload-time = "2026-09-18T01:31:52.448Z" }, ] [[package]] @@ -3225,11 +3224,11 @@ wheels = [ [[package]] name = "urllib3" -version = "2.7.0" +version = "2.8.0" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/05/b17359e1cefb4f909b5e40b1b90a496d987258916dbbf88e842c729f510e/urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63", size = 458972, upload-time = "2026-09-15T19:29:36.253Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, + { url = "https://files.pythonhosted.org/packages/92/9d/c4e665119135114480843e7ab388fa94d8480650450e6f8e26b70d323a4c/urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3", size = 135717, upload-time = "2026-09-15T19:29:34.577Z" }, ] [[package]] @@ -3259,7 +3258,7 @@ wheels = [ [[package]] name = "virtualenv" -version = "21.2.0" +version = "21.7.13" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "distlib" }, @@ -3267,9 +3266,9 @@ dependencies = [ { name = "platformdirs" }, { name = "python-discovery" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/aa/92/58199fe10049f9703c2666e809c4f686c54ef0a68b0f6afccf518c0b1eb9/virtualenv-21.2.0.tar.gz", hash = "sha256:1720dc3a62ef5b443092e3f499228599045d7fea4c79199770499df8becf9098", size = 5840618, upload-time = "2026-03-09T17:24:38.013Z" } +sdist = { url = "https://files.pythonhosted.org/packages/13/50/c9b84eb106d0db420b9878dac0a386c5791726f127ce20b06897f6e3a1e9/virtualenv-21.7.13.tar.gz", hash = "sha256:0355558b6f33619aab31347e43643b0ebc97f61ea3acf617b2b69e1f8a843d11", size = 5360306, upload-time = "2026-09-18T04:35:49.354Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/c6/59/7d02447a55b2e55755011a647479041bc92a82e143f96a8195cb33bd0a1c/virtualenv-21.2.0-py3-none-any.whl", hash = "sha256:1bd755b504931164a5a496d217c014d098426cddc79363ad66ac78125f9d908f", size = 5825084, upload-time = "2026-03-09T17:24:35.378Z" }, + { url = "https://files.pythonhosted.org/packages/9a/ce/e74453531b0c49a58d0e8a4f9bab4495705859fee4a4f7de27d58f4a791a/virtualenv-21.7.13-py3-none-any.whl", hash = "sha256:1bea5af7463f59c4719db48fe739579a2a4f569c96f26c086edda85c96da9f59", size = 5328680, upload-time = "2026-09-18T04:35:47.194Z" }, ] [[package]]