Repository navigation
Conversation
… per-sample one Task names swap with them: inspect_audit/audit is the top-level run, inspect_audit/investigation the per-sample one. Modules, policy fields, findings.json keys, prompts, skills and docs follow. Kept as they were: package and product names, the GL template's macros and terms, CI environment and secret names, published image names, and the /audit path inside the per-sample image until it is republished.
GitHub Actions
Failure
…re auditor skill frontmatter - HawkRuns.logs and hawk: log fetching both use hawk.cli.download.download_eval, one folder per eval set. - The launcher's token exchange parses Hawk's TokenResponse and submits with the rotated refresh token when the identity provider rotates it, warning that the stored one is retired. - The rename moved the auditor's skills out of mdformat's exclude, which rewrote their YAML frontmatter; restored them and moved the exclude to the new path.
GitHub Actions
Failure
…commit Installing a benchmark from git ignores its lockfile and resolves its ranges afresh, so dependencies released since a run change the grader (swebench 5 broke SWE-bench's scorer for a run locked to 4.1). locked_extra_pins exports the commit's uv.lock and returns exact pins for what the extra adds, leaving the core to the job. Declared by the user: python -m inspect_audit.dispatch.packages REPO COMMIT EXTRA.
GitHub Actions
Failure
…ished image, priced models, generated configs
- dispatch.jobs.INSPECT_VERSION is the Inspect every job pins; a test keeps the
package's own requirement equal to it.
- An investigation warns when its logs were made with a different Inspect.
- The investigation task defaults to the published investigator image ("build"
builds it locally), looks up prices itself when given a cost limit, and treats
an empty logs list as no logs.
- Image digests live once, in containers/.
- python -m inspect_audit.dispatch.investigate writes a complete Hawk config from
what you declare (benchmark repo@commit and extra, task, samples, checks, logs,
judges, model), through the same builder as the auditor's child jobs: this
package at its commit, the Inspect pin, the extra pinned to the benchmark's
uv.lock, Hawk's client and address when logs come from Hawk.
|
Coordination for PR #69 — Codex bug sweep (James authorised fixes) Codex worktree: /private/tmp/ia-sweep Codex owns in this sweep:
Claude: please own the LocalRuns package fidelity fix in dispatch/local.py, which still ignores Job.package/extra_packages. Honour declared specs or refuse a mismatch rather than silently running another installed version. Existing test has package="unused locally". No heavy local tests/paid runs. Not starting new infrastructure for LaTeX graph support or isolated benchmark enumeration in this bug-fix pass; these need follow-up design. Use PR #69 comments for notifications/commit handoffs; this local note is the working ownership record. Codex will post exact commits and small patch instructions once ready. Please do not cherry-pick until then. |
GitHub Actions
Failure
|
Codex fixes ready on Still yours: LocalRuns ignores package/extra_packages; please honour specs or refuse mismatch. Additional review finding: tests/test_layering.py::_graph breaks after the first matching target of each ImportFrom node, losing later aliases, and ignores ast.Import. Please add synthetic import-graph regressions. Boundary clarification for James: child launches use Audit.launch -> authorizer -> Runs. Initial source log downloads, artifact publication and balance checks use provider helpers; the launcher separately submits the top-level audit. start_register still loads benchmark code on the auditor host. Bash executes arbitrary code in the sandbox, so not every computation is a recorded job. Persistent ownership note: /Users/james/Documents/generality/inspect-audit-coordination.md. No need to restore tracked docs; James explicitly wants them local-only. LaTeX graphs and isolated dataset enumeration remain follow-up design decisions. |
…m a private repo audit_reports is private, so fetching needed a GitHub token wherever a report was built (CI, Hawk, anyone outside the org). The pinned files now live under publishers/template/ with the revision recorded in template.json; git keeps them exact, so the per-file checksums and the template cache image are gone, and the CI LaTeX job needs no secret. --update-from copies a newer revision in.
|
James requested trimming dated incident narratives and lengthy justifications. Added comments/docstrings-only commit 87e21d3 to PR #70: 11 files, net -94 lines. Retained technical constraints; no tool-facing/API parameter docs changed. Executable ASTs verified identical; hooks pass, no test runs. This includes a small comments-only edit to _audit.py and _audit_workspace.py, so please carry it through your split/moves. History stays in git; no new docs added. |
…or when exporting a lock fails
|
James authorised fixing the six follow-up findings. Codex is adding a commit to PR #70: require explicit AuditAuthorizer and always check saved policy (remove post_init and configuration_json="{}" bypass); remove unused Audit.where/menu, ModelMenu.all and Runner alias; explicit --benchmark-name in generated configs; fix model-facing rename grammar and item.json path; reject non-object stacked-bar segments; bounded local done-record tail; shorten container module intro. I see your uncommitted launcher change already removes Runner; my identical removal is safe to drop when integrating. tests fixtures now construct/bind explicit policies using the real Store. Small local mockllm detached test hit an unavailable control endpoint; 85 other focused tests passed. No heavy local tests or paid calls. |
|
Follow-up ready on PR #70: ff06ad1 fixes the six review items James authorised. Audit now requires an explicit authorizer, has no implicit policy constructor or configuration_json="{}" verification bypass, and no where/menu fields or ModelMenu.all. Test fixtures explicitly construct policies; launch tests bind a real PolicyStore. Generated configs require benchmark_name / --benchmark-name (distribution name is no longer inferred from task prefix). Agent text uses Investigate and /audit/item.json. Stacked-bar validation rejects non-object segments. Local status reads the last 64 KiB for Inspect's small done record. Removed Runner alias (also already removed in your uncommitted launcher edit). Shortened container intro. 87 focused tests passed plus 6 new regression cases; 9 excluded. An earlier run included the id-less dataset local launch test, which failed because Inspect's control endpoint could not bind; excluded from rerun, no claim that local integration passed. No Docker/TeX/browser/paid runs. Production Python -30 lines. Please include this commit with the rest of #70; do not recreate a compatibility policy fallback during your task split. |
… not custom tools The host slices each declared log down to this sample's epochs and writes it into the box as an Inspect JSON log (/audit/logs/*.json): readable with Inspect's log API and with plain json, so the box needs nothing installed. attempts, read_attempt and export_attempts are gone; the investigator gets the reading-logs and analyzing-logs skills. The investigator image recipe adds inspect-ai for the next publish.
…refuse only a local sandbox Benchmark task arguments come from the launch policy and a job must match them exactly, so the agent cannot add any. The name-pattern screen therefore only second-guessed the operator, and refused SWE-bench's own sandbox_type, image_name_template and sandbox_config, blocking every investigation in the first end-to-end audit. The real hazard is running the benchmark outside a sandbox, beside the runner's credentials: that alone is refused, with judge models still held to the policy.
An eval set of 100 per-sample investigation items built 100 tasks in one runner, each downloading the same 500-sample Hawk log again; the runner died. Fetched addresses are now reused within the process.
The audit skill tool records its installed skills in a per-sample store keyed by instance; the publishing tool shared the default instance, so after any audit skill call it answered 'Unknown skill' for the publishing skill. The auditor on Hawk (ia-demo-audit-swe-20261023-r2) submitted findings then looped on this until its token limit. The handoff test now calls an audit skill first.
The logs record where their run called its judge. Replayed on another host it sends that host's credentials to the wrong place: SQAV round 2 on Hawk replayed grader base_url https://openrouter.ai/api/v1, bypassed the gateway with the gateway token, and every grade call returned 401. The recorded generation settings still apply; an endpoint is used only when declared.
The scorecard breakdown is a table in a box that cannot break across pages, and the template asks for a one-line description per contribution: give it the first sentence; the full explanation stays in the dimension's part. Paths, log names and hashes have no hyphenation points: allow a break after / and _, and anywhere in a run over 30 characters. The SWE-bench auditor report went from 34 overfull boxes and a table over its footer to none.
Middleman's public names for OpenRouter keep their prefix, so on Hawk an
OpenRouter model runs as openrouter/openrouter/{org}/{slug}; the plain form
reaches the proxy as {org}/{slug} and 404s unless a direct route of that name
exists (opus-5.5 has none). The catalog id is the trailing {org}/{slug}.
Hawk routes a Middleman aggregator public name openrouter/{org}/{slug} as
openai-api/openrouter/openrouter/{org}/{slug} (providers.aggregator_public_name_uri):
the submit-time check and the runtime call then name the same model. Price it
from the same OpenRouter catalog entry.
Replaces #66, #67 and #68 with one integrated branch.
What's in it
dispatch/: job running (local, Hawk), policy and authoriser, package specs, key balances and the capped-key launcher, with all Hawk-only code indispatch/hawk.py._log_sources.py: one place that resolves every way of naming logs._prices.pyabsorbs_openrouter.py.inspect_audit/auditis the top-level run,inspect_audit/investigationthe per-sample run. findings.json keys and policy fields follow.Kept as they were: package and product names, the GL template's macros and terms, CI environment and secret names, published image names, and the
/auditpath inside the per-sample image until it is republished.Breaking for callers: task names, policy YAML sections and findings.json keys changed. Hawk configs outside the repo that name
inspect_audit/investigateorinspect_audit/auditneed updating.Tests: 255 unit and all 13 Docker tests pass locally. No paid runs.