Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
### Added
- Added Kernel as a managed remote browser runtime with live view and downloaded replay recordings. Thanks to @[rgarcia](https://github.com/rgarcia).
- Added `clawbench-analyze` entrypoint for aggregate batch error analysis.
- Added a `--browser-runtime kernel` mode to the Harbor adapter that runs each task against one Kernel cloud browser, exposing only a credential-free CDP bridge to the agent, and finalizes the replay and deletes the browser during verification.

### Changed
- Updated the harbor adaptor to support the full V2 lenient & strict and reports numeric results.

### Fixed
- Fail task setup when PurelyMail returns an API error instead of emitting credentials for an account that was not created.
- Fixed an issue where malformed per-run metadata could prevent `batch-summary.json` from being written and, when configured, uploaded.
- Fixed the issue that an invalid judge model would lose the `run-meta.json` file.

Expand Down
19 changes: 19 additions & 0 deletions docs/harbor.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,25 @@ uvx --from harbor==0.15.0 harbor run \
--jobs-dir ./harbor-jobs/hermes-deepseek-flash
```

## Kernel browser runtime (control arm)

By default each Harbor trial runs Chromium inside its own container. Pass `--browser-runtime kernel` to the adapter to run the same tasks against one Kernel cloud browser per task instead:

```bash
uv run clawbench-harbor-adapt \
--output-dir ./harbor-datasets/clawbench-v2-kernel \
--browser-runtime kernel \
--browser-runtime-options '{"stealth": true}' \
--task-ids v2-1134-chapter-finder-redcross \
--overwrite
```

During task setup, the environment creates exactly one Kernel browser and replay, starts the ClawBench runtime server against it, and exposes only the local credential-free CDP bridge (`http://127.0.0.1:7878`) to the agent — the Kernel API key is never visible to the benchmark agent. Session identity and cleanup metadata land in `/my-info/kernel_browser.json`. During verification the provider replay is finalized, `recording.mp4` is downloaded into `/data`, and the browser is deleted (idempotently, including failure paths via a setup trap).

Generated tasks register a pinned Playwright MCP package (`@playwright/mcp@0.0.79`) pointed at the CDP bridge, so Harbor's stock Claude Code and Codex agents drive the Kernel browser with native Playwright MCP tool calls — structurally identical to ClawBench's native Claude/Codex harnesses.

Export `KERNEL_API_KEY` (and optionally `KERNEL_BASE_URL` for non-production gateways) before `harbor run`; no extra flags are needed.

## Making it fast

A full V2 sweep is 129 containerized browser sessions, each capped by the task's `time_limit`. Serial, that is a very long night. What actually moves the needle, in order:
Expand Down
Loading
Loading