diff --git a/CHANGELOG.md b/CHANGELOG.md index ec9a25f2..e7250a59 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/) and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] +### Added +- Added an opt-in `tests/test_harbor_version_compatibility.py` that loads every generated Harbor task with Harbor's own loader, so a Harbor version bump can be verified instead of assumed. Skipped unless Harbor is installed. + +### Changed +- Harbor docs now pin `harbor==0.22.0` instead of the six-releases-stale `harbor==0.15.0`, and document which version the generated dataset was verified against. ## [0.10.0] - 2026-08-30 ### Added diff --git a/docs/harbor.md b/docs/harbor.md index fba41381..0ba29f7b 100644 --- a/docs/harbor.md +++ b/docs/harbor.md @@ -5,6 +5,7 @@ **Use Harbor when** you already run other benchmarks through it, you want Harbor's agent registry (`-a openclaw`, `-a hermes`, …) instead of ClawBench's own harnesses, or you need Harbor's retry/attempt semantics. **Use `clawbench-batch` instead** when you just want to score a model on ClawBench — it is the shorter path and it is what the leaderboard uses. - [Prerequisites](#prerequisites) +- [Harbor versions](#harbor-versions) - [1. Convert V2 into a Harbor dataset](#1-convert-v2-into-a-harbor-dataset) - [2. Wire up the judge](#2-wire-up-the-judge) - [3. Run it](#3-run-it) @@ -14,11 +15,26 @@ ## Prerequisites +- **Harbor 0.22.0.** Every command below pins `harbor==0.22.0`, which is the version the generated dataset is verified against — see [Harbor versions](#harbor-versions). - **Docker.** Harbor runs use Harbor's Docker provider, so Docker must be available even if you normally use Podman for native ClawBench runs. - **ClawBench installed** (`uv tool install clawbench-eval`, or a source checkout with `uv run` prefixes). - **Judge credentials.** Scoring requires both an intercepted request *and* a judge verdict; without judge credentials every intercepted task scores `0`. - **PurelyMail credentials** from `.env`, passed through with `--env-file .env`. +## Harbor versions + +The commands here pin **`harbor==0.22.0`**, the current release at the time of writing. The previous pin, `0.15.0`, was six releases stale: anyone following these docs installed an old Harbor, and anyone who already had a current Harbor found the pin fighting their install. + +**What was checked.** `clawbench-harbor-adapt` output loads under 0.22.0: all 129 generated V2 tasks construct as `harbor.models.task.task.Task` with no errors, and every flag these docs use (`-p`, `-a`, `-m`, `--ak`, `--env-file`, `--ve`, `--jobs-dir`, `-n`, `-k`, `-r`, `--timeout-multiplier` and the per-phase multipliers) still exists on `harbor run`. `tests/test_harbor_version_compatibility.py` re-runs the task-loading half of that check; it skips unless Harbor is installed, so a version bump can be re-verified with: + +```bash +uv run --with harbor== pytest tests/test_harbor_version_compatibility.py +``` + +**What was not checked.** No end-to-end `harbor run` against live sites — that needs Docker, a browser container per trial, and judge credentials. Loading proves the dataset format is accepted, not that a full sweep passes. + +**On the local-path form.** `harbor run -p ./harbor-datasets/clawbench-v2` is still supported in 0.22.0; the newer `-d org/dataset@version` form addresses hub-published datasets and is an addition, not a replacement. Publishing ClawBench to the Harbor hub is tracked separately in [#331](https://github.com/TIGER-AI-Lab/ClawBench/issues/331). + ## 1. Convert V2 into a Harbor dataset ```bash @@ -59,7 +75,7 @@ Use `deepseek-v4-pro` if you want numbers comparable to the published leaderboar ## 3. Run it ```bash -uvx --from harbor==0.15.0 harbor run \ +uvx --from harbor==0.22.0 harbor run \ -p ./harbor-datasets/clawbench-v2 \ -a "" \ -m "" \ @@ -70,7 +86,7 @@ uvx --from harbor==0.15.0 harbor run \ --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}" ``` -Drop `uvx --from harbor==0.15.0` if Harbor is already installed. +Drop `uvx --from harbor==0.22.0` if Harbor is already installed. ### OpenClaw through an OpenAI-compatible endpoint @@ -78,7 +94,7 @@ Drop `uvx --from harbor==0.15.0` if Harbor is already installed. export OPENAI_BASE_URL="https://openrouter.ai/api/v1" export OPENAI_API_KEY="$OPENROUTER_API_KEY" -uvx --from harbor==0.15.0 harbor run \ +uvx --from harbor==0.22.0 harbor run \ -p ./harbor-datasets/clawbench-v2 \ -a openclaw \ -m openai/deepseek/deepseek-v4-flash \ @@ -96,7 +112,7 @@ uvx --from harbor==0.15.0 harbor run \ ```bash export OPENROUTER_API_KEY="your-openrouter-key" -uvx --from harbor==0.15.0 harbor run \ +uvx --from harbor==0.22.0 harbor run \ -p ./harbor-datasets/clawbench-v2 \ -a hermes \ -m deepseek/deepseek-v4-flash \ @@ -115,7 +131,7 @@ A full V2 sweep is 129 containerized browser sessions, each capped by the task's **1. Raise concurrency.** `-n / --n-concurrent` is the single biggest lever: ```bash -uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 \ +uvx --from harbor==0.22.0 harbor run -p ./harbor-datasets/clawbench-v2 \ -a hermes -m deepseek/deepseek-v4-flash -n 8 --env-file .env --ve ... ``` @@ -124,7 +140,7 @@ Each trial is a full Chromium container, so budget roughly **1 CPU core and ~2 G **2. Build the image once.** The first trial builds the ClawBench environment image; parallel cold starts all build at once. Warm the cache with the smoke dataset before the real sweep: ```bash -uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2-smoke \ +uvx --from harbor==0.22.0 harbor run -p ./harbor-datasets/clawbench-v2-smoke \ -a hermes -m deepseek/deepseek-v4-flash --env-file .env --ve ... ``` diff --git a/src/README.md b/src/README.md index 8fa8f728..e2b1c8e2 100644 --- a/src/README.md +++ b/src/README.md @@ -26,10 +26,10 @@ uv run clawbench-harbor-adapt --output-dir ./harbor-datasets/clawbench-v2 --over # Example Harbor runs after adaptation: # OpenClaw via OpenRouter/OpenAI-compatible API: -uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a openclaw -m openai/deepseek/deepseek-v4-flash --ak thinking=off --env-file .env +uvx --from harbor==0.22.0 harbor run -p ./harbor-datasets/clawbench-v2 -a openclaw -m openai/deepseek/deepseek-v4-flash --ak thinking=off --env-file .env # Hermes via OpenRouter: -uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a hermes -m deepseek/deepseek-v4-flash --env-file .env +uvx --from harbor==0.22.0 harbor run -p ./harbor-datasets/clawbench-v2 -a hermes -m deepseek/deepseek-v4-flash --env-file .env ``` The console scripts are defined in the root `pyproject.toml`: diff --git a/tests/test_harbor_version_compatibility.py b/tests/test_harbor_version_compatibility.py new file mode 100644 index 00000000..770e6ecd --- /dev/null +++ b/tests/test_harbor_version_compatibility.py @@ -0,0 +1,73 @@ +"""Does the adapter's output still load in the Harbor version the docs pin? + +Skipped unless Harbor is installed -- it is not a ClawBench dependency. The +point of the test is to make a version bump checkable instead of assumed: + + uv run --with harbor== pytest tests/test_harbor_version_compatibility.py + +The docs pinned harbor==0.15.0 for six releases past upstream (#294). Nothing +in the repo could tell you whether that mattered, because nothing ever loaded +a generated task with Harbor's own loader. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from clawbench.eval import harbor_adapter + +harbor_task = pytest.importorskip( + "harbor.models.task.task", + reason="Harbor is not installed; run with `uv run --with harbor==`", +) + +# The version docs/harbor.md pins and claims to have verified. A mismatch is +# not a failure -- the point is to make the tested version visible in output. +DOCUMENTED_HARBOR_VERSION = "0.22.0" + + +@pytest.fixture(scope="module") +def generated_dataset(tmp_path_factory: pytest.TempPathFactory) -> Path: + """Convert the real V2 corpus, not a synthetic task. + + A hand-built fixture would only prove the fixture loads. What has to hold + is that the corpus we tell people to convert produces a dataset Harbor + accepts. + """ + out = tmp_path_factory.mktemp("harbor-dataset") / "clawbench-v2" + rc = harbor_adapter.main(["--output-dir", str(out), "--overwrite"]) + assert rc == 0 + return out + + +def test_installed_harbor_version_is_reported(record_property) -> None: + from importlib.metadata import version + + installed = version("harbor") + record_property("harbor_version", installed) + if installed != DOCUMENTED_HARBOR_VERSION: + pytest.skip( + f"harbor {installed} installed, docs pin {DOCUMENTED_HARBOR_VERSION}; " + "the load test below still runs" + ) + + +def test_every_generated_task_loads_with_harbors_own_loader( + generated_dataset: Path, +) -> None: + task_dirs = sorted(d for d in generated_dataset.iterdir() if d.is_dir()) + assert task_dirs + + failures: list[tuple[str, str]] = [] + for task_dir in task_dirs: + if not harbor_task.Task.is_valid_dir(task_dir): + failures.append((task_dir.name, "Harbor does not recognise the directory")) + continue + try: + harbor_task.Task(task_dir) + except Exception as exc: # noqa: BLE001 - report whatever Harbor raises + failures.append((task_dir.name, f"{type(exc).__name__}: {exc}")) + + assert not failures, "\n".join(f"{name}: {why}" for name, why in failures)