feat(harbor): strict Harbor-format packaging — registry, Hub manifests, adapter package (V1+V2) - #332
feat(harbor): strict Harbor-format packaging — registry, Hub manifests, adapter package (V1+V2)#332reacher-z wants to merge 4 commits into
Conversation
… fix duplicate Star History - harbor/: README (git-registry / Hub / upstream-adapter channels), dataset.toml manifest (tiger-ai-lab/clawbench-v2, 129 task digests), job-config.yaml, and a harbor-framework-layout adapter package (harbor/adapter/) wrapping clawbench.eval.harbor_adapter. - registry.json: Harbor git registry so `harbor run --repo TIGER-AI-Lab/ClawBench -d clawbench-v2` works without conversion. - harbor/datasets/clawbench-v2/: committed prebuilt-mode tasks (129, 7 MB) that reference ghcr.io/tiger-ai-lab/clawbench-harbor-runtime:<version>; all 129 load with Harbor 0.22 TaskConfig. - clawbench-harbor-adapt --docker-image: prebuilt-image mode (no per-task environment/ build context); test added. - scripts/harbor/: build-runtime-image.sh, regenerate.sh, build_registry.py (digests identical to `harbor add`'s Packager content hash). - CI: validate-harbor.yml (stale-dataset + TaskConfig + manifest/registry agreement), publish-harbor-image.yml (GHCR on release tags). - docs/harbor.md: zero-conversion path, harbor pin 0.15.0 -> 0.22.0. - README + zh-CN: remove duplicated '## Star History' heading, center the chart, add Harbor git-registry row; .gitignore: ._* AppleDouble sidecars; CHANGELOG.
… entry
Decisions from the review console:
- clawbench-harbor-adapt --suite v1|v2: V1 (152 tasks) now exports alongside V2;
both committed under harbor/datasets/clawbench-{v1,v2} in prebuilt mode.
- Runtime image moves to Docker Hub: clawbench/clawbench-harbor-runtime
(publish-harbor-image.yml uses DOCKERHUB_USERNAME/DOCKERHUB_TOKEN secrets).
- harbor/dataset-v1.toml Hub manifest (tiger-ai-lab/clawbench-v1); registry.json
now lists both datasets (build_registry.py merges entries by name).
- validate-harbor.yml checks both suites (stale diff, TaskConfig load 281/281,
manifest/registry agreement).
- README + zh-CN: 'Run it via Harbor' cell in the What-are-you-looking-for grid
(FAQ/issue links kept in the cell footer); llms.txt URLs unified to TIGER-AI-Lab.
|
@Perry2004 — three items here need maintainer hands (decided in review; the rest of the pipeline is automated in this PR). Could you take them? 1. Docker Hub (blocks the git-registry channel)All 281 committed tasks reference
The image also builds fine locally: 2. Harbor Hub publish (needs a human GitHub OAuth)
Full walkthrough: 3. Review focusYou wrote the original adapter (#232), so the main judgment calls to check: prebuilt-image mode ( Parity (hermes × deepseek-v4-flash, 3 runs/side, full corpus) is being set up separately and is tracked in #331 — not a blocker for this PR. |
…_dir compliance) Harbor's TaskModel.is_valid_dir requires environment/ to exist even when [environment].docker_image is set, so prebuilt-mode tasks now ship environment/Dockerfile = 'FROM clawbench/clawbench-harbor-runtime:<ver>'. Verified: all 281 committed tasks pass Task.is_valid_dir on Harbor 0.22, and a podman-as-docker 'harbor run -a nop' trial boots the environment end-to-end.
prepare-task.py reads /app/src/harbor/resume_template.json; in env-mode the adapter stages it per task, but the prebuilt image built from the raw runtime tree lacked it, so every step setup.sh died with FileNotFoundError. The build script now stages a context with the template included. Verified in-container: setup.sh exit 0 (disposable email created, CDP + noVNC ready).
|
End-to-end validation update (podman-as-docker on a 16-core box,
Two real bugs were found by these runs and fixed in this branch:
The image is built and validated locally — once the |
Perry2004
left a comment
Perry2004
left a comment
There was a problem hiding this comment.
Generated harbor configs should be git ignored and removed, even from the previous commits, otherwise it will introduce amount of changes that is freezing GitHub webpage and impossible for review.
|
Also may I get some context/justification on why the Harbor adaptor is added as a new package rather than a new entrypoint/script in the existing clawbench-eval package? |
|
Please merge this with the v0.10.0 release changes as well since that also contains some harbor changes. |
2 participants
Summary
Implements the format-compliance half of #331: ClawBench packaged as a strict-format Harbor benchmark, distributed through three channels.
registry.json+ committed prebuilt-mode datasets, soharbor run --repo TIGER-AI-Lab/ClawBench -d clawbench-v2(or-d clawbench-v1) needs no conversion step.harbor/dataset.toml(tiger-ai-lab/clawbench-v2, 129 digests) +harbor/dataset-v1.toml(152 digests), ready forharbor publish --public.harbor/adapter/in the officialharbor adapter initlayout (README template,adapter_metadata.json,parity_experiment.jsonplaceholder,run_clawbench.yaml,src/clawbench_adapter/with the standard--output-dir --limit --overwrite --task-idsflags), to be copied intoharbor-framework/harbor/adapters/clawbenchonce parity is run.Key mechanics:
clawbench-harbor-adapt --docker-image <ref>: prebuilt-image mode — tasks referenceclawbench/clawbench-harbor-runtime:<version>(Docker Hub) instead of shipping a 280 KBenvironment/build context each (35 MB → 7 MB per corpus).clawbench-harbor-adapt --suite v1|v2: V1 (152 tasks) exports alongside V2 (129).scripts/harbor/regenerate.shrebuilds datasets +registry.json+ manifests deterministically;validate-harbor.ymlfails PRs when the committed copy is stale, loads all 281 tasks with Harbor 0.22TaskConfig, and cross-checks manifests vs registry.publish-harbor-image.ymlbuilds/pushes the runtime image on release tags or manual dispatch (needsDOCKERHUB_USERNAME/DOCKERHUB_TOKENrepo secrets).## Star Historyheading removed, chart centered; “Run it via Harbor” cell added to the top grid;llms.txtURLs unified to TIGER-AI-Lab.Validation
pytest tests/test_harbor_adapter.py tests/test_cli_entrypoints.py— 11 passedruff check/ruff format --checkclean on touched filesTaskConfigharbor add --scanoutput (Packager content hash)Not in this PR (tracked in #331)
harbor auth login+harbor publishto the Hub (interactive, maintainer-run)Closes nothing; first PR toward #331.
Manual tasks (assigned to @Perry2004)
See the checklist comment: (1) Docker Hub
clawbenchnamespace +DOCKERHUB_USERNAME/DOCKERHUB_TOKENrepo secrets + onepublish-harbor-imagedispatch (blocks the git-registry channel), (2) Harbor Hub publish —harbor auth loginis a browser OAuth flow no bot can complete, (3) the review itself.Additional blocker found during setup: the team OpenRouter key in
models/models.yamlis expired (401 API key expired), which blocks the parity runs (hermes × deepseek/deepseek-v4-flash, judge deepseek-v4-pro). Needs a refreshed OpenRouter key — or the 2077AI-sponsored parity keys via the Harbor adapters team (adapters/parity_api_instructions.md, contact Lin Shi on the Harbor Discord).