ci(validate-task): validate the claw-eval suite, which the job never saw

[vaibhavdabas16]

What does this PR do?

Fixes #292.

The workflow's paths: filter listed test-cases/**/task.json. claw-eval stores one flat file per task — test-cases/claw-eval/ce-T046-cve-research.json and 18 siblings — which that pattern never matches. Editing or adding a claw-eval task ran no schema validation at all.

It is not only the trigger

The issue supposes "the gap is the trigger". It isn't only that — every selection rule inside the job was task.json-only too, so broadening paths: alone would have made the workflow fire and still validate nothing. Four places had to change:

	before
pull_request / push filter	test-cases/**/task.json
workflow_dispatch fallback	find test-cases -path "*/task.json"
schema-changed glob	repo.glob("test-cases/**/task.json")
per-changed-path test	if path.name == "task.json"

Scope, stated plainly

claw-eval was not entirely unchecked, and the issue's impact line slightly overstates it. tests/test_host_tasks.py parametrises over CASE_SUITES and its _task_files_for_suite does glob flat files — but it calls validate_task_data(), which checks only instruction, eval_schema.url_pattern, eval_schema.method and time_limit (run_support/task.py:246-265). The full task.schema.json validation and the extra_info path-existence check never reached claw-eval. That is the actual gap, and it is narrower than "no validation".

Why the logic moved to a script

The job's Python lived in a heredoc inside the YAML, where nothing could exercise it — which is precisely how four copies of the same wrong rule went unnoticed. It moves to scripts/ci/validate_tasks.py, beside the existing scripts/ci/check_markdown_links.py, and is_task_file() now states both layouts in one place and is tested directly.

Flagging this as a judgment call: it makes the diff bigger than a one-line filter change. I think it's warranted — an untestable rule is what let this survive — but say the word and I'll inline it back and keep only the four rule fixes.

Two things fixed while extracting:

• Paths are re-rooted at test-cases/ before matching, so an absolute path behaves like the repo-relative ones git reports. The old rule tested parts[0] == "test-cases", so any absolute path silently matched nothing — a live trap for the --all path.
• --all no longer lists every task as "changed JSON" as well, which double-reported a broken file.

All 19 claw-eval tasks validate cleanly against task.schema.json today, so this turns the job on without turning it red. Worth confirming explicitly — a trigger fix that surfaced 19 broken tasks would be a different PR.

Corpus

• v2
• v1
• both
• not applicable

_{CI and tooling only. No task data is modified — the point is that existing task data finally gets checked.}

Test plan

• New tests/test_validate_tasks_script.py, 20 tests: both corpus layouts recognised; the non-tasks that must not be validated as tasks (the schema itself, extra_info payloads, per-suite reports, non-JSON); deleted paths (a git diff lists deletions, and validating a vanished path would fail the job on a legitimate task removal); extra_info ownership; a schema change fanning out to all four suites; absolute vs relative paths; a malformed task actually being reported; and the workflow trigger itself.
• Control. I restored the previous */task.json-only rules — in both the script and the paths: filter — held the tests constant, and re-ran: 6 of the 20 fail, including the two that name the bug (test_the_flat_suite_is_actually_discovered, test_the_workflow_trigger_matches_flat_suite_files).
• python scripts/ci/validate_tasks.py --all run locally: the 19 claw-eval tasks pass. The only errors reported are the 20 v1-lite task.json files, which are git symlinks (mode 120000) that this Windows checkout materialises as text — the same known local artifact behind the deselection below, green on Linux.
• Full suite: 230 passed, 4 skipped. The 4 deselected are test_host_tasks.py::test_checked_task_json_files_parse_and_validate, the same Windows symlink artifact.
• ruff check / ruff format --check clean, no new findings anywhere. pyright --pythonplatform Linux clean on src/clawbench, tests, and the new script (which CI's pyright config does not cover, checked anyway).
• jsonschema is already a dev dependency and already imported by tests/test_harness_registry.py, so the new test adds no dependency to the pytest job.
• Merge-clean against main and against all six of my other open PRs.

Not verified: GitHub evaluates paths: filters server-side, so the trigger change is asserted against the workflow file rather than observed firing. The first push that touches only a claw-eval task will be the real proof.

Related issues

Fixes #292.

[Perry2004]

Thanks for the contribution. Though I'd suggest an alternative approach:

• Instead of changing the workflow it's better to use this chance to re-organize the claw-eval port's structure. It should go with the standard test-cases/claw-eval/<task-identifier>/task.json structure as native ClawBench tasks.
• After the structure change we can then simplify the TUI and batch discovery logic to a standard task.json search.
• I believe there's no need for extracting the json schema validation logic to a separate script and add tests on it given the simplicity of the logic.