ci(validate-task): validate flat suites, not just <case>/task.json

[vaibhavdabas16]

What does this PR do?

Fixes #292.

validate-task.yml fired only on test-cases/**/task.json, which cannot match claw-eval's 19 flat files (test-cases/claw-eval/ce-T046-cve-research.json, …). That suite is a first-class --cases-suite target — registered in CASE_SUITES and offered in the TUI — so a malformed task there reached run time instead of failing review.

The trigger was only half of it

The issue guessed the checker itself "likely" handled flat files. It did not. The collector inside the workflow keyed on path.name == "task.json", and the validate-everything branch globbed test-cases/**/task.json. So even after broadening the paths: filter, the workflow would have fired and then validated nothing from that suite — a green check that still proved nothing.

Both halves are fixed, and both have tests.

Recognising the two layouts

The rule is taken from what the runner already does rather than invented here: batch._flat_case_files and tests/test_host_tasks._task_files_for_suite both treat every <suite>/*.json except eligibility-report.json as a case. test_the_flat_predicate_matches_the_runners_own_rule pins the two definitions together, so CI cannot quietly start checking a different set of files than the runner executes.

Broadening the trigger to test-cases/**/*.json also means more non-task JSON reaches the collector, so the exclusions (task.schema.json, eligibility-report.json, anything under extra_info/, non-.json) are covered by their own test.

Why the logic moved to a script

scripts/ci/validate_tasks.py, following the pattern scripts/ci/check_markdown_links.py already set for validate-docs.yml. A 90-line Python heredoc embedded in YAML cannot be tested, which is precisely how this gap survived — the trigger and the collector were both wrong and nothing could have caught either. The collection rules now have coverage.

While moving it: the workflow_dispatch path used to build a file list with find only for Python to re-derive the same set; it now calls the script with --all.

Corpus

• v2
• v1
• both
• not applicable

_{CI plumbing only — no task content changed. It does newly validate claw-eval, see below.}

Test plan

• Confirmed this does not turn CI red on merge. All 19 claw-eval tasks already validate against test-cases/task.schema.json, and none of them use extra_info. Turning on a check for a suite that has never been checked is the obvious risk here, so I verified it before changing the trigger rather than after. test_the_real_claw_eval_suite_validates keeps that true.
• New tests/test_validate_tasks_script.py, 20 tests: both layouts recognised; non-task JSON excluded; the flat predicate agreeing with the runner's; changed flat task / changed nested task / changed extra_info pulling in its owner; a schema change re-validating both layouts; a deleted file not being read; a malformed flat task actually being reported; and the workflow's own paths: filter asserted for pull_request and push.
• Ran the extracted validator over the whole corpus: 300 task files excluding v1-lite, 0 errors, with the 19 flat claw-eval tasks now included where previously they were invisible. (v1-lite's task.json entries are git symlinks, mode 120000, that this Windows checkout materialises as text; they resolve normally on the ubuntu runner.)
• Full suite: 230 passed, 4 skipped.
• ruff format --check clean. For ruff check I verified against the ruleset CI actually resolves (E4,E7,E9,F — pyproject.toml sets no select): both new files pass. The # noqa: E402 on the script import is load-bearing — removing it produces E402 Module level import not at top of file under that exact ruleset, which I checked rather than assumed.
• pyright --pythonplatform Linux src/clawbench tests: 0 errors.

Note on the pyproject.toml change

[tool.pyright] gains extraPaths so the test can import the CI script. "src" is listed alongside "scripts/ci" because setting extraPaths replaces the implicit source root — with only "scripts/ci" there, pyright stops resolving clawbench itself and reports 32 errors in pre-existing test files. Flagging it since it's the one change outside CI's own files.

Related issues

Fixes #292. Related: #291's markdown link checker established the scripts/ci/ pattern this follows.

[vaibhavdabas16]

Superseded by #334, which takes @Perry2004's suggested approach: instead of teaching the CI script to validate flat suites, the claw-eval port moves to the standard <task-identifier>/task.json layout every native corpus uses. With no flat suites left there is nothing here to teach — scripts/ci/validate_tasks.py is dropped from #334 as well, and the workflow is left untouched.