MCB: introduce opt-in pinned GSM8K, SQuAD 2.0 and JSONSchemaBench suite

.github/workflows/mcb-public-benchmarks.yml

@@ -0,0 +1,58 @@
+name: MCB public benchmark integration
+
+on:
+  pull_request:
+    paths:
+      - "experiments/model-capability-benchmark/**"
+      - "packages/benchmark-core/**"
+      - ".github/workflows/mcb-public-benchmarks.yml"
+  workflow_dispatch:
+
+jobs:
+  public-benchmarks:
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    defaults:
+      run:
+        working-directory: experiments/model-capability-benchmark
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install uv
+        run: python -m pip install --disable-pip-version-check uv
+      - name: Sync locked dependencies
+        run: uv sync --frozen --extra dev
+      - name: Lint
+        run: uv run --frozen ruff check src tests/test_public_benchmarks.py
+      - name: Test public adapters and evaluators
+        run: uv run --frozen pytest -q tests/test_public_benchmarks.py
+      - name: Plan public Core without provider calls
+        run: uv run --frozen model-bench --suite suite_public.yaml plan --profile core --capabilities all
+      - name: Smoke-test real pinned upstream data without provider calls
+        run: |
+          uv run --frozen python - <<'PY'
+          from pathlib import Path
+          from benchmark_core import DatasetLoadContext
+          from model_capability_bench import load_capability_suite
+          suite = load_capability_suite(Path.cwd(), suite_filename="suite_public.yaml")
+          cache = Path.cwd() / ".cache/public-ci"
+          expected = {
+              "gsm8k-main-test": 12,
+              "squad-v2-validation": 12,
+              "jsonschemabench-github-easy-test": 12,
+          }
+          for dataset_id, count in expected.items():
+              result = suite.datasets.load(
+                  dataset_id,
+                  DatasetLoadContext(
+                      cache_dir=cache,
+                      profile=suite.profiles["smoke"],
+                      seed=42,
+                  ),
+              )
+              assert len(result.samples) == count, (dataset_id, len(result.samples))
+              assert result.source_checksums.get("source")
+              print(dataset_id, len(result.samples), result.available_count, result.selection_fingerprint)
+          PY

experiments/model-capability-benchmark/PUBLIC_BENCHMARKS.md

@@ -0,0 +1,77 @@
+# MCB public benchmark suite (opt-in)
+
+MCB has two independent suites. **The historical default stays unchanged**:
+
+- `suite.yaml` / `capability-core` v3: 468 controlled/public-label cases, including all previous benchmark revisions.
+- `suite_public.yaml` / `capability-public-benchmarks` v1: 140 small, reproducible public benchmark test/validation cases in Core.
+
+Public suite is an **MCB protocol on pinned public data**, **not** the official full dataset leaderboard protocol.
+We do not publish a 40-case GSM8K score as an official GSM8K full-test score.
+
+## Sources and provenance
+
+| MCB dataset ID | Upstream | Upstream split | License | Core |
+| --- | --- | --- | --- | ---: |
+| `gsm8k-main-test` | [openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k) | `main/test` (1,319 total) | MIT | 40 |
+| `squad-v2-validation` | [rajpurkar/squad_v2](https://huggingface.co/datasets/rajpurkar/squad_v2) | `validation` (11,873 total) | CC BY-SA 4.0 | 60 (30 answerable + 30 unanswerable) |
+| `jsonschemabench-github-easy-test` | [epfl-dlab/JSONSchemaBench](https://huggingface.co/datasets/epfl-dlab/JSONSchemaBench) | `Github_easy/test` (577 upstream) | MIT | 40 |
+
+Each source URL in `datasets.yaml` pins a **full immutable upstream commit SHA**, the adapter uses the original Parquet file, and the cached source's SHA-256 is preserved in `source_checksums`. Dataset IDs, revisions, selected source indices, source splits, profiles and selection fingerprints persist in evidence. Case IDs never depend on random order. This requires no new dependency: MCB reads Parquet with its existing DuckDB dependency.
+
+The upstream dataset may have known contamination in pretrained models. Public provenance alone does **not** prove model training independence.
+
+**SQuAD protocol:** MCB asks for JSON `{answer, abstain}`, uses the official SQuAD text normalization (lowercase, remove punctuation, articles and extra whitespace), and accepts every upstream gold alias. It scores `qa_correct`, which requires a correct answer on answerable items and a correct abstention on impossible items. It does not compute the official SQuAD v2 threshold-tuned F1/EM leaderboard metric.
+
+**GSM8K protocol:** MCB extracts the upstream final numeric gold after `####`, normalizes decimals/commas and evaluates the final answer. The original worked solution is not leaked to the prompt. This is an MCB zero-shot JSON-output evaluation, not necessarily prompt-parity with an external published number.
+
+**JSONSchemaBench protocol:** MCB tests JSON Schema compliance, **not** semantic extraction or JSONSchemaBench's official constrained-decoding engine timings/coverage. Its v1 adapter deliberately selects only object-root schemas that satisfy local validator checks, have locally resolvable references and at most 12,000 schema characters. Rejected upstream rows are counted (`source_rows`, `excluded_rows`, `eligible_rows`) to make this reduced subset explicit. Unsupported remote references are never automatically fetched. The upstream dataset has no gold data values: `expected=None` is correct.
+
+## Commands
+
+From `experiments/model-capability-benchmark`:
+
+```bash
+uv sync --frozen --extra dev
+
+# Read configuration before downloading anything
+uv run --frozen model-bench --suite suite_public.yaml capabilities
+uv run --frozen model-bench --suite suite_public.yaml plan --profile core --capabilities all
+
+# Real preflight and cost/latency pilot on one selected model
+export MODEL=qwen3.5-2b-q4km
+uv run --frozen model-bench --suite suite_public.yaml validate-config --models "$MODEL"
+uv run --frozen model-bench --suite suite_public.yaml estimate \
+  --models "$MODEL" --profile core --capabilities all --pilot-cases 3
+
+# Distinct run group: never overwrite legacy results
+uv run --frozen model-bench --suite suite_public.yaml run \
+  --run-group public-benchmarks-v1 \
+  --models "$MODEL" --capabilities all --profile core
+
+# Offline projection and dashboard export from immutable run evidence
+uv run --frozen model-bench project --rebuild --export-dashboard
+```
+
+Existing default commands without `--suite` still run historical `suite.yaml`. Use the same
+public suite filename, seed, profile, commit, source artifact and evaluator versions for every compared model.
+
+Case budgets: `smoke = 12+12+12 = 36`, `core = 40+60+40 = 140`,
+`extended = 200+200+150 = 550`. Source files are cached and downloaded only on first
+load. Estimated run time is model-dependent. `plan` does not contact models or download data;
+`estimate` and `run` do.
+
+## Science and interpretation guards
+
+- MCB stores **raw sources and actual selected case fingerprints**, so any same-case paired comparison remains reproducible.
+- Do not compare public v1 scores directly to controlled v2/v3 scores, or call them equivalent despite both measuring math or QA.
+- Do not give official benchmark leaderboard labels or cite official test totals as run sample counts.
+- Report paired `n`, confidence intervals, invalid-output rates and missing resource data separately.
+- SQuAD and JSONSchemaBench use different concepts of correctness; aggregating into one undifferentiated score is not recommended.
+- JSONSchemaBench's object/root-size/reference filtering must remain versioned and visible, because it affects measured difficulty.
+- Before external publication, perform a real-provider smoke + canonical run and review a sample of gold normalization and schema evaluator outputs.
+
+## Recommended follow-on
+
+Add a second JSONSchemaBench subset (`Github_medium`) only after the object-root compliance path passes
+real provider validation. Add structured-output **semantic extraction** from benchmark sources separately;
+schema compliance alone cannot replace the original controlled extraction suite.

experiments/model-capability-benchmark/README.md

@@ -103,6 +103,22 @@ read from the environment and is never written to the registry or manifest.
 
 See `.env.example` for the currently supported environment variables.
 
+## Public benchmark suite (opt-in)
+
+A second, independently versioned suite (`suite_public.yaml`) integrates pinned upstream
+test/validation subsets of **GSM8K**, **SQuAD 2.0** and **JSONSchemaBench**.
+It has 36 smoke / 140 core / 550 extended cases and does not replace the canonical
+468-case `suite.yaml`. The UI and persisted evidence keep the benchmark lineages distinct.
+JSONSchemaBench measures schema compliance, not semantic field extraction.
+
+See **[PUBLIC_BENCHMARKS.md](PUBLIC_BENCHMARKS.md)** for sources, data licenses,
+normalization/scoring semantics, reproducible run commands and the explicit
+differences from official full-benchmark leaderboard protocols.
+
+```bash
+uv run --frozen model-bench --suite suite_public.yaml plan --profile core --capabilities all
+```
+
 ## Capability suite
 
 `suite.yaml` now composes five capability families: BANKING77 intent classification,

experiments/model-capability-benchmark/datasets.yaml

@@ -156,3 +156,54 @@ datasets:
       path: data/math-reasoning-controlled-v3.yaml
       human_reviewed: false
       difficulty_labels: authored-provisional
+
+  # Public benchmarks: immutable Hugging Face dataset revisions and original test splits.
+  gsm8k-main-test:
+    version: "1"
+    adapter: public-parquet
+    source: openai/gsm8k
+    revision: 740312add88f781978c0658806c59bc2815b9866
+    split: test
+    license: MIT
+    sample_schema_version: "1"
+    cache_mode: download
+    options:
+      timeout_seconds: 120
+      user_agent: model-capability-bench/0.1
+      file:
+        url: https://huggingface.co/datasets/openai/gsm8k/resolve/740312add88f781978c0658806c59bc2815b9866/main/test-00000-of-00001.parquet
+        cache_path: gsm8k/740312add88f781978c0658806c59bc2815b9866/test.parquet
+
+  squad-v2-validation:
+    version: "1"
+    adapter: public-parquet
+    source: rajpurkar/squad_v2
+    revision: 3ffb306f725f7d2ce8394bc1873b24868140c412
+    split: validation
+    license: CC-BY-SA-4.0
+    sample_schema_version: "1"
+    cache_mode: download
+    options:
+      timeout_seconds: 120
+      user_agent: model-capability-bench/0.1
+      file:
+        url: https://huggingface.co/datasets/rajpurkar/squad_v2/resolve/3ffb306f725f7d2ce8394bc1873b24868140c412/squad_v2/validation-00000-of-00001.parquet
+        cache_path: squad-v2/3ffb306f725f7d2ce8394bc1873b24868140c412/validation.parquet
+
+  jsonschemabench-github-easy-test:
+    version: "1"
+    adapter: public-parquet
+    source: epfl-dlab/JSONSchemaBench
+    revision: 83df923c5b1dd05fb5c9659ad751d7c056a3cd43
+    split: test
+    license: MIT
+    sample_schema_version: "1"
+    cache_mode: download
+    options:
+      subset: Github_easy
+      max_schema_chars: 12000
+      timeout_seconds: 120
+      user_agent: model-capability-bench/0.1
+      file:
+        url: https://huggingface.co/datasets/epfl-dlab/JSONSchemaBench/resolve/83df923c5b1dd05fb5c9659ad751d7c056a3cd43/Github_easy/test-00000-of-00001.parquet
+        cache_path: jsonschemabench/83df923c5b1dd05fb5c9659ad751d7c056a3cd43/github_easy_test.parquet

experiments/model-capability-benchmark/profiles.yaml

@@ -3,6 +3,9 @@ profiles:
     max_cases: 20
     datasets:
       banking77: 77
+      gsm8k-main-test: 12
+      squad-v2-validation: 12
+      jsonschemabench-github-easy-test: 12
       clinc150-oos: 50
       structured-output-controlled-v1: null
       qa-abstention-controlled-v1: null
@@ -16,6 +19,9 @@ profiles:
     max_cases: 100
     datasets:
       banking77: 154
+      gsm8k-main-test: 40
+      squad-v2-validation: 60
+      jsonschemabench-github-easy-test: 40
       clinc150-oos: 100
       structured-output-controlled-v2: 60
       qa-abstention-controlled-v2: 60
@@ -26,6 +32,9 @@ profiles:
     max_cases: 500
     datasets:
       banking77: 770
+      gsm8k-main-test: 200
+      squad-v2-validation: 200
+      jsonschemabench-github-easy-test: 150
       clinc150-oos: 500
       structured-output-controlled-v2: 60
       qa-abstention-controlled-v2: 60

experiments/model-capability-benchmark/src/model_capability_bench/cli.py

@@ -60,7 +60,9 @@ def _selection(value: str, available: list[str]) -> tuple[str, ...]:
 
 def _select_capabilities(bundle, raw_value: str) -> tuple[str, ...]:
     canonical = [c.spec.capability_id for c in bundle.resolved_capabilities]
-    available = canonical + list(CAPABILITY_ALIASES.keys())
+    available = canonical + [
+        alias for alias, target in CAPABILITY_ALIASES.items() if target in canonical
+    ]
     selected = _selection(raw_value, available)
     return tuple(dict.fromkeys(resolve_capability_id(c) for c in selected))
 
@@ -71,6 +73,10 @@ def build_parser() -> argparse.ArgumentParser:
         description="Unified model capability benchmark runner.",
     )
     parser.add_argument("--root", type=Path, default=ROOT)
+    parser.add_argument(
+        "--suite", default="suite.yaml",
+        help="Versioned suite YAML basename (e.g. suite_public.yaml); place before command.",
+    )
     sub = parser.add_subparsers(dest="command", required=True)
 
     run = sub.add_parser("run", help="Execute a model × capability benchmark matrix.")
@@ -284,7 +290,7 @@ def main() -> int:
         output = args.output_dir or root / "results" / "runs" / args.run_id
         summary = consolidate_runs(
             sources=args.run_dir, output_dir=output, run_id=args.run_id,
-            run_group=args.run_group, suite=load_capability_suite(root), plan_only=args.plan_only,
+            run_group=args.run_group, suite=load_capability_suite(root, suite_filename=args.suite), plan_only=args.plan_only,
         )
         if not args.plan_only:
             reporting_config = load_reporting_config(root)
@@ -448,7 +454,7 @@ def main() -> int:
         )
         return 0
 
-    bundle = load_capability_suite(root)
+    bundle = load_capability_suite(root, suite_filename=args.suite)
     defaults = load_runner_defaults(root)
 
     if args.command in {"models", "tasks", "datasets", "capabilities"}:

experiments/model-capability-benchmark/src/model_capability_bench/datasets/__init__.py

@@ -1,6 +1,7 @@
 from model_capability_bench.datasets.banking77 import Banking77Dataset
 from model_capability_bench.datasets.clinc150 import Clinc150OosDataset
 from model_capability_bench.datasets.controlled_yaml import ControlledYamlDataset
+from model_capability_bench.datasets.public_benchmarks import PublicParquetBenchmark
 from model_capability_bench.datasets.registry import (
     build_dataset_plugins,
     build_dataset_registry,
@@ -13,6 +14,7 @@
     "Banking77Dataset",
     "Clinc150OosDataset",
     "ControlledYamlDataset",
+    "PublicParquetBenchmark",
     "StructuredOutputControlledDataset",
     "build_dataset_plugins",
     "build_dataset_registry",