Publish ThinkingBox-Bench viewer data

.github/workflows/code-quality.yml

@@ -31,3 +31,9 @@ jobs:
     - name: Run pre-commit hooks
       run: |
         pre-commit run --all-files
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+
+    - name: Verify Hugging Face publishing bundle
+      run: uv run scripts/export_huggingface_viewer.py --check

README.md

@@ -16,6 +16,11 @@ Each task runs in an isolated, stateful tool environment and is evaluated with
 executable checks over the final backend state and side effects. Some tasks
 also check required properties of the final response.
 
+Browse the 507 tasks, shared scenarios, and agent configuration in the
+[ThinkingBox-Bench dataset on Hugging Face](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench).
+The Hugging Face dataset is a viewer-friendly representation; this GitHub
+repository remains the executable source.
+
 If your goal is to run the published benchmark, follow the complete
 [ThinkingBox-Bench v1.0 installation and run instructions](releases/thinkingbox_bench_v1/README.md#run-the-benchmark)
 directly. The remaining setup and examples in this README are intended for

releases/thinkingbox_bench_v1/README.md

@@ -39,6 +39,52 @@ so these alternatives do not increase the task count.
 Other datasets and tests in this repository are not part of ThinkingBox-Bench
 v1.0.
 
+## Hugging Face Dataset Viewer
+
+Browse this release in the public
+[ThinkingBox-Bench dataset on Hugging Face](https://huggingface.co/datasets/microsoft/ThinkingBox-Bench).
+Its Viewer exposes three linked subsets:
+
+| Subset | Rows | Contents |
+| - | -: | - |
+| `tasks` | 507 | User goal, initial-state patch, expected tool interactions, and rubrics |
+| `scenarios` | 5 | Shared world state and available tools, linked by `scenario_id` |
+| `agents` | 1 | Agent instructions and built-in tools |
+
+### Maintainer workflow
+
+The generated [`huggingface/`](huggingface/) directory is a self-contained
+publishing bundle for a separate Hugging Face dataset repository. It provides
+the Viewer data without copying the executable benchmark.
+
+Regenerate the Parquet files from the canonical release sources:
+
+```bash
+uv run scripts/export_huggingface_viewer.py
+```
+
+Verify that the committed exports are current:
+
+```bash
+uv run scripts/export_huggingface_viewer.py --check
+```
+
+Publishing updates under `microsoft/` requires membership in the Microsoft
+Hugging Face organization with write access. Confirm the authenticated account
+before uploading:
+
+```bash
+hf auth login
+hf auth whoami
+hf upload microsoft/ThinkingBox-Bench \
+    releases/thinkingbox_bench_v1/huggingface . \
+    --repo-type dataset
+```
+
+The Hugging Face repository contains the dataset card, license, and derived
+Parquet tables. Users run the benchmark from the tagged GitHub sources
+documented below.
+
 ## Evaluation method
 
 Each task compares the final backend state with its golden expected state using

releases/thinkingbox_bench_v1/huggingface/LICENSE.txt

@@ -0,0 +1,104 @@
+This repository is dual-licensed by content type:
+
+  - Software (code under servers/ and any first-party Python in this repo)
+    is licensed under the MIT License (SPDX: MIT) — see the MIT section
+    below.
+
+  - Data (content under dataset/, support/, and releases/) is licensed under
+    the Community Data License Agreement – Permissive – Version 2.0 (SPDX:
+    CDLA-Permissive-2.0) — see the CDLA section below. The canonical text
+    is published at https://cdla.dev/permissive-2-0/.
+
+When redistributing data from this repository, include this LICENSE.txt
+alongside the redistributed data, as required by section 2.1 of the CDLA.
+
+
+================================================================================
+MIT License (applies to software / code)
+================================================================================
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+
+================================================================================
+CDLA-Permissive-2.0 (applies to data)
+================================================================================
+
+Copyright (c) 2026 Microsoft Corporation
+
+
+Community Data License Agreement – Permissive – Version 2.0
+
+This is the Community Data License Agreement – Permissive, Version 2.0 (the
+"agreement"). Data Provider(s) and Data Recipient(s) agree as follows:
+
+1. Provision of the Data
+
+1.1. A Data Recipient may use, modify, and share the Data made available by
+Data Provider(s) under this agreement if that Data Recipient follows the terms
+of this agreement.
+
+1.2. This agreement does not impose any restriction on a Data Recipient's use,
+modification, or sharing of any portions of the Data that are in the public
+domain or that may be used, modified, or shared under any other legal
+exception or limitation.
+
+2. Conditions for Sharing Data
+
+2.1. A Data Recipient may share Data, with or without modifications, so long
+as the Data Recipient makes available the text of this agreement with the
+shared Data.
+
+3. No Restrictions on Results
+
+3.1. This agreement does not impose any restriction or obligations with
+respect to the use, modification, or sharing of Results.
+
+4. No Warranty; Limitation of Liability
+
+4.1. All Data Recipients receive the Data subject to the following terms:
+
+THE DATA IS PROVIDED ON AN "AS IS" BASIS, WITHOUT REPRESENTATIONS, WARRANTIES
+OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT
+LIMITATION, ANY WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT,
+MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
+
+NO DATA PROVIDER SHALL HAVE ANY LIABILITY FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING WITHOUT
+LIMITATION LOST PROFITS), HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
+OTHERWISE) ARISING IN ANY WAY OUT OF THE DATA OR RESULTS, EVEN IF ADVISED OF
+THE POSSIBILITY OF SUCH DAMAGES.
+
+5. Definitions
+
+5.1. "Data" means the material received by a Data Recipient under this
+agreement.
+
+5.2. "Data Provider" means any person who is the source of Data provided
+under this agreement and in reliance on a Data Recipient's agreement to its
+terms.
+
+5.3. "Data Recipient" means any person who receives Data directly or
+indirectly from a Data Provider and agrees to the terms of this agreement.
+
+5.4. "Results" means any outcome obtained by computational analysis of Data,
+including for example machine learning models and models' insights.

releases/thinkingbox_bench_v1/huggingface/README.md

@@ -0,0 +1,99 @@
+---
+pretty_name: ThinkingBox-Bench
+license: cdla-permissive-2.0
+language:
+  - en
+task_categories:
+  - reinforcement-learning
+tags:
+  - agent
+  - tool-use
+  - benchmark
+  - evaluation
+configs:
+  - config_name: tasks
+    default: true
+    data_files:
+      - split: test
+        path: data/tasks.parquet
+  - config_name: scenarios
+    data_files:
+      - split: test
+        path: data/scenarios.parquet
+  - config_name: agents
+    data_files:
+      - split: test
+        path: data/agents.parquet
+---
+
+# ThinkingBox-Bench
+
+ThinkingBox-Bench is an executable benchmark for evaluating whether tool-using
+LLM agents can reliably complete stateful business workflows. Version 1.0
+contains 507 tool-agent-user tasks across retail and e-commerce, travel and
+hospitality, auto insurance, neobank support, and consulting IT/HR support.
+
+This dataset repository provides a browsable representation of the benchmark.
+The executable benchmark, tool servers, and supporting fixtures are maintained
+in the
+[`microsoft/thinkingbox-data`](https://github.com/microsoft/thinkingbox-data)
+GitHub repository.
+
+## Dataset structure
+
+| Subset | Rows | Contents |
+| - | -: | - |
+| `tasks` | 507 | User goal, initial-state patch, expected tool interactions, and rubrics |
+| `scenarios` | 5 | Shared world state and available tools, linked by `scenario_id` |
+| `agents` | 1 | Agent instructions and built-in tools |
+
+Select a subset using the Dataset Viewer dropdown. Each task references its
+shared scenario through `scenario_id`. Nested task state and expected
+interactions are serialized as JSON strings so they remain readable and
+portable in the Viewer.
+
+### Task fields
+
+| Field | Description |
+| - | - |
+| `task_ref` | Canonical `file.py:function_name` identifier from the release test list |
+| `domain` | Human-readable benchmark domain |
+| `scenario_id` | Key linking the task to its shared row in the `scenarios` subset |
+| `query` | Initial request sent by the simulated user |
+| `user_context` | Instructions and facts available to the simulated user |
+| `initial_state_patch_json` | JSON object applied to the scenario's base world state before the task starts |
+| `expected_tool_interactions_json` | Ordered golden tool calls that define the expected state changes |
+| `rubrics_json` | Additional response requirements evaluated for applicable tasks |
+| `source_url` | Tagged GitHub source containing the executable test definition |
+| `release_tag` | Immutable `thinkingbox-data` release used to generate the row |
+
+The dataset does not store a precomputed expected end state. During evaluation,
+the scenario's MCP server creates a fresh database, applies
+`initial_state_patch_json`, and replays `expected_tool_interactions_json` to
+materialize the `golden_db_state`. It then compares the stable hash of that
+state with the hash of the database modified by the evaluated agent. This
+runtime process ensures the expected state uses the same tool implementation
+and database semantics as the agent's attempt.
+
+## Intended use
+
+ThinkingBox-Bench v1.0 is intended exclusively for evaluation. Do not use its
+task content, expected outcomes, golden state, or tool trajectories for prompt
+optimization, fine-tuning, reinforcement learning, reward-model training, or
+other model optimization.
+
+## Run the benchmark
+
+The Parquet tables are for browsing and analysis; they are not the executable
+runtime. Follow the
+[ThinkingBox-Bench v1.0 instructions](https://github.com/microsoft/thinkingbox-data/blob/thinkingbox-bench-v1.0/releases/thinkingbox_bench_v1/README.md#run-the-benchmark)
+to install ThinkingBox, start the required services, and run all 507 tasks.
+
+For reproducibility, use the
+[`thinkingbox-bench-v1.0`](https://github.com/microsoft/thinkingbox-data/releases/tag/thinkingbox-bench-v1.0)
+release.
+
+## License
+
+The dataset is licensed under the Community Data License Agreement -
+Permissive - Version 2.0 (`CDLA-Permissive-2.0`). See `LICENSE.txt`.