Adding ThinkingBox to OpenEnv - #1092
Conversation
Adds an OpenEnv adapter for the ThinkingBox stateful tool-use benchmark. The server hydrates tasks from a pinned thinkingbox-data release, runs the agent through MCP Session Proxy tools, simulates the user, collects effects, and grades with native ThinkingBox assertions and judge. Private user context, credentials, and grading internals stay server-side.
eval_testlist.py provides canonical benchmark evaluation with resume, provenance, coverage validation, and native aggregation. example_usage.py demonstrates a single-task run through the OpenEnv WebSocket interface.
81 focused offline tests covering environment lifecycle, trust boundary, data integrity, batch semantics, policy isolation, coverage validation, and deployment helpers. Adds a dedicated Python 3.12 CI job since the generic suite skips ThinkingBox tests when the package is absent.
Adds README with quick start, architecture, configuration, deployment boundary, and citation. Generates the docs stub, adds the environment card, and updates the repository walkthrough.
Adds generic SKIP_HF_DEPLOYMENT opt-out to prepare_hf_deployment.sh so automatic Space discovery excludes environments requiring external topology. Adds .dockerignore to exclude development artifacts from the build context.
TuhinKundu
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
hey @TuhinKundu! 🤗
thanks a lot for opening this PR and for sharing the dataset on the Hub, really nice to see
this continue the conversation you started with my colleague Niels :)
I went through the code and I also deployed the environment to a private HF Space, because
I wanted to see it running instead of just reading it. And it worked! The image built, the
app booted, GET /health returned 200, and with a real UID from the manifest it pulled the
pinned release, extracted it and verified the bundle sha256, so GET /ready flipped
"data" to true. Provenance metadata came out complete too.
That also confirms step 6 of your test plan independently. I used your Dockerfile as is and
got exactly the three results you expected (build succeeds, /health 200, /ready 503). The
Space is deleted now, the outputs below are the record.
Overall this is careful work. The privacy boundary is real and consistently drawn, the data
loader does safe tar extraction with traversal and symlink guards plus atomic publication
under a file lock, and terminal-observation latching through a shielded task is correct.
I left the rest of my notes as inline comments on the files they belong to, so this one only
covers the cross-cutting parts. Full disclosure, I used an agent to help me work through a
diff this size, so please treat those as a checklist rather than a verdict and push back on
anything that does not apply. Thanks again for bringing this here!
Where it stops
The one thing it could not do is serve an actual episode:
{"kind": "error", "task_uid": "sandbox_external_retail_group1.py:test_case_ST002_001",
"finish_reason": "reset_error", "reward_type": "system_error", "system_error": true,
"error": "ThinkingBox reset failed during proxy_create.", "reward": 0.0, "done": true}Everything the environment controls on its own works. It dies at the first point where it
needs the Session Proxy.
The deployment boundary
The container starts only the OpenEnv API and needs an externally managed Session Proxy,
the benchmark MCP servers, Typesense, and three model endpoints. SKIP_HF_DEPLOYMENT
exists so that deploy-hf-env.yml, whose default invocation is --all, does not
auto-create a permanently broken Space in the openenv namespace and add it to the public
collection. I appreciate that you built a generic, documented, tested opt-out rather than
quietly omitting the Dockerfile. But it does make this the first catalog environment that
declares itself undeployable.
I see the marker says "disabled now, coming soon", so you are already planning to lift it.
What does that path look like, and is any of it in reach for this PR? openapp_env is the
closest precedent and went the other way: it bundles its server inside the container and
documents the manual path as a development-only alternative ("Docker mode (recommended,
fully self-contained) or Local mode"). Given tb mcp-start and
scripts/install_typesense.sh exist upstream, could the proxy, the MCP servers and
Typesense move into the image? Model endpoints can stay as environment variables, that is
normal here. Even a docker compose that stands the topology up in one command would
change the adoption story a lot.
This matters more than it looks, because of the build issue I left inline on the Dockerfile.
The moment the marker comes out, that build starts failing, and it will fail inside someone
else's release run. Worth fixing both together.
Executable data and the Hub
DATA_ARCHIVE_URL points at codeload.github.com, and roughly 440 lines re-implement
download, cache keying, locking, checksum verification, size limits and safe extraction.
huggingface_hub.snapshot_download already gives caching, resume, offline mode, HF_HOME
conventions and revision pinning.
Since the dataset on the Hub is currently the viewer representation only (3 parquet files),
the thing that actually executes is not on HF. Mirroring the executable bundle to the Hub
(same repo on another revision, or a separate data repo) and loading it through
huggingface_hub would close the loop that microsoft/thinkingbox#24 opened, and would also
let this run inside a Space or a Job without egress to GitHub.
Questions about the pins
I checked both and they hold up. The data pin matches the thinkingbox-bench-v1.0 release
tag exactly and is additionally guarded by DATA_BUNDLE_SHA256. The code pin is an
ancestor of main (11 commits behind), so it is reachable and safe from GC, and since
microsoft/thinkingbox has no tags at all, a raw SHA is currently the only option. This is
in fact the only environment in the repo pinning a third-party git dependency to a SHA.
openapp_env and unity_env track default branches, which is strictly worse.
So my questions are not about pinning:
- Any plan to publish
thinkingboxon PyPI? That removes git from the image, stops CI
cloning on every run, and would let most ofconfig.py's PEP 610 provenance machinery
go away, since it exists only because this is a VCS install. - What is the bump policy? The pin is already 11 commits stale, and if benchmark semantics
track the harness version, the adapter will drift from what you run internally. Does
bumping require re-validating the 507 tasks? - Could you tag the harness commit corresponding to
thinkingbox-bench-v1.0? Right now
the data has a citable named release while the code that executes it is an anonymous
SHA. For a benchmark whose thesis is reproducibility, the harness should be as nameable
as the data.
Our side of this
Two things that are ours to settle. Flagging them here so it is clear they are not
change requests on you.
The dataset card states the benchmark is for evaluation only and explicitly rules out
fine-tuning and RL. OpenEnv is an RL environment spec, so it is worth settling whether the
catalog takes eval-only environments (tbench2 and terminus are precedent).
One thing that might defuse this entirely: resolve_data_bundle() accepts an arbitrary
local root and validates without requiring the canonical stamp, and reset(dataset=...)
passes it through per episode. So this is really an adapter for the ThinkingBox sandbox
(MIT, unrestricted) that ships the public bench as its default dataset. If that is
intended, documenting how to point it at your own scenarios would make the framing much
cleaner: an RL environment for stateful tool use, with the public bench as a held-out eval
set.
Also, CI has not run on this PR yet, so the test plan claims are unverified on our side.
Worth approving the workflows before a deeper pass.
P.S. it would be super cool to coordinate on a blog post about this on the Hub at some
point. The gap between 65.36% pass@1 and 25.25% pass^20 is a great story and I don't think
enough people have seen it yet. Happy to pick that up once this PR is merged :)
Preserve the builder virtualenv at /app/env/.venv so console-script shebangs remain valid in the runtime image. Add regression assertions for the final-stage copy and PATH.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7f1e9cd. Configure here.
| if result.observation.user_message is None: | ||
| return record(await env.finish("agent_error")) | ||
| if agent_turns >= test_case.max_agent_sim_turns: | ||
| return record(await env.finish("agent_limit")) |
There was a problem hiding this comment.
Evaluator turn limit diverges
High Severity
run_configured_agent increments agent_turns for every decoded ParallelToolCall and visible assistant Text, then calls finish("agent_limit") when that counter reaches max_agent_sim_turns. The environment only counts direct-response texts and submitted assistant messages, so tool batches without direct responses consume harness budget only. The evaluator can therefore stop a still-legal episode early and record agent_limit instead of the native outcome.
Reviewed by Cursor Bugbot for commit 7f1e9cd. Configure here.
| ], | ||
| tool_calls_before_content=False, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
End-turn text order dropped
Medium Severity
_terminal_tail only reads the message immediately before the end-turn ParallelToolCall, and run_configured_agent always submits that turn with tool_calls_before_content=False. When the provider emits the end-turn tool first, the adjacent assistant text is dropped and the server records the opposite order, so grading sees the wrong final response.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 7f1e9cd. Configure here.
| observation = result.observation | ||
| if observation.tool_result is None: | ||
| raise RuntimeError("OpenEnv returned no ThinkingBox tool result") | ||
| return observation.tool_result |
There was a problem hiding this comment.
Parse errors execute tools
Medium Severity
_provider_call skips tool calls whose metadata has error, so call_tool falls through to a live env.call_tool that omits parse_error. The decode loop still submits the same batch with parse_error set. A malformed provider call can therefore execute for real and also be recorded as a parse failure, applying extra side effects before grading.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 7f1e9cd. Configure here.
Hi @sergiopaniego I pushed some changes, let me know in case further changes are required. Was planning to write a blog post on Hub anyways, was waiting for a few runs such as PS: upstream PR for supporting all reasoning efforts for all models WIP |
2 participants
Summary
Hi I'm Tuhin from Microsoft's ThinkingBox team. The dataset was released on huggingface as discussed, this PR further enables people already familiar with OpenEnv to easily evaluate their models on ThinkingBox-Bench. More details on our official release:
Type of Change
Alignment Checklist
Before submitting, verify:
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles.claude/docs/INVARIANTS.mdand no invariants are violated/pre-submit-pr(orbash .claude/hooks/lint.shand tests) and addressed all issuesRFC Status
Test Plan
uv run pytest tests/envs/test_thinkingbox_env.py -v # Expected: 77 passeduv run pytest tests/scripts/test_prepare_hf_deployment.py -v # Expected: 4 passedpython scripts/sync_env_docs.py --check # Expected: "All environment stubs are present and up to date."usort check envs/thinkingbox_env/ examples/thinkingbox/ tests/envs/test_thinkingbox_env.py ruff format --check envs/thinkingbox_env/ examples/thinkingbox/ tests/envs/test_thinkingbox_env.py ruff check envs/thinkingbox_env/ examples/thinkingbox/ tests/envs/test_thinkingbox_env.py # Expected: all cleanopenenv validate envs/thinkingbox_env --verbose # Expected: passes all advertised deployment modesSteps 1–5 require no external services and verify the full offline surface.
Steps 6–7 require Docker and the ThinkingBox deployment topology respectively.
Claude Code Review
Ran copilot-cli
/reviewwith modelsopus 4.6,opus 5,gpt 5.6 solandsonnet 5withmaxeffort to fix bugs; N/A for claude codeNote
Medium Risk
Large new environment with external runtime dependencies (proxy, MCP, model endpoints) and a complex evaluator, but changes are additive and keep secrets and grading server-side.
Overview
Adds a new
thinkingbox_envpackage that exposes Microsoft ThinkingBox-Bench through OpenEnv’s WebSocket reset/step API for evaluation-only runs, while keeping grading, simulated users, and MCP tool execution on the server behind an external Session Proxy.The adapter ships typed actions/observations, a lazy
ThinkingBoxEnvclient (tool listing, single/batch tool calls, assistant messages; harness-onlyfinish), a FastAPI server with/readydependency checks, and pinned benchmark data download with bundle hash verification and atomic cache publication. A packagedthinkingbox-evalharness drives native agent/user/judge orchestration, emits privacy-scrubbed canonical JSONL, supports sharding/resume/coverage profiles (canary/full), and validates execution provenance.CI gains a dedicated
test-thinkingbox-envjob (Python 3.11/3.12, env-localuv sync). Docs and the environments catalog include ThinkingBox; HF Space deploy is explicitly deferred viaSKIP_HF_DEPLOYMENT.Reviewed by Cursor Bugbot for commit 7f1e9cd. Bugbot is set up for automated code reviews on this repo. Configure here.