Skip to content

Fix quickstart: correct agent_data placement and add model adapter implementation - #70

Merged
cemde merged 1 commit into
maseval:mainfrom
ArthurFouquet:patch-1
Oct 6, 2026
Merged

cemde merged 1 commit into
maseval:mainfrom
ArthurFouquet:patch-1

Conversation

@ArthurFouquet

Copy link
Copy Markdown
Contributor

Updated the quickstart guide to include configuration of model IDs and use of a custom benchmark class.

Description

While following the quickstart guide with a fresh maseval[openai] install (v0.5.1), the "Using a default agent" and "Plugging in your own agent" examples under Existing Benchmarks did not run as written:

  • DefaultAgentTau2Benchmark cannot be instantiated directly since get_model_adapter is abstract — the example needs a subclass implementing it.
  • agent_data is not accepted by the Tau2Benchmark constructor; it belongs in .run(tasks, agent_data=...).
  • The user simulator's model_id must be set via configure_model_ids() after load_tasks(), or .run() raises a ValueError.

This PR updates both code examples to reflect the current API, verified against a real run.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code quality improvement (refactoring, formatting, etc.)

Checklist

Contribution

Documentation

  • Added/updated docstrings for new/modified functions as instructed CONTRIBUTING.md
  • Updated relevant documentation in docs/ (if applicable)
  • Tag github issue with this PR (if applicable)

Changelog

  • Added entry to CHANGELOG.md under [Unreleased] section
  • OR this is a documentation-only change (no changelog needed)

Architecture (if applicable)

  • Core/Interface separation: Changes in maseval/core/ do NOT import from maseval/interface/
  • Dependencies: New core dependencies added sparingly; framework integrations go to optional dependencies

Additional Notes

Only fixed the two Tau2Benchmark-based examples, which I tested end-to-end. Did not touch the generic MyBenchmark(Benchmark) example at the top of the page since I haven't verified whether the base Benchmark class has the same agent_data signature issue — flagging in case a maintainer wants to check that separately.

…rt: agent_data goes to .run(), not constructor

Updated the quickstart guide to include configuration of model IDs and use of a custom benchmark class.
@cemde
cemde merged commit 5f110d5 into maseval:main Oct 6, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants