[cemde]

Description

TaskProtocol.max_retries and TimeoutAction.RETRY/EXTEND were part of the public API but the execution loop never read them (#62). This PR makes them take effect. It replaces #66, which rejected the settings instead and broke task loading for tau2 and gaia2.

Retry rules

• max_retries retries failures outside the agent's control: setup_failed, environment_error, user_error, unknown_execution_error.
• agent_error and evaluation_failed are never retried. Retrying agent errors would inflate scores. Retrying evaluation failures would discard a finished agent run.
• A timeout is retried once. RETRY keeps the timeout, EXTEND doubles it. This is separate from max_retries.
• A retry reruns the full repetition with the same seeds.
• With a fail_on_* flag set, errors are raised on the first failure without retrying.

Reports

Every report has a new attempts list. Each entry holds status, error, usage, and timeout_seconds for one attempt, oldest first. Retried results and extended timeouts are therefore visible in the results, and per-attempt usage keeps the cost of failed attempts. ResultLogger writes the field.

Benchmark loaders

tau2 and gaia2 load_tasks() defaulted to max_retries=1 ("skip on first failure"). With retries now working, that default would have changed benchmark results. It is now 0, which matches the originals: neither tau2-bench v0.2.0 (src/tau2/run.py) nor ARE retries failed runs. Execution is identical to before.

Implementation

_execute_task_repetition is now a small retry loop. The previous body moved unchanged into _execute_task_attempt, which takes the attempt's timeout as an argument. Sequential and parallel runs both go through the loop. Callbacks still fire once per repetition.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Code quality improvement (refactoring, formatting, etc.)

Checklist

Contribution

• I have read the CONTRIBUTING.md guide.
• Commits follow "How to write a good git commit message"

Documentation

• Added/updated docstrings for new/modified functions as instructed CONTRIBUTING.md
• Updated relevant documentation in docs/ (if applicable)
• Tag github issue with this PR (if applicable)

Changelog

• Added entry to CHANGELOG.md under [Unreleased] section
  • Use Added section for new features
  • Use Changed section for modifications to existing functionality
  • Use Fixed section for bug fixes
  • Use Removed section for deprecated/removed features
• OR this is a documentation-only change (no changelog needed)

Architecture (if applicable)

• Core/Interface separation: Changes in maseval/core/ do NOT import from maseval/interface/
• Dependencies: New core dependencies added sparingly; framework integrations go to optional dependencies

Additional Notes

Closes #62. Supersedes #66.

Behavior changes for users

• Anyone who already set max_retries > 0 or RETRY/EXTEND will now get retries. Previously these settings were silently ignored.
• TaskProtocol(max_retries=-1) now raises ValueError.
• Reports gain the attempts key. Existing keys are unchanged.

Known limitation: if gaia2 setup fails after the ARE simulation has started, the simulation is not stopped. This leak already exists on main. With retries enabled, each retry can leak another simulation. Retries are off by default.

Tests

• New tests/test_core/test_benchmark/test_task_retries.py (17 tests) covers every retryable status, non-retried statuses, exhausted retries, all three timeout actions, separate retry budgets, seed reuse, per-attempt usage, fail-fast, and sequential plus parallel runs. 16 of them fail on main. The fail-fast test passes on both, because raising right away was already the behavior.
• uv run pytest: 2648 passed. 5 tests in tests/test_core/test_usage.py::TestFullPipeline fail identically on main because from tests.conftest import ... cannot be resolved. That is unrelated to this PR.
• ruff format, ruff check, and ty check pass.
• mkdocs build --strict reports the same 426 warnings as main and nothing new.

🤖 Generated with Claude Code

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
maseval/benchmark/gaia2
data_loader.py
maseval/benchmark/tau2
data_loader.py
maseval/core
benchmark.py
task.py
maseval/core/callbacks
result_logger.py
Project Total

This report was generated by python-coverage-comment-action