Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions hpc/harbor_yaml/eval/snowball_tasktrove_v342_terminus2.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ environment:
kwargs:
auto_snapshot: true
verifier:
override_timeout_sec: 14400
max_timeout_sec: 14400
override_timeout_sec: null
max_timeout_sec: 600
agents:
- name: terminus-2
model_name: placeholder/override-at-runtime
Expand Down
4 changes: 2 additions & 2 deletions hpc/harbor_yaml/eval/tasktrove_pi_ctx32k.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ environment:
kwargs:
auto_snapshot: true
verifier:
override_timeout_sec: 7200
max_timeout_sec: 7200
override_timeout_sec: null
max_timeout_sec: 600
agents:
- name: pi
model_name: placeholder/override-at-runtime
Expand Down
2 changes: 1 addition & 1 deletion hpc/harbor_yaml/eval/tasktrove_qwen3_coder_ctx32k.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ environment:
kwargs:
auto_snapshot: true
verifier:
max_timeout_sec: 14400
max_timeout_sec: 600
agents:
- name: terminus-2
model_name: placeholder/override-at-runtime
Expand Down
2 changes: 1 addition & 1 deletion hpc/harbor_yaml/eval/tasktrove_terminus2_ctx32k.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ environment:
kwargs:
auto_snapshot: true
verifier:
max_timeout_sec: 14400
max_timeout_sec: 600
agents:
- name: terminus-2
model_name: placeholder/override-at-runtime
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ environment:
kwargs:
auto_snapshot: true
verifier:
override_timeout_sec: 14400
max_timeout_sec: 14400
override_timeout_sec: null
max_timeout_sec: 600
agents:
- name: terminus-2
model_name: placeholder/override-at-runtime
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@ environment:
kwargs:
auto_snapshot: true
verifier:
override_timeout_sec: 14400
max_timeout_sec: 14400
override_timeout_sec: null
max_timeout_sec: 600
agents:
- name: opencode
model_name: openai/override-at-runtime
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ environment:
kwargs:
auto_snapshot: true
verifier:
override_timeout_sec: 14400
max_timeout_sec: 14400
override_timeout_sec: null
max_timeout_sec: 600
agents:
- name: terminus-2
model_name: placeholder/override-at-runtime
Expand Down
29 changes: 29 additions & 0 deletions tests/hpc/test_tasktrove_eval_configs.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
from pathlib import Path

import yaml


REPOSITORY_ROOT = Path(__file__).resolve().parents[2]
EVAL_CONFIG_DIR = REPOSITORY_ROOT / "hpc/harbor_yaml/eval"
MAX_VERIFIER_TIMEOUT = 600


def tasktrove_eval_configs() -> list[Path]:
return sorted(
{
*EVAL_CONFIG_DIR.glob("tasktrove*.yaml"),
*EVAL_CONFIG_DIR.glob("snowball_tasktrove*.yaml"),
}
)


def test_tasktrove_verifiers_are_capped_at_ten_minutes() -> None:
paths = tasktrove_eval_configs()
assert paths

for path in paths:
config = yaml.safe_load(path.read_text())
verifier = config["verifier"]
assert verifier["max_timeout_sec"] <= MAX_VERIFIER_TIMEOUT, path
override = verifier.get("override_timeout_sec")
assert override is None or override <= MAX_VERIFIER_TIMEOUT, path
Loading