pass@k is under-counted when --n-attempts is greater than 10

[eeshsaxena]

Running an eval with --n-attempts (k) above 10 silently scores the attempts past the tenth as failures, so the reported pass@k is too low.

In run_evaluation (src/anvil/evals/runner.py):

• Rollouts are only written to disk for the first keep_n = min(k, 10) attempts:

keep_n = min(k, 10)          # line 266
...
if attempt <= keep_n:        # line 304
    result_dir = base_out / iid / f"attempt_{attempt}" / "rollout"
    write_single_result(result, result_dir, eval_id)

• But the evaluation phase then reads a prediction for every attempt up to k:

for attempt in range(1, k + 1):     # line 331
    ...
    pred_path = base_out / iid / f"attempt_{attempt}" / "rollout" / f"{iid}.pred"
    patch = ""
    if pred_path.exists():
        ...

For k > 10, attempts 11..k never had their rollout (and .pred) written, so pred_path does not exist, patch stays "", and those attempts are evaluated with an empty patch (a guaranteed failure). The final aggregation in compute_pass_at_k_summary then averages over all k attempts, so every task loses (k - 10) attempts to forced failures and pass@k is under-counted. The per-attempt success accounting (for attempt in range(1, k + 1)) is skewed the same way.

So either the cap is intentional (in which case the eval and aggregation loops should also stop at keep_n, and pass@k should be reported as pass@keep_n), or the cap is unintended and all k rollouts should be written. Right now the write side caps at 10 while the read/aggregate side uses the full k, which is the inconsistency.

Repro sketch: run any dataset with --n-attempts 12; the two extra attempts per task are counted as failures regardless of what the agent produced.