fix: guard _classify_completed_outcomes against non-dict rewards

[VaisakhiMishra]

Previously, a truthy non-dict value in the 'rewards' field of result.json (e.g. bare float 1.0, int, bool, list) caused an uncaught AttributeError in _classify_completed_outcomes, crashing the entire evaluation run before any new tasks could execute.

---

How to reproduce the crash (pre-fix)

# 1. Create a jobs directory with a result.json whose rewards is a bare float
mkdir -p /tmp/bf_repro/my-job/my-task__abc123
cat > /tmp/bf_repro/my-job/my-task__abc123/result.json <<'EOF'
{
  "task_name": "my-task",
  "rollout_name": "my-task__abc123",
  "rewards": 1.0,
  "agent": "oracle",
  "model": null,
  "n_tool_calls": 5
}
EOF

# 2. Resume — crashes before any tasks run
bench eval run --tasks-dir tasks/my-task --jobs-dir /tmp/bf_repro
# AttributeError: 'float' object has no attribute 'get'

---

Can we make this bug fix as part of https://github.com/benchflow-ai/FrontierPhysics/issues/125 or should I open a separate issue to track this fix?

Root cause:

    rewards = r.get('rewards') if isinstance(r, dict) else None
    reward = rewards.get('reward') if rewards else None

only guards against falsy values; a truthy non-dict (e.g. 1.0) passes the if rewards check and then fails on .get().

Fixes:

1. Import extract_reward from benchflow._utils.scoring — it already handles None, dict, and non-dict values safely.
2. Replace the two-liner in _classify_completed_outcomes with:

reward = extract_reward(r) 
if isinstance(r, dict) else None

3. Add a logger.warning in _get_completed_tasks when rewards is a non-dict truthy value, so operators can identify malformed result files without a crash. The fix is backwards-compatible: valid dict-shaped rewards are unchanged; falsy non-dict values were already silently treated as errored and still are (with a warning now); truthy non-dict values no longer crash.

4. Minimal fix for uvx ruff check . failure for src/benchflow/contracts/user.py

---

[devin-ai-integration]

Devin Review found 1 potential issue.

[VaisakhiMishra]

/devin review

[VaisakhiMishra]

@bingran-you can you take a look at this PR (and the issue that it fixes) and if it counts as infra against the associated miscellaneous issue? If yes, how do we get devin to retrigger the automated review / tests?

[Galius5136]

@VaisakhiMishra Reproduced the issue A/B against the merge-base and verified the fix end to end.

With a malformed rewards: 1.0:

• the merge-base crashes in _classify_completed_outcomes() with AttributeError;
• this PR handles it correctly, resumes the job, classifies the rollout as errored, and writes the summary without crashing;
• well-formed reward payloads keep the same behaviour.

I also verified this using a real Docker oracle rollout: after changing only the produced result.json from {"reward": 1.0} to 1.0, the base crashes while this branch handles the malformed artifact correctly.

One blocker before merge:

uv run ruff format --check src tests tools currently fails on the PR head:

Would reformat: src/benchflow/evaluation.py

This is the same command used by the CI format-check step, so the branch needs to be formatted before approval.

The equivalent malformed-rewards crash still exists in bench eval metrics, but that behaviour predates this PR and is not introduced by the change here, so I don't consider it a blocker for this fix.

Once the formatting gate is clean, this looks good from my side.

[VaisakhiMishra]

@Galius5136 thank you for the quick review! I have confirmed what you see uv run ruff format --check src tests tools and fixed it with uv run ruff format src/benchflow/evaluation.py, please review when you can!

[Galius5136]

Confirmed on 1fa95a2 — uv run ruff format --check src tests tools is clean now, and the malformed-rewards checks give the same results as before. Good from my side.

[VaisakhiMishra]

@Galius5136 I resolved the conflicts, what would be the next step to merge? (I don't have permissions to merge or run workflows)