feat(classification): add classify_risk utility for multi-run results (#349)

[lifelmy]

Summary

Adds a standalone, post-hoc risk classification utility that turns repeated
(case, evaluator) runs into a single stability label, addressing #349.

When the same case is evaluated N times, the raw pass/fail counts don't say what kind of
problem a failure is. A case that fails every run is a deterministic bug; one that fails
intermittently is flaky. These need different responses, but today both surface as "some
failures". This adds a pure utility that names the difference.

What's added

• src/strands_evals/classification.py
  • RiskLabel(str, Enum) — BUG / FLAKY / CNE / PASS
  • classify_risk(results) — classifies one (case, evaluator) pair's repeated runs
  • classify_task_risk(evaluator_risks) — worst-of rollup across evaluators for one case
• Public exports for all three from strands_evals
• tests/strands_evals/test_classification.py — 20 unit tests

Design notes (matching the issue and repo conventions)

• Standalone module, not a change to EvaluationOutput/EvaluationReport/Experiment.
  Classification is invoked only when a caller already holds multi-run results, exactly as the
  issue proposes. Nothing forces multi-run onto existing single-run callers.
• Not-applicable handling reuses the existing contract. Rather than a bespoke status
  field, gradable rows are selected via the shipped EvaluationOutput.not_applicable property
  (the NOT_APPLICABLE label). This is the same "drop non-verdict rows" rule
  EvaluationReport already applies, so a case can't be tipped toward pass/fail by rows that
  had nothing to judge. CNE is returned when there are no gradable rows.
• Worst-of rollup: BUG > FLAKY > PASS > CNE. A case with one deterministic failure and
  some flaky evaluators is a bug — the deterministic signal is the actionable one.

The issue's example sketched a status-based gradable filter with a getattr fallback; since
EvaluationOutput already ships not_applicable, this implementation keys off that instead,
which is the current source of truth for "nothing to judge" and needs no new field.

Testing

Run in a fresh worktree venv against origin/main (base abf7b91):

• hatch test tests/strands_evals/test_classification.py -> 20 passed
• hatch test tests/strands_evals/types/ tests/strands_evals/test_classification.py -> 124 passed (no regressions)
• hatch fmt --formatter --check -> 362 files already formatted
• hatch fmt --linter -> all checks passed
• mypy -p src (hatch-static-analysis env) -> Success, no issues in 236 source files

Note on authorship

I use a coding assistant to help implement, and I review and take responsibility for the final change.

[JackYPCOnline]

Overall, the purpose and intension of this PR is clear. On the other side, the naming and DEVX remain unclear to me, I would suggest give this more thought and how can we make it more explicit, what apis we want to expose. With said, the purposed in the issue ticket need refinement.

[lifelmy]

Thanks for the review. I agree that RiskLabel / classify_risk is too broad and can be confused with the red-team risk APIs, while classify_task_risk does not make it obvious that it aggregates evaluator-level stability verdicts.

Before changing the public API, I propose making the concepts explicit:

• EvaluationStability: the result across repeated runs of one case/evaluator pair
• classify_evaluation_stability(outputs): classify that repeated-run result
• combine_evaluation_stability(verdicts): apply the documented worst-of ordering across evaluators

I would also revise issue #349 to define the input grouping contract, the handling of not_applicable, the empty-input behavior, and why this is a post-hoc helper rather than an Experiment/EvaluationReport method. Does that API direction match what you have in mind, or would you prefer this to live on a report/analysis object instead of as standalone functions? I have not churned the public API yet so that the next revision can follow the intended DX.