Skip to content

Fix/python remote eval normalize list score dicts#595

Open
Everett Smith (evrtt) wants to merge 2 commits into
mainfrom
fix/python-remote-eval-normalize-list-score-dicts
Open

Fix/python remote eval normalize list score dicts#595
Everett Smith (evrtt) wants to merge 2 commits into
mainfrom
fix/python-remote-eval-normalize-list-score-dicts

Conversation

@evrtt

Copy link
Copy Markdown

Context

Valid multi-score remote scorer results from remote eval scorers are rejected. Remote eval scorers can return list[Score], but that result crosses an HTTP/JSON boundary and the caller receives list[dict] rather than real Score objects. await_or_run_scorer normalizes a single score dict with Score.from_dict(), but is missing normalization for list[dict] responses.

Changes

  • Normalize dict entries inside scorer result lists with Score.from_dict().
  • Keep existing validation for malformed score entries.
  • Add regression coverage for a scorer returning a list of score dicts.
  • Verify both per-result scores and summary scores are populated correctly.

Testing

  • Added test_run_evaluator_normalizes_list_of_dict_scores

    • creates an evaluator with two data rows
    • uses a scorer that returns a list of two score dicts:
      • {"name": "per_result", "score": 1.0}
      • "name": "summary_only", "score": 1.0}
    • runs the evaluator locally with run_evaluator(...)
    • asserts each eval result contains both scores
    • asserts the final summary contains both score names with score 1.0
  • Ran the focused core regression test

  • Ran a local smoke test where a scorer returns multiple score dicts, confirming the eval completes and emits both expected scores.

@AbhiPrasad

Copy link
Copy Markdown
Member

does this issue effect the other sdks?

@evrtt

Copy link
Copy Markdown
Author

Abhijeet Prasad (@AbhiPrasad) yeah Go and Ruby both similarly fail with a list of scores currently

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants