From 2cad7a8e2a6beef6c63e1e192370589afae7f1c6 Mon Sep 17 00:00:00 2001 From: FAQ Bot Date: Sat, 15 Aug 2026 18:20:15 +0000 Subject: [PATCH 1/2] NEW: How can I detect when an LLM judge gives inconsistent scores compared to --- ...-llm-judge-inconsistent-reasoning-score.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 _questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md diff --git a/_questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md b/_questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md new file mode 100644 index 00000000..984145a1 --- /dev/null +++ b/_questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md @@ -0,0 +1,28 @@ +--- +id: b47f096063 +question: How can I detect when an LLM judge gives inconsistent scores compared to + its own reasoning in LLM-as-judge evaluation? +sort_order: 7 +--- + +When using LLM-as-a-judge, don’t rely only on the numeric score—also read the judge’s written reasoning next to the score, especially during spot-checks on a small sample (e.g., 5–10 items). + +A distinct failure mode is when the judge’s justification says one thing, but the numeric score contradicts it (e.g., reasoning concludes the answer is correct, yet it assigns a low score). This can be invisible in aggregate statistics over large batches because it may “average out,” so you need case-level inspection. + +To make this easier to check, structure the judge output so the reasoning and score always appear together, for example with a structured schema like: + +```python +class JudgeScore(BaseModel): + reasoning: str + score: int # e.g. 1-3 +``` + +Then after scoring, manually review a small sample: + +```python +for result in sample: + print(f"Reasoning: {result.reasoning}") + print(f"Score: {result.score}") +``` + +If your judge returns only a bare number (no accompanying reasoning text), this inconsistency check becomes much harder or impossible. \ No newline at end of file From d7465871e70932ceb9e0725bcdce381b8a357680 Mon Sep 17 00:00:00 2001 From: Alexey Grigorev Date: Wed, 9 Sep 2026 14:42:08 +0200 Subject: [PATCH 2/2] Fix LLM-judge FAQ: pydantic import, sources, shorter question (#360) --- ...t-llm-judge-inconsistent-reasoning-score.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/_questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md b/_questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md index 984145a1..ee22035b 100644 --- a/_questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md +++ b/_questions/llm-zoomcamp/module-4/007_b47f096063_detect-llm-judge-inconsistent-reasoning-score.md @@ -1,28 +1,28 @@ --- id: b47f096063 -question: How can I detect when an LLM judge gives inconsistent scores compared to - its own reasoning in LLM-as-judge evaluation? +question: How do I catch an inconsistent score from an LLM-as-judge evaluation? sort_order: 7 --- -When using LLM-as-a-judge, don’t rely only on the numeric score—also read the judge’s written reasoning next to the score, especially during spot-checks on a small sample (e.g., 5–10 items). +When using LLM-as-a-judge, don't rely only on the numeric score — also read the judge's written reasoning next to the score. A distinct failure mode is a justification that says one thing while the numeric score contradicts it (e.g. the reasoning concludes the answer is correct, yet the score is low). This is invisible in aggregate statistics over large batches because it "averages out" — it only surfaces through case-level inspection, so spot-check a small sample (5-10 items) by hand. -A distinct failure mode is when the judge’s justification says one thing, but the numeric score contradicts it (e.g., reasoning concludes the answer is correct, yet it assigns a low score). This can be invisible in aggregate statistics over large batches because it may “average out,” so you need case-level inspection. - -To make this easier to check, structure the judge output so the reasoning and score always appear together, for example with a structured schema like: +To make the check easy, structure the judge output so reasoning and score always appear together: ```python +from pydantic import BaseModel + class JudgeScore(BaseModel): reasoning: str score: int # e.g. 1-3 ``` -Then after scoring, manually review a small sample: +Then review a small sample manually: ```python -for result in sample: +for result in sample: # 5-10 scored items, checked by hand print(f"Reasoning: {result.reasoning}") print(f"Score: {result.score}") + # does the reasoning's conclusion actually match the score? ``` -If your judge returns only a bare number (no accompanying reasoning text), this inconsistency check becomes much harder or impossible. \ No newline at end of file +If the judge returns only a bare number with no reasoning text, this check is impossible. See the [Module 4 LLM-as-judge lesson](https://github.com/DataTalksClub/llm-zoomcamp/blob/main/cohorts/2026/04-evaluation/13-llm-as-judge.md) and [Hamel's LLM-judge guide](https://hamel.dev/blog/posts/llm-judge/).