Foundation: Box evaluation report and gold Box label format (#311)

docs/evaluation.md

@@ -736,6 +736,38 @@ numbers, for the reason "Outcomes versus the confidence-only view" gives general
 `confidence` from `pool_label_decisions` fed to `evaluate_selective` agree with `accepted` and
 `correct` from `pool_label_outcomes` fed to `evaluate_outcomes`, on every field but `threshold`.
 
+### Box localization
+
+Box answers are scored in `jev_cookbook.vision`, not here, because they need rectangle
+geometry. `vision.localization_report(gold, outcomes, iou_threshold=0.5)` takes the gold Box
+labels (`{"present": true, "box": {...}}` or `{"present": false}`, see
+[fixtures](fixtures.md#box-labels)) and one `("located", box)`, `("absent", None)` or
+`("review", None)` outcome per example. Like every evaluation in this toolkit it states its
+denominators and lets no case disappear: reviews and absent targets are counted, never
+dropped from the report.
+
+| Number | Denominator |
+| --- | --- |
+| mean IoU over located present targets | present targets with a returned box |
+| mean IoU over all present targets (missed and reviewed count 0) | all present targets |
+| hit rate at the IoU threshold | all present targets |
+| false positive rate (a box for an absent target) | all absent targets |
+| correct absent rate | all absent targets |
+
+A present target sent to review therefore counts as 0 in the second mean and as a miss in
+the hit rate, and is simply not one of the first mean's located targets: the printed report
+shows both denominators. Only
+located boxes on present targets contribute an IoU; a box returned for an absent target is a
+false positive, because a rectangle never proves the target exists. The worked example, and
+the test that runs it, are in [vision](vision.md#the-localization-report).
+Use `evaluate_outcomes` above for the review rate of the presence question alone.
+
+In a tiled or hierarchical recipe the headline report is per image: gold is the full-image
+Box label and each outcome is the final rectangle after `tile_to_image`, with a stage-1 "no
+match" on present gold reported as `absent` and an uncertain choice as `review`. A per-tile
+report counts tiles, is a stage diagnostic and says so, and states its rule for a target a
+tile edge cuts; see [vision](vision.md#tiled-and-hierarchical-recipes).
+
 ### Calibration
 
 `reliability_table(probability, outcome, n_bins=10)` bins stated probability in equal widths

docs/fixtures.md

@@ -102,6 +102,46 @@ here too.
 labels). Every `train`, `validation` and `test` example needs exactly one label; a label whose
 `id` is not in the inputs is an error.
 
+### Box labels
+
+A recipe whose question asks for a `Box` labels each example with a gold Box label, so ten
+recipes do not invent ten formats. An object with a boolean `present`:
+
+```json
+{"id": "t07", "label": {"present": true, "box": {"top": 0.25, "bottom": 0.75, "left": 0.125, "right": 0.625}}}
+{"id": "t08", "label": {"present": false}}
+```
+
+- `present: true` has exactly `present` and `box`. `box` has exactly `top`, `bottom`, `left`
+  and `right`: finite numbers in [0, 1] (normalized to the image, as in
+  [`docs/vision.md`](vision.md)), with `left < right` and `top < bottom`, and an area that
+  does not underflow to zero.
+- `present: false` has exactly `present`. It means the target is not in the image. It carries
+  no `box`, not even a `null` one; `null` is never a label.
+- `present` is a JSON boolean. `"yes"`, `1` and `null` are errors.
+
+`vision.gold_box(label)` decodes one (the rectangle, or `None` for absent) and raises
+`ValueError` on anything else; `vision.localization_report` takes these labels directly.
+
+The validator treats a label that is an object with a `present` key as a Box label: that is
+how a recipe declares the format, so `present` is reserved at the top level of a label object.
+Every such label must pass `gold_box`, and once any label in the file is a Box label, a label
+that is not (a bare box, a string) is an error. A recipe whose label holds a Box label among
+other values nests it under its own key and calls `gold_box(label["target"])` in its own test;
+the validator does not look inside other labels, so that test must cover every split the
+recipe ships (`train` included), not only the one it reports:
+
+```json
+{"id": "t12", "label": {"target": {"present": true, "box": {"top": 0.25, "bottom": 0.75, "left": 0.125, "right": 0.625}}, "rubric": 3}}
+```
+
+Store the gold box once, for the whole image, as a full-image Box label. A recipe that tiles
+the image does not store a tile choice beside it: a stored tile name can disagree with the box
+(for an absent target there is no right tile at all, only "no match"), and with overlapping
+tiles a target near a seam lies in more than one tile, so one string cannot hold the gold.
+Python derives which tiles count as correct from the box and the tile bounds, and the
+headline report stays per image ([vision](vision.md#tiled-and-hierarchical-recipes)).
+
 ## Nesting limit
 
 No JSON value in a fixture file nests deeper than **64 levels**, counting the line (or, in a
@@ -366,7 +406,7 @@ content. Recipe tests must also check image content and lineage across splits, s
 paths can contain identical pixels and the fixture validator does not open image files.
 
 The input row schema is unchanged; do not add a top-level `assets` field. Labels can contain
-normalized gold boxes or an explicit absent-target label. A response's `BoxAnswer.to_dict()`
+the gold Box label above (a normalized box, or the explicit `{"present": false}`). A response's `BoxAnswer.to_dict()`
 stores `type`, the four-edge `box`, nullable `confidence`, and the ordinary provenance.
 `DecisionResult` and the fixture validator reject malformed geometry and confidence.
 Missing confidence means unknown, never certainty. A Box is still returned for an absent

docs/vision.md

@@ -123,7 +123,8 @@ touching rectangles and one for identical rectangles. A valid but vanishingly sm
 (for example 1e-200 wide) has an area that underflows to zero; when both do, the union is
 zero and `box_iou` (and `localization_metrics`) raise `ValueError` rather than invent a score.
 Because the union is at least as large as each area, this happens only when both boxes
-underflow, so in `localization_metrics` it means a degenerate gold label: fix the fixture.
+underflow, so against a gold box it means a degenerate gold label: `gold_box` rejects
+such a label up front, so fix the fixture.
 
 ## Presence, confidence and evaluation
 
@@ -153,12 +154,117 @@ def localization_outcome(answer, presence, threshold):
 
 Choose the presence and confidence thresholds on validation examples and freeze
 them before evaluating the test split. Keep reviews in the recipe's review queue
-and account for them with `evaluation.evaluate_outcomes`; a review is not a claim
-that the target is absent.
+and account for them in `localization_report` (and, for the overall review rate,
+`evaluation.evaluate_outcomes`); a review is not a claim that the target is absent.
+
+### Gold labels
+
+A Box recipe labels every example with a gold Box label (specified in
+[Box labels](fixtures.md#box-labels)): `{"present": true, "box": {"top": ..., "bottom": ...,
+"left": ..., "right": ...}}` when the target is in the image, `{"present": false}` when it is
+not. `gold_box(label)` decodes one: it returns the validated normalized rectangle for a
+present target and `None` for an absent one, and raises `ValueError` for anything malformed,
+including a `box` on an absent label, a non-boolean `present`, edges outside [0, 1], zero
+width or height, and a box so small its area underflows to zero (which `box_iou` cannot score).
+The fixture validator applies the same check to every label file that uses the format.
+
+### The localization report
+
+`localization_report(gold, outcomes, *, iou_threshold=0.5)` is the entry every Box recipe
+reports through. `gold` is one gold Box label per example; `outcomes` is one pair per example
+from the localization policy above: `("located", box)`, `("absent", None)` or
+`("review", None)`. Nothing is filtered, so no example can leave a denominator: each lands in
+exactly one cell of a present table or an absent table.
+
+| Gold | `located` (a box was returned) | `absent` | `review` |
+| --- | --- | --- | --- |
+| present | IoU is scored | missed (`present_missed`) | `present_review` |
+| absent | false positive (`false_positives`) | correct absent (`correct_absents`) | `absent_review` |
+
+A returned rectangle never proves a target exists, so a `located` box on absent gold is a
+false positive, never an IoU. A review is never an absent prediction. The report states:
+
+- `n_total`, `n_present`, `n_absent`;
+- `mean_iou_located`: mean IoU over located present targets, denominator `present_located`;
+- `mean_iou_present`: mean IoU over all present targets with missed and reviewed targets
+  counted as 0, denominator `n_present`;
+- `hit_rate`: located present targets with IoU at or above `iou_threshold` (default 0.5),
+  over `n_present`;
+- `false_positives` and `false_positive_rate` (over `n_absent`), `correct_absents` and
+  `correct_absent_rate` (over `n_absent`), `absent_review`.
+
+A mean or rate whose denominator is zero is NaN (and prints `n/a`), never 0. `str(report)`
+names every denominator, and `report.denominators` returns them by metric name.
 
-`localization_metrics(gold, predicted)` accepts equally sized, nonempty sequences.
-Here `None` explicitly labels absence. Apply it to resolved outcomes and report
-review counts separately:
+```python
+from jev_cookbook.vision import localization_report
+
+G = {"top": 0.0, "bottom": 0.5, "left": 0.0, "right": 0.5}
+HALF = {"top": 0.0, "bottom": 0.25, "left": 0.0, "right": 0.5}  # IoU 0.5 with G
+present, absent = {"present": True, "box": G}, {"present": False}
+
+report = localization_report(
+    gold=[present, present, present, present, absent, absent, absent],
+    outcomes=[
+        ("located", G),  # IoU 1.0
+        ("located", HALF),  # IoU 0.5
+        ("review", None),  # a present target in review keeps its place in the denominator
+        ("absent", None),  # a present target called absent: missed
+        ("located", G),  # a box returned for an absent target: a false positive
+        ("absent", None),  # correct absent
+        ("review", None),
+    ],
+)
+assert (report.n_total, report.n_present, report.n_absent) == (7, 4, 3)
+assert report.present_table == {"located": 2, "absent": 1, "review": 1}
+assert report.absent_table == {"located": 1, "absent": 1, "review": 1}
+assert report.mean_iou_located == 0.75  # (1.0 + 0.5) / 2 located present targets
+assert report.mean_iou_present == 0.375  # (1.0 + 0.5) / 4 present targets
+assert report.hit_rate == 0.5  # 2 of 4 present targets at IoU >= 0.5
+print(report)
+```
+
+```text
+Localization report: 7 examples (4 present + 3 absent), each counted once
+Present targets (n = 4): box returned 2, said absent (missed) 1, sent to review 1
+  mean IoU over located present targets: 0.7500 (denominator 2: located present targets)
+  mean IoU over all present targets, missed and review counted as 0: 0.3750 (denominator 4: all present targets)
+  hit rate at IoU >= 0.5: 2/4 = 0.5000 (denominator 4: all present targets)
+Absent targets (n = 3): box returned (false positive) 1, correct absent 1, sent to review 1
+  false positive rate: 1/3 = 0.3333 (denominator 3: all absent targets)
+  correct absent rate: 1/3 = 0.3333 (denominator 3: all absent targets)
+```
+
+`tests/test_vision.py` runs this example and compares the printed text. Malformed gold,
+an unknown outcome kind, a `located` outcome without a valid box, a box attached to `absent`
+or `review`, empty or unequal-length inputs and an `iou_threshold` outside (0, 1] raise
+`ValueError` naming the example index. Choose `iou_threshold` and the presence policy on
+validation examples and freeze them before scoring the test split.
+
+### Tiled and hierarchical recipes
+
+When a recipe tiles the image (a Choice picks the tile, then a Box locates the target inside
+it), one example is one image, not one tile. A target can be present in the full image and
+absent from most tiles, so counting tiles would make `n_total` a tile count and
+`correct_absent_rate` mostly a measure of empty tiles.
+
+- The headline report is per image. `gold` is the full-image gold Box label (stored once,
+  see [Box labels](fixtures.md#box-labels)) and each outcome is the final rectangle after
+  `tile_to_image`.
+- Map the stage-1 result to an outcome before reporting: a stage-1 "no match" on an image
+  whose gold is present is `("absent", None)` (a miss); an uncertain tile choice, or an
+  uncertain Box in the chosen tile, is `("review", None)`.
+- A per-tile report is a stage diagnostic. Label it as counting tiles, never quote it as the
+  recipe's result, and state its rule for a target a tile edge cuts (the minimum visible
+  fraction of the box that makes the target "present" in that tile), because that rule
+  changes the per-tile denominators.
+
+### The older `localization_metrics`
+
+`localization_metrics(gold, predicted)` is kept for compatibility. It takes `None` for an
+absent target, so the caller has to drop reviewed examples before calling it, and a present
+target sent to review then silently leaves the `mean_iou` denominator. It also reports no
+correct-absent count and no hit rate. Use `localization_report` in new work.
 
 ```python
 from jev_cookbook.vision import localization_metrics

src/jev_cookbook/fixtures/__init__.py

@@ -553,6 +553,35 @@ def _check_examples(
     return problems
 
 
+def _check_box_labels(
+    labels: dict[str, Any], label_lines: dict[str, int], name: str
+) -> list[Problem]:
+    """A recipe whose labels are Box labels must use the gold Box label format everywhere.
+
+    A top-level label object with a ``present`` key declares the format (``docs/fixtures.md``,
+    "Box labels"): every such label must decode with ``vision.gold_box``, and once one label
+    declares it, a label that does not (a bare box, a string) is an error, not a quiet skip.
+    A label that nests a Box label under another key is not seen here.
+    """
+    declared = {i for i, v in labels.items() if isinstance(v, Mapping) and "present" in v}
+    if not declared:
+        return []
+    from ..vision import gold_box  # deferred: validating other recipes needs no Pillow
+
+    problems: list[Problem] = []
+    for ident, label in labels.items():
+        if ident not in declared:
+            msg = "label is not a Box label ({'present': ...}) but this recipe's other labels are"
+        else:
+            try:
+                gold_box(label)
+                continue
+            except ValueError as exc:
+                msg = f"malformed Box label: {exc}"
+        problems.append(Problem(name, msg, label_lines[ident], ident))
+    return problems
+
+
 def _check_responses(
     examples: list[Example], responses: Mapping[str, Any], name: str
 ) -> list[Problem]:
@@ -692,6 +721,8 @@ def validate_recipe(recipe_dir: PathLike | None = None) -> Report:
     if inputs_ok:
         names = (_display(inputs_path), _display(labels_path))
         problems += _check_examples(examples, labels if labels_ok else None, label_lines, names)
+        if labels_ok:
+            problems += _check_box_labels(labels, label_lines, names[1])
     if replays and None not in responses_files:
         # Listed keys need the default file; with no key listed it is simply not needed.
         problems.append(Problem(_display(folder / RESPONSES_FILE), "file is missing"))

src/jev_cookbook/fixtures/schema.json

@@ -53,7 +53,7 @@
           "pattern": "^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$"
         },
         "label": {
-          "description": "The gold label. Any JSON value except null: a string, a number, a boolean, a list, or an object (for several labels).",
+          "description": "The gold label. Any JSON value except null: a string, a number, a boolean, a list, or an object (for several labels, or a Box label: see docs/fixtures.md).",
           "type": ["string", "number", "boolean", "array", "object"]
         }
       }