[virtue039]

Why

#1136 needs a fixed measurement before another namesake policy is written. The review asked for cases where production code receives distinguishing evidence, can succeed without breaking a control, and is checked against a named list of today's failures.

What changes

Add four cases × both seed orders through the real extract_document or adjudicate_entities, using one local scripted request per case. The independent checks are:

Case	Evidence available to production	Check	Current result
resolver_passage_over_profile	Two Alex Lee candidates have different department facts; the passage says Finance, while the profile prefers Platform Engineering	The incoming fact belongs to the pre-existing Finance holder's exact ID	Named failure in both orders
resolver_passage_agrees_with_profile	The same candidates; passage and profile both point to Platform Engineering	The incoming fact belongs to that pre-existing holder's exact ID	Pass in both orders
adjudicator_literal_clues_reach_request	Each record has its own stored scalar employee number, E17 or F42, and a supporting source quote	Each actual HTTP Record A/B section contains its own employee number → value, without the other record's clue	Named failure in both orders; values currently become ?
adjudicator_entity_clues_reach_request	Each record has an open fact pointing to its own department entity	Each actual HTTP Record A/B section preserves the department fact and its side	Pass in both orders

Both resolver profiles are viable (cosine 1.0 and 0.8); the seed reversal changes insertion order, not the clue or profile roles. Exact-ID checks reject creating another entity or declining to attach. Adjudicator replies are the same fixed different reply in both cases, solely to complete the caller: these cases measure information reaching the request, not scripted model judgement or identity accuracy. Assertions inspect each real user-message section separately, so a fact on the wrong side does not count.

KNOWN_FAILURES names the failing case/order pairs inside the test. Each listed check must fail and every unlisted check must pass. An unexpected pass requires removing its name with the fix; an unlisted failure fails the test. Script, request-count, database, extraction and Human-queue errors are hard failures, never accepted as known defects.

The test runs in ordinary cargo test --workspace. The extra workflow step, summary script, strict flag and standalone notes file have been removed; usage and interpretation are in the module comment. The final PR diff contains only the test module and its cfg(test) registration, with no production policy, schema, dependency or workflow changes. #1192 and #1199 remain held. These synthetic checks do not replace the real-model identity bench.

How it was checked

On macOS arm64, Rust 1.99.0, on head 8050a8c against updated dev 259931a, using a dedicated PostgreSQL 16/pgvector test database migrated through 0111:

• Focused run: 8 cases, exactly the 4 named failures, 4 passing controls, one local scripted request and zero pending Human reviews per case.
• Calibration temporarily changed production code to retain scalar values and consult existing passage corroboration before high-score attachment. All 8 checks then passed, and the unchanged named list correctly failed on 4 unexpected passes; temporarily removing those names made the aggregate test pass.
• A temporary never-attach implementation failed both resolver controls. Omitting entity fact values from requests failed both adjudicator controls. Each produced the corresponding unlisted failures.
• All calibration edits and temporary list changes were restored before the final validation; no prototype fix is included in this PR.
• Workspace: 1515 passed / 0 failed / 6 existing ignored, with UTOPIA_TEST_REQUIRE_DB=1 and UTOPIA_TEST_REQUIRE_PDFTOTEXT=1.
• Format, clippy with warnings denied, full workspace build and diff checks pass.
• Repository-pinned pnpm 10.2.1 frozen install, web tests/build and bench scoring tests pass. No external model requests were made.

Latest CI run 38416809082 is success on 47cf8eb, after the maintainer merged current dev into the PR branch. All six jobs pass; the guarded workspace step reports 1521 passed / 0 failed / 6 existing ignored and executes the named-failure test. The two test files are byte-identical to locally checked 8050a8c; the newer count includes upstream #1216/#1217.

Before review

• Every contribution commit is signed off with Virtue039 <virtue039@qq.com>.
• Format, clippy and workspace tests pass with DB/PDF guards.
• Cases run in the existing workspace CI test step, without an extra flag or ignored test.
• Base is dev; the final diff contains only tests.

[WaylandYang]

Thank you for turning this around so quickly, and for keeping it free of both production changes. The harness is the part we were missing: the real extract_document and adjudicate_entities against a local scripted model, both orders, one request each. I ran it on a fresh database and got the same 4 of 8.

Before it lands, the four failing rows need another look. As built, no change to the code could turn them green without breaking a control:

• Adjudicator. All three adjudicator cases send the model the same prompt (both records read "Alex Lee" (untyped) with works for → ?); only the scripted verdict differs. distinct_same and true_same give the code identical inputs and opposite gold, so the row measures the script. A scripted reply cannot test the model's judgement, but it can test what the model is shown: a case asserting that the request carries the facts that tell the two people apart is one a fix could be judged against.
• Resolver. The mention's vector equals the holder's profile (cosine 1.0) and the other candidate scores 0, so everything the resolver can see points at the holder, and there is no same-person control: a resolver that never attaches when a namesake exists passes. The question in #1136 is whether the passage is consulted. A pair would cover it: the passage names a clue for one candidate while the profile prefers the other, and the control where the two agree.

Two smaller things. Please drop the checks.yml step, the summary script and the strict flag: cargo test --workspace already runs the test, and the named list from my note on #1136 replaces the ceiling. And what is needed from the notes file can live in the module comment.

Thank you again. The harness is right, and with cases the code can win or lose this is what we need.

[virtue039]

Thank you for the precise review. The cases are reworked in 8050a8c; the final diff against updated dev is only the test module and its registration.

• Resolver: both bare-name holders now have distinct department facts pointing to existing entities. The passage uniquely names Finance while the profile prefers Platform Engineering; its paired control names Platform Engineering and agrees with the profile. Both check the exact pre-existing holder ID, not just separation. Seed reversal preserves clue/profile roles.
• Adjudicator: the failing case asserts the real request preserves each record's stored E17/F42 scalar fact in its own Record A/B section. The control checks the already-preserved entity-valued department facts and their sides. Both use the same fixed different transport reply; neither treats that reply as model-quality evidence.
• The named (case, order) list replaces the ceiling: listed checks must fail, every other check must pass. Extra CI steps, the summary script, strict flag and notes file are removed; ordinary workspace CI runs the test and the module comment explains it.

I checked that the code can actually win: temporary scalar-preservation and passage-corroboration edits made all eight checks pass without breaking controls. The unchanged list correctly rejected four unexpected passes, and temporarily removing their names made the test pass. A never-attach mutation and an omit-department-values mutation each failed the corresponding two unlisted controls. All temporary production/list edits were restored before the final checks and are excluded from this PR.

Final local results: 1515 passed / 0 failed / 6 existing ignored, with DB/PDF guards, fmt, clippy, workspace build, frozen pnpm 10.2.1 web tests/build and bench scoring checks passing. The focused run completes all eight: exactly the four named failures, four controls pass, one local scripted request and zero pending Human reviews each. CI run 38410602919 is complete: all six jobs passed, with DB/PDF guards and the named-failure test running in the ordinary workspace step. Its merge run reports 1516 passed / 0 failed / 6 existing ignored, including the additional upstream #1216 rule-combination regression. No external model was called.

[WaylandYang]

Thank you for the rework. This is the measurement we were missing, and every point from the review is met: the cases give production something to decide with, each failing case has a control that a split-everything fix would break, the known failures are a named list in the test, and the workflow step, the flag and the notes file are gone.

I ran it on a fresh database: eight rows, exactly the four named failures. To check that a listed case is one a fix can win, I rendered literal values in the adjudicator's fact lines locally (one line in entity_fact_lines_many): both adjudicator_literal_clues_reach_request rows turned into "unexpected pass: remove this name with its fix", and the test failed as it should. So the list works in both directions, and that case has found a real defect: a value such as a title or an employee number reaches the adjudicator as ?.

Merging. We will follow up on the literal values with the identity bench before and after; the resolver case stays listed for whoever takes the passage-over-profile question next. Thanks again.