Skip to content

Require source-grounded identity evidence before automatic exact-name merges - #1199

Closed
virtue039 wants to merge 3 commits into
deeplethe:devfrom
virtue039:fix/namesake-adjudication-evidence
Closed

virtue039 wants to merge 3 commits into
deeplethe:devfrom
virtue039:fix/namesake-adjudication-evidence

Conversation

@virtue039

Copy link
Copy Markdown
Contributor

Why

A model’s high-confidence same is not a checkable identity link for two records with the same exact name. This is the narrow adjudication follow-up agreed in #1193, separate from the mention-resolution change in #1192.

What changes

Before an automatic exact-name merge, re-read the current pair’s source-grounded facts. A proof needs an explicitly eternal inverse-functional relation pointing to the same entity or concrete attribute value, or an eternal functional incoming relation from the same subject. Both facts must be active, directly evidenced, without time bounds or unknown ending times, and each quote must occur in its current source chunk with matching KB, document and version. Name facts do not count. Ordinary shared employer, role and topic matches do not count without the required uniqueness declaration. Successful merge reasons retain the fact and chunk IDs.

If no proof exists, close the pair as kept with kept_apart|no_checkable_identity_evidence; add no human review and no model call. Recheck cached same verdicts using current evidence. Apply the same rule in the governance automatic path; explicit human decisions remain authoritative. Decision and audit records report the executed keep rather than an applied merge. When the original human review identifies an unresolved left namesake, an automatic keep retains that role in its reason; #1192 uses it to bound later mentions without reopening human work. A deterministic DB regression checks 20 subsequent mentions and the real caller. The two real-model experiments evaluate each PR independently, not their combined effect. Existing execution, contradiction and rollback safeguards remain in place. No schema, ontology-field, HTTP API or prompt changes.

How it was checked

macOS / Apple Silicon; Rust 1.99.0. Baseline da2c45e, candidate 883e440. Required DB/PDF guards enabled: workspace 1501 passed / 0 failed / 6 existing ignored; focused adjudication tests 7 passed (4 new gate regressions). fmt, workspace/all-targets clippy (-D warnings), frozen-lockfile web install/build and diff check passed. Every commit is signed off.

Removing only the execution check reproduces two actual failures: employer-only records merge, and a cached same still merges after a source is superseded. With the check restored both pass. Known-ground-truth fixtures preserve two supported true merges (outgoing and incoming unique identity links) and intentionally lose one true merge supported only by a common employer; that denial adds zero human reviews. Source-missing, unrelated, invalidated, null/empty identifier, mismatched-document, unknown-ending and stale-source cases are rejected. The supported incoming merge still rolls back.

Same-session evaluation

Both sides use qwen3.7-flash-2026-07-15 via Alibaba Cloud Model Studio (Beijing), thinking disabled, and pinned local BAAI/BGE-M3 5617a9f61b028005a4858fdac845db406aefb181 (CPU float32, normalized CLS, 1024 dimensions). Two rounds, eight fresh KBs per side, 12 documents and 21 truth anchors per KB. The upstream corpus, scorer and ontology options are unchanged; all 16 KBs have completed documents and jobs. This is an independent evaluation against dev, not a comparison using the older #1192 baseline. Each round has two independent forward/reverse invocations per side; round two waits for round one. Round-one baseline ingestion began while the final candidate was being frozen; no run was selected or restarted by score.

Official bench, eight KBs each Baseline Candidate
Mean F1 0.5246 0.5285
Mean surviving entities 27.6250 27.1250
Mean pending human reviews 0.5000 0.1250
Mean false-merge anchor pairs 2.0000 2.3750
Mean false-split anchor pairs 6.8750 7.2500
Mean missing anchors 5.0000 4.5000

Round 1 mean F1: 0.5290 → 0.5192.

Round 2 mean F1: 0.5202 → 0.5377.

Observed gate-kept pairs supported by singleton official gold groups on both records: 3 true same-entity merges withheld, 0 different-entity pairs kept. 0 pairs are unanchored/missing/untyped/mixed and remain unknown; 0 later merged/redirected pairs are reported separately. Classification uses current active source quotes, ontology type and the official anchor’s role/other-side constraint, without inferring truth from names. These are observed withheld pairs, not a causal loss estimate obtained by subtracting F1. The controlled fixture’s one lost true merge is reported separately above.

Observed chat calls: baseline 536, candidate 516; tokens 762315 and 703263. Gateway transport errors: 0; all associated jobs completed. The gate itself makes no additional model request. Shared embedding-cache timings are not an isolated speed comparison.

Limits

This first slice relies on the existing ontology’s uniqueness declarations and current extracted facts; checking provenance does not prove extraction semantics are correct. It deliberately does not recognize arbitrary prose co-reference or match on employer/title alone. The stock identity corpus declares state relations and no eternal identity key, so exact-name duplicates without a qualifying link are conservatively kept; source-derived true-merge losses and missing anchors must be read beside F1. Non-exact names, singleton resolution and qualification/alias recall remain separate paths. These small repeated measurements are descriptive, not a significance or end-to-end identity-fix claim.

All 16 fresh KBs
Run Order KB F1 Entities Pending human reviews False-merge pairs False-split pairs Missing
baseline-r1-b1 forward 01a1294e-f70f-7ad1-acd0-aea57c5b5bf8 0.5330 29 0 3 4 7
baseline-r1-b1 reverse 01a12952-25da-7403-9587-114c25cea50d 0.5000 26 0 3 7 4
baseline-r1-b2 forward 01a1294e-f70f-7ad1-acd0-aebcf1e614a1 0.5830 26 1 3 7 4
baseline-r1-b2 reverse 01a12952-85ce-7192-9b01-c2a7f46baafa 0.5000 27 0 1 9 5
baseline-r2-b1 forward 01a1295c-9496-7f20-9d42-926fcd78fffd 0.5560 27 1 2 6 6
baseline-r2-b1 reverse 01a1295f-6de6-7bc3-bd50-cb08b9b2d8d1 0.3330 28 2 1 11 4
baseline-r2-b2 forward 01a1295c-9496-7f20-9d42-927ebe28c957 0.6920 28 0 3 5 4
baseline-r2-b2 reverse 01a1295f-aa7f-7ad3-b046-3be8fcfa7196 0.5000 30 0 0 6 6
candidate-r1-b1 forward 01a12956-3b81-7092-9f89-c0df7e88f333 0.6000 27 0 3 5 6
candidate-r1-b1 reverse 01a12959-9b9b-7743-ad4d-33dc682f1ef8 0.3160 30 1 1 12 3
candidate-r1-b2 forward 01a12956-3b80-7bf1-ab75-27018a68d970 0.7060 27 0 0 5 7
candidate-r1-b2 reverse 01a12959-5d73-7e93-b355-081eab7e0d0e 0.4550 27 0 3 9 4
candidate-r2-b1 forward 01a1295c-9461-76c3-b773-2d6e3bce606d 0.6090 26 0 3 6 4
candidate-r2-b1 reverse 01a1295f-6de6-7b61-8186-85edb075385f 0.4350 28 0 3 10 3
candidate-r2-b2 forward 01a1295c-9461-76c3-b773-2d54c40dd93f 0.4710 27 0 3 6 5
candidate-r2-b2 reverse 01a1295e-ea1f-73c1-8a3f-298e25b3917d 0.6360 25 0 3 5 4

Withheld true same-entity pairs

Run/order KB Official gold group Kept review
candidate-r1-b2 / forward 01a12956-3b80-7bf1-ab75-27018a68d970 ZW_ENG 01a12958-5dbc-7b11-b42c-35e806647a40
candidate-r1-b2 / reverse 01a12959-5d73-7e93-b355-081eab7e0d0e LN 01a1295a-e68b-76a3-8482-a56609acc0be
candidate-r1-b1 / reverse 01a12959-9b9b-7743-ad4d-33dc682f1ef8 LN 01a1295b-2cbd-7123-b450-e418ad2a88bc

Before review

  • Every commit is signed off.
  • fmt, workspace/all-targets clippy and workspace tests pass.
  • Store SQL was exercised with the database and required DB guard.

…amesakes

Signed-off-by: Virtue039 <virtue039@qq.com>
…views

Signed-off-by: Virtue039 <virtue039@qq.com>
@WaylandYang

Copy link
Copy Markdown
Contributor

Thank you for building this so quickly and for stating its cost plainly.

We would like to hold this one as well, and part of the reason is mine: my reply on #1193 was too loose about what counts as evidence, and I did not answer your 04:40 note in time. As built, a merge needs a link the ontology declares unique. An open graph declares none, so every exact-name pair the adjudicator would merge is kept apart, and a kept pair is not asked again. Exact-name pairs are the most common duplicates. Your run shows it: three true merges withheld, no wrong merge prevented.

The same step as on #1192 would help here first: a fixed, scripted set of namesake cases in CI. With it we can agree what evidence is worth requiring before any gate is written.

Thank you again, and sorry for the late answer on the design.

@virtue039

Copy link
Copy Markdown
Contributor Author

Agreed; this gate remains held. #1202 is the separate fixed scripted measurement from dev, with no gate or ontology uniqueness requirement.

It includes an open-graph true-same control alongside the distinct-person high-confidence same failure, both source orders. The current adjudicator merges both, so the true-same control passes while the distinct-person case fails the independent source gold. Strict mode fails on the disagreements; CI displays them and enforces only a proposed known-failure ceiling. It does not label every kept pair correct or treat absence of declared uniqueness as distinctness. We can agree which source evidence should decide these cases before another production gate.

@WaylandYang

Copy link
Copy Markdown
Contributor

Thank you for the careful work here, @virtue039, and for agreeing to the hold. I'm closing this one rather than keeping it open. As measured in the description, a gate that only accepts links the ontology declares unique and timeless keeps every exact-name pair apart in an open graph: F1 moved within noise, three true merges were withheld and no wrong one was prevented, and the lower pending count came from closing pairs as kept, which are not proposed again.

The harness this led to (#1202) is in, and the literal values it found reaching the adjudicator as ? were fixed in #1220, so the adjudicator now sees what tells two namesakes apart. #1251 is the right place to continue, and #1193 to agree what counts as identity evidence before another gate is written. The provenance checks you wrote here were thorough; thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants