Skip to content

BackgroundCheck: merge people on identifiers first, never on a name alone across sources #474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
StephenAbbott merged 2 commits into from
Oct 21, 2026
Merged

BackgroundCheck: merge people on identifiers first, never on a name alone across sources #474

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f32001
BackgroundCheck: merge people on identifiers first, never on a name a…
Oct 20, 2026
09b44ce
Merge origin/main into fix/backgroundcheck-person-merge
claude Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .phases/pending/backgroundcheck-person-merge.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
title: BackgroundCheck merges people on identifiers first, and never on a name alone across sources.
readme: BackgroundCheck now merges people on a shared identifier first and on name plus birth year second, and never merges two Wikidata items or two Estonian register person keys into one card.
tests: vitest +13, vitest_files +1
---
From the "BackgroundCheck person-source data quality" ticket (section 2, Wikidata bare-name records), and the precondition Stephen set on 20 Oct 2026 for any new BackgroundCheck person source (Phase F). `extractConnectedPeople` keyed every person on order-insensitive name + birth year, ignoring identifiers, so two same-name records with no birth year always merged: Unilever's Wikidata "Fernando Fernández" card held a Mexican actor's item and another person's, and since the Estonia work two Estonians with the same name, which the register keeps apart by `isikukood_hash`, would have shared a card (the open data carries no birth dates). The new `groupPersonRecords` decides which statements are one person: records sharing any identifier (read from `identifiers` and the Phase 255 `identifying` annotations) are one; a same-name pair merges when both carry a birth year and the years agree, or when neither does and both come from the same source (one register naming one person on several lists, which kept NZ and the Estonian page path merging as before); a name alone never merges across sources, and those pairs stay separate cards that clustering offers at Medium. Nothing merges across two values of a scheme that names one person per value (`ONE_PERSON_PER_VALUE_SCHEMES`: `EE-ARIREGISTER-HASH`, `WIKIDATA`), and `clusterPeople.scorePair` no longer offers such a pair for review. Companies House officer ids are deliberately not in that set: one person can hold several. A birth year on one side and none on the other still does not merge, as before. A card's key stays `name|year` and gains the first statementId only when two people now share it. Measured on production exports of 20 Oct 2026 against the previous rule: Unilever 12 → 13 people (the two Wikidata items split, each still a Medium candidate against the real CFO, never against each other); HSBC unchanged (19 people, no clusters); Tallink unchanged at 14 people, and its two Medium clusters, the Hanschmidt family (four people) and Enn and Eve Pant, are gone because the register gives each a different person key. `frontend/scripts/cluster-eval.mjs` now imports the real modules (run with `npx vite-node`) instead of a copy that had already fallen behind. `docs/background-check.md` states the rule. Section 1 of the ticket (Estonian coverage) was closed by the Estonia plan without code: Tallink now shows 14 people where it showed 4. Option 3 on the ticket (requiring more than a bare name for a Medium cluster against a Wikidata name-only record) was not taken. After deploy: open BackgroundCheck on Unilever and Tallink and confirm the cards and clusters above.
17 changes: 17 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1738,6 +1738,23 @@ control per person on BackgroundCheck, the shape of the Estonian

---

## BackgroundCheck merges people on identifiers first (20 Oct 2026)

`groupPersonRecords` in `frontend/src/lib/backgroundCheck.ts`, pinned by
`personIdentityMerge.test.ts`. Stephen's decision (20 Oct 2026, the
"person-source data quality" ticket, the precondition for Phase F): a shared
identifier merges; name + birth year merges only when both sides have a year;
a name alone merges only within one source; nothing merges across two values
of `ONE_PERSON_PER_VALUE_SCHEMES` (`ee-ariregister-hash`, `wikidata`), and
`clusterPeople.scorePair` never offers such a pair. Companies House officer
ids stay out of that set on purpose: one person can hold several. A new
person source that publishes a register-issued person key should add its
scheme there. `frontend/scripts/cluster-eval.mjs` runs the real modules
(`npx vite-node`) against captured bundles — use it to measure a change on
Unilever, HSBC and Tallink before shipping.

---

## The Estonia dataset is built from local data (sweep_estonia.py)

`backend/scripts/sweep_estonia.py` builds the `dataset-estonia-<cut>` release
Expand Down
20 changes: 17 additions & 3 deletions docs/background-check.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,23 @@ error so "no hit" renders honestly. Tests: `tests/test_person_check.py`.
**Frontend** — a third check mode alongside QuickCheck / FullCheck:

- `src/lib/backgroundCheck.ts` — extracts connected people from the assembled
BODS bundle (person statements + their relationship statements), merging the
same person across sources by normalised name + birth year while keeping
every statementId for traceability. Tested in `backgroundCheck.test.ts`.
BODS bundle (person statements + their relationship statements), merging
records that are one person while keeping every statementId for
traceability. Tested in `backgroundCheck.test.ts` and
`personIdentityMerge.test.ts`. The merge rule (20 Oct 2026,
`groupPersonRecords`):
1. **Identifiers first.** Records sharing an identifier (the Estonian
register's person key, a Wikidata item, a Companies House officer id, an
OpenSanctions id…) are one person, whatever the spelling.
2. **Name + birth year** when both sides carry a year and the years agree.
3. **Name alone only within one source** (one register naming one person on
several lists). Across sources a same-name pair with no birth year stays
two cards and is offered for review as a Medium cluster.
4. **Never across two values of a one-person-per-value scheme**
(`ONE_PERSON_PER_VALUE_SCHEMES`: the Estonian person key and Wikidata).
Such pairs are not offered as clusters either: the source has already
said they are different people. Companies House officer ids are left out
on purpose: one person can hold several.
- `src/components/cdd/BackgroundCheckPanel.tsx` — lists connected people with
their roles/sources; per-person "Run background check" (plus a capped
"check all") calls `/person-check` on demand and renders risk chips,
Expand Down
231 changes: 18 additions & 213 deletions frontend/scripts/cluster-eval.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,228 +1,33 @@
#!/usr/bin/env node
/**
* Phase 3 evaluation harness for the Cluster UI ticket — SELF-CONTAINED.
* Phase 3 evaluation harness for the Cluster UI ticket.
*
* Runs the real clustering logic over one or more /lookup bundles and prints,
* per bundle: the connected people, the clusters formed (confidence + evidence),
* and summary metrics. High-confidence clusters are called out for scrutiny (a
* false High pairing two different people is the failure we most care about).
* Runs the real extraction and clustering over one or more /lookup bundles and
* prints, per bundle: the clusters formed (confidence + evidence) and summary
* metrics. High-confidence clusters are called out for scrutiny (a false High
* pairing two different people is the failure we most care about).
*
* The extraction + clustering below mirror frontend/src/lib/backgroundCheck.ts
* (order-insensitive nameMatchKey) and clusterPeople.ts verbatim, so verdicts
* match the live UI. No imports — drop it anywhere and run.
* It imports frontend/src/lib/backgroundCheck.ts and clusterPeople.ts
* directly, so it cannot drift from the UI. (Until the person-merge fix of
* 20 Oct 2026 it carried a verbatim copy, which had already fallen behind:
* it read no identifiers from annotations.) Run it with vite-node, which
* vitest ships:
*
* Usage (from the repo root; bundles are captured /lookup responses, never
* committed — make one first):
* curl 'http://127.0.0.1:8000/lookup?lei=<LEI>&refresh=true' > company.json
* node frontend/scripts/cluster-eval.mjs company.json [more.json ...]
* cd frontend && npx vite-node scripts/cluster-eval.mjs ../company.json [more.json ...]
*
* A bundle is a captured /lookup response ({bods: [...]}), an export
* ({statements: [...]}) or a bare statement array — never committed.
*/

import { readFileSync } from "node:fs";

// ---- name matching (mirror of backgroundCheck.ts + clusterPeople.ts) -------
function normaliseName(name) {
return name
.normalize("NFKD")
.replace(/[̀-ͯ]/g, "")
.toLowerCase()
.replace(/[^\p{L}\p{N}\s]/gu, " ")
.replace(/\s+/g, " ")
.trim();
}
const HONORIFICS = new Set([
"mr", "mrs", "ms", "miss", "mx", "dr", "prof", "professor", "sir", "dame",
"lord", "lady", "rev", "reverend", "hon", "honourable", "honorable",
]);
function nameTokens(name) {
const toks = normaliseName(name).split(" ").filter(Boolean);
const stripped = toks.filter((t) => !HONORIFICS.has(t));
return stripped.length > 0 ? stripped : toks;
}
function nameMatchKey(name) {
return [...nameTokens(name)].sort().join(" ");
}
function tokens(name) {
return nameTokens(name);
}
function diceCoefficient(a, b) {
if (a === b) return 1;
if (a.length < 2 || b.length < 2) return 0;
const bigrams = (s) => {
const m = new Map();
for (let i = 0; i < s.length - 1; i++) {
const bg = s.slice(i, i + 2);
m.set(bg, (m.get(bg) ?? 0) + 1);
}
return m;
};
const A = bigrams(a), B = bigrams(b);
let inter = 0, sizeA = 0, sizeB = 0;
for (const v of A.values()) sizeA += v;
for (const v of B.values()) sizeB += v;
for (const [bg, ca] of A) { const cb = B.get(bg); if (cb) inter += Math.min(ca, cb); }
return (2 * inter) / (sizeA + sizeB);
}
function tokensCoveredByInitials(few, many) {
const used = new Array(many.length).fill(false);
return few.every((t) => {
for (let i = 0; i < many.length; i++) {
if (used[i]) continue;
const m = many[i];
if (m === t || (t.length === 1 && m[0] === t) || (m.length === 1 && t[0] === m)) {
used[i] = true; return true;
}
}
return false;
});
}
function nameSimilarity(aName, bName) {
const a = tokens(aName).join(" "), b = tokens(bName).join(" ");
if (a === b) return { score: 1, note: "identical name" };
const ta = tokens(aName), tb = tokens(bName);
if ([...ta].sort().join(" ") === [...tb].sort().join(" ")) {
return { score: 0.97, note: "same name, order differs" };
}
const shorter = ta.length <= tb.length ? ta : tb;
const longer = ta.length <= tb.length ? tb : ta;
const sameFirstLast =
shorter.length >= 2 &&
(shorter[0] === longer[0] || shorter[0][0] === longer[0][0]) &&
shorter[shorter.length - 1] === longer[longer.length - 1];
if (sameFirstLast && tokensCoveredByInitials(shorter, longer)) {
return { score: 0.9, note: "differs only by a middle name or initial" };
}
const dice = diceCoefficient(a, b);
if (dice >= 0.8) return { score: 0.8, note: "names very similar" };
if (dice >= 0.6) return { score: 0.6, note: "names loosely similar" };
return { score: dice, note: "names differ" };
}

// ---- pairwise signals + scoring --------------------------------------------
function birthYearRelation(a, b) {
if (a.birthYear === undefined && b.birthYear === undefined) return "both-missing";
if (a.birthYear === undefined || b.birthYear === undefined) return "one-missing";
return a.birthYear === b.birthYear ? "match" : "conflict";
}
function overlap(a = [], b = []) {
const B = new Set(b.map((x) => String(x).toLowerCase()));
return a.filter((x) => B.has(String(x).toLowerCase()));
}
function companiesOf(p) {
return (p.roles ?? []).map((r) => r.subjectName).filter(Boolean);
}
const MEDIUM = 0.62;
function scorePair(a, b) {
const ids = overlap(a.identifiers ?? [], b.identifiers ?? []);
const name = nameSimilarity(a.name, b.name);
const byr = birthYearRelation(a, b);
const nats = overlap(a.nationalities, b.nationalities);
const cos = overlap(companiesOf(a), companiesOf(b));
let score = name.score;
if (byr === "match") score += 0.1;
if (nats.length) score += 0.05;
if (cos.length) score += 0.05;
if (byr === "conflict") score = Math.min(score, 0.7);
score = Math.min(score, 1);
const identifierDominant = ids.length > 0;
if (identifierDominant) score = Math.max(score, 0.95);
if (score < MEDIUM && !identifierDominant) return null;
const highOnMerit = name.score >= 0.9 && byr === "match" && (nats.length > 0 || cos.length > 0);
const confidence = identifierDominant || highOnMerit ? "high" : "medium";
const byrNote =
byr === "match" ? `both born ${a.birthYear}`
: byr === "conflict" ? `birth years differ (${a.birthYear} vs ${b.birthYear})`
: byr === "one-missing" ? "birth year missing on one record"
: "no birth year on either record";
const parts = [];
if (ids.length) parts.push(`shared identifier ${ids.join(", ")}`);
parts.push(name.note);
parts.push(byrNote);
if (nats.length) parts.push(`both ${nats.join("/")}`);
if (cos.length) parts.push(`shared role in ${cos.join(", ")}`);
return { aKey: a.key, bKey: b.key, score: Number(score.toFixed(3)), confidence, evidence: parts.join("; ") };
}
function clusterConnectedPeople(people) {
const parent = people.map((_, i) => i);
const find = (i) => { while (parent[i] !== i) { parent[i] = parent[parent[i]]; i = parent[i]; } return i; };
const union = (i, j) => { parent[find(i)] = find(j); };
const pairs = [];
for (let i = 0; i < people.length; i++)
for (let j = i + 1; j < people.length; j++) {
const pair = scorePair(people[i], people[j]);
if (!pair) continue;
pairs.push(pair); union(i, j);
}
const groups = new Map();
for (let i = 0; i < people.length; i++) {
const root = find(i);
(groups.get(root) ?? groups.set(root, []).get(root)).push(people[i].key);
}
const clusters = [], clustered = new Set();
for (const keys of groups.values()) {
if (keys.length < 2) continue;
keys.forEach((k) => clustered.add(k));
const memberSet = new Set(keys);
const clusterPairs = pairs.filter((p) => memberSet.has(p.aKey) && memberSet.has(p.bKey));
const confidence = clusterPairs.some((p) => p.confidence === "high") ? "high" : "medium";
clusters.push({ keys, size: keys.length, confidence, pairs: clusterPairs });
}
const singletons = people.map((p) => p.key).filter((k) => !clustered.has(k));
return { clusters, singletons };
}

// ---- person extraction (mirror of extractConnectedPeople merge) ------------
const str = (v) => (typeof v === "string" && v ? v : undefined);
const rec = (v) => (v && typeof v === "object" && !Array.isArray(v) ? v : {});
const arr = (v) => (Array.isArray(v) ? v : []);
function personName(rd) {
const names = arr(rd.names).map(rec);
const chosen = names.find((n) => str(n.type) === "individual") ?? names.find((n) => str(n.fullName));
if (!chosen) return undefined;
const full = str(chosen.fullName);
if (full) return full;
const joined = `${str(chosen.givenName) ?? ""} ${str(chosen.familyName) ?? ""}`.trim();
return joined || undefined;
}
function birthYearOf(bd) { const m = bd?.match(/^(\d{4})/); return m ? Number(m[1]) : undefined; }
function extractPersonIdentifiers(rd) {
const out = [];
for (const id of arr(rd.identifiers).map(rec)) {
const scheme = str(id.scheme) ?? str(id.schemeName);
const value = str(id.id);
if (scheme && value) out.push(`${scheme}:${value}`.toLowerCase());
else if (value) out.push(String(value).toLowerCase());
}
return out;
}
function extractPeople(statements) {
const people = new Map();
for (const s of statements) {
const t = str(s.recordType) ?? str(s.statementType);
if (t !== "person" && t !== "personStatement") continue;
const rd = rec(s.recordDetails);
const pt = str(rd.personType);
if (pt && pt !== "knownPerson") continue;
const name = personName(rd);
if (!name) continue;
const birthDate = str(rd.birthDate);
const birthYear = birthYearOf(birthDate);
const key = `${nameMatchKey(name)}|${birthYear ?? ""}`;
let p = people.get(key);
if (!p) { p = { key, name, birthDate, birthYear, nationalities: [], identifiers: [], roles: [], sources: [], statementIds: [] }; people.set(key, p); }
for (const n of arr(rd.nationalities).map(rec)) { const l = str(n.name) ?? str(n.code); if (l && !p.nationalities.includes(l)) p.nationalities.push(l); }
for (const id of extractPersonIdentifiers(rd)) if (!p.identifiers.includes(id)) p.identifiers.push(id);
const src = str(rec(s.source).description);
if (src && !p.sources.includes(src)) p.sources.push(src);
const sid = str(s.statementId);
if (sid && !p.statementIds.includes(sid)) p.statementIds.push(sid);
}
return Array.from(people.values());
}
import { extractConnectedPeople } from "../src/lib/backgroundCheck.ts";
import { clusterConnectedPeople } from "../src/lib/clusterPeople.ts";

function evalBundle(path) {
const raw = JSON.parse(readFileSync(path, "utf8"));
const statements = Array.isArray(raw) ? raw : arr(raw.bods);
const people = extractPeople(statements);
const statements = Array.isArray(raw) ? raw : (raw.bods ?? raw.statements ?? []);
const people = extractConnectedPeople(statements);
const { clusters, singletons } = clusterConnectedPeople(people);
console.log(`\n########## ${path} ##########`);
console.log(`people=${people.length} clusters=${clusters.length} singletons=${singletons.length}`);
Expand Down
22 changes: 18 additions & 4 deletions frontend/src/lib/backgroundCheck.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,11 +296,25 @@ describe("nzRoleHolder — the New Zealand register's anchor for a person", () =
};

it("carries the company number and the name as the NZ filing gives it", () => {
// The same person from another source, spelt differently, merges first.
// The same register naming the person twice, spelt differently, merges
// first (same source, no birth year on either); the anchor keeps the
// spelling of the statement the role is filed against.
const other = { ...nzPerson, statementId: "nz-p2", recordDetails: { ...nzPerson.recordDetails, names: [{ fullName: "Mark Julian VERBIEST" }] } };
const people = extractConnectedPeople([nzCompany, other, nzPerson, nzRel]);
expect(people).toHaveLength(1);
expect(people[0].name).toBe("Mark Julian VERBIEST");
expect(people[0].nzRoleHolder).toEqual({ companyNumber: "1166320", name: "Mark Julian Verbiest" });
});

it("keeps a same-name record from another source with no birth year on its own card", () => {
const other = person("w-p", "Mark Julian VERBIEST", {}, "Wikidata");
const [p] = extractConnectedPeople([nzCompany, other, nzPerson, nzRel]);
expect(p.name).toBe("Mark Julian VERBIEST");
expect(p.nzRoleHolder).toEqual({ companyNumber: "1166320", name: "Mark Julian Verbiest" });
const people = extractConnectedPeople([nzCompany, other, nzPerson, nzRel]);
expect(people).toHaveLength(2);
const nz = people.find((p) => p.statementIds.includes("nz-p"));
const wd = people.find((p) => p.statementIds.includes("w-p"));
expect(nz?.nzRoleHolder).toEqual({ companyNumber: "1166320", name: "Mark Julian Verbiest" });
expect(wd?.nzRoleHolder).toBeUndefined();
expect(nz?.key).not.toBe(wd?.key);
});

it("falls back to the NZBN when the filing has no company number", () => {
Expand Down
Loading
Loading