diff --git a/.phases/pending/backgroundcheck-person-merge.md b/.phases/pending/backgroundcheck-person-merge.md new file mode 100644 index 00000000..f4f9bfa8 --- /dev/null +++ b/.phases/pending/backgroundcheck-person-merge.md @@ -0,0 +1,6 @@ +--- +title: BackgroundCheck merges people on identifiers first, and never on a name alone across sources. +readme: BackgroundCheck now merges people on a shared identifier first and on name plus birth year second, and never merges two Wikidata items or two Estonian register person keys into one card. +tests: vitest +13, vitest_files +1 +--- +From the "BackgroundCheck person-source data quality" ticket (section 2, Wikidata bare-name records), and the precondition Stephen set on 20 Oct 2026 for any new BackgroundCheck person source (Phase F). `extractConnectedPeople` keyed every person on order-insensitive name + birth year, ignoring identifiers, so two same-name records with no birth year always merged: Unilever's Wikidata "Fernando Fernández" card held a Mexican actor's item and another person's, and since the Estonia work two Estonians with the same name, which the register keeps apart by `isikukood_hash`, would have shared a card (the open data carries no birth dates). The new `groupPersonRecords` decides which statements are one person: records sharing any identifier (read from `identifiers` and the Phase 255 `identifying` annotations) are one; a same-name pair merges when both carry a birth year and the years agree, or when neither does and both come from the same source (one register naming one person on several lists, which kept NZ and the Estonian page path merging as before); a name alone never merges across sources, and those pairs stay separate cards that clustering offers at Medium. Nothing merges across two values of a scheme that names one person per value (`ONE_PERSON_PER_VALUE_SCHEMES`: `EE-ARIREGISTER-HASH`, `WIKIDATA`), and `clusterPeople.scorePair` no longer offers such a pair for review. Companies House officer ids are deliberately not in that set: one person can hold several. A birth year on one side and none on the other still does not merge, as before. A card's key stays `name|year` and gains the first statementId only when two people now share it. Measured on production exports of 20 Oct 2026 against the previous rule: Unilever 12 → 13 people (the two Wikidata items split, each still a Medium candidate against the real CFO, never against each other); HSBC unchanged (19 people, no clusters); Tallink unchanged at 14 people, and its two Medium clusters, the Hanschmidt family (four people) and Enn and Eve Pant, are gone because the register gives each a different person key. `frontend/scripts/cluster-eval.mjs` now imports the real modules (run with `npx vite-node`) instead of a copy that had already fallen behind. `docs/background-check.md` states the rule. Section 1 of the ticket (Estonian coverage) was closed by the Estonia plan without code: Tallink now shows 14 people where it showed 4. Option 3 on the ticket (requiring more than a bare name for a Medium cluster against a Wikidata name-only record) was not taken. After deploy: open BackgroundCheck on Unilever and Tallink and confirm the cards and clusters above. diff --git a/CLAUDE.md b/CLAUDE.md index 9e7fb900..f7297e6c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1738,6 +1738,23 @@ control per person on BackgroundCheck, the shape of the Estonian --- +## BackgroundCheck merges people on identifiers first (20 Oct 2026) + +`groupPersonRecords` in `frontend/src/lib/backgroundCheck.ts`, pinned by +`personIdentityMerge.test.ts`. Stephen's decision (20 Oct 2026, the +"person-source data quality" ticket, the precondition for Phase F): a shared +identifier merges; name + birth year merges only when both sides have a year; +a name alone merges only within one source; nothing merges across two values +of `ONE_PERSON_PER_VALUE_SCHEMES` (`ee-ariregister-hash`, `wikidata`), and +`clusterPeople.scorePair` never offers such a pair. Companies House officer +ids stay out of that set on purpose: one person can hold several. A new +person source that publishes a register-issued person key should add its +scheme there. `frontend/scripts/cluster-eval.mjs` runs the real modules +(`npx vite-node`) against captured bundles — use it to measure a change on +Unilever, HSBC and Tallink before shipping. + +--- + ## The Estonia dataset is built from local data (sweep_estonia.py) `backend/scripts/sweep_estonia.py` builds the `dataset-estonia-` release diff --git a/docs/background-check.md b/docs/background-check.md index 06f0e1a1..7c850d65 100644 --- a/docs/background-check.md +++ b/docs/background-check.md @@ -29,9 +29,23 @@ error so "no hit" renders honestly. Tests: `tests/test_person_check.py`. **Frontend** — a third check mode alongside QuickCheck / FullCheck: - `src/lib/backgroundCheck.ts` — extracts connected people from the assembled - BODS bundle (person statements + their relationship statements), merging the - same person across sources by normalised name + birth year while keeping - every statementId for traceability. Tested in `backgroundCheck.test.ts`. + BODS bundle (person statements + their relationship statements), merging + records that are one person while keeping every statementId for + traceability. Tested in `backgroundCheck.test.ts` and + `personIdentityMerge.test.ts`. The merge rule (20 Oct 2026, + `groupPersonRecords`): + 1. **Identifiers first.** Records sharing an identifier (the Estonian + register's person key, a Wikidata item, a Companies House officer id, an + OpenSanctions id…) are one person, whatever the spelling. + 2. **Name + birth year** when both sides carry a year and the years agree. + 3. **Name alone only within one source** (one register naming one person on + several lists). Across sources a same-name pair with no birth year stays + two cards and is offered for review as a Medium cluster. + 4. **Never across two values of a one-person-per-value scheme** + (`ONE_PERSON_PER_VALUE_SCHEMES`: the Estonian person key and Wikidata). + Such pairs are not offered as clusters either: the source has already + said they are different people. Companies House officer ids are left out + on purpose: one person can hold several. - `src/components/cdd/BackgroundCheckPanel.tsx` — lists connected people with their roles/sources; per-person "Run background check" (plus a capped "check all") calls `/person-check` on demand and renders risk chips, diff --git a/frontend/scripts/cluster-eval.mjs b/frontend/scripts/cluster-eval.mjs index 6b54e8de..c342edf0 100644 --- a/frontend/scripts/cluster-eval.mjs +++ b/frontend/scripts/cluster-eval.mjs @@ -1,228 +1,33 @@ #!/usr/bin/env node /** - * Phase 3 evaluation harness for the Cluster UI ticket — SELF-CONTAINED. + * Phase 3 evaluation harness for the Cluster UI ticket. * - * Runs the real clustering logic over one or more /lookup bundles and prints, - * per bundle: the connected people, the clusters formed (confidence + evidence), - * and summary metrics. High-confidence clusters are called out for scrutiny (a - * false High pairing two different people is the failure we most care about). + * Runs the real extraction and clustering over one or more /lookup bundles and + * prints, per bundle: the clusters formed (confidence + evidence) and summary + * metrics. High-confidence clusters are called out for scrutiny (a false High + * pairing two different people is the failure we most care about). * - * The extraction + clustering below mirror frontend/src/lib/backgroundCheck.ts - * (order-insensitive nameMatchKey) and clusterPeople.ts verbatim, so verdicts - * match the live UI. No imports — drop it anywhere and run. + * It imports frontend/src/lib/backgroundCheck.ts and clusterPeople.ts + * directly, so it cannot drift from the UI. (Until the person-merge fix of + * 20 Oct 2026 it carried a verbatim copy, which had already fallen behind: + * it read no identifiers from annotations.) Run it with vite-node, which + * vitest ships: * - * Usage (from the repo root; bundles are captured /lookup responses, never - * committed — make one first): * curl 'http://127.0.0.1:8000/lookup?lei=&refresh=true' > company.json - * node frontend/scripts/cluster-eval.mjs company.json [more.json ...] + * cd frontend && npx vite-node scripts/cluster-eval.mjs ../company.json [more.json ...] + * + * A bundle is a captured /lookup response ({bods: [...]}), an export + * ({statements: [...]}) or a bare statement array — never committed. */ import { readFileSync } from "node:fs"; - -// ---- name matching (mirror of backgroundCheck.ts + clusterPeople.ts) ------- -function normaliseName(name) { - return name - .normalize("NFKD") - .replace(/[̀-ͯ]/g, "") - .toLowerCase() - .replace(/[^\p{L}\p{N}\s]/gu, " ") - .replace(/\s+/g, " ") - .trim(); -} -const HONORIFICS = new Set([ - "mr", "mrs", "ms", "miss", "mx", "dr", "prof", "professor", "sir", "dame", - "lord", "lady", "rev", "reverend", "hon", "honourable", "honorable", -]); -function nameTokens(name) { - const toks = normaliseName(name).split(" ").filter(Boolean); - const stripped = toks.filter((t) => !HONORIFICS.has(t)); - return stripped.length > 0 ? stripped : toks; -} -function nameMatchKey(name) { - return [...nameTokens(name)].sort().join(" "); -} -function tokens(name) { - return nameTokens(name); -} -function diceCoefficient(a, b) { - if (a === b) return 1; - if (a.length < 2 || b.length < 2) return 0; - const bigrams = (s) => { - const m = new Map(); - for (let i = 0; i < s.length - 1; i++) { - const bg = s.slice(i, i + 2); - m.set(bg, (m.get(bg) ?? 0) + 1); - } - return m; - }; - const A = bigrams(a), B = bigrams(b); - let inter = 0, sizeA = 0, sizeB = 0; - for (const v of A.values()) sizeA += v; - for (const v of B.values()) sizeB += v; - for (const [bg, ca] of A) { const cb = B.get(bg); if (cb) inter += Math.min(ca, cb); } - return (2 * inter) / (sizeA + sizeB); -} -function tokensCoveredByInitials(few, many) { - const used = new Array(many.length).fill(false); - return few.every((t) => { - for (let i = 0; i < many.length; i++) { - if (used[i]) continue; - const m = many[i]; - if (m === t || (t.length === 1 && m[0] === t) || (m.length === 1 && t[0] === m)) { - used[i] = true; return true; - } - } - return false; - }); -} -function nameSimilarity(aName, bName) { - const a = tokens(aName).join(" "), b = tokens(bName).join(" "); - if (a === b) return { score: 1, note: "identical name" }; - const ta = tokens(aName), tb = tokens(bName); - if ([...ta].sort().join(" ") === [...tb].sort().join(" ")) { - return { score: 0.97, note: "same name, order differs" }; - } - const shorter = ta.length <= tb.length ? ta : tb; - const longer = ta.length <= tb.length ? tb : ta; - const sameFirstLast = - shorter.length >= 2 && - (shorter[0] === longer[0] || shorter[0][0] === longer[0][0]) && - shorter[shorter.length - 1] === longer[longer.length - 1]; - if (sameFirstLast && tokensCoveredByInitials(shorter, longer)) { - return { score: 0.9, note: "differs only by a middle name or initial" }; - } - const dice = diceCoefficient(a, b); - if (dice >= 0.8) return { score: 0.8, note: "names very similar" }; - if (dice >= 0.6) return { score: 0.6, note: "names loosely similar" }; - return { score: dice, note: "names differ" }; -} - -// ---- pairwise signals + scoring -------------------------------------------- -function birthYearRelation(a, b) { - if (a.birthYear === undefined && b.birthYear === undefined) return "both-missing"; - if (a.birthYear === undefined || b.birthYear === undefined) return "one-missing"; - return a.birthYear === b.birthYear ? "match" : "conflict"; -} -function overlap(a = [], b = []) { - const B = new Set(b.map((x) => String(x).toLowerCase())); - return a.filter((x) => B.has(String(x).toLowerCase())); -} -function companiesOf(p) { - return (p.roles ?? []).map((r) => r.subjectName).filter(Boolean); -} -const MEDIUM = 0.62; -function scorePair(a, b) { - const ids = overlap(a.identifiers ?? [], b.identifiers ?? []); - const name = nameSimilarity(a.name, b.name); - const byr = birthYearRelation(a, b); - const nats = overlap(a.nationalities, b.nationalities); - const cos = overlap(companiesOf(a), companiesOf(b)); - let score = name.score; - if (byr === "match") score += 0.1; - if (nats.length) score += 0.05; - if (cos.length) score += 0.05; - if (byr === "conflict") score = Math.min(score, 0.7); - score = Math.min(score, 1); - const identifierDominant = ids.length > 0; - if (identifierDominant) score = Math.max(score, 0.95); - if (score < MEDIUM && !identifierDominant) return null; - const highOnMerit = name.score >= 0.9 && byr === "match" && (nats.length > 0 || cos.length > 0); - const confidence = identifierDominant || highOnMerit ? "high" : "medium"; - const byrNote = - byr === "match" ? `both born ${a.birthYear}` - : byr === "conflict" ? `birth years differ (${a.birthYear} vs ${b.birthYear})` - : byr === "one-missing" ? "birth year missing on one record" - : "no birth year on either record"; - const parts = []; - if (ids.length) parts.push(`shared identifier ${ids.join(", ")}`); - parts.push(name.note); - parts.push(byrNote); - if (nats.length) parts.push(`both ${nats.join("/")}`); - if (cos.length) parts.push(`shared role in ${cos.join(", ")}`); - return { aKey: a.key, bKey: b.key, score: Number(score.toFixed(3)), confidence, evidence: parts.join("; ") }; -} -function clusterConnectedPeople(people) { - const parent = people.map((_, i) => i); - const find = (i) => { while (parent[i] !== i) { parent[i] = parent[parent[i]]; i = parent[i]; } return i; }; - const union = (i, j) => { parent[find(i)] = find(j); }; - const pairs = []; - for (let i = 0; i < people.length; i++) - for (let j = i + 1; j < people.length; j++) { - const pair = scorePair(people[i], people[j]); - if (!pair) continue; - pairs.push(pair); union(i, j); - } - const groups = new Map(); - for (let i = 0; i < people.length; i++) { - const root = find(i); - (groups.get(root) ?? groups.set(root, []).get(root)).push(people[i].key); - } - const clusters = [], clustered = new Set(); - for (const keys of groups.values()) { - if (keys.length < 2) continue; - keys.forEach((k) => clustered.add(k)); - const memberSet = new Set(keys); - const clusterPairs = pairs.filter((p) => memberSet.has(p.aKey) && memberSet.has(p.bKey)); - const confidence = clusterPairs.some((p) => p.confidence === "high") ? "high" : "medium"; - clusters.push({ keys, size: keys.length, confidence, pairs: clusterPairs }); - } - const singletons = people.map((p) => p.key).filter((k) => !clustered.has(k)); - return { clusters, singletons }; -} - -// ---- person extraction (mirror of extractConnectedPeople merge) ------------ -const str = (v) => (typeof v === "string" && v ? v : undefined); -const rec = (v) => (v && typeof v === "object" && !Array.isArray(v) ? v : {}); -const arr = (v) => (Array.isArray(v) ? v : []); -function personName(rd) { - const names = arr(rd.names).map(rec); - const chosen = names.find((n) => str(n.type) === "individual") ?? names.find((n) => str(n.fullName)); - if (!chosen) return undefined; - const full = str(chosen.fullName); - if (full) return full; - const joined = `${str(chosen.givenName) ?? ""} ${str(chosen.familyName) ?? ""}`.trim(); - return joined || undefined; -} -function birthYearOf(bd) { const m = bd?.match(/^(\d{4})/); return m ? Number(m[1]) : undefined; } -function extractPersonIdentifiers(rd) { - const out = []; - for (const id of arr(rd.identifiers).map(rec)) { - const scheme = str(id.scheme) ?? str(id.schemeName); - const value = str(id.id); - if (scheme && value) out.push(`${scheme}:${value}`.toLowerCase()); - else if (value) out.push(String(value).toLowerCase()); - } - return out; -} -function extractPeople(statements) { - const people = new Map(); - for (const s of statements) { - const t = str(s.recordType) ?? str(s.statementType); - if (t !== "person" && t !== "personStatement") continue; - const rd = rec(s.recordDetails); - const pt = str(rd.personType); - if (pt && pt !== "knownPerson") continue; - const name = personName(rd); - if (!name) continue; - const birthDate = str(rd.birthDate); - const birthYear = birthYearOf(birthDate); - const key = `${nameMatchKey(name)}|${birthYear ?? ""}`; - let p = people.get(key); - if (!p) { p = { key, name, birthDate, birthYear, nationalities: [], identifiers: [], roles: [], sources: [], statementIds: [] }; people.set(key, p); } - for (const n of arr(rd.nationalities).map(rec)) { const l = str(n.name) ?? str(n.code); if (l && !p.nationalities.includes(l)) p.nationalities.push(l); } - for (const id of extractPersonIdentifiers(rd)) if (!p.identifiers.includes(id)) p.identifiers.push(id); - const src = str(rec(s.source).description); - if (src && !p.sources.includes(src)) p.sources.push(src); - const sid = str(s.statementId); - if (sid && !p.statementIds.includes(sid)) p.statementIds.push(sid); - } - return Array.from(people.values()); -} +import { extractConnectedPeople } from "../src/lib/backgroundCheck.ts"; +import { clusterConnectedPeople } from "../src/lib/clusterPeople.ts"; function evalBundle(path) { const raw = JSON.parse(readFileSync(path, "utf8")); - const statements = Array.isArray(raw) ? raw : arr(raw.bods); - const people = extractPeople(statements); + const statements = Array.isArray(raw) ? raw : (raw.bods ?? raw.statements ?? []); + const people = extractConnectedPeople(statements); const { clusters, singletons } = clusterConnectedPeople(people); console.log(`\n########## ${path} ##########`); console.log(`people=${people.length} clusters=${clusters.length} singletons=${singletons.length}`); diff --git a/frontend/src/lib/backgroundCheck.test.ts b/frontend/src/lib/backgroundCheck.test.ts index b5f7ae89..65077141 100644 --- a/frontend/src/lib/backgroundCheck.test.ts +++ b/frontend/src/lib/backgroundCheck.test.ts @@ -296,11 +296,25 @@ describe("nzRoleHolder — the New Zealand register's anchor for a person", () = }; it("carries the company number and the name as the NZ filing gives it", () => { - // The same person from another source, spelt differently, merges first. + // The same register naming the person twice, spelt differently, merges + // first (same source, no birth year on either); the anchor keeps the + // spelling of the statement the role is filed against. + const other = { ...nzPerson, statementId: "nz-p2", recordDetails: { ...nzPerson.recordDetails, names: [{ fullName: "Mark Julian VERBIEST" }] } }; + const people = extractConnectedPeople([nzCompany, other, nzPerson, nzRel]); + expect(people).toHaveLength(1); + expect(people[0].name).toBe("Mark Julian VERBIEST"); + expect(people[0].nzRoleHolder).toEqual({ companyNumber: "1166320", name: "Mark Julian Verbiest" }); + }); + + it("keeps a same-name record from another source with no birth year on its own card", () => { const other = person("w-p", "Mark Julian VERBIEST", {}, "Wikidata"); - const [p] = extractConnectedPeople([nzCompany, other, nzPerson, nzRel]); - expect(p.name).toBe("Mark Julian VERBIEST"); - expect(p.nzRoleHolder).toEqual({ companyNumber: "1166320", name: "Mark Julian Verbiest" }); + const people = extractConnectedPeople([nzCompany, other, nzPerson, nzRel]); + expect(people).toHaveLength(2); + const nz = people.find((p) => p.statementIds.includes("nz-p")); + const wd = people.find((p) => p.statementIds.includes("w-p")); + expect(nz?.nzRoleHolder).toEqual({ companyNumber: "1166320", name: "Mark Julian Verbiest" }); + expect(wd?.nzRoleHolder).toBeUndefined(); + expect(nz?.key).not.toBe(wd?.key); }); it("falls back to the NZBN when the filing has no company number", () => { diff --git a/frontend/src/lib/backgroundCheck.ts b/frontend/src/lib/backgroundCheck.ts index a3c88839..8c85f4bb 100644 --- a/frontend/src/lib/backgroundCheck.ts +++ b/frontend/src/lib/backgroundCheck.ts @@ -8,12 +8,14 @@ * ownership, shareholding…), each with the source that asserted it and * the company it is held in. * - * People are merged across sources by normalised name + birth year — - * the same person typically appears in both the live Companies House - * bundle and the Open Ownership UK PSC bulk dataset (and, for Estonia, - * as officer + beneficial owner + shareholder). Merging is - * presentational only: every underlying statementId is kept so claims - * remain traceable to their statements. + * People are merged on a shared identifier first (the Estonian register's + * person key, a Wikidata item, a Companies House officer id…), then on + * normalised name + birth year — the same person typically appears in both + * the live Companies House bundle and the Open Ownership UK PSC bulk dataset. + * A name alone never merges records from different sources, and two + * different values of a one-person-per-value scheme are never merged (see + * `groupPersonRecords`). Merging is presentational only: every underlying + * statementId is kept so claims remain traceable to their statements. */ import { personIdentifierNotes } from "./annotations"; @@ -38,7 +40,8 @@ export interface ConnectedPersonRole { } export interface ConnectedPerson { - /** Merge key — normalised name + birth year. */ + /** Merge key — normalised name + birth year, made unique with the first + * statementId when two people share both. */ key: string; name: string; birthDate?: string; @@ -224,6 +227,190 @@ export function extractPersonSubgraph( return [...entities, ...persons, ...relationships]; } +/** + * Identifier schemes whose values name exactly one person each: two different + * values in one of these schemes are two different people, whatever their + * names. The Estonian register's person key is one per person by + * construction; a Wikidata item is one per human (Unilever's "Fernando + * Fernández" card merged a Mexican actor's item with another person's). + * + * Companies House officer ids are deliberately NOT here: Companies House can + * give one person several officer ids, and a veto on them would split people + * the name + birth-year rule has always merged correctly. + */ +export const ONE_PERSON_PER_VALUE_SCHEMES: ReadonlySet = new Set([ + "ee-ariregister-hash", + "wikidata", +]); + +interface PersonRecord { + stmt: Stmt; + id: string; + name: string; + nameKey: string; + birthDate?: string; + birthYear?: number; + /** opencheckSourceId, else the source description. */ + source?: string; + /** Normalised "scheme:value" keys. */ + identifiers: string[]; +} + +function personRecord(s: Stmt): PersonRecord | undefined { + const id = str(s.statementId); + if (!id) return undefined; + const rd = rec(s.recordDetails); + const personType = str(rd.personType); + // anonymousPerson / unknownPerson carry no screenable identity. + if (personType && personType !== "knownPerson") return undefined; + const name = personName(rd); + if (!name) return undefined; + const birthDate = str(rd.birthDate); + const identifiers: string[] = []; + // Phase 255: a Wikidata Q-id or record id now rides in an `identifying` + // annotation; read it back so people still group on the same key. + for (const idObj of [...arr(rd.identifiers).map(rec), ...personIdentifierNotes(s).map(rec)]) { + const scheme = str(idObj.scheme) ?? str(idObj.schemeName); + const value = str(idObj.id); + const idKey = (scheme && value ? `${scheme}:${value}` : value)?.toLowerCase(); + if (idKey && !identifiers.includes(idKey)) identifiers.push(idKey); + } + const source = rec(s.source); + return { + stmt: s, + id, + name, + nameKey: nameMatchKey(name), + birthDate, + birthYear: birthYearOf(birthDate), + source: str(source.opencheckSourceId) ?? str(source.description), + identifiers, + }; +} + +function schemeOf(idKey: string): string | undefined { + const at = idKey.indexOf(":"); + return at > 0 ? idKey.slice(0, at) : undefined; +} + +interface PersonGroup { + years: Set; + sources: Set; + /** scheme → values, for ONE_PERSON_PER_VALUE_SCHEMES only. */ + strict: Map>; +} + +function strictConflict(a: PersonGroup, b: PersonGroup): boolean { + for (const [scheme, values] of a.strict) { + const other = b.strict.get(scheme); + if (!other) continue; + for (const v of other) if (!values.has(v)) return true; + for (const v of values) if (!other.has(v)) return true; + } + return false; +} + +function intersects(a: Set, b: Set): boolean { + for (const v of a) if (b.has(v)) return true; + return false; +} + +/** + * Decide which person statements are one person. Returns, for each record, + * the index of its group's root. + * + * 1. Records sharing an identifier (any scheme) are one person. + * 2. Records with the same order-insensitive name are one person when + * - both carry a birth year and the years agree, or + * - neither group carries a birth year and both come from the same source + * (one register naming one person on several lists of one company); + * never on a name alone across sources — those stay separate candidates + * for the clustering step to surface for review. + * 3. Nothing is merged across two different values of a scheme that names one + * person per value (ONE_PERSON_PER_VALUE_SCHEMES). + * + * A birth year on one side and none on the other was never a merge and still + * is not: clustering offers it as a possible match. + */ +export function groupPersonRecords( + records: ReadonlyArray<{ nameKey: string; birthYear?: number; source?: string; identifiers: string[] }> +): number[] { + const parent = records.map((_, i) => i); + const groups: PersonGroup[] = records.map((r) => { + const strict = new Map>(); + for (const idKey of r.identifiers) { + const scheme = schemeOf(idKey); + if (!scheme || !ONE_PERSON_PER_VALUE_SCHEMES.has(scheme)) continue; + const set = strict.get(scheme) ?? new Set(); + set.add(idKey); + strict.set(scheme, set); + } + return { + years: new Set(r.birthYear !== undefined ? [r.birthYear] : []), + sources: new Set(r.source ? [r.source] : []), + strict, + }; + }); + const find = (i: number): number => { + while (parent[i] !== i) { + parent[i] = parent[parent[i]]; + i = parent[i]; + } + return i; + }; + const union = (i: number, j: number): boolean => { + const a = find(i); + const b = find(j); + if (a === b) return true; + if (strictConflict(groups[a], groups[b])) return false; + const [keep, drop] = a < b ? [a, b] : [b, a]; + parent[drop] = keep; + const g = groups[keep]; + const d = groups[drop]; + d.years.forEach((y) => g.years.add(y)); + d.sources.forEach((v) => g.sources.add(v)); + for (const [scheme, values] of d.strict) { + const set = g.strict.get(scheme) ?? new Set(); + values.forEach((v) => set.add(v)); + g.strict.set(scheme, set); + } + return true; + }; + + // 1. Shared identifiers. + const firstWithId = new Map(); + records.forEach((r, i) => { + for (const idKey of r.identifiers) { + const seen = firstWithId.get(idKey); + if (seen === undefined) firstWithId.set(idKey, i); + else union(seen, i); + } + }); + + // 2. Same name, with a birth year to agree on — or one source, no years. + const byName = new Map(); + records.forEach((r, i) => { + const list = byName.get(r.nameKey) ?? []; + list.push(i); + byName.set(r.nameKey, list); + }); + for (const members of byName.values()) { + for (let x = 1; x < members.length; x++) { + for (let y = 0; y < x; y++) { + const a = groups[find(members[x])]; + const b = groups[find(members[y])]; + if (find(members[x]) === find(members[y])) continue; + const agree = + (a.years.size > 0 && b.years.size > 0 && intersects(a.years, b.years)) || + (a.years.size === 0 && b.years.size === 0 && intersects(a.sources, b.sources)); + if (agree) union(members[x], members[y]); + } + } + } + + return records.map((_, i) => find(i)); +} + /** * A person is a *current* connection to the entity if they hold at least one * role with no end date. People whose every recorded role has ended are @@ -263,23 +450,37 @@ export function extractConnectedPeople(statements: Stmt[]): ConnectedPerson[] { const people = new Map(); const keyByStatementId = new Map(); - const addPerson = (s: Stmt): ConnectedPerson | undefined => { - const id = str(s.statementId); - if (!id) return undefined; - const rd = rec(s.recordDetails); - const personType = str(rd.personType); - // anonymousPerson / unknownPerson carry no screenable identity. - if (personType && personType !== "knownPerson") return undefined; - const name = personName(rd); - if (!name) return undefined; - const birthDate = str(rd.birthDate); - const birthYear = birthYearOf(birthDate); - const key = `${nameMatchKey(name)}|${birthYear ?? ""}`; - let person = people.get(key); + // ---- 1. One record per screenable person statement ---------------------- + const records: PersonRecord[] = []; + for (const s of statements) { + const recordType = str(s.recordType) ?? str(s.statementType); + if (recordType !== "person" && recordType !== "personStatement") continue; + const r = personRecord(s); + if (r) records.push(r); + } + + // ---- 2. Group the records: identifiers first, then name (+ birth year) -- + const groupOf = groupPersonRecords(records); + + // ---- 3. One ConnectedPerson per group, in order of first appearance ----- + const keyByGroup = new Map(); + for (let i = 0; i < records.length; i++) { + const r = records[i]; + const root = groupOf[i]; + let key = keyByGroup.get(root); + let person = key ? people.get(key) : undefined; if (!person) { + const first = records.find((x, j) => groupOf[j] === root && x.birthYear !== undefined); + const birthDate = first?.birthDate; + const birthYear = first?.birthYear; + key = `${r.nameKey}|${birthYear ?? ""}`; + // Two people can share a name and a (missing) birth year now that a + // name alone no longer merges them: keep their keys apart. + if (people.has(key)) key = `${key}|${r.id}`; + keyByGroup.set(root, key); person = { key, - name, + name: r.name, birthDate, birthYear, nationalities: [], @@ -290,27 +491,18 @@ export function extractConnectedPeople(statements: Stmt[]): ConnectedPerson[] { }; people.set(key, person); } - if (!person.birthDate && birthDate) { - person.birthDate = birthDate; - person.birthYear = birthYear; - } + const s = r.stmt; + const rd = rec(s.recordDetails); for (const n of arr(rd.nationalities).map(rec)) { const label = str(n.name) ?? str(n.code); if (label && !person.nationalities.includes(label)) { person.nationalities.push(label); } } - // Phase 255: a Wikidata Q-id or record id now rides in an `identifying` - // annotation; read it back so people still group on the same key. - for (const idObj of [...arr(rd.identifiers).map(rec), ...personIdentifierNotes(s).map(rec)]) { - const scheme = str(idObj.scheme) ?? str(idObj.schemeName); - const value = str(idObj.id); - const idKey = (scheme && value ? `${scheme}:${value}` : value)?.toLowerCase(); - if (idKey && !person.identifiers.includes(idKey)) { - person.identifiers.push(idKey); - } + for (const idKey of r.identifiers) { + if (!person.identifiers.includes(idKey)) person.identifiers.push(idKey); } - if (!person.statementIds.includes(id)) person.statementIds.push(id); + if (!person.statementIds.includes(r.id)) person.statementIds.push(r.id); // Merged rows can hold one person from two filings (a PSC notification and // a directorship). Keep the record that says more: an ACSP verification // names who verified and when; the direct route publishes neither. @@ -324,15 +516,7 @@ export function extractConnectedPeople(statements: Stmt[]): ConnectedPerson[] { } const source = str(rec(s.source).description); if (source && !person.sources.includes(source)) person.sources.push(source); - keyByStatementId.set(id, key); - return person; - }; - - for (const s of statements) { - const recordType = str(s.recordType) ?? str(s.statementType); - if (recordType === "person" || recordType === "personStatement") { - addPerson(s); - } + keyByStatementId.set(r.id, key as string); } for (const s of statements) { diff --git a/frontend/src/lib/clusterPeople.ts b/frontend/src/lib/clusterPeople.ts index 211ed689..fa9acf79 100644 --- a/frontend/src/lib/clusterPeople.ts +++ b/frontend/src/lib/clusterPeople.ts @@ -14,6 +14,8 @@ * never a silent "different people" veto; * - a shared hard identifier (Companies House officer id, Wikidata Q-id) * DOMINATES and can carry a pair to High even across a birth-year conflict; + * - two different values of a one-person-per-value scheme (the Estonian + * register's person key, a Wikidata item) are never a candidate pair; * - name similarity ALONE never reaches High — that needs an identifier, or a * near-exact name plus corroboration (matching birth year AND shared * nationality or company). @@ -21,7 +23,11 @@ * Pure and deterministic — unit-tested in clusterPeople.test.ts. */ -import { nameTokens, type ConnectedPerson } from "./backgroundCheck"; +import { + ONE_PERSON_PER_VALUE_SCHEMES, + nameTokens, + type ConnectedPerson, +} from "./backgroundCheck"; /** A ConnectedPerson optionally carrying normalised "scheme:value" identifiers. */ export type ClusterablePerson = ConnectedPerson & { identifiers?: string[] }; @@ -158,6 +164,36 @@ function sharedIdentifiers(a: ClusterablePerson, b: ClusterablePerson): string[] return overlap(a.identifiers ?? [], b.identifiers ?? []); } +/** + * The two records carry different values of a scheme that names one person per + * value (the Estonian register's person key, a Wikidata item): the source has + * already said they are different people, so they are never a candidate pair. + */ +export function distinctByIdentifier(a: ClusterablePerson, b: ClusterablePerson): boolean { + const bySchemes = (p: ClusterablePerson): Map> => { + const m = new Map>(); + for (const id of p.identifiers ?? []) { + const key = id.toLowerCase(); + const at = key.indexOf(":"); + if (at <= 0) continue; + const scheme = key.slice(0, at); + if (!ONE_PERSON_PER_VALUE_SCHEMES.has(scheme)) continue; + const set = m.get(scheme) ?? new Set(); + set.add(key); + m.set(scheme, set); + } + return m; + }; + const A = bySchemes(a); + const B = bySchemes(b); + for (const [scheme, values] of A) { + const other = B.get(scheme); + if (!other) continue; + if (![...values].some((v) => other.has(v))) return true; + } + return false; +} + function companiesOf(p: ClusterablePerson): string[] { return (p.roles ?? []).map((r) => r.subjectName).filter((s): s is string => !!s); } @@ -171,6 +207,7 @@ export function scorePair( b: ClusterablePerson ): PersonClusterPair | null { const ids = sharedIdentifiers(a, b); + if (ids.length === 0 && distinctByIdentifier(a, b)) return null; const name = nameSimilarity(a.name, b.name); const byr = birthYearRelation(a, b); const nats = overlap(a.nationalities, b.nationalities); diff --git a/frontend/src/lib/personIdentityMerge.test.ts b/frontend/src/lib/personIdentityMerge.test.ts new file mode 100644 index 00000000..a0d982be --- /dev/null +++ b/frontend/src/lib/personIdentityMerge.test.ts @@ -0,0 +1,175 @@ +import { describe, expect, it } from "vitest"; +import { + extractConnectedPeople, + groupPersonRecords, + type Stmt, +} from "./backgroundCheck"; +import { clusterConnectedPeople } from "./clusterPeople"; + +/** + * The person-merge rules from the "BackgroundCheck person-source data quality" + * ticket (20 Oct 2026): identifiers first; name + birth year only when there + * is a birth year to agree on; a name alone never merges across sources; two + * different values of a one-person-per-value scheme are never merged. + */ + +function person( + id: string, + fullName: string, + opts: { + year?: number; + source?: string; + sourceId?: string; + ids?: { scheme: string; id: string }[]; + notes?: { scheme: string; id: string }[]; + } = {} +): Stmt { + return { + statementId: id, + recordType: "person", + source: { description: opts.source ?? "UK Companies House", ...(opts.sourceId ? { opencheckSourceId: opts.sourceId } : {}) }, + recordDetails: { + personType: "knownPerson", + names: [{ fullName }], + ...(opts.year ? { birthDate: String(opts.year) } : {}), + ...(opts.ids ? { identifiers: opts.ids } : {}), + }, + ...(opts.notes + ? { + annotations: opts.notes.map((n) => ({ + statementPointerTarget: "/recordDetails", + motivation: "identifying", + description: `Identifier ${n.id} (scheme ${n.scheme}; register person key). BODS keeps a person's identifiers for identity documents, so this one is published here: it says which record the statement was matched on, not who the person is.`, + })), + } + : {}), + }; +} + +const EE = { source: "Estonian e-Business Register", sourceId: "ariregister" }; +const hash = (n: number) => ({ scheme: "EE-ARIREGISTER-HASH", id: `00000000-0000-4000-8000-00000000000${n}` }); + +describe("Wikidata bare-name records (Unilever's Fernando Fernández)", () => { + // Two Wikidata items found by name — a Mexican actor and another person — + // plus the real CFO from Companies House and OpenCorporates, with a year. + const bundle = [ + person("oc", "FERNANDO FERNANDEZ", { year: 1966, source: "OpenCorporates" }), + person("ch", "FERNANDEZ, Fernando", { year: 1966 }), + person("wd-1", "Fernando Fernández", { source: "Wikidata", sourceId: "wikidata", ids: [{ scheme: "WIKIDATA", id: "Q5444704" }] }), + person("wd-2", "Fernando Fernández", { source: "Wikidata", sourceId: "wikidata", ids: [{ scheme: "WIKIDATA", id: "Q204277" }] }), + ]; + const people = extractConnectedPeople(bundle); + + it("does not merge two Wikidata items into one card", () => { + expect(people).toHaveLength(3); + const wd = people.filter((p) => p.sources.includes("Wikidata")); + expect(wd).toHaveLength(2); + expect(wd.map((p) => p.identifiers)).toEqual([["wikidata:q5444704"], ["wikidata:q204277"]]); + expect(new Set(people.map((p) => p.key)).size).toBe(3); + }); + + it("still merges the register records that agree on the birth year", () => { + const cfo = people.find((p) => p.birthYear === 1966); + expect(cfo?.statementIds).toEqual(["oc", "ch"]); + }); + + it("never offers the two Wikidata items as a possible match for each other", () => { + const { clusters } = clusterConnectedPeople(people); + const pairs = clusters.flatMap((c) => c.pairs); + const wdKeys = people.filter((p) => p.sources.includes("Wikidata")).map((p) => p.key); + expect(pairs.some((p) => wdKeys.includes(p.aKey) && wdKeys.includes(p.bKey))).toBe(false); + // Each is still offered, at Medium, against the real CFO for review. + expect(pairs.length).toBe(2); + expect(clusters.every((c) => c.confidence === "medium")).toBe(true); + }); + + it("does not merge different Wikidata items even when the birth years agree", () => { + const out = extractConnectedPeople([ + person("a", "Jane Example", { year: 1970, source: "Wikidata", ids: [{ scheme: "WIKIDATA", id: "Q1" }] }), + person("b", "Jane Example", { year: 1970, source: "Wikidata", ids: [{ scheme: "WIKIDATA", id: "Q2" }] }), + ]); + expect(out).toHaveLength(2); + }); +}); + +describe("a name alone does not merge across sources", () => { + it("keeps same-name records with no birth year from two sources apart, for review", () => { + const people = extractConnectedPeople([ + person("oc", "Jane Example", { source: "OpenCorporates" }), + person("wd", "Jane Example", { source: "Wikidata" }), + ]); + expect(people).toHaveLength(2); + expect(people[0].key).not.toBe(people[1].key); + const { clusters } = clusterConnectedPeople(people); + expect(clusters).toHaveLength(1); + expect(clusters[0].confidence).toBe("medium"); + expect(clusters[0].pairs[0].evidence).toMatch(/no birth year on either record/); + }); + + it("still merges one source naming one person twice with no birth year", () => { + const people = extractConnectedPeople([ + person("a", "Jane Example", { source: "New Zealand Companies Register (NZBN)" }), + person("b", "EXAMPLE, Jane", { source: "New Zealand Companies Register (NZBN)" }), + ]); + expect(people).toHaveLength(1); + expect(people[0].statementIds).toEqual(["a", "b"]); + }); + + it("does not merge a birth year on one side with none on the other (unchanged)", () => { + const people = extractConnectedPeople([ + person("a", "Jane Example", { year: 1980 }), + person("b", "Jane Example", { source: "Wikidata" }), + ]); + expect(people).toHaveLength(2); + }); +}); + +describe("the Estonian register's person key", () => { + it("keeps two Estonians with the same name and different keys as two cards", () => { + const people = extractConnectedPeople([ + person("ee-1", "Mari Tamm", { ...EE, notes: [hash(1)] }), + person("ee-2", "Mari Tamm", { ...EE, notes: [hash(2)] }), + ]); + expect(people).toHaveLength(2); + expect(clusterConnectedPeople(people).clusters).toHaveLength(0); + }); + + it("merges statements carrying the same key whatever the spelling", () => { + const people = extractConnectedPeople([ + person("ee-1", "Mari Tamm", { ...EE, notes: [hash(1)] }), + person("ee-2", "MARI-LIIS TAMM", { ...EE, notes: [hash(1)] }), + ]); + expect(people).toHaveLength(1); + expect(people[0].identifiers).toEqual(["ee-ariregister-hash:00000000-0000-4000-8000-000000000001"]); + }); +}); + +describe("identifiers first", () => { + it("merges records sharing an identifier even when the names differ", () => { + const officer = { scheme: "GB-COH-OFFICER", id: "abc123" }; + const people = extractConnectedPeople([ + person("a", "SMITH, John Edward", { year: 1960, ids: [officer] }), + person("b", "John Smith", { ids: [officer] }), + ]); + expect(people).toHaveLength(1); + expect(people[0].birthYear).toBe(1960); + }); + + it("does not let a shared non-strict identifier bridge two different person keys", () => { + const groups = groupPersonRecords([ + { nameKey: "a", identifiers: ["ee-ariregister-hash:1", "gb-coh-officer:x"] }, + { nameKey: "b", identifiers: ["ee-ariregister-hash:2", "gb-coh-officer:x"] }, + ]); + expect(groups[0]).not.toBe(groups[1]); + }); + + it("does not let a name merge join a group to a different Wikidata item", () => { + const groups = groupPersonRecords([ + { nameKey: "jane example", birthYear: 1970, identifiers: ["wikidata:q1"] }, + { nameKey: "jane example", birthYear: 1970, identifiers: [] }, + { nameKey: "jane example", birthYear: 1970, identifiers: ["wikidata:q2"] }, + ]); + expect(groups[0]).toBe(groups[1]); + expect(groups[2]).not.toBe(groups[0]); + }); +});