Skip to content

Measure source reads and graph outcomes along namesake adjudication paths - #1377

Draft
virtue039 wants to merge 1 commit into
deeplethe:devfrom
virtue039:codex/adjudication-evidence-measurement
Draft

virtue039 wants to merge 1 commit into
deeplethe:devfrom
virtue039:codex/adjudication-evidence-measurement

Conversation

@virtue039

@virtue039 virtue039 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

An exact-name same can reach an automatic merge without reading either source. Add a test-only decision-path measurement for #1193 that distinguishes no investigation, investigation without source reads, source reads followed by a verdict, and a low-confidence split left for a person.

The 48 cases exercise ordinary adjudication and governance, both creation orders, a source-defined true duplicate and a namesake, batch same at 0.8 and 0.9, a seeded ordinary cache, both-side fact/quote tools and a final different(0.5). Governance does not read the ordinary cache: its cache-seeded case still asks the model. No ontology uniqueness declaration is needed for the true-merge control.

The tests assert transport and state consistency: A/B ownership in actual HTTP inputs/tool results, verbatim source delivery, graph/review agreement and no repeated work on a terminal pair. They separately print source gold, actual identity, gold errors, persisted investigation trace/calls and human count. Scripted conclusions deliberately disagree with some gold labels; green tests do not claim these errors are fixed or estimate model accuracy. The original eight inputs/gold, all 34 existing checks and the empty known-failure list are retained. There is no new gold-error ceiling.

Current-path observations:

  • Ordinary exact-name batch/cache same(0.8) can apply without investigation; governance's unsupported threshold is 0.85, so its 0.8 investigates and its 0.9 can apply directly.
  • Investigation can decide without reading sources. Even when both source passages arrive correctly, an incorrect scripted same(0.95) is applied.
  • A low-confidence different(0.5) leaves a true pair unmerged and pending human review; it is not a formal decision that they are distinct.

This is the measurement described in the independent next step. It adds no production gate, prompt, schema, provider call or lifecycle state. The production evidence standard remains for discussion in #1193; #1251 is already on dev.

Validation on macOS arm64, from dev 02611ae, with DB/PDF require guards enabled: workspace 1,659 passed / 0 failed / 6 existing ignored; all 34 original checks and the 48 new path measurements complete; workspace/all-target clippy (-D warnings), formatting and a frozen frontend install/build pass. No paid provider requests were made. A temporary negative control swapped the real governance lookup's A/B sides; the new measurement failed on evidence ownership, and that mutation was restored. The control log is retained locally.

Refs #1193, #1136, #1202, #1220.

Scripted observations

These are deliberate path probes, not observed provider error rates. Each row covers both creation orders and a true/false gold pair (four fresh fixture KBs). Unmerged true in the last path is pending human review, not a formal different.

Path Governance Model calls per KB Investigation calls Read sources / 4 Wrong merges Unmerged true Pending human total
Batch false 1 0 0 2 0 0
BatchHigh false 1 0 0 2 0 0
Cached false 0 0 0 2 0 0
Facts false 3 2 0 2 0 0
Quotes false 3 2 4 2 0 0
QuotesSplit false 3 2 4 0 2 4
Batch true 2 1 0 2 0 0
BatchHigh true 1 0 0 2 0 0
Cached true 2 1 0 2 0 0
Facts true 3 2 0 2 0 0
Quotes true 3 2 4 2 0 0
QuotesSplit true 3 2 4 0 2 4

The 100 local scripted requests use no provider. The fixture exposes 20 wrong merges and four unmerged true pairs; it does not register them as passing identity-gold regressions or add them to a known-failure list. A policy fix must remove its errors while protecting true merges; this test-only PR supplies the observation and transport checks.

CI 39787269383 passes all six checks at signed commit 13e68db941752d57c791fa1625ae577370b7e483. Kept as a draft measurement for review of the next-step scope; the production evidence standard is still open.

…nges the graph

Signed-off-by: Virtue039 <virtue039@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant