Skip to content

fix: keep dated cessations in the ended extraction slot - #1205

Closed
virtue039 wants to merge 7 commits into
deeplethe:devfrom
virtue039:fix/extraction-values-and-endings
Closed

virtue039 wants to merge 7 commits into
deeplethe:devfrom
virtue039:fix/extraction-values-and-endings

Conversation

@virtue039

@virtue039 virtue039 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

A dated cessation such as “from May 7, no longer managed” could be extracted as a beginning. Keep the relationship phrase, including “no longer”, and put the stated date in the existing ended slot. Contrast it with the same date wording beginning a positive relationship. The production diff is five added prompt lines, with no parser, data-model or API changes. A machine-readable comparison record accompanies the measurements.

The identifier clarification has been removed. In response to the review, each candidate was measured independently against the same frozen dev base 259931a, in one session with Qwen qwen3.7-flash-2026-07-15, thinking disabled, and the same pinned BGE-M3 CPU float32 normalized-CLS embeddings. Every group used fresh knowledge bases. Repository scripts, corpora and truth were unchanged; only the relevant prompt fragment differed.

Repository temporal bench — final ending-only candidate 69c1ee2:

Attempt Original prompt Ending-only World / record
1 (r3) 31/37 32/37 Original 17/23 + 14/14; candidate 18/23 + 14/14
2 (r4) 31/37 Failed before scoring Candidate extraction omitted the initial Zhang San lead date, so the scripted correction could not find its source fact
3 (r5, additional unchanged-code pair after the failure) 32/37 32/37 Both 18/23 + 14/14

The failed attempt is retained and has no invented score. These results do not establish a general accuracy improvement or blanket non-regression. Two supplementary rounds of dated Chinese/English cessation and a positive-start control gave original 1/3 each, candidate 3/3 each; they support the specific date-slot fix, not the whole-document claim. The shorter, explicit contrast replaced an initial verbose ending-only candidate that scored 25/37 versus 32/37; an intermediate abstract rule did not fix the dated controls. Those discarded candidates and results are retained too.

Repository recall bench — identifier-only candidate f11f06d, not included in this PR:

Round Original prompt Identifier-only
1 29/52 34/52
2 33/52 28/52
Mean 31/52 31/52

All four recall runs extracted all 55 chunks and finished settling. The identifier rule showed no stable recall benefit, and the historical identifier failure had already failed to reproduce on a fresh baseline, so that rule is not being submitted separately.

Execution notes: the first temporal pair stalled while waiting for remaining work; both failures are retained. An early suspicion about future governance jobs was not established, so no bench waiting or scoring logic was changed. Three distinct binaries were checked for the correct embedded prompt fragments after rejecting shared-cache reuse. The session used the full schema.org pack and prefilled ontology/chunk embeddings for recall; known stayed enabled. One upstream HTTP 429 was handled by the existing retry path; all recall runs completed. Recorded usage includes preparation, discarded candidates, controls and bench settling, not just scored extraction.

Validation on macOS arm64 at final head 69c1ee2: locked workspace tests with required PostgreSQL/PDF tools 1,514 passed, 0 failed, 6 existing ignored; format, locked all-target Clippy with -D warnings, server build, frozen-lockfile web install and web production build passed. Bench binaries were captured separately before later builds. Temporary credentials were deleted and all benchmark services stopped.

Existing facts are not rewritten; affected documents need ordinary re-extraction after merge. This PR retains a narrow ending fix for review and explicitly leaves the whole-document measurement limits visible.

…on slots

Signed-off-by: Virtue039 <virtue039@qq.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the traces, and for testing with controls on both sides.

Two things before we can take this:

  • Measurement. This is the one prompt every document goes through, and eight synthetic cases are the only measure so far. Please run the repository's benches before and after in one session: scripts/bench/temporal.mjs for the ending rule and recall.mjs for the identifier rule. A cheap model is fine as long as both sides use it.
  • Two changes. The identifier rule and the ending rule are independent; please split them so each is judged by its own numbers. You note that the identifier failure did not reproduce on a fresh baseline. If it still does not, that half may not be needed.

Thanks again for the careful write-up.

… base

Signed-off-by: Virtue039 <virtue039@qq.com>
…eir own

Signed-off-by: Virtue039 <virtue039@qq.com>
…er extraction rules

Signed-off-by: Virtue039 <virtue039@qq.com>
@virtue039 virtue039 changed the title fix: keep identifier values and dated endings in their extraction slots fix: keep dated cessations in the ended extraction slot Oct 13, 2026
…s for review

Signed-off-by: Virtue039 <virtue039@qq.com>
…tempt

Signed-off-by: Virtue039 <virtue039@qq.com>
@WaylandYang

Copy link
Copy Markdown
Contributor

Thank you for splitting this and for measuring it the way we asked, including the failed attempt.

We will leave the prompt as it is. On the temporal bench the ending-only candidate scored 32/37 against 31/37 in one pair and 32/37 against 32/37 in the other, and its second attempt failed before scoring because a lead date was not extracted. That is within what a rerun moves, and the prompt is read by every document, so we would rather not add lines that the measurement does not separate from noise.

If you find a corpus where dated cessations go wrong often enough to show a clear difference, a pull request with that bench would be very welcome. Thanks again for the careful work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants