Repository navigation
Conversation
…on slots Signed-off-by: Virtue039 <virtue039@qq.com>
WaylandYang
left a comment
There was a problem hiding this comment.
Thank you for the traces, and for testing with controls on both sides.
Two things before we can take this:
- Measurement. This is the one prompt every document goes through, and eight synthetic cases are the only measure so far. Please run the repository's benches before and after in one session:
scripts/bench/temporal.mjsfor the ending rule andrecall.mjsfor the identifier rule. A cheap model is fine as long as both sides use it. - Two changes. The identifier rule and the ending rule are independent; please split them so each is judged by its own numbers. You note that the identifier failure did not reproduce on a fresh baseline. If it still does not, that half may not be needed.
Thanks again for the careful write-up.
… base Signed-off-by: Virtue039 <virtue039@qq.com>
…eir own Signed-off-by: Virtue039 <virtue039@qq.com>
Signed-off-by: Virtue039 <virtue039@qq.com>
…er extraction rules Signed-off-by: Virtue039 <virtue039@qq.com>
…s for review Signed-off-by: Virtue039 <virtue039@qq.com>
…tempt Signed-off-by: Virtue039 <virtue039@qq.com>
|
Thank you for splitting this and for measuring it the way we asked, including the failed attempt. We will leave the prompt as it is. On the temporal bench the ending-only candidate scored 32/37 against 31/37 in one pair and 32/37 against 32/37 in the other, and its second attempt failed before scoring because a lead date was not extracted. That is within what a rerun moves, and the prompt is read by every document, so we would rather not add lines that the measurement does not separate from noise. If you find a corpus where dated cessations go wrong often enough to show a clear difference, a pull request with that bench would be very welcome. Thanks again for the careful work. |
A dated cessation such as “from May 7, no longer managed” could be extracted as a beginning. Keep the relationship phrase, including “no longer”, and put the stated date in the existing
endedslot. Contrast it with the same date wording beginning a positive relationship. The production diff is five added prompt lines, with no parser, data-model or API changes. A machine-readable comparison record accompanies the measurements.The identifier clarification has been removed. In response to the review, each candidate was measured independently against the same frozen dev base
259931a, in one session with Qwenqwen3.7-flash-2026-07-15, thinking disabled, and the same pinned BGE-M3 CPU float32 normalized-CLS embeddings. Every group used fresh knowledge bases. Repository scripts, corpora and truth were unchanged; only the relevant prompt fragment differed.Repository temporal bench — final ending-only candidate
69c1ee2:r3)r4)r5, additional unchanged-code pair after the failure)The failed attempt is retained and has no invented score. These results do not establish a general accuracy improvement or blanket non-regression. Two supplementary rounds of dated Chinese/English cessation and a positive-start control gave original 1/3 each, candidate 3/3 each; they support the specific date-slot fix, not the whole-document claim. The shorter, explicit contrast replaced an initial verbose ending-only candidate that scored 25/37 versus 32/37; an intermediate abstract rule did not fix the dated controls. Those discarded candidates and results are retained too.
Repository recall bench — identifier-only candidate
f11f06d, not included in this PR:All four recall runs extracted all 55 chunks and finished settling. The identifier rule showed no stable recall benefit, and the historical identifier failure had already failed to reproduce on a fresh baseline, so that rule is not being submitted separately.
Execution notes: the first temporal pair stalled while waiting for remaining work; both failures are retained. An early suspicion about future governance jobs was not established, so no bench waiting or scoring logic was changed. Three distinct binaries were checked for the correct embedded prompt fragments after rejecting shared-cache reuse. The session used the full schema.org pack and prefilled ontology/chunk embeddings for recall;
knownstayed enabled. One upstream HTTP 429 was handled by the existing retry path; all recall runs completed. Recorded usage includes preparation, discarded candidates, controls and bench settling, not just scored extraction.Validation on macOS arm64 at final head
69c1ee2: locked workspace tests with required PostgreSQL/PDF tools 1,514 passed, 0 failed, 6 existing ignored; format, locked all-target Clippy with-D warnings, server build, frozen-lockfile web install and web production build passed. Bench binaries were captured separately before later builds. Temporary credentials were deleted and all benchmark services stopped.Existing facts are not rewritten; affected documents need ordinary re-extraction after merge. This PR retains a narrow ending fix for review and explicitly leaves the whole-document measurement limits visible.