Retained sampled suffix can carry logprobs under a different causal prefix

[bradhilton]

Native multi-history tokenization can mark a retained response suffix
SAMPLED|EXACT and select it for training after earlier sampled tokens have been
removed from its conditioning context. Its stored logprob still belongs to the
original longer prefix. This is independent of the terminal renderer-tail work
in #882 and does not require the experimental raw-capture adapter.

Minimal synthetic capture:

Exchange	Captured prompt	Captured output
First	[1]	[2, 3]
Second	[1, 3, 4]	[5]

The second request retains the first assistant's message text, but its prompt
tokens omit token 2. With multi_history=True and
reconcile_text_equivalent_tokenizations=False, ART emits a second history
[1, 3, 4, 5], marks token 3 as SAMPLED|EXACT|ASSISTANT|OUTPUT (23), and gives
it the first response's logprob -0.3. first_occurrence_masks(where=SAMPLED)
selects it: prefix [1, 3] differs from the actual sampled prefix [1, 2, 3].
The probability was observed for 3 | [1, 2], not 3 | [1].

A CPU regression using the repository's existing synthetic exchange fixture:

import art
from art.trajectories import TokenFlag
from tests.unit.trajectories.test_tokenize import _chat_exchange

def test_retained_suffix_requires_its_original_causal_prefix():
    trajectory = art.Trajectory(
        exchanges=art.trajectories.TrajectoryExchanges(
            chat_completions=[
                _chat_exchange([1], [2, 3]),
                _chat_exchange([1, 3, 4], [5], offset=1),
            ]
        )
    )
    tokenized = trajectory.tokenize(
        multi_history=True,
        reconcile_text_equivalent_tokenizations=False,
    )
    masks = tokenized.tensorize().first_occurrence_masks(where=TokenFlag.SAMPLED)
    captured_prefixes = {(1, 2), (1, 2, 3), (1, 3, 4, 5)}
    for history, mask in zip(tokenized.histories, masks, strict=True):
        for index, selected in enumerate(mask.tolist()):
            if selected:
                assert tuple(history.tokens[:index + 1]) in captured_prefixes

This reproduces on ART 1ce5d7213e2d9cea2dd8a04a608966c24f4629a0 and current
077abf6fa7a3a51f9ff255c37f8886c6c8256082 (both _tokenize.py SHA256
cf8745b44edf3fb22a377246ccb81fc476153f4d96d5680e9c953180c4cff134), and the
reviewed terminal-boundary overlay 7bcd560cc117e593bbc8fad17a856ffc8dcbf667
(9922a80bafeccc89d1439b4c04bc08f63daccf047b37084e01d819e6d2ec0dd2).
The existing first-occurrence trie behaves as designed: it keys model-visible
prefixes, so it cannot repair an earlier attribution error.

The source is _retained_output_suffix, which searches every output[start:]
against the later prompt. _tokenize_exact_projected_chat_history then attaches
the suffix's original logprobs and sampled/exact flags at len(prompt).
Its complete-output requirement only applies to earlier finish_reason=length
sources; a non-length source can therefore lose its original causal conditioning.

Independent offline analysis also found this shape in two real captured
rollouts: 321 and 896 positions respectively remained first-occurrence-selected
after two leading sampled tokens were omitted. Raw content is private; the
synthetic regression above contains no research payload and reproduces the
same native behavior. This establishes attribution risk, not a measured GPU
training-quality effect.

Expected behavior: every sampled position selected for importance weighting
must retain the complete conditioning under which its stored logprob was
captured. A changed-context suffix can remain ordinary conditioning, but must
not reuse the old probability as an exact sampled probability under the new
prefix. Preserve the full original sampled source in another history or refuse
when exact attribution cannot be established; avoid silently dropping observed
samples. The experiment-local adapter is being tightened separately while the
appropriate shared history/attribution behavior is reviewed.

[bradhilton]

McCarthy: CLEAR for the identity-only correction at local 9fe17e39db8ad237ae304920b158b69f2f66ebf9, against reviewed da4f165a0a98c5d9539c74d854cce1d15e6810f7. No blocking finding in this scope.

Only _sampled_evidence_fingerprint changes at runtime. Captured prompts now distinguish otherwise identical source evidence for Chat, Messages, Responses generations and Completions. Chat/Completions use the existing token extractor's choice-first, response-fallback precedence; null/missing choice prompts fall back, while a supplied choice prompt shadows response metadata. Selected-source identity stays stable through deep/JSON copies, moving the same effective prompt to the fallback, and unrelated choices/generations. The change reaches both trace keys and the source signatures used to deduplicate projected sources; it does not add a causal validator or alter rendering policy.

Independent verification in a clean exact-head worktree:

• 484 trajectory tests passed in one run. The final 16 new controls on unchanged da4f165 give 10 failures / 6 passes, reproducing prompt aliasing and the dropped sampled-prefix witness.
• 21 additional controls passed, covering public multi-history same-source selection once across identity/deep/JSON copies for all four protocols, distinct-prompt retention with duplicate copies for Chat/Responses/Completions, fallback relocation and generation isolation.
• My direct captured-prefix/logprob oracle rechecked the two real value1/residual1 captures: 7,445 selections / 3 histories and 11,685 / 2, with no missing or wrong-prefix selections. Inputs and immutable renderer metadata were hash-checked; no tokenizer download was allowed.
• The seven preserved policy controls pass, including the three assertions demonstrating that explicit reconciliation/manual source substitution/rendered repair remain unsafe. Those are confirmed limitations, not safety passes.
• AST comparison confirms exactly one changed runtime function; _tokenize.py SHA256 is 1e48cadbfca55d3440fcbd1484f63a31ffbf71ca3520df189b625c5f168cdc92. Ruff check/format and whitespace checks pass.

The added prompt serialization is proportional to captured-prompt length, as disclosed. This clearance does not close #883, establish arbitrary manual-history correctness, or authorize adoption by the separately hash-pinned research drivers. Local candidate remains held for root's PR/integration decision; no shared edits, allocation, merge or deployment.

Evidence: /home/brad/.local/share/mccarthy/art883-identity-review-20260910/REVIEW.md and associated logs/oracle.