Repository navigation
[grammar] Trailing NBSP / combining marks excluded from mention spans #181
Description
Activity
Adjacent to PR #185 — not blocked, no dependency
Noting a nearby change only, because both surface through
result.span:- PR feat(gtin): GTIN capability (GS1 GenSpecs 26.0 + Prefix + Verified snapshot) #185 (GTIN + engine presentation-after-identity) changed the source of the single-value
spanin one narrow case: it is now taken from the first surviving candidate on the canonical-identity basis. For identity/injective formats this is bit-for-bit the old behavior; it only diverges for notation-keyed formats where one canonical renders differently per mention (e.g.rfc3966/native). - This issue lives entirely on the recognition side — boundary/guard treatment of trailing NBSP (U+00A0) / combining marks (U+0301) deciding the mention's
[start, end). That machinery (paxman/core/grammar/) was not touched by feat(gtin): GTIN capability (GS1 GenSpecs 26.0 + Prefix + Verified snapshot) #185, and the(0, 12)repro above behaves identically on the PR branch.
Relationship: adjacent, not blocked. Neither work item gates the other — a decision here ((a) document current behavior vs (b) void/join the mention) flows through recognition → candidate spans → surfaced span with no interaction with the presentation-vs-identity ordering, and #185's outcome doesn't constrain the span-boundary contract this issue must pin.
- PR feat(gtin): GTIN capability (GS1 GenSpecs 26.0 + Prefix + Verified snapshot) #185 (GTIN + engine presentation-after-identity) changed the source of the single-value
Triage 2026-09-30: keeping open, labeled v1.0.0 (tentative) + pending decision + followup (observed during PR #177). Plain English: when an ID is followed by an invisible space or accent mark, the ID is accepted but the invisible character is silently left out of the reported span; this issue asks for a recorded decision — keep and document that, or treat it as no-match/extend the match — with tests at the shared layer. Decision needed before any code change. No consolidation needed.
Observed during ISIN work (#177); pre-existing shared-layer behavior, not ISIN-caused. Same trait reproduces on sibling capabilities sharing the boundary machinery.
Repro (on dev):
Trailing NBSP (U+00A0) and combining acute (U+0301) are silently left out of the span while the mention still succeeds.
Ask: decide the intended contract and pin it: (a) current behavior is correct — boundary guards treat these as non-word trailing context, document it; or (b) trailing NBSP/combining marks should void or join the mention. Likely touches shared boundary/guard code in
paxman/core/grammar/, so the decision and its tests belong at the shared layer with per-capability parity runs, not in any single capability.Acceptance sketch: explicit unit tests for NBSP/combining-mark adjacency (at least word_only-guarded capabilities); no silent behavior change for existing vectors.