Repository navigation
docs: add segmentation recipe for multi-entity input #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
7c33fa3
docs: add segmentation recipe for multi-entity input
azaharizaman d7b1732
docs: link the segmentation recipe from README and ARCHITECTURE
azaharizaman 28fca13
fix(docs): address oracle and thermo review findings on segmentation …
azaharizaman 343f1c0
fix: address review comments on segmentation recipe and README
azaharizaman File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,161 @@ | ||
| # Segmentation Recipe — Multi-Entity Input | ||
|
|
||
| > One `paxman.canonicalize()` call resolves one presumed entity. Multi-entity | ||
| > input is caller-owned segmentation: split, then canonicalize per mention. | ||
|
|
||
| This recipe is the sanctioned pattern for "find all X in this text" demand | ||
| without bending Paxman's scope. See [ADR-0004](../adr/0004-single-value-invariant.md) | ||
| and the architecture review [§8 M1](../reports/2026-08-17-architecture-review.md) for the charter. | ||
|
|
||
| --- | ||
|
|
||
| ## 1. The invariant | ||
|
|
||
| One entity per `canonicalize()` call is the product contract ([ADR-0004](../adr/0004-single-value-invariant.md)). | ||
| Paxman operates at the *mention* level: the caller ensures the slice passed to | ||
| each call contains one presumed entity (or none). | ||
|
|
||
| * **`AMBIGUOUS`** means a genuine single-mention spec conflict — one recognized | ||
| span, two authorities disagreeing on its canonical value (e.g. `01/02/2026` | ||
| as `2026-01-02` vs `2026-02-01`). | ||
| * **`MultipleMentionsError`** means your input contained two or more separate | ||
| mentions that resolved to different values. It is a segmentation-usage signal, | ||
| not a domain result — it fails fast instead of masquerading as ambiguity. | ||
|
|
||
| Segmentation is **caller-owned by charter**, not a missing feature. This is | ||
| mandate [M1 in the architecture review §8](../reports/2026-08-17-architecture-review.md) | ||
| and the core decision of ADR-0004: multi-entity extraction belongs outside the | ||
| library. For the four resolution statuses see [README — Resolution Status](../../README.md#resolution-status). | ||
|
|
||
| --- | ||
|
|
||
| ## 2. The recipe | ||
|
|
||
| Segment → canonicalize per mention → reassemble. | ||
|
|
||
| Your segmenter finds mention *candidates*; Paxman canonicalizes each candidate | ||
| and tells you whether it is `SUCCESS`, `INVALID`, `MISSING`, or `AMBIGUOUS`. | ||
| Spans and `MultipleMentionsError` make the loop robust — the error fires when | ||
| your segmenter let two mentions through. | ||
|
|
||
| ```python | ||
| import re | ||
|
|
||
| import paxman | ||
| from paxman.capabilities import Email | ||
| from paxman.core.discovery import register_capability | ||
| from paxman.core.domain import Resolution | ||
| from paxman.core.errors import MultipleMentionsError | ||
|
|
||
| register_capability(Email()) | ||
| contract = Email.create_contract() | ||
|
|
||
| # Caller-owned segmentation: a coarse pattern finds mention candidates… | ||
| EMAIL_LIKE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}") | ||
|
|
||
| def canonicalize_emails(text: str) -> list[tuple[str, Resolution, str | None, tuple[int, int] | None]]: | ||
| """Return (raw, status, canonical, absolute_span) for every email candidate.""" | ||
| out: list[tuple[str, Resolution, str | None, tuple[int, int] | None]] = [] | ||
| for m in EMAIL_LIKE.finditer(text): | ||
| try: | ||
| result = paxman.canonicalize(m.group(0), contract) | ||
| except MultipleMentionsError: | ||
| # Your segmenter let two mentions through — tighten it. | ||
| raise | ||
| abs_span = ( | ||
| (m.start() + result.span[0], m.start() + result.span[1]) | ||
| if result.span is not None | ||
| else None | ||
| ) | ||
| out.append((m.group(0), result.status, result.canonicalized_value, abs_span)) | ||
| return out | ||
| ``` | ||
|
|
||
| The regex above is **deliberately coarse** — it is a caller-owned candidate | ||
| finder, not an RFC 5322 validator. Treat it as a cheap pre-filter; Paxman's | ||
| grammars and rules remain the authority on whether a candidate is `SUCCESS`, | ||
| `INVALID`, or `AMBIGUOUS`. This is pitfall (a) below: a coarse, | ||
| capability-shaped pattern beats naive splitting, but must not pretend to | ||
| replace capability validation. | ||
|
|
||
| --- | ||
|
|
||
| ## 3. Span mechanics | ||
|
|
||
| Every `ExecutionResult.span` and `Candidate.span` is a half-open `[start, end)` | ||
| offset into **the slice passed to THAT `canonicalize()` call**, not into the | ||
| original document. With per-mention calls, `result.span` is relative to the | ||
| SLICE (the single-mention string you handed to `canonicalize()`), e.g. `0` | ||
| means "start of this candidate string." | ||
|
|
||
| To reassemble document positions, add the segmenter's offset: | ||
|
|
||
| * `m.start()` / `m.end()` from your segmenter — document-absolute. | ||
| * `result.span` / `candidate.span` — mention-local, slice-relative. | ||
|
|
||
| So the document position of a resolved mention is `m.start() + result.span[0]` | ||
| when you passed `m.group(0)` as the slice. For `SUCCESS` there is a single | ||
| resolved entity and `result.span` is set; for `MISSING`/`INVALID`/`AMBIGUOUS` | ||
| there is no single resolved entity and `result.span` is `None` — locate | ||
| mentions via per-`Candidate.span` on `AMBIGUOUS` instead. | ||
|
|
||
| --- | ||
|
|
||
| ## 4. Signals, not failures | ||
|
|
||
| Per mention, the four statuses keep their exact meanings (see | ||
| [README — Resolution Status](../../README.md#resolution-status)): | ||
|
|
||
| * `SUCCESS` — one canonical value resolved. | ||
| * `INVALID` — recognized, but no authority validates it. | ||
| * `MISSING` — nothing recognized. | ||
| * `AMBIGUOUS` — one mention, multiple authorities disagree. | ||
|
|
||
| `MultipleMentionsError` is **not** a Paxman status (it is a `PaxmanError` exception, not a `Resolution` status). It is a segmenter bug | ||
| detector: two mentions landed in one slice and they disagree on value. It never | ||
| represents Paxman state; it tells you to tighten the segmenter so each slice | ||
| holds at most one mention. Handle it as an invariant violation in the caller, | ||
| not as a domain outcome to branch on. | ||
|
|
||
| --- | ||
|
|
||
| ## 5. Pitfalls | ||
|
|
||
| (a) **Naive splitting vs capability-shaped patterns.** Splitting on commas, | ||
| newlines, or whitespace is brittle — addresses, display names, and surrounding | ||
| punctuation break naive delimiters. Prefer a capability-shaped coarse pattern | ||
| (like `EMAIL_LIKE` above) that approximates the capability's own grammars. Keep | ||
| it coarse and let Paxman decide validity; a too-strict pre-filter silently | ||
| drops mentions that would have been `INVALID`/`AMBIGUOUS` honestly. | ||
|
|
||
| (b) **Segmenter vs grammar boundary disagreement.** Your segmenter and Paxman's | ||
| grammars may disagree on where a mention starts or ends. Always feed the | ||
| segmenter's slice to `canonicalize()` and **trust the returned status** — an | ||
| honest `INVALID` or `AMBIGUOUS` beats pre-filtering or trimming the slice to | ||
| force a `SUCCESS`. Only `MultipleMentionsError` indicates a segmentation error; `INVALID` and `AMBIGUOUS` are valid per-mention outcomes that tell you the candidate was recognized but not validated or was ambiguous, so handle them as domain results rather than discarding them as segmentation failures. | ||
|
|
||
| (c) **Don't widen a segment to "give context."** Adding surrounding words to | ||
| help Paxman understand a mention backfires: extra text that contains another | ||
| mention with a *different* canonical value triggers `MultipleMentionsError` | ||
| (identical values still coalesce to `SUCCESS` per ADR-0004). Keep slices tight | ||
| to one presumed entity; Paxman is stateless per call and needs no surrounding | ||
| document context. | ||
|
|
||
| --- | ||
|
|
||
| ## 6. Scope statement | ||
|
|
||
| Extraction stays **caller-owned forever** ([M1](../reports/2026-08-17-architecture-review.md), | ||
| [ADR-0004](../adr/0004-single-value-invariant.md)). This recipe is the sanctioned | ||
| pattern for multi-entity input, and requests for built-in document extraction | ||
| are out of scope by charter, not by limitation. Paxman will not ship a | ||
| "find all emails/phones/dates in this document" API — the split-then-canonicalize | ||
| loop above is the intended interface. | ||
|
|
||
| --- | ||
|
|
||
| ## References | ||
|
|
||
| * [ADR-0004: Single-Value Invariant](../adr/0004-single-value-invariant.md) | ||
| * [Architecture Review §8 M1](../reports/2026-08-17-architecture-review.md) — "One entity per call, forever" | ||
| * [README — Resolution Status](../../README.md#resolution-status) — `MISSING` / `INVALID` / `SUCCESS` / `AMBIGUOUS` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.