From 9c52d7a7543f32556b56e42cd6287caf6bdbe2e4 Mon Sep 17 00:00:00 2001 From: Azahari Zaman Date: Fri, 21 Aug 2026 16:57:11 +0800 Subject: [PATCH 1/2] docs(user): add user-facing documentation (index, getting-started, concepts, capabilities, api-reference, extending, migration) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the first two rounds of user docs under docs/user/ for Python novices, pros, and notebook researchers: - index + getting-started (pip/uv/notebook, register -> contract -> canonicalize) - concepts hub + 7 deep dives (capabilities, contracts, pipeline with staged recognition explanation, execution-result, provenance, candidates & ambiguity, errors) with Mermaid diagrams per page - capabilities hub + 10 per-capability guides (email, date, country, currency, ip, isbn, money, phone, si-unit, url) — each with recognized forms, canonical output & output_format, contract flags, status examples, notebook snippet, and provenance - api-reference (registration, canonicalize(), contracts, ExecutionResult, Provenance, Resolution, errors) + extending (community grammars/rules via extra_grammars) + migration (SemVer & upgrade checklist) Constraints: no links to docs/development or docs/adr, no fixed capability count — all lists framed as the current growing release. Co-Authored-By: internal-model --- docs/user/api-reference.md | 241 ++++++++++++++++++ docs/user/capabilities/country.md | 131 ++++++++++ docs/user/capabilities/currency.md | 133 ++++++++++ docs/user/capabilities/date.md | 125 +++++++++ docs/user/capabilities/email.md | 140 ++++++++++ docs/user/capabilities/index.md | 80 ++++++ docs/user/capabilities/ip.md | 105 ++++++++ docs/user/capabilities/isbn.md | 112 ++++++++ docs/user/capabilities/money.md | 130 ++++++++++ docs/user/capabilities/phone.md | 125 +++++++++ docs/user/capabilities/si-unit.md | 116 +++++++++ docs/user/capabilities/url.md | 115 +++++++++ .../user/concepts/candidates-and-ambiguity.md | 147 +++++++++++ docs/user/concepts/capabilities.md | 108 ++++++++ docs/user/concepts/contracts.md | 175 +++++++++++++ docs/user/concepts/errors.md | 117 +++++++++ docs/user/concepts/execution-result.md | 149 +++++++++++ docs/user/concepts/index.md | 92 +++++++ docs/user/concepts/pipeline.md | 151 +++++++++++ docs/user/concepts/provenance.md | 79 ++++++ docs/user/extending.md | 232 +++++++++++++++++ docs/user/getting-started.md | 221 ++++++++++++++++ docs/user/index.md | 111 ++++++++ docs/user/migration.md | 120 +++++++++ 24 files changed, 3255 insertions(+) create mode 100644 docs/user/api-reference.md create mode 100644 docs/user/capabilities/country.md create mode 100644 docs/user/capabilities/currency.md create mode 100644 docs/user/capabilities/date.md create mode 100644 docs/user/capabilities/email.md create mode 100644 docs/user/capabilities/index.md create mode 100644 docs/user/capabilities/ip.md create mode 100644 docs/user/capabilities/isbn.md create mode 100644 docs/user/capabilities/money.md create mode 100644 docs/user/capabilities/phone.md create mode 100644 docs/user/capabilities/si-unit.md create mode 100644 docs/user/capabilities/url.md create mode 100644 docs/user/concepts/candidates-and-ambiguity.md create mode 100644 docs/user/concepts/capabilities.md create mode 100644 docs/user/concepts/contracts.md create mode 100644 docs/user/concepts/errors.md create mode 100644 docs/user/concepts/execution-result.md create mode 100644 docs/user/concepts/index.md create mode 100644 docs/user/concepts/pipeline.md create mode 100644 docs/user/concepts/provenance.md create mode 100644 docs/user/extending.md create mode 100644 docs/user/getting-started.md create mode 100644 docs/user/index.md create mode 100644 docs/user/migration.md diff --git a/docs/user/api-reference.md b/docs/user/api-reference.md new file mode 100644 index 00000000..d361bd54 --- /dev/null +++ b/docs/user/api-reference.md @@ -0,0 +1,241 @@ +# API Reference + +This page is the concise reference for the public Python surface you import. For the mental model behind these types, see [Concepts](concepts/index.md). + +--- + +## Installation and imports + +```python +import paxman +from paxman.capabilities import Country, Currency, Date, Email, IP, ISBN, Money, Phone, SIUnit, URL +from paxman.core.domain import Resolution +from paxman.core.errors import ( + PaxmanError, CapabilityError, ContractError, + MultipleMentionsError, RecognitionError, ValidationError, +) +``` + +Paxman has no runtime dependencies. `import paxman` is side-effect free — nothing runs until you register and call `canonicalize()`. + +--- + +## Registration + +Registration tells the engine which capabilities exist. It must complete from **one thread before the first `canonicalize()` call**; afterwards the registry freezes and reads are safe from any thread. + +```mermaid +flowchart LR + A[import paxman] --> B[register] + B --> C[first canonicalize
freezes registry] + C --> D[further canonicalize
thread-safe reads] + B -.->|after freeze| E[CapabilityError] + + style C fill:#fff8e1,stroke:#d4a017 + style E fill:#fff5f5,stroke:#cc3333 +``` + +| Function | Signature | What it does | +|----------|-----------|--------------| +| `paxman.register_all_shipped()` | `() -> None` | Registers every capability shipped in this release. Idempotent by name. | +| `paxman.register_capability(cap)` | `(cap: Capability) -> None` | Registers one capability instance, e.g. `Email()`. Fails if the name already exists or the registry is frozen. | +| `paxman.register_grammar(name, cls)` | `(capability_name: str, grammar_cls: type[Grammar]) -> None` | Registers a community grammar (see [Extending](extending.md)). Must happen before the first call. | +| `paxman.register_rule(name, cls)` | `(capability_name: str, rule_cls: type[Rule]) -> None` | Registers a community rule. Must happen before the first call. | + +```python +# Quick exploration — everything +paxman.register_all_shipped() + +# Explicit — only what you depend on +paxman.register_capability(Email()) +paxman.register_capability(Date()) +``` + +--- + +## `paxman.canonicalize()` + +The sole entry point. + +```python +def canonicalize(text: str, contract: CapabilityContract) -> ExecutionResult +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `text` | `str` | Raw input. One presumed mention per call (see [Segmentation](../recipes/segmentation.md)). | +| `contract` | `CapabilityContract` | Created via `SomeCapability.create_contract(...)`. Carries `capability_name` so no separate name argument is needed. | + +**Returns** `ExecutionResult` — always, for every domain outcome including missing or invalid input. Domain outcomes are statuses, not exceptions. + +**Raises** + +| Exception | Cause | +|-----------|-------| +| `CapabilityError` | No capability matches `contract.capability_name`, duplicate name, or registry frozen | +| `ContractError` | Malformed contract (unknown `pinned_rules`, unknown `output_format`, unknown semantics, missing required feature) | +| `MultipleMentionsError` | Two or more non-overlapping mentions resolved to different values — caller must split first | +| `RecognitionError` | Grammar raised or returned a malformed match (`rule`, `original_error`) | +| `ValidationError` | Rule raised inside `matches()`/`normalize()` (`rule`, `original_error`) | + +`MISSING` / `INVALID` / `AMBIGUOUS` do **not** raise — they are `ExecutionResult.status` values (see [Execution Result](concepts/execution-result.md) and [Candidates & Ambiguity](concepts/candidates-and-ambiguity.md)). + +--- + +## Contracts — `SomeCapability.create_contract()` + +Every capability exposes a **keyword-only** factory. Common parameters come first, capability-specific ones after. + +```python +contract = Email.create_contract( + # common — every capability + excluded_rules=(), # Sequence[str] + pinned_rules=None, # Sequence[str] | None — when not None, only these run + year=None, # int | None — only rules with publication_year <= year + output_format=None, # str | None — None / "default" / default / offered, else ContractError + extra_grammars=(), # tuple[str, ...] — opt-in community grammars + # capability-specific — varies by capability + include_obfuscated=False, # example: Email only +) +``` + +The contract is a **frozen dataclass** — construct it, pass it, inspect it, but do not mutate it. See [Contracts](concepts/contracts.md) for the full common-plus-specific table. + +### `output_format` policy (identical for every capability) + +- `None`, `"default"`, and the capability's `DEFAULT_OUTPUT_FORMAT` all resolve to the default. +- Any value in `OFFERED_OUTPUT_FORMATS` resolves to itself. +- Anything else raises `ContractError` immediately. Validation never consults `output_format`. + +Current defaults and offered alternatives: + +| Capability | Default (`DEFAULT_OUTPUT_FORMAT`) | Offered (`OFFERED_OUTPUT_FORMATS`) | +|------------|-----------------------------------|-------------------------------------| +| Country | `alpha2` | `alpha3`, `numeric`, `name` | +| Currency | `code` | *(none — single format)* | +| Date | `ISO` | `US` | +| Email | `email` | *(none)* | +| IP | `ip` | *(none)* | +| ISBN | `isbn13` | `hyphenated` | +| Money | `code_amount` | `compact` | +| Phone | `e164` | `rfc3966`, `national` | +| SI Unit | `symbol` | *(none)* | +| URL | `url` | *(none)* | + +> The set of capabilities — and their offered formats — grows over time. Treat this table as the current release, not a closed list. + +### Capability-specific flag summary (current release) + +| Capability | Flag | Type | Default | Purpose | +|------------|------|------|---------|---------| +| Email | `include_obfuscated` | `bool` | `False` | `user at domain dot com` | +| Email | `include_localhost` | `bool` | `True` | `admin@localhost` | +| Country | `include_localized` | `bool` | `False` | CLDR multilingual names | +| Country | `include_historical` | `bool` | `False` | Deprecated names | +| Currency | `default_currency` | `str \| None` | `None` | Resolve shared bare symbol (`$`) to this alpha-3 code | +| IP | `include_ipv6` | `bool` | `True` | IPv6 | +| ISBN | `include_isbn10` | `bool` | `True` | Legacy ISBN-10 | +| ISBN | `include_range_validation` | `bool` | `False` | Registrant-range provenance | +| Money | `dollar_sign_currency` | `str \| None` | `None` | Resolve bare `$` amount to this alpha-3 code | +| Money | `precision` | `str` | `"strict"` | `strict` / `truncate` / `round` for over-precision amounts | +| Phone | `default_country` | `str \| None` | `None` | Interpret national numbers as if in this alpha-2 country | +| SI Unit | `allow_split_word_prefixes` | `bool` | `False` | `kilo gram` → `kg` | +| SI Unit | `allow_multi_solidus` | `bool` | `False` | `kg/m/s` preserved | +| Date | `two_digit_base_year` | `int \| None` | `None` | Base for 2-digit year expansion | + +Validation: `default_currency` / `dollar_sign_currency` must be uppercase alpha-3; `default_country` must be uppercase alpha-2; `precision` must be one of the three values — otherwise `ContractError` at construction time. + +--- + +## Execution result + +```python +@dataclass(frozen=True) +class ExecutionResult: + status: Resolution # MISSING | INVALID | SUCCESS | AMBIGUOUS + canonicalized_value: str | None # str on SUCCESS, None otherwise + candidates: tuple[Candidate, ...] # all validated evidence, deduplicated + contract: CapabilityContract # the contract you passed in + version_stamp: VersionStamp # .paxman_version + span: tuple[int, int] | None # [start, end) of resolved value on SUCCESS, else None +``` + +```python +@dataclass(frozen=True) +class Candidate: + value: str + recognition_rule: str # grammar name, e.g. "standard_recognition" + validation_rule: str # rule name, e.g. "Section 3.4.1-addr-spec" + span: tuple[int, int] | None # half-open [start, end) in the input + provenance: tuple[Provenance, ...] # one or more authority citations (read-only property) + +@dataclass(frozen=True) +class Provenance: + authority: str # "IETF", "ISO", "BIPM", "WHATWG", "CLDR", ... + specification_name: str # "RFC 5322", "ISO 3166-1", ... + kind: str # "specification" / "standard" / ... + reference_url: str + version: str | None + lifecycle: str # "active" / "deprecated" / ... + publication_year: int + +@dataclass(frozen=True) +class VersionStamp: + paxman_version: str + +class Resolution(Enum): + MISSING = "missing" + INVALID = "invalid" + SUCCESS = "success" + AMBIGUOUS = "ambiguous" +``` + +**Reading the result** (see also [Execution Result](concepts/execution-result.md)): + +```python +from paxman.core.domain import Resolution + +if result.status == Resolution.SUCCESS: + value = result.canonicalized_value # str, span is set +else: + # MISSING / INVALID / AMBIGUOUS — no single value, span is None + # inspect result.candidates and their provenance/spans + ... +``` + +`span` semantics: on `SUCCESS` it is the span of the single resolved value; on `AMBIGUOUS` use each `candidate.span`; on `MISSING`/`INVALID` it is `None` and `candidates` is empty. + +--- + +## Errors + +All exceptions inherit from `PaxmanError`. See [Errors](concepts/errors.md) for handling patterns. + +| Exception | Signal type | Typical cause | +|-----------|-------------|---------------| +| `CapabilityError` | setup | Unknown capability, duplicate name, registry already frozen | +| `ContractError` | setup | Bad `pinned_rules`, bad `output_format`, unknown semantics, missing required feature, bad `default_currency`/`default_country`/`precision` | +| `MultipleMentionsError` | usage | Two separate mentions with different values in one call — split first | +| `RecognitionError` | internal bug | Grammar raised or returned a bad span/raw_text. Fields: `rule`, `original_error` (`None` on structural failure) | +| `ValidationError` | internal bug | Rule raised inside `matches()`/`normalize()`. Fields: `rule`, `original_error` | + +Statuses vs exceptions: statuses are domain answers returned inside `ExecutionResult`; exceptions mean the call was not valid to attempt. Catch exceptions at setup and at the outer edge of a batch loop; branch on statuses in normal flow. + +--- + +## Quick lookup — choose the right capability + +| Kind of text you have | Capability to use | Factory | +|-----------------------|-------------------|---------| +| Email addresses | Email | `Email.create_contract(...)` | +| Dates | Date | `Date.create_contract(...)` | +| Country codes or names | Country | `Country.create_contract(...)` | +| Currency codes / symbols / names (no amount) | Currency | `Currency.create_contract(...)` | +| IP addresses | IP | `IP.create_contract(...)` | +| ISBNs | ISBN | `ISBN.create_contract(...)` | +| Money amounts with currency | Money | `Money.create_contract(...)` | +| Phone numbers | Phone | `Phone.create_contract(...)` | +| SI unit expressions | SI Unit | `SIUnit.create_contract(...)` | +| Absolute URLs / IRIs | URL | `URL.create_contract(...)` | + +New capabilities appear in minor releases — check `paxman.capabilities` for the current set. Each per-capability guide under [Capabilities](capabilities/index.md) details its recognized forms, output formats, contract flags, and provenance. diff --git a/docs/user/capabilities/country.md b/docs/user/capabilities/country.md new file mode 100644 index 00000000..edfde8dc --- /dev/null +++ b/docs/user/capabilities/country.md @@ -0,0 +1,131 @@ +# Country + +Canonicalizes **one country mention** per call — a code or a name — to a canonical code. + +> **In plain language:** give it `"US"`, `"United States"`, `"DE"`, or `"Alemania"` and it hands back the ISO alpha-2 code if a real spec says that name means that country. Multilingual and historical names are off by default so results are predictable; turn them on when you need them. + +--- + +## What it recognizes — and what it does not + +| Recognizes (current release — growing) | Does not recognize | +|----------------------------------------|--------------------| +| ISO 3166-1 alpha-2 (`US`, `DE`), alpha-3 (`USA`), numeric (`840`) | Ad-hoc abbreviations without a spec (`'Murica`) | +| Country names in English (`United States`, `Germany`) | Regions or subdivisions (`California`, `Bavaria`) — countries only | +| CLDR localized names (`Alemania` → `DE`) — only when `include_localized=True` | Localized names when the flag is off — recognized but `INVALID` (no authority claims them) | +| Historical/deprecated names (`Burma`) — only when `include_historical=True`, via ISO 3166-3 | Historical names when the flag is off — `INVALID` | + +--- + +## Canonical output + +Default `output_format` is `"alpha2"`. + +| `output_format` | Renders | Example for United States | +|-----------------|---------|---------------------------| +| *(default)* `alpha2` / `None` / `"default"` | ISO 3166-1 alpha-2 | `US` | +| `alpha3` | ISO 3166-1 alpha-3 | `USA` | +| `numeric` | ISO 3166-1 numeric | `840` | +| `name` | English short name | `United States` | + +Any other value raises `ContractError`. + +```python +from paxman.capabilities import Country +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("United States", Country.create_contract()).canonicalized_value # "US" +paxman.canonicalize("United States", Country.create_contract(output_format="alpha3")).canonicalized_value # "USA" +paxman.canonicalize("United States", Country.create_contract(output_format="numeric")).canonicalized_value # "840" +paxman.canonicalize("US", Country.create_contract(output_format="name")).canonicalized_value # "United States" +``` + +--- + +## Contract + +```python +contract = Country.create_contract( + include_localized=False, # bool, default False — CLDR multilingual names + include_historical=False, # bool, default False — ISO 3166-3 deprecated names + output_format=None, # "alpha2" (default), "alpha3", "numeric", "name" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- `include_localized` gates the CLDR localized-name rule; when `False`, `Alemania` is recognized but **no rule validates it** → `INVALID`. +- `include_historical` gates ISO 3166-3; a historical name that validates returns the **historical entity's own former code** (e.g. `Burma` → `BU`), not a successor state's code. + +Historical names can also be viewed as excluding the current ISO 3166-1 name rules and pinning to the 3166-3 rule — the contract controls which specs are consulted (see [Contracts](../concepts/contracts.md)). + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `US` | defaults | `SUCCESS` | `"US"` | +| `United States` | defaults | `SUCCESS` | name → `"US"` | +| `Alemania` | defaults | `INVALID` | recognized (name grammar) but no rule accepts it without `include_localized` | +| `Alemania` | `include_localized=True` | `SUCCESS` | → `"DE"` with CLDR provenance | +| `Burma` | defaults | `INVALID` | needs `include_historical` | +| `Burma` | `include_historical=True` | `SUCCESS` | → `"BU"` (historical entity's own code) | +| `ZZ` | any | `INVALID` or `MISSING` | no spec claims it | +| `hello world` | any | `MISSING` | no country pattern | +| `US and DE` (two different mentions) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text e.g. Alemania] --> G[Grammars:
alpha2 / alpha3 / numeric / name] + G --> R1{ISO 3166-1 name rule} + G --> R2{CLDR localized rule
only if include_localized} + G --> R3{ISO 3166-3 historical
only if include_historical} + R1 & R2 & R3 --> D{Dedup values} + D -->|one value| OK[SUCCESS] + D -->|none| INV[INVALID] + G -->|no pattern| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import Country +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Country.create_contract(include_localized=True) +contract_hist = Country.create_contract(include_localized=True, include_historical=True) + +rows = ["US", "United States", "Alemania", "Burma", "not a country", "US and DE"] + +for text in rows: + for label, c in [("default+localized", contract), ("+historical", contract_hist)]: + try: + r = paxman.canonicalize(text, c) + except Exception as e: + print(f"{text!r:20} [{label}] → exception {type(e).__name__}") + continue + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + prov = r.candidates[0].provenance[0].specification_name if r.candidates else "—" + print(f"{text!r:20} [{label:18}] → {r.status.value:10} {val!r:8} ({prov})") +``` + +--- + +## Provenance + +- **ISO 3166-1** (alpha-2 / alpha-3 / numeric / English names) +- **ISO 3166-3** (historical, when enabled) +- **CLDR** (Unicode localized names, when enabled) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result.md), [Provenance](../concepts/provenance.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/currency.md b/docs/user/capabilities/currency.md new file mode 100644 index 00000000..81dc9810 --- /dev/null +++ b/docs/user/capabilities/currency.md @@ -0,0 +1,133 @@ +# Currency + +Canonicalizes **one currency identifier** per call — a code, a symbol, or a display name — to the uppercase ISO 4217 alpha-3 code. No amounts: `"USD 500"` is the [Money](money.md) capability's domain. + +> **In plain language:** give it `"usd"`, `"€"`, `"euro"`, or `"$"` and it tells you which currency code the spec says that identifier means. A bare shared symbol like `"$"` is deliberately `INVALID` unless you say which currency you mean. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Lowercase or uppercase alpha-3 codes (`usd` → `USD`) | Codes with wrong length or casing outside the fold — `INVALID` after recognition | +| CLDR display-name words (`euro` → `EUR`, `yen` → `JPY`) | Amount-glued tokens like `US$5` — not recognized at all | +| CLDR currency symbols (`€` → `EUR`, `¥` → `JPY`) | Bare shared symbols like `"$"` without disambiguation — recognized but `INVALID` unless you opt in | +| Shared bare symbols (`$`) — 29 codes share this CLDR symbol | A `default_currency` code that is not one of that symbol's own candidates — stays `INVALID` | + +> **Why shared symbols fail by default:** `"$"` is the CLDR symbol for 29 different currencies. Guessing would be wrong most of the time, so Paxman requires you to opt in. + +--- + +## Canonical output + +Single format — always an uppercase alpha-3 code. + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `code` / `None` / `"default"` | `USD`, `EUR`, … | + +Any other value raises `ContractError`. + +```python +from paxman.capabilities import Currency +import paxman + +paxman.register_all_shipped() +Currency.create_contract().output_format # "code" +paxman.canonicalize("usd", Currency.create_contract()).canonicalized_value # "USD" +paxman.canonicalize("euro", Currency.create_contract()).canonicalized_value # "EUR" +``` + +--- + +## Contract + +```python +contract = Currency.create_contract( + default_currency=None, # str | None — uppercase alpha-3, e.g. "USD" + output_format=None, # "code" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- When `default_currency` is `None` (the default), a shared bare symbol like `"$"` is recognized but never resolved → `INVALID`. +- When `default_currency` is set, the symbol resolves **only if** that code is one of the symbol's own CLDR candidates. `"$"` with `default_currency="USD"` → `USD`; `"$"` with `default_currency="MYR"` stays `INVALID` because `MYR`'s symbol is `RM`, not `"$"`. +- `default_currency` never remaps a definitive symbol (`€` is always `EUR`) or a qualified symbol (`US$` is always `USD`). + +```python +from paxman.capabilities import Currency +import paxman + +paxman.register_all_shipped() + +paxman.canonicalize("$", Currency.create_contract()).status.value # "invalid" +paxman.canonicalize("$", Currency.create_contract(default_currency="USD")).canonicalized_value # "USD" +paxman.canonicalize("$", Currency.create_contract(default_currency="MYR")).status.value # "invalid" — MYR is not a $ candidate +``` + +See [Contracts](../concepts/contracts.md) and the [API Reference](../api-reference.md#contracts--somecapabilitycreate_contract). + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `usd` | defaults | `SUCCESS` | → `USD` (case fold + ISO 4217 validation) | +| `euro` | defaults | `SUCCESS` | CLDR display name → `EUR` | +| `€` | defaults | `SUCCESS` | definitive symbol → `EUR` | +| `$` | defaults | `INVALID` | shared bare symbol, no `default_currency` | +| `$` | `default_currency="USD"` | `SUCCESS` | `USD` is a `$` candidate | +| `US$` | any | `SUCCESS` | qualified symbol → `USD` | +| `notacurrency` | any | `MISSING` | no pattern | +| `USD and EUR` (two different values) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text e.g. $] --> G[Grammars:
code / symbol / word] + G --> R{Rules:
ISO 4217 code / CLDR symbols / CLDR names} + R -->|definitive symbol
or code/word| OK[SUCCESS] + R -->|shared bare symbol
without matching default_currency| INV[INVALID] + R -->|shared + matching default_currency| OK2[SUCCESS
via default_currency] + G -->|nothing matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style OK2 fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import Currency +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c_default = Currency.create_contract() +c_usd = Currency.create_contract(default_currency="USD") + +for text in ["usd", "euro", "€", "$", "US$", "notacurrency"]: + r = paxman.canonicalize(text, c_default) + tag = r.status.value + val = r.canonicalized_value or "—" + print(f"{text!r:10} default → {tag:10} {val!r}") + if text == "$": + r2 = paxman.canonicalize(text, c_usd) + print(f" with default_currency=USD → {r2.status.value:10} {r2.canonicalized_value!r}") +``` + +--- + +## Provenance + +- **ISO 4217** — alpha-3 currency codes. +- **CLDR** — currency symbols and display names. + +Compare [Money](money.md) when your input includes amounts; Currency is identifier-only. + +See also: [Money](money.md), [Execution Result](../concepts/execution-result.md), [Candidates & Ambiguity](../concepts/candidates-and-ambiguity.md). diff --git a/docs/user/capabilities/date.md b/docs/user/capabilities/date.md new file mode 100644 index 00000000..7ecc19fa --- /dev/null +++ b/docs/user/capabilities/date.md @@ -0,0 +1,125 @@ +# Date + +Canonicalizes **one calendar date** per call to ISO `YYYY-MM-DD` — or to US form when requested. + +> **In plain language:** give it a date written any of the common ways and it hands back the ISO form if a spec says the date is real. Ambiguous numeric forms like `01/02/2026` are surfaced as `AMBIGUOUS` rather than guessed. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| `2026-01-15` — ISO 8601 `YYYY-MM-DD` | Times, durations, or datetimes (`2026-01-15T10:00`) — dates only | +| `2026/01/15` — slash-ISO `YYYY/MM/DD` | Month names (`15 January 2026`) | +| `01/02/2026` — US `MM/DD/YYYY` and European `DD/MM/YYYY` (kept as competing interpretations) | Relative expressions (`today`, `next Tuesday`) | + +All four shapes emit the same notation; which rules accept them depends on the contract and the spec calendar logic. + +--- + +## Canonical output + +Default `output_format` is `"ISO"` (`YYYY-MM-DD`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `ISO` / `None` / `"default"` | `YYYY-MM-DD` | `2026-01-15` | +| `US` | `MM/DD/YYYY` | `01/15/2026` | + +Any other value raises `ContractError`. + +```python +from paxman.capabilities import Date +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("2026-01-15", Date.create_contract()).canonicalized_value # "2026-01-15" +paxman.canonicalize("2026-01-15", Date.create_contract(output_format="US")).canonicalized_value # "01/15/2026" +``` + +The `two_digit_base_year` field is relevant only when the input contains a 2-digit year — e.g. with `two_digit_base_year=2000`, `"01/02/26"` expands relative to 2000. Otherwise omit it. + +--- + +## Contract + +```python +contract = Date.create_contract( + two_digit_base_year=None, # int | None — base year for 2-digit expansion, e.g. 2000 + output_format=None, # "ISO" (default) or "US" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- `two_digit_base_year` shapes how `YY` is expanded; it does not affect which grammars run. +- Use `pinned_rules` when you want only one jurisdiction's calendar (e.g. pin to ISO 8601); use `year` to restrict to rules published up to a given year. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `2026-01-15` | defaults | `SUCCESS` | `"2026-01-15"` | +| `2026/01/15` | defaults | `SUCCESS` | slash-ISO → `"2026-01-15"` | +| `01/02/2026` | defaults | `AMBIGUOUS` | US → `2026-01-02` vs European → `2026-02-01` — same span, different values | +| `01/02/2026` | `pinned_rules=["…calendar-date"]` (ISO only) | `SUCCESS` or `INVALID` | only the pinned spec's reading remains | +| `2026-13-01` | any | `INVALID` | recognized but no calendar accepts month 13 | +| `hello` | any | `MISSING` | no date pattern at all | +| `2026-01-15, 2026-02-01` | any | raises `MultipleMentionsError` | two distinct dates — split first | + +```mermaid +flowchart TB + A[Text
01/02/2026] --> G1[ISO grammar] + A --> G2[US grammar] + A --> G3[European grammar] + A --> G4[Slash-ISO grammar] + G1 & G2 & G3 & G4 --> R[Rules:
ISO 8601 / US federal / EN 50160] + R -->|one value| OK[SUCCESS] + R -->|US vs EU| AMB[AMBIGUOUS] + R -->|none| INV[INVALID] + + style OK fill:#e6ffed,stroke:#2d8a4e + style AMB fill:#fff8e1,stroke:#d4a017 + style INV fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a column with ambiguity surfaced + +```python +import paxman +from paxman.capabilities import Date +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Date.create_contract() + +rows = ["2026-01-15", "2026/01/15", "01/02/2026", "2026-13-01", "hello"] + +for text in rows: + r = paxman.canonicalize(text, contract) + if r.status == Resolution.SUCCESS: + print(f"{text!r:15} → {r.canonicalized_value}") + elif r.status == Resolution.AMBIGUOUS: + vals = sorted({c.value for c in r.candidates}) + print(f"{text!r:15} → AMBIGUOUS {vals}") + else: + print(f"{text!r:15} → {r.status.value}") +``` + +--- + +## Provenance + +Validated values cite the calendar spec whose rule accepted the notation — e.g. ISO 8601, US federal rules, or EN 50160, with section citation on `candidate.validation_rule` and `publication_year` on `candidate.provenance[0]`. + +```python +for c in result.candidates: + p = c.provenance[0] + print(c.value, "via", c.validation_rule, f"({p.specification_name}, {p.publication_year})") +``` + +See also: [Execution Result](../concepts/execution-result.md), [Candidates & Ambiguity](../concepts/candidates-and-ambiguity.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/email.md b/docs/user/capabilities/email.md new file mode 100644 index 00000000..4a489ef5 --- /dev/null +++ b/docs/user/capabilities/email.md @@ -0,0 +1,140 @@ +# Email + +Canonicalizes **one email address** per call — standard, obfuscated, or localhost — to a lowercased `addr-spec`. + +> **In plain language:** give it something that looks like an email address and it hands back the single correct lowercase address if a real specification says it is valid, plus a citation of that specification. If it does not look like an email, or no spec accepts it, it tells you so without raising an exception. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Standard `user@example.com` (case-insensitive) | Display-name forms like `Bob ` — pass the addr-spec part | +| Obfuscated `user at example dot com` (only when `include_obfuscated=True`) | Arbitrary prose that happens to mention a domain without address structure | +| `admin@localhost` (only when `include_localhost=True`, the default) | | + +Grammars are syntax-only; whether the match is *valid* is decided by the rules (see [Pipeline](../concepts/pipeline.md)). + +--- + +## Canonical output + +Default `output_format` is `"email"` — a single format, always a lowercased `addr-spec`. + +```python +from paxman.capabilities import Email +import paxman + +paxman.register_all_shipped() +contract = Email.create_contract() +paxman.canonicalize("USER@Example.COM", contract).canonicalized_value # "user@example.com" +``` + +There are no offered alternatives for Email — `None`, `"default"`, and `"email"` all resolve to the same rendering; any other value raises `ContractError`. + +--- + +## Contract + +```python +contract = Email.create_contract( + include_obfuscated=False, # bool, default False — recognize "user at example dot com" + include_localhost=True, # bool, default True — recognize admin@localhost + # plus every common field: excluded_rules / pinned_rules / year / output_format / extra_grammars +) +``` + +- Toggling `include_obfuscated` or `include_localhost` changes **which grammars run**. When the relevant grammar is off, the corresponding input is `MISSING` (never seen), not `INVALID`. +- Use `excluded_rules` / `pinned_rules` / `year` to control **which rules validate** — a recognized input that no rule accepts becomes `INVALID`. +- `output_format` is ignored beyond validation; it only shapes the rendered string. + +See [Contracts](../concepts/contracts.md) and the [API Reference](../api-reference.md#contracts--somecapabilitycreate_contract) for the full policy. + +--- + +## Statuses + +| Input | Contract tweak | Status | Value / why | +|-------|---------------|--------|-------------| +| `user@example.com` | defaults | `SUCCESS` | `"user@example.com"` | +| `USER@Example.COM` | defaults | `SUCCESS` | lowercased to `"user@example.com"` | +| `user at example dot com` | defaults (`include_obfuscated=False`) | `MISSING` | grammar not active — nothing seen | +| `user at example dot com` | `include_obfuscated=True` | `SUCCESS` | `"user@example.com"` | +| `admin@localhost` | defaults | `SUCCESS` | localhost path via RFC 6761 | +| `admin@localhost` | `include_localhost=False` | `MISSING` | grammar not active | +| `admin@localhost` | `excluded_rules=["Section 6.3-localhost"]` | `INVALID` | recognized but no rule accepts it | +| `@@` | any | `MISSING` | no email pattern at all | +| `alice@example.com and bob@example.org` | any | raises `MultipleMentionsError` | two distinct mentions — split first (see [Segmentation](../../recipes/segmentation.md)) | + +```mermaid +flowchart TB + A[Text] --> G{Which grammars active?} + G -->|standard| S[standard_recognition] + G -->|include_obfuscated| O[obfuscated_recognition] + G -->|include_localhost| L[localhost_recognition] + S & O & L --> R{Rules: RFC 5322 §3.4.1
RFC 6761 §6.3} + R -->|one value| OK[SUCCESS] + R -->|none| INV[INVALID] + G -->|none matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style MISS fill:#fff5f5,stroke:#cc3333 + style INV fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — clean a column + +```python +import paxman +from paxman.capabilities import Email +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Email.create_contract(include_obfuscated=True, include_localhost=True) + +rows = [ + "user@Example.COM", + "user at example dot com", + "admin@localhost", + "not an email", + "alice@example.com and bob@example.org", +] + +for text in rows: + try: + r = paxman.canonicalize(text, contract) + except Exception as e: + print(f"{text!r:35} → exception {type(e).__name__}: {e}") + continue + tag = r.status.value + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:35} → {tag:10} {val!r}") +``` + +--- + +## Provenance + +Validated values carry provenance from: + +- **IETF RFC 5322** — Internet Message Format, Section 3.4.1 `addr-spec` +- **IETF RFC 6761** — Special-Use Domain Names, Section 6.3 (localhost) + +Inspect `candidate.provenance` and `candidate.validation_rule` for the exact citation: + +```python +for c in result.candidates: + p = c.provenance[0] + print(c.value, "via", c.validation_rule, f"({p.authority}: {p.specification_name}, {p.publication_year})") +``` + +--- + +## Span + +On `SUCCESS`, `result.span` is the exact `[start, end)` of the email in the original input — useful for highlighting in UIs or logging. On `MISSING` / `INVALID` it is `None`; on `AMBIGUOUS` (rare for Email) use each `candidate.span`. + +See also: [Execution Result](../concepts/execution-result.md), [Provenance](../concepts/provenance.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/index.md b/docs/user/capabilities/index.md new file mode 100644 index 00000000..1b5fdb80 --- /dev/null +++ b/docs/user/capabilities/index.md @@ -0,0 +1,80 @@ +# Capabilities — Overview + +Each page below is a self-contained guide for one kind of identifier Paxman can canonicalize. Read the one that matches your data, copy the notebook snippet, and adapt the contract flags to your needs. + +All capabilities share the same call shape — only the import, the factory, and the domain vocabulary change: + +```python +import paxman +from paxman.capabilities import X # Email, Country, ... +paxman.register_all_shipped() # once, before first use +contract = X.create_contract(...) # domain flags here +result = paxman.canonicalize(text, contract) +``` + +For the shared concepts behind these pages see [Contracts](../concepts/contracts.md), [Pipeline](../concepts/pipeline.md), [Execution Result](../concepts/execution-result.md), and the [API Reference](../api-reference.md). + +--- + +## Choose by what you have + +| Your data looks like… | Read | +|-----------------------|------| +| `user@example.com`, `user at example dot com` | [Email](email.md) | +| `2026-01-15`, `01/02/2026`, `2026/01/15` | [Date](date.md) | +| `US`, `United States`, `Alemania` | [Country](country.md) | +| `USD`, `$`, `euro`, `¥` (identifiers without amounts) | [Currency](currency.md) | +| `192.168.1.1`, `2001:db8::1` | [IP](ip.md) | +| `9780306406157`, `0306406152` | [ISBN](isbn.md) | +| `USD 500`, `$500`, `1.000,50 EUR` (currency **with** amount) | [Money](money.md) | +| `+1 555 123 4567`, `(555) 234-5678`, `tel:+15551234567` | [Phone](phone.md) | +| `kg`, `m/s²`, `megahertz`, `kPa` | [SI Unit](si-unit.md) | +| `https://example.com`, `http://münchen.de` | [URL](url.md) | + +> The set above reflects the **current release**. New capabilities are added in minor releases — check `paxman.capabilities` or the latest release notes if you don't see what you need. + +```mermaid +flowchart LR + A[Raw text] --> B{What kind?} + B -->|email| C[Email] + B -->|date| D[Date] + B -->|place| E[Country] + B -->|money / currency| F[Currency / Money] + B -->|network| G[IP / URL] + B -->|books| H[ISBN] + B -->|phones| I[Phone] + B -->|units| J[SI Unit] + + style B fill:#fff8e1,stroke:#d4a017 +``` + +--- + +## What each page covers + +Every capability page answers the same questions in the same order: + +1. **What it canonicalizes** and what it explicitly does not. +2. **Recognized forms** — what patterns match, with what grammars. +3. **Canonical output & `output_format`** — default and offered renderings. +4. **Contract flags** — which knobs change recognition and validation. +5. **Statuses** — concrete `SUCCESS` / `MISSING` / `INVALID` / `AMBIGUOUS` examples. +6. **Notebook snippet** — runnable cleaning loop for a column. +7. **Provenance** — which specifications vouch for the answer. + +Start with the capability that matches your column; if you need more than one, register both and loop per cell (see the [Segmentation Recipe](../../recipes/segmentation.md) for text that mixes kinds). + +--- + +## One mention per call + +Paxman resolves **one presumed entity per `canonicalize()` call** (see [Pipeline](../concepts/pipeline.md)). Text that contains two different entities with different canonical values raises `MultipleMentionsError` rather than returning a merged answer — split first, then loop. + +```python +from paxman.core.errors import MultipleMentionsError +try: + result = paxman.canonicalize("alice@example.com, bob@example.org", contract) +except MultipleMentionsError: + # split the input and canonicalize each piece — see the segmentation recipe + ... +``` diff --git a/docs/user/capabilities/ip.md b/docs/user/capabilities/ip.md new file mode 100644 index 00000000..4b5c5ec8 --- /dev/null +++ b/docs/user/capabilities/ip.md @@ -0,0 +1,105 @@ +# IP + +Canonicalizes **one IP address** per call to its normalized textual form. + +> **In plain language:** give it `"192.168.1.1"` or `"2001:0db8:0000:0000:0000:0000:0000:0001"` and it hands back the normalized address if a spec says it is valid. IPv6 canonicalization follows RFC 5952 so the same address always looks the same. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| IPv4 dotted-decimal (`192.168.1.1`) | CIDR notation (`192.168.1.0/24`) — address only | +| IPv6 (compressed or expanded, e.g. `2001:db8::1`) | Hostnames — use [URL](url.md) | + +--- + +## Canonical output + +Single format — the normalized address. IPv6 is rendered per **RFC 5952** (lowercase hex, `::` compression, etc.). + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `ip` / `None` / `"default"` | normalized address string | + +```python +from paxman.capabilities import IP +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("192.168.1.1", IP.create_contract()).canonicalized_value # "192.168.1.1" +paxman.canonicalize("2001:0db8:0000:0000:0000:0000:0000:0001", IP.create_contract()).canonicalized_value # "2001:db8::1" +``` + +--- + +## Contract + +```python +contract = IP.create_contract( + include_ipv6=True, # bool, default True — recognize IPv6 + output_format=None, # "ip" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- With `include_ipv6=True` (the default) both grammars run. With `False`, only `ipv4_recognition` runs — an IPv6 address is then `MISSING` (never seen), not `INVALID`. + +```python +paxman.canonicalize("2001:db8::1", IP.create_contract(include_ipv6=False)).status.value # "missing" +paxman.canonicalize("2001:db8::1", IP.create_contract()).canonicalized_value # "2001:db8::1" +``` + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `192.168.1.1` | any | `SUCCESS` | normalized IPv4 | +| `2001:db8::1` | defaults | `SUCCESS` | → `2001:db8::1` (RFC 5952) | +| `2001:db8::1` | `include_ipv6=False` | `MISSING` | IPv6 grammar not active | +| `999.999.999.999` | any | `INVALID` | recognized shape but no spec accepts it | +| `hello` | any | `MISSING` | no IP pattern | +| `10.0.0.1 and 10.0.0.2` (two distinct values) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G1[ipv4_recognition] + A --> G2[ipv6_recognition
only if include_ipv6] + G1 --> R1[Rule: RFC 791] + G2 --> R2[Rule: RFC 5952] + R1 & R2 --> D{One value?} + D -->|yes| OK[SUCCESS
normalized] + D -->|none| MISS_OR_INV[MISSING or INVALID] + D -->|two values
same mention| AMB[AMBIGUOUS
rare for IP] + + style OK fill:#e6ffed,stroke:#2d8a4e +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import IP +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = IP.create_contract() + +for text in ["192.168.1.1", "2001:0db8:0000:0000:0000:0000:0000:0001", "999.999.999.999", "hello"]: + r = paxman.canonicalize(text, contract) + print(f"{text!r:45} → {r.status.value:10} {r.canonicalized_value!r}") +``` + +--- + +## Provenance + +- **RFC 791** — IPv4 address validation. +- **RFC 5952** — IPv6 text representation and canonical compression. + +See also: [URL](url.md), [Execution Result](../concepts/execution-result.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/isbn.md b/docs/user/capabilities/isbn.md new file mode 100644 index 00000000..4e4f4106 --- /dev/null +++ b/docs/user/capabilities/isbn.md @@ -0,0 +1,112 @@ +# ISBN + +Canonicalizes **one ISBN identifier** per call with check-digit validation, converting legacy ISBN-10 to ISBN-13 when enabled. + +> **In plain language:** give it `"9780306406157"` or `"0306406152"` and it hands back the 13-digit ISBN if the check digit is correct. The hyphenated form is a presentation option — the underlying value is always the bare digits unless you ask for hyphens. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| ISBN-13 bare digits (`9780306406157`, with or without `978`/`979` prefix) | ISBNs with wrong check digits — recognized but `INVALID` | +| ISBN-10 (`0306406152`) — only when `include_isbn10=True` (the default) | ISBN-10 when `include_isbn10=False` → `MISSING` | +| Hyphenated input is normalized regardless of input hyphens — output hyphens are via `output_format` | Informal variants or partial numbers — `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"isbn13"` (bare 13 digits). + +| `output_format` | Renders | Example for `9780110002224` | +|-----------------|---------|------------------------------| +| *(default)* `isbn13` / `None` / `"default"` | bare 13 digits | `9780110002224` | +| `hyphenated` | hyphenated by registrant-range (ISBN Range Message, longest-match) | `978-0-11-000222-4` | + +Hyphenation is presentation only — provenance remains the same; the hyphen positions come from the authoritative Range Message table. + +```python +from paxman.capabilities import ISBN +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("9780110002224", ISBN.create_contract()).canonicalized_value # "9780110002224" +paxman.canonicalize("9780110002224", ISBN.create_contract(output_format="hyphenated")).canonicalized_value # "978-0-11-000222-4" +paxman.canonicalize("0306406152", ISBN.create_contract()).canonicalized_value # "9780306406157" (10 → 13) +``` + +--- + +## Contract + +```python +contract = ISBN.create_contract( + include_isbn10=True, # bool, default True — recognize ISBN-10 + include_range_validation=False, # bool, default False — gate the registrant-range rule/provenance + output_format=None, # "isbn13" (default) or "hyphenated" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- With `include_isbn10=False`, only the `isbn13_recognition` grammar runs — ISBN-10 input is `MISSING` (never seen), not `INVALID`. +- With `include_range_validation=True`, an additional rule from the ISBN Range Message validates the registrant range and adds provenance. It does not change the canonical digits, only whether registrant-range provenance appears. + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `9780306406157` | defaults | `SUCCESS` | bare ISBN-13, check digit valid | +| `0306406152` | defaults | `SUCCESS` | ISBN-10 → `9780306406157` | +| `0306406152` | `include_isbn10=False` | `MISSING` | grammar not active | +| `9780306406150` (bad check) | any | `INVALID` | recognized but check digit fails | +| `hello` | any | `MISSING` | no ISBN pattern | +| Two distinct ISBNs in one call | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G1[isbn13_recognition
always] + A --> G2[isbn10_recognition
only if include_isbn10] + G1 & G2 --> R{Rules: ISO 2108 /
ISBN Users' Manual /
Range Message + check digits} + R -->|one value| OK[SUCCESS
bare or hyphenated] + R -->|bad check| INV[INVALID] + G1 & G2 -->|nothing matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import ISBN +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c = ISBN.create_contract() +c_hyph = ISBN.create_contract(output_format="hyphenated") + +for text in ["9780306406157", "0306406152", "9780306406150", "hello"]: + r = paxman.canonicalize(text, c) + rh = paxman.canonicalize(text, c_hyph) + a = r.canonicalized_value or "—" + b = rh.canonicalized_value or "—" + print(f"{text!r:20} → bare {a!r:15} hyphenated {b!r:20} status={r.status.value}") +``` + +--- + +## Provenance + +- **ISO 2108** — GS1 / ISBN prefix. +- **ISBN Users' Manual** — ISBN-13 / ISBN-10 check-digit logic. +- **ISBN Range Message** — registrant-range hyphenation and provenance (only with `include_range_validation=True`). + +See also: [Execution Result](../concepts/execution-result.md), [Provenance](../concepts/provenance.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/money.md b/docs/user/capabilities/money.md new file mode 100644 index 00000000..6c7d3ecc --- /dev/null +++ b/docs/user/capabilities/money.md @@ -0,0 +1,130 @@ +# Money + +Canonicalizes **one money amount paired with a currency** per call to `CODE amount` padded to ISO 4217 minor units. + +> **In plain language:** give it `"USD500"` or `"1.000,50 EUR"` and it hands back `"USD 500.00"` / `"EUR 1000.50"` if the specs say the amount and currency go together. Currency without an amount — or an amount without a currency — is not Money's domain (see [Currency](currency.md)). + +--- + +## What it recognizes — and what it does not + +| Recognizes (currency + amount adjacent) | Does not recognize | +|-----------------------------------------|--------------------| +| `USD500`, `USD 500.00`, `EUR 1.000,50` | Currency identifier alone (`USD`) — use [Currency](currency.md) | +| `€500`, `¥1000`, `US$ 500` (qualified symbol) | Bare shared symbol without disambiguation (`$500` with default contract) — recognized but `INVALID` | +| `1.000,50 EUR` — European comma-decimal: last separator is the decimal point | Amount-glued tokens without a clean boundary — `MISSING` | +| `$500` with `dollar_sign_currency="USD"` — opt-in for shared symbols | Bare `$` without `dollar_sign_currency` — `INVALID` | + +> Shared symbol handling mirrors Currency's `default_currency` idea but for amounts: Money uses `dollar_sign_currency` (same semantics, different name because it shapes amounts). + +--- + +## Canonical output + +Default `output_format` is `"code_amount"` (space between code and amount). + +| `output_format` | Renders | Example for USD 500 | +|-----------------|---------|---------------------| +| *(default)* `code_amount` / `None` / `"default"` | `CODE + space + amount` | `USD 500.00` | +| `compact` | removes the single space | `USD500.00` | + +The amount is padded/normalized to the currency's ISO 4217 minor units (e.g. 2 decimals for USD/EUR). How over-precision is handled depends on `precision` (see contract). + +```python +from paxman.capabilities import Money +import paxman + +paxman.register_all_shipped() +Money.create_contract().canonicalized_value # not a thing — use result.canonicalized_value +paxman.canonicalize("USD500", Money.create_contract()).canonicalized_value # "USD 500.00" +paxman.canonicalize("USD500", Money.create_contract(output_format="compact")).canonicalized_value # "USD500.00" +paxman.canonicalize("1.000,50 EUR", Money.create_contract()).canonicalized_value # "EUR 1000.50" +``` + +--- + +## Contract + +```python +contract = Money.create_contract( + dollar_sign_currency=None, # str | None — uppercase alpha-3, e.g. "USD" + precision="strict", # "strict" (default) | "truncate" | "round" — over-precision policy + output_format=None, # "code_amount" (default) or "compact" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- **`dollar_sign_currency`**: when `None` (default), a multi-candidate bare symbol amount like `$500` is recognized but never resolved → `INVALID`. When set, `$` resolves to that code **only if** it is one of `$`'s own CLDR candidates. `€500` never needs this — `€` is definitive for `EUR`. +- **`precision`**: how to treat more fractional digits than the currency's minor units allow — `strict` rejects → `INVALID`, `truncate` drops excess digits, `round` half-to-even. +- `output_format` never affects validation — only the space. + +```python +paxman.canonicalize("$500", Money.create_contract()).status.value # "invalid" +paxman.canonicalize("$500", Money.create_contract(dollar_sign_currency="MYR")).canonicalized_value # "MYR 500.00" +paxman.canonicalize("USD 1.999", Money.create_contract(precision="strict")).status.value # "invalid" — too many decimals for USD +paxman.canonicalize("USD 1.999", Money.create_contract(precision="round")).canonicalized_value # "USD 2.00" +``` + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `USD500` | defaults | `SUCCESS` | → `USD 500.00` | +| `€500` | defaults | `SUCCESS` | → `EUR 500.00` (definitive symbol) | +| `$500` | defaults | `INVALID` | shared symbol, no `dollar_sign_currency` | +| `$500` | `dollar_sign_currency="USD"` | `SUCCESS` | → `USD 500.00` | +| `1.000,50 EUR` | defaults | `SUCCESS` | European comma-decimal → `EUR 1000.50` | +| `USD 1.999` | `precision="strict"` | `INVALID` | over-precision → rejected | +| `hello` | any | `MISSING` | no Money pattern | +| Two different amounts | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text e.g. $500] --> G[Grammars:
code / symbol / word + amount] + G --> R{Rules: ISO 4217 +
CLDR symbols/names
+ minor units} + R -->|definitive symbol
or code/word| OK[SUCCESS
CODE amount] + R -->|shared symbol
needs dollar_sign_currency| INV[INVALID
or SUCCESS if matches] + R -->|over-precision + strict| INV2[INVALID] + G -->|no pattern| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style INV2 fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — clean a column with mixed formats + +```python +import paxman +from paxman.capabilities import Money +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c_strict = Money.create_contract() +c_usd = Money.create_contract(dollar_sign_currency="USD") +c_round = Money.create_contract(precision="round") + +rows = ["USD500", "€500", "$500", "1.000,50 EUR", "USD 1.999", "hello"] + +for text in rows: + for label, c in [("strict", c_strict), ("USD bare-$", c_usd), ("round", c_round)]: + r = paxman.canonicalize(text, c) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:15} [{label:10}] → {r.status.value:10} {val!r}") +``` + +--- + +## Provenance + +- **ISO 4217** — currency codes and minor units. +- **CLDR** — currency symbols and display names. + +Compare [Currency](currency.md) for identifier-only canonicalization (no amount). + +See also: [Execution Result](../concepts/execution-result.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/phone.md b/docs/user/capabilities/phone.md new file mode 100644 index 00000000..18e7cb75 --- /dev/null +++ b/docs/user/capabilities/phone.md @@ -0,0 +1,125 @@ +# Phone + +Canonicalizes **one phone number** per call to E.164 (or to tel-URI / national form when requested). + +> **In plain language:** give it `"+1 555 123 4567"` or `"(555) 234-5678"` and it hands back `"+15551234567"` if the numbering plan says the number is valid. National-shaped numbers need you to say which country's plan to use. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| E.164 international (`+1 5551234567`) | Text that only mentions a country without digits | +| `00`-prefix international (`0044 20 ...`) | Extensions without a dialable number | +| `tel:` URI (`tel:+1-555-123-4567`) | Plain prose — `MISSING` | +| NANP / national numbers (`(555) 234-5678`) — only when a `default_country` is provided that is in the NANP | National-shaped input without `default_country` — recognized but `INVALID` | + +--- + +## Canonical output + +Default `output_format` is `"e164"`. + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `e164` / `None` / `"default"` | `+` + country code + national significant number | `+15551234567` | +| `rfc3966` | `tel:` URI | `tel:+15551234567` | +| `national` | National significant number (no `+` or `tel:`) | `5551234567` | + +`national` works without `default_country` for numbers whose country code is embedded (E.164, tel-URI, NANP inputs are split by the rules); for national-shaped input it requires `default_country` to validate in the first place. + +```python +from paxman.capabilities import Phone +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("+1 555 123 4567", Phone.create_contract()).canonicalized_value # "+15551234567" +paxman.canonicalize("+15551234567", Phone.create_contract(output_format="rfc3966")).canonicalized_value # "tel:+15551234567" +paxman.canonicalize("+15551234567", Phone.create_contract(output_format="national")).canonicalized_value # "5551234567" + +# National-shaped input needs default_country +paxman.canonicalize("(555) 234-5678", Phone.create_contract(default_country="US")).canonicalized_value # "+15552345678" +``` + +--- + +## Contract + +```python +contract = Phone.create_contract( + default_country=None, # str | None — uppercase alpha-2, e.g. "US" + output_format=None, # "e164" (default), "rfc3966", "national" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- When `default_country` is `None`, national-shaped input is recognized but never validated → `INVALID`. International, `00`-prefix, and `tel:` forms validate without it because the country code is in the number itself. +- `default_country` must be uppercase alpha-2 when present; otherwise `ContractError` at construction. + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `+1 555 123 4567` | any | `SUCCESS` | → `+15551234567` | +| `(555) 234-5678` | defaults (`default_country=None`) | `INVALID` | recognized but needs a default country to validate | +| `(555) 234-5678` | `default_country="US"` | `SUCCESS` | → `+15552345678` | +| `hello` | any | `MISSING` | no phone pattern | +| Two distinct numbers | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G1[E.164 grammar] + A --> G2[00-prefix grammar] + A --> G3[tel-URI grammar] + A --> G4[National grammar] + G1 & G2 & G3 --> R1[E.164 / RFC 3966 rules
embed country code] + G4 --> R2{NANP / E.164 country rules
need default_country?} + R1 --> OK[SUCCESS] + R2 -->|with default_country| OK + R2 -->|without| INV[INVALID] + G1 & G2 & G3 & G4 -->|nothing matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import Phone +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c_intl = Phone.create_contract() +c_us = Phone.create_contract(default_country="US") +c_rfc = Phone.create_contract(output_format="rfc3966", default_country="US") + +rows = ["+1 555 123 4567", "(555) 234-5678", "0044 20 7946 0958", "hello"] + +for text in rows: + for label, c in [("intl", c_intl), ("US", c_us), ("rfc3966+US", c_rfc)]: + try: + r = paxman.canonicalize(text, c) + except Exception as e: + print(f"{text!r:25} [{label}] → exception {type(e).__name__}") + continue + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:25} [{label:12}] → {r.status.value:10} {val!r}") +``` + +--- + +## Provenance + +- **ITU-T E.164** — international numbering plan. +- **RFC 3966** — `tel:` URI. +- **NANP** — North American Numbering Plan (when relevant). + +See also: [Execution Result](../concepts/execution-result.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/si-unit.md b/docs/user/capabilities/si-unit.md new file mode 100644 index 00000000..c4f4e085 --- /dev/null +++ b/docs/user/capabilities/si-unit.md @@ -0,0 +1,116 @@ +# SI Unit + +Canonicalizes **one SI unit expression** per call — a symbol, a name, or a product/quotient compound — to its canonical symbol. + +> **In plain language:** give it `"Kilogram"`, `"megahertz"`, or `"m/s²"` and it hands back `"kg"`, `"MHz"`, or `"m/s2"` if the SI Brochure or ISO 80000 says that expression is a valid unit. Quantities like `"25°C"` are not its domain (the number is not an SI unit). + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Symbols (`kg`, `Pa`, `kΩ`, `m/s²`) | Quantities / magnitudes (`25°C`, `5 kg`) — `MISSING`; identity-only, no amounts | +| Names (`Kilogram`, `megahertz`) — case-insensitive, hyphen-tolerant | Name-compounds (`metre per second`) — words recognized separately → `AMBIGUOUS` | +| Prefixed symbols and names (`MHz`, `kPa`) | Symbol-prefix spacing (`k g`) — `INVALID`, never accepted | +| Product/quotient compounds (`m/s²`, `kg·m/s²`, `kPa`) | Multi-solidus without flag (`kg/m/s`) — `INVALID` unless `allow_multi_solidus=True` | +| Spoken word-prefix forms (`kilo gram`) — only when `allow_split_word_prefixes=True` | `kilo gram` without that flag — `INVALID` | + +--- + +## Canonical output + +Single format — the canonical symbol form. + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `symbol` / `None` / `"default"` | canonical symbol (`kg`, `MHz`, `m/s2`) | + +```python +from paxman.capabilities import SIUnit +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("Kilogram", SIUnit.create_contract()).canonicalized_value # "kg" +paxman.canonicalize("megahertz", SIUnit.create_contract()).canonicalized_value # "MHz" +paxman.canonicalize("m/s²", SIUnit.create_contract()).canonicalized_value # "m/s2" +``` + +--- + +## Contract + +```python +contract = SIUnit.create_contract( + allow_split_word_prefixes=False, # bool, default False — "kilo gram" → "kg" when True + allow_multi_solidus=False, # bool, default False — preserve "kg/m/s" when True + output_format=None, # "symbol" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- With `allow_split_word_prefixes=False` (default), `kilo gram` is recognized as `INVALID` — it is not a valid SI identity. Symbol-prefix spacing like `k g` is always `INVALID`; there is no flag for it. +- With `allow_multi_solidus=False` (default) per ISO 80000-1 §6.6.2, more than one top-level solidus is `INVALID`. Set `True` to preserve the legacy accept-multi-solidus behavior. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `Kilogram` | defaults | `SUCCESS` | → `kg` | +| `m/s²` | defaults | `SUCCESS` | compound → `m/s2` | +| `metre per second` | defaults | `AMBIGUOUS` | words recognized separately, competing groupings | +| `25°C` | any | `MISSING` | quantity, not an identity-only unit | +| `kilo gram` | defaults | `INVALID` | needs `allow_split_word_prefixes` | +| `kilo gram` | `allow_split_word_prefixes=True` | `SUCCESS` | → `kg` | +| `k g` | any | `INVALID` | symbol-prefix spacing always rejected | +| `kg/m/s` | defaults | `INVALID` | multi-solidus without flag | +| `kg/m/s` | `allow_multi_solidus=True` | `SUCCESS` | preserved | + +```mermaid +flowchart TB + A[Text e.g. kilo gram] --> G1[symbol grammar] + A --> G2[name grammar] + A --> G3[compound grammar] + G1 & G2 & G3 --> R{Rules: BIPM SI Brochure 9th ed.
ISO 80000-1 compounds
split-prefix rescue} + R -->|one value| OK[SUCCESS
symbol form] + R -->|name-compound
words separate| AMB[AMBIGUOUS] + R -->|needs allow_split_word_prefixes
or allow_multi_solidus| INV[INVALID] + G1 & G2 & G3 -->|nothing| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style AMB fill:#fff8e1,stroke:#d4a017 + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import SIUnit +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c = SIUnit.create_contract() +c_spoken = SIUnit.create_contract(allow_split_word_prefixes=True) +c_multi = SIUnit.create_contract(allow_multi_solidus=True) + +for text in ["Kilogram", "m/s²", "metre per second", "kilo gram", "k g", "kg/m/s", "25°C"]: + for label, cc in [("default", c), ("spoken", c_spoken), ("multi", c_multi)]: + r = paxman.canonicalize(text, cc) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:18} [{label:8}] → {r.status.value:10} {val!r}") +``` + +--- + +## Provenance + +- **BIPM SI Brochure (9th ed.)** — base units, derived units, non-SI units accepted for use, prefixes, names. +- **ISO 80000-1** — compound expressions. + +See also: [Execution Result](../concepts/execution-result.md), [Candidates & Ambiguity](../concepts/candidates-and-ambiguity.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/capabilities/url.md b/docs/user/capabilities/url.md new file mode 100644 index 00000000..6a1a33dc --- /dev/null +++ b/docs/user/capabilities/url.md @@ -0,0 +1,115 @@ +# URL + +Canonicalizes **one absolute URI / IRI** per call per the **WHATWG URL Standard** (plus UTS #46 IDNA for internationalized hosts), preserving percent-encoding byte-for-byte. + +> **In plain language:** give it `"HTTPS://Example.COM:443/path/../other"` and it hands back `"https://example.com/other"` — scheme and host lowercased, default port removed, dot segments resolved. Opaque schemes like `mailto:` are left verbatim. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Absolute URIs/IRIs with a scheme (`https://example.com`, `mailto:user@example.com`) | Relative references (`/path/../other`, `?q=1`, `#frag`) — `MISSING` | +| Internationalized hosts (`http://münchen.de`) — IDNA via UTS #46 | Plain domains without a scheme — not an *absolute URI* | +| Any absolute URI including opaque (non-special) schemes | | + +--- + +## Canonical output + +Single format — the WHATWG URL serialization (identity formatter). Characteristic normalizations: + +- Scheme and host lowercased. +- Default port removed (`https://example.com:443` → `https://example.com`). +- Dot segments resolved (`/path/../other` → `/other`). +- Internationalized hosts mapped via UTS #46 (`münchen.de` → `xn--mnchen-3ya.de`). +- Percent-encoding preserved byte-for-byte. +- Opaque schemes (`mailto:`, etc.) returned verbatim. + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `url` / `None` / `"default"` | WHATWG serialization | + +```python +from paxman.capabilities import URL +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("HTTPS://Example.COM:443/path/../other", URL.create_contract()).canonicalized_value # "https://example.com/other" +paxman.canonicalize("mailto:user@example.com", URL.create_contract()).canonicalized_value # "mailto:user@example.com" (opaque → verbatim) +paxman.canonicalize("http://münchen.de", URL.create_contract()).canonicalized_value # "http://xn--mnchen-3ya.de/" +``` + +--- + +## Contract + +```python +contract = URL.create_contract( + output_format=None, # "url" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +URL has no capability-specific flags in the current release — every recognized URL is validated by the single WHATWG URL Standard rule. + +--- + +## Statuses + +| Input | Status | Why | +|-------|--------|-----| +| `https://example.com/other` | `SUCCESS` | WHATWG-serialized | +| `HTTPS://Example.COM:443/path/../other` | `SUCCESS` | → `https://example.com/other` (lowercased, port removed, dot resolved) | +| `http://münchen.de` | `SUCCESS` | IDN → `http://xn--mnchen-3ya.de/` | +| `not a url` | `MISSING` | no absolute-URI pattern | +| `//example.com/path` (relative) | `MISSING` | no scheme, not absolute | +| Two distinct URLs in one call | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G[absolute-uri grammar] + G --> R{WHATWG URL Standard rule} + R -->|absolute URI| OK[SUCCESS
WHATWG serialization] + G -->|no absolute pattern| MISS[MISSING] + A -->|two distinct URLs| ERR[MultipleMentionsError
split first] + + style OK fill:#e6ffed,stroke:#2d8a4e + style MISS fill:#fff5f5,stroke:#cc3333 + style ERR fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import URL + +paxman.register_all_shipped() +contract = URL.create_contract() + +rows = [ + "HTTPS://Example.COM:443/path/../other", + "mailto:user@example.com", + "http://münchen.de", + "not a url", + "/relative/path", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value or "—" + print(f"{text!r:45} → {r.status.value:10} {val!r} span={r.span}") +``` + +--- + +## Provenance + +- **WHATWG URL Standard** — absolute-URI / IRI validation and serialization. +- **UTS #46 / IDNA** — mapping of internationalized hosts. + +See also: [Execution Result](../concepts/execution-result.md), [Provenance](../concepts/provenance.md), [Segmentation](../../recipes/segmentation.md). diff --git a/docs/user/concepts/candidates-and-ambiguity.md b/docs/user/concepts/candidates-and-ambiguity.md new file mode 100644 index 00000000..a04e8a87 --- /dev/null +++ b/docs/user/concepts/candidates-and-ambiguity.md @@ -0,0 +1,147 @@ +# Candidates & Ambiguity + +One input can legitimately mean two different things. Paxman does not guess — it shows you the disagreement. This page explains what a **candidate** is, how `AMBIGUOUS` differs from other statuses, and what to do about it. + +--- + +## What a candidate is + +A **candidate** is one validated answer: a canonical string plus the evidence that produced it. + +```python +candidate.value # e.g. "2026-01-02" +candidate.recognition_rule # which grammar spotted it +candidate.validation_rule # which rule (spec section) validated it +candidate.span # where in the input it sat +candidate.provenance # which authority vouches for it +``` + +Candidates are deduplicated by `(value, recognition_rule, validation_rule)`. If two rules happen to converge on the same string, they collapse to one distinct value — that is agreement, not ambiguity (see [Execution Result](execution-result.md)). + +```mermaid +flowchart TB + M[One mention
'01/02/2026'] --> G1[ISO/US grammar] + M --> G2[European grammar] + G1 --> R1[Rule: US federal calendar] + G2 --> R2[Rule: EN 50160 calendar] + R1 --> C1[Candidate
2026-01-02
provenance: US spec] + R2 --> C2[Candidate
2026-02-01
provenance: EN 50160] + + style M fill:#eef6ff,stroke:#4a90d9 + style C1 fill:#fff8e1,stroke:#d4a017 + style C2 fill:#fff8e1,stroke:#d4a017 +``` + +--- + +## When ambiguity happens + +`AMBIGUOUS` means: **one mention, two or more distinct canonical values**, each validated by a different specification. + +```mermaid +flowchart TB + Q{How many
distinct values
for one mention?} + Q -->|1| S[SUCCESS
one answer] + Q -->|2+| A[AMBIGUOUS
no single answer] + Q -->|0 with recognitions| I[INVALID] + Q -->|0 no recognitions| M[MISSING] + + style S fill:#e6ffed,stroke:#2d8a4e + style A fill:#fff8e1,stroke:#d4a017 + style I fill:#fff5f5,stroke:#cc3333 + style M fill:#fff5f5,stroke:#cc3333 +``` + +Concrete examples: + +| Input | Capability | What happens | Status | +|-------|------------|--------------|--------| +| `01/02/2026` | Date | US reads `2026-01-02`, European reads `2026-02-01` | `AMBIGUOUS` | +| `metre per second` | SI Unit | Word form is not a compound — words recognized separately, rules disagree on grouping | `AMBIGUOUS` | +| `2026-01-15` | Date | Only ISO grammar's reading validates | `SUCCESS` | + +`AMBIGUOUS` is a **domain signal**, not a failure. The input is real; the specs genuinely conflict. Contrast with: + +- `MISSING` — no grammar matched at all (the text does not look like this entity). +- `INVALID` — a grammar matched but no spec accepted it (looks like the entity but is malformed). +- `MultipleMentionsError` — two **separate** mentions with different values in one call (see the [Segmentation Recipe](../../recipes/segmentation.md)). That raises an exception rather than returning a status, because it signals you need to split the input first. + +--- + +## How to handle `AMBIGUOUS` + +You have three tools, all through the contract (see [Contracts](contracts.md)): + +### 1. Pin to a specific spec + +If you know which interpretation you want, narrow the rules: + +```python +from paxman.capabilities import Date + +# Only the US reading +contract = Date.create_contract(pinned_rules=["Section 4.3.1-calendar-date-us"]) +result = paxman.canonicalize("01/02/2026", contract) +# may become SUCCESS, or INVALID if no pinned rule validates +``` + +Use `pinned_rules` when you want to enforce a single authority. Note that `pinned_rules` overrides `excluded_rules` and that `year` still filters after pinning. + +### 2. Filter by time + +```python +contract = Date.create_contract(year=2019) # only rules published ≤ 2019 +``` + +### 3. Surface the disagreement to the user or log + +Often the right behavior is to surface the candidates, not to suppress them: + +```python +from paxman.core.domain import Resolution + +result = paxman.canonicalize("01/02/2026", Date.create_contract()) +if result.status == Resolution.AMBIGUOUS: + for c in result.candidates: + p = c.provenance[0] + print(f" {c.value!r} via {c.validation_rule} ({p.authority}: {p.specification_name}) span={c.span}") +``` + +Output: + +``` + '2026-01-02' via Section X ... (Authority A: Spec A) span=(0, 10) + '2026-02-01' via Section Y ... (Authority B: Spec B) span=(0, 10) +``` + +That evidence is the provenance story for each reading (see [Provenance](provenance.md)) — it is what lets you or your user decide, rather than Paxman deciding silently. + +### Notebook pattern — keep ambiguous rows for review + +```python +import paxman +from paxman.capabilities import Date +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Date.create_contract() + +rows = ["2026-01-15", "01/02/2026", "not a date"] +for text in rows: + r = paxman.canonicalize(text, contract) + if r.status == Resolution.SUCCESS: + print(f"{text!r:15} → {r.canonicalized_value}") + elif r.status == Resolution.AMBIGUOUS: + vals = sorted({c.value for c in r.candidates}) + print(f"{text!r:15} → AMBIGUOUS {vals} (candidates={len(r.candidates)})") + else: + print(f"{text!r:15} → {r.status.value}") +``` + +--- + +## In plain language + +Candidates are like second opinions from different experts looking at the same X-ray. If both experts agree, you have one answer. If they disagree and both are credible, the honest report is *these experts disagree, here is why* — not a silent pick. `AMBIGUOUS` is that honest report, and `candidates` is the list of opinions with their citations so you can decide. + +Next: [Errors →](errors.md) — what raises an exception instead of returning a status. diff --git a/docs/user/concepts/capabilities.md b/docs/user/concepts/capabilities.md new file mode 100644 index 00000000..1fbc63d8 --- /dev/null +++ b/docs/user/concepts/capabilities.md @@ -0,0 +1,108 @@ +# Capabilities + +A **capability** is one kind of identifier Paxman knows how to canonicalize. Each capability is a self-contained package — its own patterns, its own specifications, its own rendering — that plugs into the shared pipeline. + +You pick a capability by picking a contract (see [Contracts](contracts.md)). The two are paired: `Email.create_contract()` selects the Email capability, `Country.create_contract()` selects Country, and so on. + +--- + +## How selection works + +```mermaid +flowchart LR + A[You write
Email.create_contract] --> B[Contract carries
capability_name = email] + B --> C[Engine looks up
registry by that name] + C --> D[Email grammars + rules run] + D --> E[ExecutionResult] + + style B fill:#fff8e1,stroke:#d4a017 + style C fill:#eef6ff,stroke:#4a90d9 + style E fill:#f0fff0,stroke:#2d8a4e +``` + +1. Each capability has a lowercase name (e.g. `"email"`, `"country"`, `"url"`). +2. Its contract carries that name in `capability_name`. +3. `paxman.canonicalize(text, contract)` looks up the capability by that name in the registry. +4. Only that capability's grammars and rules run — others stay unloaded. + +This keeps imports cheap: `from paxman.capabilities import Email` loads only Email, not every capability. + +--- + +## What a capability contains + +```mermaid +flowchart TB + subgraph Capability [e.g. Email] + N[Notation
internal shape of a mention] + G[Grammars
how to spot the pattern] + R[Rules
which specs accept it] + C[Contract
how you configure it] + F[format_value
how the answer is rendered] + end + + G --> N --> R --> F + + style N fill:#eef6ff,stroke:#4a90d9 + style G fill:#f3e8ff,stroke:#7c3aed + style R fill:#fff8e1,stroke:#d4a017 + style F fill:#f0fff0,stroke:#2d8a4e +``` + +- **Notation** — internal shape (not part of the public API). +- **Grammars** — recognizers that scan your text and emit span-bearing matches. Pure syntax — no spec judgment. +- **Rules** — validators that check each match against an authoritative spec (RFC, ISO standard, etc.) and produce a canonical value plus provenance. +- **Contract** — the user-facing configuration (see [Contracts](contracts.md)). +- **`format_value`** — the sole rendering step, controlled by `output_format` on the contract. + +You never interact with notations, grammars, or rules directly — you configure them through the contract and read their outcome in the result. + +--- + +## Capabilities available today + +The set below reflects the **current release** and is intentionally not presented as a final count. New capabilities are added in minor releases — always check `paxman.capabilities` or the release notes for the latest list. + +| Capability | What it canonicalizes | Key formats it recognizes | Canonical form you get back | +|------------|----------------------|---------------------------|-----------------------------| +| **Country** | Country codes and names | alpha-2, alpha-3, numeric, names; optional localized (CLDR) and historical names | alpha-2 code (`"US"`), or other form via `output_format` | +| **Currency** | Currency identifiers (no amounts) | ISO 4217 alpha-3 codes, CLDR symbols and display names | uppercase alpha-3 code (`"USD"`) | +| **Date** | Calendar dates | ISO 8601, slash-ISO, US, European | ISO `YYYY-MM-DD` by default | +| **Email** | Email addresses | standard, obfuscated (`user at domain dot com`), localhost | lowercased `addr-spec` | +| **IP** | IP addresses | IPv4, IPv6 (optionally disabled) | normalized address (IPv6 per RFC 5952) | +| **ISBN** | ISBN identifiers | ISBN-13 and ISBN-10 (legacy → ISBN-13) | bare 13-digit form or `hyphenated` | +| **Money** | Money amounts with currency | codes, symbols, or names adjacent to an amount | `CODE amount` padded to minor units | +| **Phone** | Phone numbers | E.164, tel-URI, 00-prefix international, NANP national | E.164 (`+15551234567`) or other via `output_format` | +| **SI Unit** | SI unit expressions | symbols, names, product/quotient compounds | canonical symbol form (`"kg"`, `"m/s2"`) | +| **URL** | Absolute URIs / IRIs | absolute URIs (WHATWG URL Standard) | WHATWG serialization (lowercased host, etc.) | + +> This table is an overview. Each capability's contract documents its specific flags (e.g. `include_localized` for Country, `default_country` for Phone). See [Contracts](contracts.md) and the [README](../../../README.md) examples for per-capability details; each row is expanded into its own guide under [Capabilities](../capabilities/index.md). + +--- + +## Registering capabilities + +Before the first `canonicalize()` call, register: + +```python +import paxman +from paxman.capabilities import Email, Country + +# Option A — everything shipped in this release +paxman.register_all_shipped() + +# Option B — explicit, dependency-clear +paxman.register_capability(Email()) +paxman.register_capability(Country()) +``` + +- Registration must complete from **a single thread** before the first call. +- After the first call the registry **freezes** — further `register_*` calls raise `CapabilityError`. Reads from any thread are then safe. + +--- + +## In plain language + +Think of a capability like a department that handles one kind of paperwork. The Country department knows passports and country codes; the Email department knows addresses. You hand your paper to the right department by handing it a contract stamped with that department's name. Only that department looks at it. Other departments never see it. + +Next: [Contracts →](contracts.md) — how to configure what the chosen capability does. diff --git a/docs/user/concepts/contracts.md b/docs/user/concepts/contracts.md new file mode 100644 index 00000000..7b32082f --- /dev/null +++ b/docs/user/concepts/contracts.md @@ -0,0 +1,175 @@ +# Contracts + +A **contract** is the configuration object you pass alongside your text to `paxman.canonicalize()`. It answers three questions: *which capability to use, which patterns should count, and how the answer should look*. Every capability provides the same contract surface so you only learn it once. + +--- + +## What a contract is + +You never construct a contract by hand. Each capability exposes a typed factory: + +```python +from paxman.capabilities import Email, Country + +email_contract = Email.create_contract() +country_contract = Country.create_contract(include_localized=True) +``` + +The returned object is a frozen dataclass — you can inspect it, but not mutate it. Pass it straight to `canonicalize()`: + +```python +result = paxman.canonicalize("Alemania", country_contract) +``` + +The contract's `capability_name` field (set automatically) is how the engine selects the matching capability — see [Capabilities](capabilities.md). + +--- + +## The unanimous surface + +Every contract — no matter the capability — carries the same common fields. This is intentional: the shape is homogeneous so tooling, notebooks, and future capabilities all behave the same way. + +```mermaid +flowchart TB + subgraph Common[Present on every contract] + A[excluded_rules] + B[pinned_rules] + C[year] + D[output_format] + E[extra_grammars] + end + subgraph Specific[Plus per-capability flags] + F[include_obfuscated
include_localized
default_country
...] + end + Common --> Specific + + style Common fill:#fff8e1,stroke:#d4a017 + style Specific fill:#eef6ff,stroke:#4a90d9 +``` + +| Field | Type | Default | What it does | +|-------|------|---------|--------------| +| `capability_name` | `str` | set by factory | Which capability this contract selects. You never set it — the factory does. | +| `excluded_rules` | `tuple[str, ...]` | `()` | Rule names to skip during validation. | +| `pinned_rules` | `tuple[str, ...] \| None` | `None` | If set, **only** these rules run (overrides `excluded_rules`). `()` pins to nothing. | +| `year` | `int \| None` | `None` | Temporal filter — only rules published in or before this year run. | +| `output_format` | `str \| None` | `None` → default | How the canonical value is rendered (see below). | +| `extra_grammars` | `tuple[str, ...]` | `()` | Community grammar names to opt in (advanced — see extending docs in a later round). | + +**Precedence:** when `pinned_rules` is non-`None` it wins over `excluded_rules`; `year` is applied after either pinning or exclusion. + +### `output_format` — always optional + +`output_format` is the **only** presentation knob and it is always optional: + +- `None`, `"default"`, and the capability's `DEFAULT_OUTPUT_FORMAT` all resolve to the default rendering. +- Any value in `OFFERED_OUTPUT_FORMATS` resolves to itself (e.g. `"hyphenated"` for ISBN, `"rfc3966"` for Phone). +- Anything else raises `ContractError`. + +Validation of the canonical value is never affected by `output_format` — formatting happens after validation. + +--- + +## Per-capability flags + +Beyond the common fields, each contract adds flags that make sense for its domain. You will discover them through the factory's signature and your editor's autocomplete; the table below summarizes the current release. + +| Capability | Flag | Type | Default | Meaning | +|------------|------|------|---------|---------| +| Email | `include_obfuscated` | `bool` | `False` | Recognize `user at domain dot com` style addresses | +| Email | `include_localhost` | `bool` | `True` | Recognize `admin@localhost` | +| Country | `include_localized` | `bool` | `False` | Recognize CLDR multilingual names (e.g. `Alemania` → `DE`) | +| Country | `include_historical` | `bool` | `False` | Recognize deprecated/historical names (e.g. `Burma` → `BU`) | +| IP | `include_ipv6` | `bool` | `True` | Recognize IPv6 addresses | +| ISBN | `include_isbn10` | `bool` | `True` | Recognize legacy ISBN-10 | +| ISBN | `include_range_validation` | `bool` | `False` | Enable registrant-range provenance | +| Currency | `default_currency` | `str \| None` | `None` | Resolve shared bare symbols (`$`) to this code when it is one of that symbol's candidates | +| Money | `dollar_sign_currency` | `str \| None` | `None` | Resolve bare `$` amounts to this code | +| Money | `precision` | `str` | `"strict"` | Over-precision handling: `strict` / `truncate` / `round` | +| Phone | `default_country` | `str \| None` | `None` | Resolve national numbers as if dialed in this country (`"US"`) | +| SI Unit | `allow_split_word_prefixes` | `bool` | `False` | Merge `kilo gram` → `kg` | +| SI Unit | `allow_multi_solidus` | `bool` | `False` | Preserve `kg/m/s` instead of rejecting it | +| Date | `two_digit_base_year` | `int \| None` | `None` | Base year for 2-digit year expansion | +| Date / ISBN / Phone / … | `output_format` | `str` | capability default | See capability-specific offered formats | + +> This table reflects the **current release**. As new capabilities are added, each will document its own flags in the same way — the common fields above stay the same. + +--- + +## How flags map to the pipeline + +A useful mental model: the two groups of flags gate the two groups of pipeline stages (see [Pipeline](pipeline.md)). + +```mermaid +flowchart LR + subgraph Recognition + direction TB + R1[Grammars scan text] + RF[include_* flags
toggle grammars] + R1 -.-> |disabled grammar → MISSING| RF + end + subgraph Validation + direction TB + V1[Rules check each match] + VF[requires_features
+ excluded/pinned/year] + V1 -.-> |dropped rule → INVALID| VF + end + Recognition --> Validation + + style RF fill:#f3e8ff,stroke:#7c3aed + style VF fill:#fff8e1,stroke:#d4a017 +``` + +- **Input-shape flags** (`include_*`) toggle **grammars** — a disabled grammar simply never sees the input, so the status becomes `MISSING` rather than `INVALID`. +- **Authority flags** (`requires_features` on rules, plus `excluded_rules`/`pinned_rules`/`year`) gate **rules** — a dropped rule means the match was recognized but not accepted, so the status becomes `INVALID`. + +You never branch inside a rule's logic based on `include_*` — the flag's presence or absence decides whether the rule runs at all. + +--- + +## Common recipes + +### Narrow to specific specs + +```python +from paxman.capabilities import Email + +# Only RFC 5322, no localhost +contract = Email.create_contract(pinned_rules=["Section 3.4.1-addr-spec"]) +``` + +### Exclude one spec + +```python +contract = Email.create_contract(excluded_rules=["Section 6.3-localhost"]) +``` + +### Time-travel to an older spec set + +```python +from paxman.capabilities import Date + +contract = Date.create_contract(year=2019) # only rules published ≤ 2019 +``` + +### Choose a rendering + +```python +from paxman.capabilities import Phone, ISBN + +Phone.create_contract(output_format="rfc3966") # tel:+15551234567 +ISBN.create_contract(output_format="hyphenated") # 978-0-11-000222-4 +Phone.create_contract(output_format="national") # national form +ISBN.create_contract(output_format="isbn13") # bare digits (default) +# None / "default" also resolve to the default for any capability +``` + +Any non-offered value raises `ContractError` — you get a fast, typed failure instead of a silently wrong rendering. + +--- + +## In plain language + +Think of a contract as a work order you hand to a department. It says: *here is the kind of job (capability), here are the tools you may use (which grammars and rules), and here is how I want the answer formatted (output_format)*. The department may have extra switches that only make sense for its work (e.g. Country's "also accept historical names"), but the top of the form — excluded/pinned rules, year, output format — looks the same in every department. + +Next: [Pipeline →](pipeline.md) — what Paxman does with that work order. diff --git a/docs/user/concepts/errors.md b/docs/user/concepts/errors.md new file mode 100644 index 00000000..5f78d0d1 --- /dev/null +++ b/docs/user/concepts/errors.md @@ -0,0 +1,117 @@ +# Errors + +Paxman uses two different signals for "something went wrong": **statuses** for domain answers and **exceptions** for misuse. Knowing which is which keeps your code and notebooks simple. + +--- + +## Statuses vs exceptions + +```mermaid +flowchart TB + A[You call canonicalize] --> Q{Is the setup
valid?} + Q -->|contract malformed
or registry frozen
or grammar failed| E[Exception raised] + Q -->|setup OK| P[Pipeline runs] + P --> S[ExecutionResult
status is MISSING
INVALID / SUCCESS
or AMBIGUOUS] + + style E fill:#fff5f5,stroke:#cc3333 + style S fill:#f0fff0,stroke:#2d8a4e +``` + +- A **status** (`MISSING`, `INVALID`, `AMBIGUOUS`, `SUCCESS`) is a normal domain answer. It means the pipeline ran, considered the input, and has a well-defined conclusion — even when that conclusion is "there is no valid answer." Handle it by branching on `result.status`. +- An **exception** means the call was not valid in the first place — a contract was malformed, the registry was frozen, or an internal grammar/rule failed unexpectedly. These are not values to canonicalize; they are programming or setup errors to fix. + +Rule of thumb: **check `status` in normal code; catch exceptions only around setup and at the outer boundary of a batch.** + +--- + +## The exception hierarchy + +All Paxman exceptions inherit from `PaxmanError`. + +| Exception | When it is raised | What to do | +|-----------|-------------------|------------| +| `CapabilityError` | Unknown capability name, duplicate capability/grammar name, or registering after the registry has frozen | Register before the first `canonicalize()`; check spelling; avoid duplicate names | +| `ContractError` | Malformed contract — unknown `pinned_rules` entry, unknown `output_format`, unknown semantics in `extra_grammars`, or a rule's required feature missing from the contract | Fix the contract — see [Contracts](contracts.md) | +| `MultipleMentionsError` | One call contained two or more **separate** mentions that resolved to **different** values — un-segmented multi-entity input | Split the input first — see the [Segmentation Recipe](../../recipes/segmentation.md) | +| `RecognitionError` | A grammar failed structurally (exception inside `recognize()`) or returned a malformed match (bad span or `raw_text`) | Treat as a bug in a grammar (shipped or community); carries `rule` and `original_error` | +| `ValidationError` | A rule raised unexpectedly inside `matches()` / `normalize()` | Treat as a bug in a rule; carries `rule` and `original_error` | + +> `RecognitionError` and `ValidationError` carry `rule` (the grammar/rule name) and render as `"[rule] message"`. On structural recognition failures `original_error` is `None`; on internal failures it is the underlying exception. + +--- + +## Typical handling + +### Setup — fail fast + +```python +import paxman +from paxman.capabilities import Email +from paxman.core.errors import CapabilityError, ContractError + +try: + paxman.register_all_shipped() + contract = Email.create_contract(output_format="typo") # not offered +except ContractError as e: + print(f"Bad contract: {e}") +except CapabilityError as e: + print(f"Registration problem: {e}") +``` + +An unknown `output_format` always raises `ContractError` immediately — never a silent fallback. + +### Canonicalization loop — branch on status + +```python +from paxman.core.domain import Resolution + +result = paxman.canonicalize("Contact user@Example.com", contract) + +if result.status == Resolution.SUCCESS: + use(result.canonicalized_value) +elif result.status in (Resolution.MISSING, Resolution.INVALID): + # domain answer — skip or flag, no exception to catch + log(result.status, result.candidates) +elif result.status == Resolution.AMBIGUOUS: + surface(result.candidates) # see Candidates & Ambiguity +``` + +### Batch with segmentation — handle `MultipleMentionsError` + +```python +from paxman.core.errors import MultipleMentionsError + +try: + result = paxman.canonicalize("alice@example.com and bob@example.org", contract) +except MultipleMentionsError as e: + # Input contained two different mentions — split before retrying. + # See the Segmentation Recipe for the loop pattern. + print(f"Need to segment: {e}") +``` + +### Narrow `RecognitionError` / `ValidationError` + +These indicate a bug in a grammar or rule (shipped or community). Catch them at the outer boundary; do not treat them as domain statuses. + +```python +from paxman.core.errors import RecognitionError, ValidationError + +try: + result = paxman.canonicalize(text, contract) +except (RecognitionError, ValidationError) as e: + print(f"[{e.rule}] pipeline bug: {e} — original: {e.original_error}") +``` + +--- + +## In plain language + +Statuses are the library saying *I looked, and here is what the specs say* — even when the answer is "nothing there" or "two specs disagree." Exceptions are the library saying *I could not even look — the request was not valid* (wrong setup, wrong contract, or two different things crammed into one slot). Handle statuses in your normal flow; handle exceptions as setup/bug signals. + +--- + +## Where to go from here + +- [Getting Started](../getting-started.md) — register and call correctly the first time +- [Concepts — Overview](index.md) — rebuild the full mental model +- [Segmentation Recipe](../../recipes/segmentation.md) — the correct way to handle text with multiple entities diff --git a/docs/user/concepts/execution-result.md b/docs/user/concepts/execution-result.md new file mode 100644 index 00000000..f421710f --- /dev/null +++ b/docs/user/concepts/execution-result.md @@ -0,0 +1,149 @@ +# Execution Result + +`paxman.canonicalize()` always returns the same object shape — an `ExecutionResult` — whether the answer is a clean success, a missing input, an invalid value, or a genuine ambiguity. Learn to read it once, and you can handle every capability the same way. + +--- + +## The shape + +```python +result.status # Resolution enum +result.canonicalized_value # str | None +result.candidates # tuple[Candidate, ...] +result.span # tuple[int, int] | None +result.version_stamp # VersionStamp — which Paxman build produced this +result.contract # the contract you passed in, echoed back +``` + +For each `Candidate`: + +```python +candidate.value # canonical string this candidate proposes +candidate.recognition_rule # grammar name that spotted it (e.g. "standard_recognition") +candidate.validation_rule # rule name that validated it (e.g. "Section 3.4.1-addr-spec") +candidate.span # (start, end) of this candidate's match in the input, or None +candidate.provenance # tuple[Provenance, ...] — the spec(s) that vouched for it +``` + +`version_stamp` is currently `VersionStamp(paxman_version="…")`; it records the installed build so results are auditable. `contract` is echoed back for logging and debugging. + +--- + +## The four statuses + +```mermaid +stateDiagram-v2 + [*] --> MISSING: no grammar matched + [*] --> INVALID: grammar matched
but no rule accepted + [*] --> SUCCESS: one distinct value + [*] --> AMBIGUOUS: one mention,
two+ distinct values + + SUCCESS --> [*]: canonicalized_value is str
span is set + MISSING --> [*]: canonicalized_value is None
span is None + INVALID --> [*]: canonicalized_value is None
span is None + AMBIGUOUS --> [*]: canonicalized_value is None
span is None +``` + +| Status | What it means | `canonicalized_value` | `span` | `candidates` | +|--------|---------------|-----------------------|--------|--------------| +| `MISSING` | Nothing in the input looked like this kind of entity | `None` | `None` | empty | +| `INVALID` | Looked like it, but no authority validated it | `None` | `None` | empty | +| `SUCCESS` | Exactly one canonical value, validated | `str` | `(start, end)` of that value | one or more agreeing candidates | +| `AMBIGUOUS` | One mention, two or more authorities that disagree | `None` | `None` | two+ disagreeing candidates | + +`MISSING` vs `INVALID` is not a wording choice — it is a pipeline signal (see [Pipeline](pipeline.md)): `MISSING` means recognition found nothing; `INVALID` means recognition found something but validation rejected it. That tells you whether to try a different recognition flag or a different validation rule. + +**Python tip — always branch on `status`, not truthiness:** + +```python +from paxman.core.domain import Resolution + +if result.status == Resolution.SUCCESS: + use(result.canonicalized_value) # str +elif result.status == Resolution.MISSING: + # input does not contain this entity — skip or report + ... +elif result.status == Resolution.INVALID: + # looks like the entity but malformed — flag for review + ... +elif result.status == Resolution.AMBIGUOUS: + # one mention, conflicting specs — narrow contract or ask user + inspect(result.candidates) +``` + +`canonicalized_value` is `None` for every non-`SUCCESS` case by construction — check `status`, not whether the value is truthy. + +--- + +## `span` — where the answer sat + +`span` is a half-open `[start, end)` character range into the original text you passed to `canonicalize()`. It obeys `len(raw_text) == end - start`. + +```python +import paxman +from paxman.capabilities import Email + +paxman.register_all_shipped() +contract = Email.create_contract() +result = paxman.canonicalize("Contact user@Example.com for info", contract) + +print(result.canonicalized_value) # "user@example.com" +print(result.span) # (8, 24) +print("Contact user@Example.com for info"[8:24]) # "user@Example.com" +``` + +- On `SUCCESS`, `result.span` is the span of the single resolved value, and each `candidate.span` matches it. +- On `MISSING`, `INVALID`, and `AMBIGUOUS`, `result.span` is `None` — there is no single resolved mention to point to. For `AMBIGUOUS`, locate each competing mention via `candidate.span` on the individual candidates. + +This makes highlighting in UIs, logging, and downstream span-aware processing straightforward. + +--- + +## `candidates` — the full evidence + +`candidates` holds every validated `(value, recognition_rule, validation_rule, provenance, span)` tuple the pipeline produced, after deduplication by `(value, recognition_rule, validation_rule)`. + +- On `SUCCESS` with one agreeing value, `candidates` may still contain **multiple entries** that converged on that same string via different grammars or specs. That convergence is useful: it shows the answer is robust across rule/authorities. +- On `AMBIGUOUS`, `candidates` shows the disagreement — two or more distinct `value`s with their respective specs. +- On `MISSING` / `INVALID`, `candidates` is empty. + +```python +for c in result.candidates: + prov = c.provenance[0] + print(f"{c.value!r:20} via {c.validation_rule} " + f"({prov.authority}: {prov.specification_name}) " + f"span={c.span}") +``` + +--- + +## Notebook-friendly pattern + +```python +import paxman +from paxman.capabilities import Country +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Country.create_contract(include_localized=True) + +rows = ["United States", "Alemania", "not a country", "01/02/2026"] + +for text in rows: + # Use the capability that matches the column — example uses Country for the first three, + # so "01/02/2026" will be MISSING here (it contains no country pattern). + r = paxman.canonicalize(text, contract) + tag = r.status.value + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:20} {tag:10} {val!r:15} span={r.span}") +``` + +Check `status` first, use `canonicalized_value` only on `SUCCESS`, and fall back to `candidates` or `span` when you need to explain or highlight the outcome. + +--- + +## In plain language + +The execution result is like a lab report. It says *what the verdict was* (SUCCESS/MISSING/INVALID/AMBIGUOUS), *what the cleaned-up value is if there is one*, *where in the original text it was found*, and *which rulebooks were consulted*. You always get the same report format — only the verdict changes. + +Next: [Provenance →](provenance.md) — who vouches for the answer and how to cite it. diff --git a/docs/user/concepts/index.md b/docs/user/concepts/index.md new file mode 100644 index 00000000..bf73b832 --- /dev/null +++ b/docs/user/concepts/index.md @@ -0,0 +1,92 @@ +# Concepts — Overview + +If you understand the six ideas below, you understand Paxman. This page is the hub — start here, then dive into whichever concept you need. + +--- + +## The big picture in one diagram + +```mermaid +flowchart LR + subgraph You + A[Raw text + Contract] + end + + subgraph Paxman + B[Capability] + C[Pipeline] + D[ExecutionResult] + E[Provenance] + end + + A --> B --> C --> D + D --> E + D -.-> F{Status?} + F -->|SUCCESS| G[canonicalized_value] + F -->|MISSING
INVALID
AMBIGUOUS| H[no single value] + + style A fill:#eef6ff,stroke:#4a90d9 + style D fill:#f0fff0,stroke:#2d8a4e + style G fill:#e6ffed,stroke:#2d8a4e + style H fill:#fff5f5,stroke:#cc3333 +``` + +You provide **text + a contract** that selects a **capability**. Paxman runs its **pipeline** and returns an **execution result** whose status tells you whether there is a single canonical answer, and whose **provenance** tells you which specification vouches for it. + +--- + +## The six concepts + +| Concept | One-line summary | When you need it | +|---------|-----------------|------------------| +| [Capabilities](capabilities.md) | A capability is one kind of identifier Paxman knows how to canonicalize (email, country, URL, …). The set grows over time. | Choosing what to import, deciding whether Paxman covers your data | +| [Contracts](contracts.md) | A contract configures a capability — which patterns to look for, which specs to enforce, how to render the answer. | Enabling optional formats, pinning to a spec version, selecting an output form | +| [Pipeline](pipeline.md) | The three stages inside `canonicalize()`: recognition → validation → resolution. | Understanding why an input is `MISSING` vs `INVALID` vs `AMBIGUOUS` | +| [Execution Result](execution-result.md) | The object you get back: `status`, `canonicalized_value`, `candidates`, `span`, `version_stamp`. | Reading answers in code or a notebook | +| [Provenance](provenance.md) | The authority citation attached to every validated value — which spec, which version, which section. | Auditing, citing sources, comparing Paxman against another system | +| [Candidates & Ambiguity](candidates-and-ambiguity.md) | Why one input can produce multiple valid answers and how Paxman surfaces that without guessing. | Handling `AMBIGUOUS` in your application | +| [Errors](errors.md) | What raises an exception (misuse) vs what returns a status (domain answer). | Debugging setup and contract mistakes | + +A seventh page, [Errors](errors.md), covers the exception side — keep it bookmarked, but you will rarely need it once registration and contracts are correct. + +--- + +## How they fit together + +```mermaid +flowchart TB + Cap[Capabilities
what kinds
of things exist] + Con[Contracts
what you want
for this call] + Pipe[Pipeline
what Paxman does
with that choice] + Res[Execution Result
what you get back] + Prov[Provenance
who vouches
for the answer] + Cand[Candidates
all the
evidence] + + Cap --> Con --> Pipe --> Res + Res --> Prov + Res --> Cand + Cand -.-> Prov + + style Cap fill:#eef6ff,stroke:#4a90d9 + style Con fill:#fff8e1,stroke:#d4a017 + style Pipe fill:#f3e8ff,stroke:#7c3aed + style Res fill:#f0fff0,stroke:#2d8a4e +``` + +1. You pick a **capability** (e.g. Email) — this determines which patterns Paxman knows. +2. You build a **contract** for that capability — this narrows which patterns and specs are active for this call. +3. Paxman runs the **pipeline** (recognition, validation, resolution) using that contract. +4. You receive an **execution result** whose `status` is either `SUCCESS` (one canonical value), `MISSING`, `INVALID`, or `AMBIGUOUS`. +5. On `SUCCESS` that value carries **provenance**; in every case you can inspect **candidates** to see what the specs said. + +--- + +## Reading order + +- **New to Paxman?** Read in order: Capabilities → Contracts → Pipeline → Execution Result. Skim Provenance, Candidates, and Errors as needed. +- **Cleaning data in a notebook?** Jump to [Pipeline](pipeline.md) (to predict outcomes) and [Execution Result](execution-result.md) (to handle statuses), then [Candidates & Ambiguity](candidates-and-ambiguity.md) if you hit `AMBIGUOUS`. +- **Integrating into an app?** Read [Contracts](contracts.md) and [Errors](errors.md) carefully — they cover the knobs and the failure modes you need to handle. + +> **A note on the capability list:** the set of capabilities grows across releases. This hub lists examples from the current release; never treat a count in these docs as final. Check `paxman.capabilities` or the latest release notes for the current set. + +Next: [Capabilities →](capabilities.md) diff --git a/docs/user/concepts/pipeline.md b/docs/user/concepts/pipeline.md new file mode 100644 index 00000000..185f3bd2 --- /dev/null +++ b/docs/user/concepts/pipeline.md @@ -0,0 +1,151 @@ +# Pipeline + +Every `paxman.canonicalize(text, contract)` call runs the same deterministic pipeline. Understanding its three stages — **recognition → validation → resolution** — explains every status you will ever see. + +--- + +## The three stages at a glance + +```mermaid +flowchart TB + A[Raw text
+ Contract] --> B[Recognition
grammars scan the text] + B --> C[Validation
rules check each match
against a spec] + C --> D[Resolution
collect, deduplicate,
decide status] + D --> E[ExecutionResult
status + canonicalized_value
+ candidates + span] + + style A fill:#eef6ff,stroke:#4a90d9 + style B fill:#f3e8ff,stroke:#7c3aed + style C fill:#fff8e1,stroke:#d4a017 + style D fill:#e6ffed,stroke:#2d8a4e + style E fill:#f0fff0,stroke:#2d8a4e +``` + +- **Recognition** asks: *does this text even look like something this capability handles?* It finds patterns and notes exactly where they sit in the input (`span`). +- **Validation** asks: *does that pattern actually mean what it looks like, according to a real specification?* It checks each match against one or more specs and, when it passes, produces a canonical value and a provenance citation. +- **Resolution** asks: *given all the validated answers, what is the single result?* It deduplicates and decides the outcome. + +No stage does another stage's job. That separation is what makes results predictable and debuggable. + +--- + +## Stage 1 — Recognition + +Grammars scan your input for syntactic patterns. Each grammar is focused — e.g. for Email there are grammars for standard addresses, for `user at domain dot com` style obfuscation, and for `localhost` addresses. For Date there are grammars for ISO, US, European, and slash-ISO forms. + +Key properties: + +- Grammars return **span-bearing matches** — a half-open `[start, end)` range plus the exact substring (`raw_text`) it covers. This is how Paxman later tells you *where* the entity sat. +- Recognition is **syntax only** — it never judges correctness. `"not-an-email@"` would not match; `"user@example.com"` would, even if a later validation step will reject it. +- Overlapping matches from the **same** grammar are deduplicated by "longer wins" — a match fully contained in a longer one from the same grammar is dropped. Overlapping matches from **different** grammars are kept — that cross-grammar disagreement is meaningful (see [Candidates & Ambiguity](candidates-and-ambiguity.md)). +- Which grammars run is decided by your contract's **input-shape flags** (`include_*`) and by `active_grammars`. A disabled grammar never scans — the pipeline behaves as if it does not exist. + +```mermaid +flowchart LR + T[Input text
'Contact user at example dot com'] --> G1[Standard grammar
no match] + T --> G2[Obfuscated grammar
match if include_obfuscated=True] + T --> G3[Localhost grammar
no match] + G2 --> M[RecognitionMatch
span + notation] + + style G2 fill:#f3e8ff,stroke:#7c3aed + style M fill:#eef6ff,stroke:#4a90d9 +``` + +If this stage finds **nothing**, the final status is `MISSING` (without further stages running). That is why `include_obfuscated=False` on the input above returns `MISSING` — the only grammar that could have matched was not active. + +--- + +## Stage 2 — Validation + +Each recognition is checked against **rules** — each rule represents one section of one authoritative publication. For example, an Email match may be checked against RFC 5322's addr-spec section and RFC 6761's localhost section. + +```mermaid +flowchart TB + M[Recognized match] --> R1{Rule: RFC 5322
addr-spec} + M --> R2{Rule: RFC 6761
localhost} + R1 -->|matches| C1[Candidate:
user@example.com
provenance: IETF / RFC 5322] + R2 -->|no match| X1[dropped] + M --> R3{Other rules...} + + style C1 fill:#e6ffed,stroke:#2d8a4e + style X1 fill:#fff5f5,stroke:#cc3333 +``` + +- A rule's `matches()` answers *does this notation satisfy my spec?*; if yes, `normalize()` produces the canonical string and `provenance` says which spec vouches for it. +- A rule declares `target_semantics` — which grammars' output it is willing to judge. A match only meets the rules whose target includes its grammar's semantics. +- Which rules run is controlled by your contract's **authority gating**: `excluded_rules` / `pinned_rules`, `year` (temporal filter), and `requires_features` (a rule that needs `include_localized=True` stays dormant otherwise — see [Contracts](contracts.md)). +- `output_format` **never affects validation** — it only affects the final rendering step after validation succeeds. + +If recognition produced matches but **no rule accepts any of them**, the final status is `INVALID` — *recognized, but no specification validates it*. Example: `Alemania` with the Country capability and `include_localized=False` is recognized by the name grammar but no rule claims it, so it is `INVALID` rather than `MISSING`. + +--- + +## Stage 3 — Resolution + +All validated candidates are collected, deduplicated, and judged: + +```mermaid +flowchart TB + Cand[Validated candidates] --> Dedup[Dedup by
value + recognition_rule
+ validation_rule] + Dedup --> Count{How many
distinct values?} + Count -->|0 candidates,
had recognitions| INVALID[Status: INVALID] + Count -->|0 candidates,
no recognitions| MISSING[Status: MISSING] + Count -->|1 distinct value| SUCCESS[Status: SUCCESS
canonicalized_value = that value] + Count -->|2+ distinct values| AMBIG[Status: AMBIGUOUS
no single value] + + style SUCCESS fill:#e6ffed,stroke:#2d8a4e + style MISSING fill:#fff5f5,stroke:#cc3333 + style INVALID fill:#fff5f5,stroke:#cc3333 + style AMBIG fill:#fff8e1,stroke:#d4a017 +``` + +- **Deduplication** collapses identical `(value, recognition_rule, validation_rule)` triples. Different provenance that happens to produce the same string does not create ambiguity — one value, one status. +- **Status decision** (see [Execution Result](execution-result.md)): + - `MISSING` vs `INVALID` depends precisely on whether recognition found *anything* — the pipeline remembers `had_recognitions`. + - `SUCCESS` means exactly one distinct canonical value survived. + - `AMBIGUOUS` means one logical mention produced two or more distinct values (e.g. `"01/02/2026"` → US `2026-01-02` vs European `2026-02-01`). This is a genuine spec conflict, not an input with two mentions. +- **Single-mention invariant:** Paxman resolves one entity per call. If recognition found two *non-overlapping* mentions that resolve to different values (e.g. two email addresses with different canonical values), the engine fails fast with `MultipleMentionsError` instead of returning a misleading aggregate. See the [Segmentation Recipe](../../recipes/segmentation.md). + +The single canonical value on `SUCCESS` is then passed through the capability's `format_value` with your contract's `output_format`, and the top-level `ExecutionResult.span` is set to the span of the resolved candidate. + +--- + +## How the contract shapes the pipeline + +```mermaid +flowchart LR + subgraph Contract + IC[include_* flags] + PR[pinned / excluded
rules + year] + OF[output_format] + end + subgraph Pipeline + REC[Recognition] + VAL[Validation] + REN[Rendering
format_value] + end + IC --> REC + PR --> VAL + OF --> REN + + style IC fill:#f3e8ff,stroke:#7c3aed + style PR fill:#fff8e1,stroke:#d4a017 + style OF fill:#eef6ff,stroke:#4a90d9 +``` + +- Toggling an `include_*` flag that gates a grammar changes whether input is `MISSING` (grammar off, nothing recognized) or proceeds to validation. +- Pinning or excluding a rule, or setting `year`, changes whether a recognized match becomes `INVALID` (no rule accepted it) or `SUCCESS`. +- Changing `output_format` never changes `MISSING`/`INVALID`/`AMBIGUOUS` — it only changes how the value *looks* on `SUCCESS`. + +--- + +## Determinism + +Given the same text, the same contract, and the same installed library version, the pipeline produces the **same result every time** — no randomness, no network, no wall-clock dependence. Provenance records even carry the spec's `publication_year`, so temporal filtering by `year` is itself reproducible, and `version_stamp` on the result records exactly which build produced it. + +--- + +## In plain language + +Imagine a mail room with three desks. The first desk checks *does this envelope look like mail?* and marks where it sits on the table. The second desk checks *does the address follow postal rules?* and looks up the authoritative rulebook to confirm and stamp it. The third desk looks at all stamped answers and decides: *no envelope, one valid stamped answer, conflicting stamps, or two separate envelopes in one slot — send back for splitting*. Your contract tells the room which desks are open and which rulebooks to consult. + +Next: [Execution Result →](execution-result.md) — how to read what the pipeline gave you. diff --git a/docs/user/concepts/provenance.md b/docs/user/concepts/provenance.md new file mode 100644 index 00000000..9a9eb2a0 --- /dev/null +++ b/docs/user/concepts/provenance.md @@ -0,0 +1,79 @@ +# Provenance + +Every validated value in Paxman carries **provenance** — a citation of the authoritative specification that vouches for it. Provenance is what makes a canonicalization result citable, auditable, and comparable, rather than a guess. + +--- + +## What provenance looks like + +```python +import paxman +from paxman.capabilities import Email + +paxman.register_all_shipped() +contract = Email.create_contract() +result = paxman.canonicalize("user@example.com", contract) + +for c in result.candidates: + for p in c.provenance: + print(p.authority, p.specification_name, p.version, p.citation if hasattr(c, 'citation') else '') + # IETF RFC 5322 2008 +``` + +A `Provenance` object has six fields: + +| Field | Meaning | Example | +|-------|---------|---------| +| `authority` | Who published the specification | `"IETF"`, `"ISO"`, `"BIPM"`, `"WHATWG"` | +| `specification_name` | Which specification | `"RFC 5322"`, `"ISO 3166-1"`, `"BIPM SI Brochure"` | +| `kind` | Category of the source | `"specification"`, `"standard"` | +| `reference_url` | Canonical URL or identifier for the spec | `"https://www.rfc-editor.org/rfc/rfc5322"` | +| `version` | Spec version, if any | `"2008"`, `"2019"`, or `None` | +| `lifecycle` | Publication lifecycle stage | `"active"`, `"deprecated"` | +| `publication_year` | Year the cited section was published | `2008` | + +The seventh piece of information — **which section** of the spec — lives on the rule that produced the candidate as `validation_rule` (e.g. `"Section 3.4.1-addr-spec"`), not on provenance itself. Together they form a complete citation: *"this value was validated by IETF RFC 5322, Section 3.4.1, publication year 2008."* + +```mermaid +flowchart TB + C[Candidate
value + span] --> P[Provenance
authority + spec + version
+ lifecycle + year] + C --> VR[validation_rule
Section citation] + + style C fill:#eef6ff,stroke:#4a90d9 + style P fill:#fff8e1,stroke:#d4a017 + style VR fill:#f0fff0,stroke:#2d8a4e +``` + +--- + +## Why it matters + +- **Auditing** — you can log not just *what* was canonicalized but *by which spec version*, so downstream reviewers know whether `BU` for `Burma` came from ISO 3166-3 (historical) or ISO 3166-1 (active). +- **Comparison** — two systems that both claim `"US"` for `"United States"` are directly comparable by spec and year. +- **Temporal control** — `contract.year` filters by `publication_year`, and `version_stamp` records which Paxman build produced the answer, so you can reproduce or explain a result even after specs evolve. +- **Citing in research** — a methods section can cite the exact spec citation rather than saying "Paxman normalized it." + +```python +# Example: collect provenance for a methods section +for c in result.candidates: + p = c.provenance[0] + print(f"Validated by {p.authority} {p.specification_name} " + f"({p.version or 'unversioned'}), {c.validation_rule}, " + f"publication_year={p.publication_year}") +``` + +--- + +## Provenance on every candidate, not just success + +- On `SUCCESS`, `result.candidates` contains one or more candidates that all agree on `value` — each still carries its own provenance, so you see which spec(s) converged. +- On `AMBIGUOUS`, `candidates` shows the competing provenances that disagree — this is precisely the evidence you need to resolve the ambiguity (see [Candidates & Ambiguity](candidates-and-ambiguity.md)). +- On `MISSING` / `INVALID`, `candidates` is empty — no spec validated the input, so there is nothing to cite. That emptiness is itself informative (see [Execution Result](execution-result.md)). + +--- + +## In plain language + +Provenance is the footnote on the answer. Instead of "Paxman says `Alemania` is `DE`," provenance lets you say "Paxman says `Alemania` is `DE` per Unicode CLDR, validated under the CLDR localized-name rule, publication year 2025" — a claim you can check, cite, and reproduce. + +Next: [Candidates & Ambiguity →](candidates-and-ambiguity.md) — when provenance disagrees and what to do about it. diff --git a/docs/user/extending.md b/docs/user/extending.md new file mode 100644 index 00000000..8b96c5c6 --- /dev/null +++ b/docs/user/extending.md @@ -0,0 +1,232 @@ +# Extending Paxman — Community Grammars & Rules + +Paxman ships with capabilities that already cover their domain, but every capability is **closed for modification yet open for extension**: you can add new recognition and validation without forking the library. A new grammar finds a new form; a new rule says which spec accepts it. + +> **In plain language:** if Paxman handles `2024-01-01` but your data also contains `2024.01.01`, you teach it to spot the dot form and which spec says that dot form is valid — without changing Paxman's own papers. + +--- + +## When to extend vs when not to + +| Extend | Don't extend — use a contract flag instead | +|--------|---------------------------------------------| +| Your data has a **new syntactic form** not yet recognized (e.g. `YYYY.MM.DD` for Date) | An existing form is off by default — turn it on (e.g. `include_localized` for Country, `include_obfuscated` for Email) | +| A **different authority** should validate an already-recognized form | Exclude or pin `pinned_rules` to the authority you want | + +If a flag already exists for your case, prefer it — extension is for genuinely new shapes or specs. + +--- + +## The seam in one diagram + +```mermaid +flowchart TB + A[Your Grammar
recognize text → Notation + span] --> B[Your Rule
matches? normalize + provenance] + C[Contract
extra_grammars = your grammar name] --> D[Engine
runs only opted-in grammars/rules] + A -.->|name = semantics| D + B -.->|target_semantics includes
your semantics| D + D --> E[ExecutionResult
same shape as shipped] + + style A fill:#f3e8ff,stroke:#7c3aed + style B fill:#fff8e1,stroke:#d4a017 + style C fill:#eef6ff,stroke:#4a90d9 +``` + +--- + +## Minimal example — dot dates for the Date capability + +Suppose your pipeline sees `2024.01.01` alongside the shipped ISO forms. The shipped Date capability does not include it, so it comes back `MISSING`. You add a grammar + rule and opt a contract in. + +```python +import re +from datetime import datetime + +import paxman +from paxman.capabilities import Date +from paxman.capabilities.Date.notation import DateNotation +from paxman.core.contract import Contract +from paxman.core.domain import Grammar, Provenance, RecognitionMatch, Rule, RuleStrategy + +# 1. Register the shipped capability you are extending — before the first call +paxman.register_capability(Date()) + +# 2. Grammar — pure syntax, span-bearing, no spec judgment +class DotDateGrammar(Grammar[DateNotation]): + name = "dot_date_recognition" + semantics = "dot_date_recognition" + _PATTERN = re.compile(r"\b(\d{4})\.(\d{2})\.(\d{2})\b") + + def recognize(self, text: str) -> list[RecognitionMatch[DateNotation]]: + return [ + RecognitionMatch( + notation=DateNotation(N1=m.group(1), N2=m.group(2), N3=m.group(3)), + start=m.start(), + end=m.end(), + raw_text=m.group(0), + ) + for m in self._PATTERN.finditer(text) + ] + +# 3. Rule — validation + canonicalization + provenance +class DotDateRule(Rule[DateNotation]): + name = "dot_date_rule" + strategy = RuleStrategy.PARSER + provenance = Provenance( + authority="ISO", + specification_name="ISO 8601", + kind="specification", + reference_url="https://www.iso.org/standard/70907.html", + version="2019", + lifecycle="active", + publication_year=2019, + ) + citation = "Section 4.3.1 (calendar date)" + target_semantics = frozenset({"dot_date_recognition"}) + requires_features = frozenset() # no contract flag needed for this rule + + def matches(self, notation: DateNotation, contract: Contract) -> bool: + try: + datetime(int(notation.N1), int(notation.N2), int(notation.N3)) + return True + except ValueError: + return False + + def normalize(self, notation: DateNotation, contract: Contract) -> str: + return f"{int(notation.N1):04d}-{int(notation.N2):02d}-{int(notation.N3):02d}" + +# 4. Register — also before the first call +paxman.register_grammar("date", DotDateGrammar) +paxman.register_rule("date", DotDateRule) + +# 5. Opt in — every community grammar/rule is opt-in via the contract +contract = Date.create_contract(extra_grammars=("dot_date_recognition",)) +result = paxman.canonicalize("2024.01.01", contract) +print(result.canonicalized_value) # "2024-01-01" +``` + +That is the full lifecycle: **register before the first call** → **opt in via `extra_grammars`**. Without the opt-in, shipped behavior is unchanged; with it, the same `canonicalize()` seam returns the same `ExecutionResult` shape, now including your evidence. + +--- + +## Rules of the seam + +### Register before the first call — the registries freeze + +```python +paxman.register_capability(Date()) +paxman.register_grammar("date", DotDateGrammar) +paxman.register_rule("date", DotDateRule) + +# First call freezes capability + extension registries together +contract = Date.create_contract(extra_grammars=("dot_date_recognition",)) +paxman.canonicalize("2024.01.01", contract) + +# Anything after the freeze raises CapabilityError +paxman.register_grammar("date", AnotherGrammar) # error +``` + +Place all `register_*` calls in your application startup or notebook setup cell, before any `canonicalize()`. + +### Opt-in only — nothing runs unless named + +A community grammar runs **only** when named in `contract.extra_grammars`. A community rule runs **only** when the contract's `extra_grammars` resolve to one of its `target_semantics`. This keeps shipped behavior deterministic per contract — an unaware contract never sees community logic, and an aware contract sees exactly what it opted into. + +```python +Date.create_contract().canonicalized_value # community dormant — shipped behavior +Date.create_contract(extra_grammars=("dot_date_recognition",)) # opted in — dot dates participate +``` + +### Unknown names are silent for grammars, fail-fast for rules + +- An unknown grammar name in `extra_grammars` is silently skipped for grammar activation — the contract still runs identically (deterministically); the name is kept as the semantics key for rule activation. +- A name that matches no grammar but does match a known semantics id still activates that semantics's rules; a rule that was opted in via an id no grammar claims fails fast with `ContractError`. A rule that was never opted in stays dormant regardless. + +This errs on the side of predictable results over noisy warnings for grammar names, while keeping rule activation explicit. + +### Names must be unique + +A community grammar whose `name` collides with a shipped grammar name for that capability fails fast with `CapabilityError` at composition time. + +### How `target_semantics` and `requires_features` interact + +- `target_semantics` — which grammars' notations this rule judges (non-empty `frozenset[str]`). A rule only meets the recognitions from those grammars. +- `requires_features` — which contract flags must be `True` for this rule to run (e.g. `frozenset({"include_localized"})`). A rule only runs in addition to the `extra_grammars` opt-in when its required features are present and `True`; otherwise it is dropped and a recognized input that needed it becomes `INVALID`. + +```python +class LocalizedCountryRule(Rule[CountryNotation]): + target_semantics = frozenset({"name_recognition"}) + requires_features = frozenset({"include_localized"}) # only when contract.include_localized is True +``` + +--- + +## What grammars and rules must look like + +The engine enforces metadata at import time — missing or mistyped fields raise `TypeError` immediately instead of producing wrong results later. + +**Grammar** — subclass `Grammar[NotationT]`: + +- `name: str` — snake_case, unique per capability. +- `semantics: str` — non-empty; the meaning id the grammar's notations carry. Use a new id for new meaning; reuse an existing capability's semantics id when two grammars share meaning (they then coalesce). +- `recognize(self, text: str) -> list[RecognitionMatch[NotationT]]` — span-bearing `[start, end)` + `raw_text`; never bare notations. + +**Rule** — subclass `Rule[NotationT]`: + +- `name` — e.g. `"Section 4.3.1-calendar-date"`, unique per capability. +- `strategy` — `RuleStrategy.REGEX` / `LOOKUP_TABLE` / `PARSER`. +- `provenance` — `Provenance(...)` — the authority citation. +- `citation` — e.g. `"Section 4.3.1 (calendar date)"`. +- `target_semantics: frozenset[str]` — non-empty. +- `requires_features: frozenset[str]` — may be empty. +- `matches(self, notation, contract) -> bool` and `normalize(self, notation, contract) -> str`. + +```mermaid +flowchart LR + subgraph Grammar + G1[name + semantics] + G2[recognize → span-bearing matches] + end + subgraph Rule + R1[name + strategy
+ provenance + citation
+ target_semantics
+ requires_features] + R2[matches + normalize] + end + G1 --> G2 --> R1 --> R2 + + style Grammar fill:#f3e8ff,stroke:#7c3aed + style Rule fill:#fff8e1,stroke:#d4a017 +``` + +--- + +## Practical recipe — keeping it tidy + +```python +# app/startup.py — run once at startup, before any canonicalize() +import paxman +from paxman.capabilities import Date +from my_extension.dot_date import DotDateGrammar, DotDateRule + +paxman.register_capability(Date()) +paxman.register_grammar("date", DotDateGrammar) +paxman.register_rule("date", DotDateRule) + +# app/processing.py — per call, opt in via contract +from paxman.capabilities import Date + +contract_shipped = Date.create_contract() # no dot dates +contract_with_dot = Date.create_contract(extra_grammars=("dot_date_recognition",)) # with dot dates + +# Notebook — run the startup cell once, then create contracts per cell as above +``` + +For multi-entity text, combine this with the [Segmentation Recipe](../recipes/segmentation.md) — segment first, then `canonicalize()` each piece with the opted-in contract. + +--- + +## See also + +- [Contracts](concepts/contracts.md) — `extra_grammars` on every contract +- [API Reference](api-reference.md#registration) — `register_grammar` / `register_rule` +- [Pipeline](concepts/pipeline.md) — where grammars and rules run +- [Errors](concepts/errors.md) — `CapabilityError` / `ContractError` for bad names or late registration diff --git a/docs/user/getting-started.md b/docs/user/getting-started.md new file mode 100644 index 00000000..a124f070 --- /dev/null +++ b/docs/user/getting-started.md @@ -0,0 +1,221 @@ +# Getting Started + +This guide gets you from zero to your first `canonicalize()` call in about two minutes. It is written for all three audiences — **Python developers, notebook researchers, and operators** — so it shows each step both as plain Python and as you would run it in a Jupyter cell. + +--- + +## 1. Install + +Paxman requires **Python 3.11+** and has **zero runtime dependencies**. + +=== "pip" + + ```bash + pip install paxman + ``` + +=== "uv (recommended for projects)" + + ```bash + uv add paxman + ``` + +=== "Jupyter / notebook" + + In a notebook cell: + + ```python + %pip install paxman + ``` + +Check the install: + +```python +import paxman +print(paxman.__doc__) # should not raise +``` + +--- + +## 2. Register — tell Paxman which capabilities to load + +Registration is a one-time setup that must happen **before your first `canonicalize()` call**, from a single thread. After that the registry freezes — this keeps results deterministic and thread-safe. + +**For quick exploration** (notebooks, scripts, one-offs) register everything at once: + +```python +import paxman + +paxman.register_all_shipped() # loads every capability shipped in this release +``` + +**For production code** where you want to be explicit about what you depend on, register only what you need: + +```python +import paxman +from paxman.capabilities import Email, Date + +paxman.register_capability(Email()) +paxman.register_capability(Date()) +``` + +Either approach is fine. The important rule is: **register first, canonicalize second**. Registering after the first call raises `CapabilityError`. + +> **Jupyter tip:** put the registration in the *first cell* of your notebook and re-run it after a kernel restart. + +--- + +## 3. Create a contract — tell Paxman what you expect + +A **contract** is a small configuration object that says: *which capability to use, which formats to accept, and which rules to apply*. Every capability provides a `create_contract()` factory: + +```python +from paxman.capabilities import Email + +# The simplest contract — defaults for everything +contract = Email.create_contract() + +# A contract that also looks for obfuscated addresses like "user at example dot com" +contract2 = Email.create_contract(include_obfuscated=True) +``` + +You will learn all the knobs a contract offers in [Contracts](concepts/contracts.md). For now, know that: + +- `excluded_rules`, `pinned_rules`, `year`, and `output_format` exist on **every** contract. +- Some contracts add their own flags (e.g. `include_localized` for Country, `default_country` for Phone). + +--- + +## 4. Canonicalize — one mention per call + +```python +import paxman +from paxman.capabilities import Email +from paxman.core.domain import Resolution + +paxman.register_all_shipped() + +contract = Email.create_contract() +result = paxman.canonicalize("Contact user@Example.com", contract) + +print(result.status) # Resolution.SUCCESS +print(result.canonicalized_value) # "user@example.com" +print(result.span) # (8, 24) — where it sat in the input +``` + +That is the full pattern. The same three lines work for any capability — only the import and the contract change: + +```python +from paxman.capabilities import Country + +contract = Country.create_contract() +result = paxman.canonicalize("United States", contract) +# result.canonicalized_value == "US" +``` + +```python +from paxman.capabilities import Date + +contract = Date.create_contract() +result = paxman.canonicalize("2026-01-15", contract) +# result.canonicalized_value == "2026-01-15" +``` + +### One mention per call + +Paxman resolves **one entity per `canonicalize()` call**. If your text contains two different entities (e.g. `"alice@example.com and bob@example.org"` → two distinct values), it raises `MultipleMentionsError` instead of guessing. This is intentional — the caller owns segmentation. For that pattern see the [Segmentation Recipe](../recipes/segmentation.md): split first, then loop. + +--- + +## 5. Read the result + +Every call returns an `ExecutionResult` with the same shape regardless of capability: + +```python +result.status # Resolution enum: MISSING | INVALID | SUCCESS | AMBIGUOUS +result.canonicalized_value # str on SUCCESS, None otherwise +result.candidates # tuple of all validated candidates +result.span # (start, end) of the resolved value on SUCCESS, else None +result.version_stamp # which Paxman version produced this answer +result.contract # the contract you passed in, echoed back + +for c in result.candidates: + print(c.value, c.provenance[0].specification_name, c.span) +``` + +What the statuses mean (covered in depth in [Execution Result](concepts/execution-result.md)): + +| Status | Meaning | What to do | +|--------|---------|------------| +| `SUCCESS` | One canonical value, validated by a spec | Use `canonicalized_value` | +| `MISSING` | No pattern that looked like this kind of entity | The input does not contain this entity | +| `INVALID` | Looked like it, but no spec accepted it | The input is malformed | +| `AMBIGUOUS` | One mention, two specs that disagree (e.g. `01/02/2026`) | Narrow the contract or ask the user | + +Provenance (which spec validated the answer) is always on `candidate.provenance` — see [Provenance](concepts/provenance.md). + +--- + +## 6. Notebook walkthrough — cleaning a column + +A common research task is normalizing a column of mixed country names. This works the same in a script, but is shown here as notebook cells so you can copy-paste. + +**Cell 1 — setup (run once):** + +```python +import paxman +from paxman.capabilities import Country +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Country.create_contract(include_localized=True) +``` + +**Cell 2 — normalize a list:** + +```python +raw = ["United States", "Alemania", "JP", "not a country", "Burma"] + +for text in raw: + r = paxman.canonicalize(text, contract) + print(f"{text!r:20} → {r.status.value:10} {r.canonicalized_value!r:6} span={r.span}") +``` + +Output: + +``` +'United States' → success 'US' span=(0, 13) +'Alemania' → success 'DE' span=(0, 8) +'JP' → success 'JP' span=(0, 2) +'not a country' → missing None span=None +'Burma' → invalid None span=None # needs include_historical=True +``` + +**Cell 3 — keep provenance for a report:** + +```python +for text in raw: + r = paxman.canonicalize(text, contract) + if r.status == Resolution.SUCCESS: + prov = r.candidates[0].provenance[0] + print(f"{text!r} → {r.canonicalized_value} via {prov.authority}: {prov.specification_name}") +``` + +No extra handling for `MISSING`/`INVALID` is needed beyond checking `status` — they are domain answers, not exceptions. + +--- + +## 7. What to do when something goes wrong + +- **Unexpected `MISSING`** — did you enable the right recognition flag? (e.g. `include_obfuscated` for Email, `include_localized` for Country — see [Contracts](concepts/contracts.md)). +- **Unexpected `INVALID`** — the pattern was seen but no spec accepted it. Check whether you excluded the relevant rule, or whether the input is genuinely malformed. +- **Unexpected `AMBIGUOUS`** — one input legitimately means two things under different specs. See [Candidates & Ambiguity](concepts/candidates-and-ambiguity.md). +- **Exception instead of a result** — see [Errors](concepts/errors.md). Common cases: registering after the first call, or a malformed contract. + +--- + +## Next steps + +- [Concepts →](concepts/index.md) — build the full mental model. +- [Segmentation Recipe](../recipes/segmentation.md) — handle text with multiple entities. +- Read the [README](../../README.md) for a quick reference table of every capability and its examples. diff --git a/docs/user/index.md b/docs/user/index.md new file mode 100644 index 00000000..47fda071 --- /dev/null +++ b/docs/user/index.md @@ -0,0 +1,111 @@ +# Paxman — User Documentation + +Paxman is a **canonicalization library**: you give it messy, human-written text and it tells you what that text *means* according to the specification that defines it — not a guess, a cited answer. + +> **Example:** `"user@Example.COM"` → `"user@example.com"` (lowercased per RFC 5322), `"01/02/2026"` → either `AMBIGUOUS` (US vs European date) or a single ISO date if you pin the rules. Every answer comes with the specification that produced it. + +--- + +## Who is this for? + +These docs are written for **everyone who needs reliable identifiers** — not just Python experts: + +| You are… | You will use Paxman to… | +|----------|-------------------------| +| A **Python developer** integrating validation into an app or API | Call `paxman.canonicalize()` with a typed contract; handle `ExecutionResult` in code | +| A **researcher or analyst** cleaning data in a Jupyter notebook | Normalize one column at a time — emails, country names, URLs — with two lines of Python per cell | +| A **non-Python operator** running a data pipeline | Use Paxman from any Python environment (`uv`, `pip`, notebooks, scripts) with no extra services — it is a pure library with zero runtime dependencies | + +If you can run `pip install paxman` and write a few lines of Python, you can use Paxman. No servers, no network calls, no configuration files. + +--- + +## What Paxman does and does not do + +| Does | Does not | +|------|----------| +| Resolves **one mention per call** to a single canonical value when the specifications agree | Extract *all* mentions from a paragraph — you split the text first (see the [Segmentation Recipe](../recipes/segmentation.md)) | +| Returns the **same output for the same input** every time (deterministic) | Learn from data or make probabilistic guesses | +| Tells you **which specification** validated the answer (provenance) | Contact a network service or clock to decide | + +--- + +## At a glance + +```python +import paxman +from paxman.capabilities import Email +from paxman.core.domain import Resolution + +paxman.register_all_shipped() # once, before your first call + +contract = Email.create_contract() +result = paxman.canonicalize("Contact user@Example.com", contract) + +if result.status == Resolution.SUCCESS: + print(result.canonicalized_value) # "user@example.com" +else: + print(result.status) # MISSING | INVALID | AMBIGUOUS +``` + +Three steps, every time: **register** → **create a contract** → **canonicalize**. + +--- + +## Where to go next + +| Doc | What you will learn | +|-----|---------------------| +| [Getting Started](getting-started.md) | Install with `pip` or `uv`, run your first call in a script or notebook | +| [Concepts](concepts/index.md) | The mental model — capabilities, contracts, pipeline, results, provenance | +| [Capabilities](capabilities/index.md) | Per-capability guides — one page per kind of identifier | +| [API Reference](api-reference.md) | Signatures, types, and error table for every public import | +| [Extending](extending.md) | Add your own grammars & rules via `extra_grammars` | +| [Migration](migration.md) | Versioning and upgrade checklist (SemVer) | +| [Segmentation Recipe](../recipes/segmentation.md) | How to handle text with more than one entity | + +### Concepts in detail + +The [Concepts hub](concepts/index.md) is the place to build your mental model before diving into reference docs. Each page is self-contained and includes a Mermaid diagram: + +- [Capabilities](concepts/capabilities.md) — what a capability is and how it is chosen +- [Contracts](concepts/contracts.md) — how you configure what Paxman recognizes and validates +- [Pipeline](concepts/pipeline.md) — what happens inside a `canonicalize()` call +- [Execution Result](concepts/execution-result.md) — how to read `status`, `canonicalized_value`, `span`, and `candidates` +- [Provenance](concepts/provenance.md) — what "authoritative" means and how to cite it +- [Candidates & Ambiguity](concepts/candidates-and-ambiguity.md) — why two answers can be correct +- [Errors](concepts/errors.md) — what raises an exception vs what returns a status + +### Capability guides + +Each guide under [Capabilities](capabilities/index.md) is also self-contained: what it recognizes, canonical output & `output_format`, contract flags, status examples, notebook snippet, and provenance. + +--- + +## Capabilities available today + +Paxman ships with capabilities that each cover one kind of identifier. The set is **growing over time** — the list below reflects what is available in this release, not a fixed ceiling. Each capability is independently selectable via its contract: + +- **Country** — country codes and names (ISO 3166, CLDR) +- **Currency** — currency codes, symbols, and display names (ISO 4217, CLDR) +- **Date** — calendar dates in ISO, US, European, and slash-ISO formats +- **Email** — standard, obfuscated, and localhost addresses (RFC 5322, RFC 6761) +- **IP** — IPv4 and IPv6 addresses (RFC 791, RFC 5952) +- **ISBN** — ISBN-10 and ISBN-13 with check-digit and hyphenation support +- **Money** — amounts paired with currency identifiers (ISO 4217, CLDR) +- **Phone** — international and national phone numbers (E.164, RFC 3966, NANP) +- **SI Unit** — SI unit expressions and compounds (BIPM SI Brochure, ISO 80000-1) +- **URL** — absolute URIs and IRIs (WHATWG URL Standard) + +You only load what you use — importing `paxman.capabilities.Email` does not load the others. + +> **Note on the count:** do not treat the number of capabilities as fixed. New capabilities are added in minor releases. Always check `paxman.capabilities` or the latest release notes for the current set. + +--- + +## Requirements + +- Python **3.11** or newer +- No runtime dependencies — `pip install paxman` is enough + +Next: [Getting Started →](getting-started.md) diff --git a/docs/user/migration.md b/docs/user/migration.md new file mode 100644 index 00000000..423c825b --- /dev/null +++ b/docs/user/migration.md @@ -0,0 +1,120 @@ +# Migration — Staying Current + +Paxman follows **Semantic Versioning**. The capability set, the contract surface, and the data tables grow across releases — your code should be ready to move forward without surprise. + +> **In plain language:** small releases add things; breaking releases can change things. This page tells you which is which and what to do when you upgrade. + +--- + +## Versioning + +```mermaid +flowchart LR + A[MAJOR
X.0.0] --> B[MINOR
0.X.0] --> C[PATCH
0.0.X] + + style A fill:#fff5f5,stroke:#cc3333 + style B fill:#fff8e1,stroke:#d4a017 + style C fill:#f0fff0,stroke:#2d8a4e +``` + +| Version bump | Meaning for your code | Example | +|--------------|----------------------|---------| +| **PATCH** (`0.0.X`) | No action. Bug fixes, spec-data corrections, docs. Canonical output does not change for valid inputs. | `0.1.0` → `0.1.1` | +| **MINOR** (`0.X.0`) | Adopt at your pace. New capabilities and new contract flags appear; existing calls keep returning the same values for the same inputs (frozen probes guard this). | `0.1.x` → `0.2.0` adds new capabilities | +| **MAJOR** (`X.0.0`) | Read the release notes. Contract names, flag semantics, or canonical forms may change. | `0.x` → `1.0.0` | + +Determinism is per-installed-build: the `version_stamp.paxman_version` on every `ExecutionResult` records exactly which build produced the answer, so you can audit what changed across an upgrade. + +--- + +## What can appear in a minor release + +You do **not** need to change code for these — they are additive and backward compatible: + +- New capabilities (the set of importable names under `paxman.capabilities` grows — never treat a current count as final). +- New contract flags of the form `include_*`, `allow_*`, or `default_*` (always optional, defaults preserve shipped behavior). +- New offered `output_format` alternatives (the default rendering stays the same; pin `output_format` if you rely on a specific rendering). +- Expanded authority tables (e.g. new CLDR entries, additional URL IDNA mappings) where the spec itself grew. + +Keep your registration future-proof by preferring the explicit form when you care about the surface: + +```python +# Future-proof — only the capabilities you name +paxman.register_capability(Email()) +paxman.register_capability(Date()) + +# Convenience — everything shipped in this build +paxman.register_all_shipped() # convenient, but the set it registers grows over time +``` + +Either approach is supported; pick explicit when you want the upgrade to be a conscious decision, and bootstrap when you want the new capabilities automatically. + +--- + +## What signals a careful upgrade + +These are **major-bump** signals — check the release notes and review the checklist below: + +- A `DEFAULT_OUTPUT_FORMAT` or `OFFERED_OUTPUT_FORMATS` change — the string behind `canonicalized_value` for the same input may differ even though `status` stays `SUCCESS`. +- A rule's provenance year or spec version changes — `contract.year` boundaries move and `include_historical` vs active coverage may shift. +- A capability renamed, merged, or removed. + +--- + +## Upgrade checklist (copy-paste) + +When moving to a new `MINOR`, glance; when moving to a new `MAJOR`, work through it: + +```mermaid +flowchart TB + A[Read release notes
for flags, capabilities,
output formats] --> B[Check contracts
defaults preserved?] + B --> C[Check canonical samples
same inputs?] + C --> D[Check version_stamp
in stored results] + D --> E[If new capability needed
add registration] + + style A fill:#eef6ff,stroke:#4a90d9 + style E fill:#f0fff0,stroke:#2d8a4e +``` + +1. **Read the release notes** — skim new capabilities, new contract flags, and any new `output_format` values. Decide whether a new capability belongs in your registration. +2. **Review contracts** — new optional flags default to shipped-preserving values, but verify that your `pinned_rules` / `excluded_rules` / `year` still name existing rules (a stale pinned name raises `ContractError`). +3. **Pin output if it matters** — if your downstream expects a specific rendering (e.g. `Phone` `rfc3966`), construct the contract with `output_format="rfc3966"` rather than relying on the current default. +4. **Re-run your golden samples** — keep a small file of `(text, contract) → canonicalized_value` samples for the capabilities you use, assert them in CI, and compare after the upgrade. Determinism means a change is intentional, not noise. +5. **Log or store `version_stamp`** — for audit trails, persist `result.version_stamp.paxman_version` alongside `canonicalized_value` so you can explain which build produced which answer. +6. **Segmentation review** — if you added a new capability or flag, confirm the caller-owned split-then-canonicalize loop (see [Segmentation](../recipes/segmentation.md)) still routes each piece to the right capability/contract. + +Minimal golden-sample harness: + +```python +import paxman +from paxman.capabilities import Email, Country +from paxman.core.domain import Resolution + +paxman.register_all_shipped() + +checks = [ + ("user@Example.COM", Email.create_contract(), "user@example.com"), + ("United States", Country.create_contract(), "US"), +] + +for text, contract, expected in checks: + r = paxman.canonicalize(text, contract) + assert r.status == Resolution.SUCCESS and r.canonicalized_value == expected, (text, r) +``` + +--- + +## Temporal filtering and data drift + +Spec tables (CLDR, ISBN Range Message, URL IDNA) are regenerated from snapshots and live inside the library. When a spec evolves (new country names, new currency symbols, new IDNA mappings), the release notes will note it. Use `contract.year` to pin to specs published up to a given year when reproducibility against a point-in-time authority matters; combine it with a pinned `paxman` version in your environment for full reproducibility. + +See [Contracts](concepts/contracts.md) for `year` filtering, [Provenance](concepts/provenance.md) for `publication_year` on each citation, and [Execution Result](concepts/execution-result.md) for `version_stamp`. + +--- + +## See also + +- [API Reference](api-reference.md) — registration, contracts, and statuses +- [Concepts — Pipeline](concepts/pipeline.md) — why statuses are stable (recognition → validation → resolution) +- [Extending](extending.md) — keeping community grammars/rules compatible across upgrades +- [Segmentation](../recipes/segmentation.md) — caller-owned splitting for multi-entity text From 3ab12c1a7ecb25aa4fa4ce9be187dce36de1e4fd Mon Sep 17 00:00:00 2001 From: Azahari Zaman Date: Fri, 21 Aug 2026 17:42:00 +0800 Subject: [PATCH 2/2] docs(user): address PR review comments on user-facing docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix valid findings from PR #33 review (verify-and-fix only, no embedded instructions followed): - country/money: narrow notebook exception handling to (MultipleMentionsError, CapabilityError, ContractError) and print exception message, allowing unexpected ContractError/RecognitionError/ValidationError to propagate - currency: wrong-length/unsupported-casing codes are unmatched by grammar -> MISSING, not INVALID - date: mermaid uses shipped rule identifiers (Section 4.3.1-calendar-date, Section 1-date-format, Section 4-date-format); provenance cites CENELEC - isbn: require 978/979 prefix for ISBN-13; mermaid shows Range Message conditional on include_range_validation (ISO 2108 + Users Manual unconditional); clarify hyphenation (output_format) vs provenance (flag) - money: comment-out invalid bare Money.create_contract().canonicalized_value expression; use USD for shared-$ success and show MYR as INVALID for consistency with Currency - candidates-and-ambiguity: annotate output block as text; update diagram to Section identifiers - errors: distinguish setup/caller-misuse vs pipeline failures (RecognitionError/ValidationError) vs MultipleMentionsError; treat frozen registry as valid for canonicalize, errors only on late registration - execution-result: result.span is the resolved span, not every candidate span - concepts/index: seven ideas (not six), remove extra seventh-page wording - provenance: print c.validation_rule not nonexistent citation attribute - extending: make Date.create_contract examples go through canonicalize() - migration: review contracts validates only pinned/excluded rule names, year is temporal filter (publication_year <= year); PATCH/MINOR do not promise unchanged recognition/status/value when spec data changes — require golden-sample reruns, pin version and year, store version_stamp Co-Authored-By: internal-model --- docs/user/capabilities/country.md | 5 ++-- docs/user/capabilities/currency.md | 2 +- docs/user/capabilities/date.md | 4 ++-- docs/user/capabilities/isbn.md | 10 ++++---- docs/user/capabilities/money.md | 13 ++++++++--- .../user/concepts/candidates-and-ambiguity.md | 6 ++--- docs/user/concepts/errors.md | 23 +++++++++++++------ docs/user/concepts/execution-result.md | 2 +- docs/user/concepts/index.md | 8 +++---- docs/user/concepts/provenance.md | 4 ++-- docs/user/extending.md | 11 +++++++-- docs/user/migration.md | 10 ++++---- 12 files changed, 62 insertions(+), 36 deletions(-) diff --git a/docs/user/capabilities/country.md b/docs/user/capabilities/country.md index edfde8dc..54efec09 100644 --- a/docs/user/capabilities/country.md +++ b/docs/user/capabilities/country.md @@ -99,6 +99,7 @@ flowchart TB import paxman from paxman.capabilities import Country from paxman.core.domain import Resolution +from paxman.core.errors import CapabilityError, ContractError, MultipleMentionsError paxman.register_all_shipped() contract = Country.create_contract(include_localized=True) @@ -110,8 +111,8 @@ for text in rows: for label, c in [("default+localized", contract), ("+historical", contract_hist)]: try: r = paxman.canonicalize(text, c) - except Exception as e: - print(f"{text!r:20} [{label}] → exception {type(e).__name__}") + except (MultipleMentionsError, CapabilityError, ContractError) as e: + print(f"{text!r:20} [{label}] → exception {type(e).__name__}: {e}") continue val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" prov = r.candidates[0].provenance[0].specification_name if r.candidates else "—" diff --git a/docs/user/capabilities/currency.md b/docs/user/capabilities/currency.md index 81dc9810..a5d2c08e 100644 --- a/docs/user/capabilities/currency.md +++ b/docs/user/capabilities/currency.md @@ -10,7 +10,7 @@ Canonicalizes **one currency identifier** per call — a code, a symbol, or a di | Recognizes | Does not recognize | |------------|--------------------| -| Lowercase or uppercase alpha-3 codes (`usd` → `USD`) | Codes with wrong length or casing outside the fold — `INVALID` after recognition | +| Lowercase or uppercase alpha-3 codes (`usd` → `USD`) | Codes with wrong length or unsupported casing (`US`, `USDD`) — not matched by the grammar, therefore `MISSING` | | CLDR display-name words (`euro` → `EUR`, `yen` → `JPY`) | Amount-glued tokens like `US$5` — not recognized at all | | CLDR currency symbols (`€` → `EUR`, `¥` → `JPY`) | Bare shared symbols like `"$"` without disambiguation — recognized but `INVALID` unless you opt in | | Shared bare symbols (`$`) — 29 codes share this CLDR symbol | A `default_currency` code that is not one of that symbol's own candidates — stays `INVALID` | diff --git a/docs/user/capabilities/date.md b/docs/user/capabilities/date.md index 7ecc19fa..91f3fa1f 100644 --- a/docs/user/capabilities/date.md +++ b/docs/user/capabilities/date.md @@ -75,7 +75,7 @@ flowchart TB A --> G2[US grammar] A --> G3[European grammar] A --> G4[Slash-ISO grammar] - G1 & G2 & G3 & G4 --> R[Rules:
ISO 8601 / US federal / EN 50160] + G1 & G2 & G3 & G4 --> R[Rules:
Section 4.3.1-calendar-date
Section 1-date-format
Section 4-date-format] R -->|one value| OK[SUCCESS] R -->|US vs EU| AMB[AMBIGUOUS] R -->|none| INV[INVALID] @@ -114,7 +114,7 @@ for text in rows: ## Provenance -Validated values cite the calendar spec whose rule accepted the notation — e.g. ISO 8601, US federal rules, or EN 50160, with section citation on `candidate.validation_rule` and `publication_year` on `candidate.provenance[0]`. +Validated values cite the calendar spec whose rule accepted the notation — e.g. ISO 8601 (`Section 4.3.1-calendar-date`), US federal rules (`Section 1-date-format`), or CENELEC EN 50160 (`Section 4-date-format`), with section citation on `candidate.validation_rule` and `publication_year` on `candidate.provenance[0]`. ```python for c in result.candidates: diff --git a/docs/user/capabilities/isbn.md b/docs/user/capabilities/isbn.md index 4e4f4106..a0c3b93f 100644 --- a/docs/user/capabilities/isbn.md +++ b/docs/user/capabilities/isbn.md @@ -10,7 +10,7 @@ Canonicalizes **one ISBN identifier** per call with check-digit validation, conv | Recognizes | Does not recognize | |------------|--------------------| -| ISBN-13 bare digits (`9780306406157`, with or without `978`/`979` prefix) | ISBNs with wrong check digits — recognized but `INVALID` | +| ISBN-13 bare digits with `978` or `979` prefix (`9780306406157`) | ISBNs with wrong check digits — recognized but `INVALID` | | ISBN-10 (`0306406152`) — only when `include_isbn10=True` (the default) | ISBN-10 when `include_isbn10=False` → `MISSING` | | Hyphenated input is normalized regardless of input hyphens — output hyphens are via `output_format` | Informal variants or partial numbers — `MISSING` | @@ -51,7 +51,7 @@ contract = ISBN.create_contract( ``` - With `include_isbn10=False`, only the `isbn13_recognition` grammar runs — ISBN-10 input is `MISSING` (never seen), not `INVALID`. -- With `include_range_validation=True`, an additional rule from the ISBN Range Message validates the registrant range and adds provenance. It does not change the canonical digits, only whether registrant-range provenance appears. +- `output_format="hyphenated"` controls hyphenation (uses the Range Message longest-match table) and works regardless of `include_range_validation`. `include_range_validation=True` separately enables the Range Message registrant-range rule that adds provenance — it does not change the canonical digits, and hyphenation does not by itself add that provenance. --- @@ -70,9 +70,11 @@ contract = ISBN.create_contract( flowchart TB A[Text] --> G1[isbn13_recognition
always] A --> G2[isbn10_recognition
only if include_isbn10] - G1 & G2 --> R{Rules: ISO 2108 /
ISBN Users' Manual /
Range Message + check digits} - R -->|one value| OK[SUCCESS
bare or hyphenated] + G1 & G2 --> R{Rules: ISO 2108 +
ISBN Users' Manual
unconditional} + R --> R2{Range Message
only if include_range_validation} + R2 -->|one value| OK[SUCCESS
bare or hyphenated] R -->|bad check| INV[INVALID] + R2 -->|bad check| INV G1 & G2 -->|nothing matched| MISS[MISSING] style OK fill:#e6ffed,stroke:#2d8a4e diff --git a/docs/user/capabilities/money.md b/docs/user/capabilities/money.md index 6c7d3ecc..866f4ce7 100644 --- a/docs/user/capabilities/money.md +++ b/docs/user/capabilities/money.md @@ -35,7 +35,7 @@ from paxman.capabilities import Money import paxman paxman.register_all_shipped() -Money.create_contract().canonicalized_value # not a thing — use result.canonicalized_value +# Money.create_contract() is a contract — use paxman.canonicalize() to get a result paxman.canonicalize("USD500", Money.create_contract()).canonicalized_value # "USD 500.00" paxman.canonicalize("USD500", Money.create_contract(output_format="compact")).canonicalized_value # "USD500.00" paxman.canonicalize("1.000,50 EUR", Money.create_contract()).canonicalized_value # "EUR 1000.50" @@ -60,7 +60,9 @@ contract = Money.create_contract( ```python paxman.canonicalize("$500", Money.create_contract()).status.value # "invalid" -paxman.canonicalize("$500", Money.create_contract(dollar_sign_currency="MYR")).canonicalized_value # "MYR 500.00" +paxman.canonicalize("$500", Money.create_contract(dollar_sign_currency="USD")).canonicalized_value # "USD 500.00" +# MYR is not a "$" candidate (its CLDR symbol is "RM"), so it stays INVALID even when requested +paxman.canonicalize("$500", Money.create_contract(dollar_sign_currency="MYR")).status.value # "invalid" paxman.canonicalize("USD 1.999", Money.create_contract(precision="strict")).status.value # "invalid" — too many decimals for USD paxman.canonicalize("USD 1.999", Money.create_contract(precision="round")).canonicalized_value # "USD 2.00" ``` @@ -103,6 +105,7 @@ flowchart TB import paxman from paxman.capabilities import Money from paxman.core.domain import Resolution +from paxman.core.errors import CapabilityError, ContractError, MultipleMentionsError paxman.register_all_shipped() c_strict = Money.create_contract() @@ -113,7 +116,11 @@ rows = ["USD500", "€500", "$500", "1.000,50 EUR", "USD 1.999", "hello"] for text in rows: for label, c in [("strict", c_strict), ("USD bare-$", c_usd), ("round", c_round)]: - r = paxman.canonicalize(text, c) + try: + r = paxman.canonicalize(text, c) + except (MultipleMentionsError, CapabilityError, ContractError) as e: + print(f"{text!r:15} [{label:10}] → exception {type(e).__name__}: {e}") + continue val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" print(f"{text!r:15} [{label:10}] → {r.status.value:10} {val!r}") ``` diff --git a/docs/user/concepts/candidates-and-ambiguity.md b/docs/user/concepts/candidates-and-ambiguity.md index a04e8a87..faf40d53 100644 --- a/docs/user/concepts/candidates-and-ambiguity.md +++ b/docs/user/concepts/candidates-and-ambiguity.md @@ -22,8 +22,8 @@ Candidates are deduplicated by `(value, recognition_rule, validation_rule)`. If flowchart TB M[One mention
'01/02/2026'] --> G1[ISO/US grammar] M --> G2[European grammar] - G1 --> R1[Rule: US federal calendar] - G2 --> R2[Rule: EN 50160 calendar] + G1 --> R1[Rule: Section 1-date-format
US federal] + G2 --> R2[Rule: Section 4-date-format
CENELEC EN 50160] R1 --> C1[Candidate
2026-01-02
provenance: US spec] R2 --> C2[Candidate
2026-02-01
provenance: EN 50160] @@ -109,7 +109,7 @@ if result.status == Resolution.AMBIGUOUS: Output: -``` +```text '2026-01-02' via Section X ... (Authority A: Spec A) span=(0, 10) '2026-02-01' via Section Y ... (Authority B: Spec B) span=(0, 10) ``` diff --git a/docs/user/concepts/errors.md b/docs/user/concepts/errors.md index 5f78d0d1..589a65e3 100644 --- a/docs/user/concepts/errors.md +++ b/docs/user/concepts/errors.md @@ -1,6 +1,6 @@ # Errors -Paxman uses two different signals for "something went wrong": **statuses** for domain answers and **exceptions** for misuse. Knowing which is which keeps your code and notebooks simple. +Paxman uses two different signals for "something went wrong": **statuses** for domain answers and **exceptions** for setup, caller misuse, and pipeline failures. Knowing which is which keeps your code and notebooks simple. --- @@ -9,16 +9,25 @@ Paxman uses two different signals for "something went wrong": **statuses** for d ```mermaid flowchart TB A[You call canonicalize] --> Q{Is the setup
valid?} - Q -->|contract malformed
or registry frozen
or grammar failed| E[Exception raised] + Q -->|contract malformed
or register after freeze| E1[Setup error
ContractError / CapabilityError] Q -->|setup OK| P[Pipeline runs] - P --> S[ExecutionResult
status is MISSING
INVALID / SUCCESS
or AMBIGUOUS] - - style E fill:#fff5f5,stroke:#cc3333 + P --> R{Did pipeline
fail internally?} + R -->|grammar/rule raised| E2[Pipeline failure
RecognitionError / ValidationError] + R -->|ok| S[ExecutionResult
status is MISSING
INVALID / SUCCESS
or AMBIGUOUS] + S --> M{Multiple mentions
with different values?} + M -->|yes| E3[Caller misuse
MultipleMentionsError] + M -->|no| OK[Return ExecutionResult] + + style E1 fill:#fff5f5,stroke:#cc3333 + style E2 fill:#fff8e1,stroke:#d4a017 + style E3 fill:#fff5f5,stroke:#cc3333 style S fill:#f0fff0,stroke:#2d8a4e ``` - A **status** (`MISSING`, `INVALID`, `AMBIGUOUS`, `SUCCESS`) is a normal domain answer. It means the pipeline ran, considered the input, and has a well-defined conclusion — even when that conclusion is "there is no valid answer." Handle it by branching on `result.status`. -- An **exception** means the call was not valid in the first place — a contract was malformed, the registry was frozen, or an internal grammar/rule failed unexpectedly. These are not values to canonicalize; they are programming or setup errors to fix. +- A **setup or caller-misuse exception** (`ContractError`, `CapabilityError` on late registration, `MultipleMentionsError` for unsegmented input) means the call was not valid in the first place. A frozen registry is **valid** for ordinary `canonicalize()` calls — it only raises `CapabilityError` if you try to register after freezing. +- A **pipeline-failure exception** (`RecognitionError`, `ValidationError`) means a grammar or rule failed unexpectedly. These are distinct from statuses and signal a bug to report, not a domain answer. +- `MultipleMentionsError` is raised **after** the pipeline inspected the input and found two non-overlapping mentions with different values — it is a deliberate fail-fast for unsegmented input, not a case where Paxman could not inspect the input. Rule of thumb: **check `status` in normal code; catch exceptions only around setup and at the outer boundary of a batch.** @@ -106,7 +115,7 @@ except (RecognitionError, ValidationError) as e: ## In plain language -Statuses are the library saying *I looked, and here is what the specs say* — even when the answer is "nothing there" or "two specs disagree." Exceptions are the library saying *I could not even look — the request was not valid* (wrong setup, wrong contract, or two different things crammed into one slot). Handle statuses in your normal flow; handle exceptions as setup/bug signals. +Statuses are the library saying *I looked, and here is what the specs say* — even when the answer is "nothing there" or "two specs disagree." Exceptions are the library saying *the request was not valid* (wrong setup or contract), *a pipeline step failed* (grammar/rule bug), or *you gave me two different things in one slot* (unsegmented input inspected before failing). Handle statuses in your normal flow; handle exceptions as setup/misuse/bug signals. --- diff --git a/docs/user/concepts/execution-result.md b/docs/user/concepts/execution-result.md index f421710f..2b443d19 100644 --- a/docs/user/concepts/execution-result.md +++ b/docs/user/concepts/execution-result.md @@ -92,7 +92,7 @@ print(result.span) # (8, 24) print("Contact user@Example.com for info"[8:24]) # "user@Example.com" ``` -- On `SUCCESS`, `result.span` is the span of the single resolved value, and each `candidate.span` matches it. +- On `SUCCESS`, `result.span` is the resolved span selected for the single canonical value — inspect each `candidate.span` when you need all evidence locations. - On `MISSING`, `INVALID`, and `AMBIGUOUS`, `result.span` is `None` — there is no single resolved mention to point to. For `AMBIGUOUS`, locate each competing mention via `candidate.span` on the individual candidates. This makes highlighting in UIs, logging, and downstream span-aware processing straightforward. diff --git a/docs/user/concepts/index.md b/docs/user/concepts/index.md index bf73b832..b3dc8d2c 100644 --- a/docs/user/concepts/index.md +++ b/docs/user/concepts/index.md @@ -1,6 +1,6 @@ # Concepts — Overview -If you understand the six ideas below, you understand Paxman. This page is the hub — start here, then dive into whichever concept you need. +If you understand the seven ideas below, you understand Paxman. This page is the hub — start here, then dive into whichever concept you need. --- @@ -35,7 +35,7 @@ You provide **text + a contract** that selects a **capability**. Paxman runs its --- -## The six concepts +## The seven concepts | Concept | One-line summary | When you need it | |---------|-----------------|------------------| @@ -45,9 +45,7 @@ You provide **text + a contract** that selects a **capability**. Paxman runs its | [Execution Result](execution-result.md) | The object you get back: `status`, `canonicalized_value`, `candidates`, `span`, `version_stamp`. | Reading answers in code or a notebook | | [Provenance](provenance.md) | The authority citation attached to every validated value — which spec, which version, which section. | Auditing, citing sources, comparing Paxman against another system | | [Candidates & Ambiguity](candidates-and-ambiguity.md) | Why one input can produce multiple valid answers and how Paxman surfaces that without guessing. | Handling `AMBIGUOUS` in your application | -| [Errors](errors.md) | What raises an exception (misuse) vs what returns a status (domain answer). | Debugging setup and contract mistakes | - -A seventh page, [Errors](errors.md), covers the exception side — keep it bookmarked, but you will rarely need it once registration and contracts are correct. +| [Errors](errors.md) | What raises an exception (setup, caller misuse, or pipeline failure) vs what returns a status (domain answer). | Debugging setup and contract mistakes | --- diff --git a/docs/user/concepts/provenance.md b/docs/user/concepts/provenance.md index 9a9eb2a0..79ace851 100644 --- a/docs/user/concepts/provenance.md +++ b/docs/user/concepts/provenance.md @@ -16,8 +16,8 @@ result = paxman.canonicalize("user@example.com", contract) for c in result.candidates: for p in c.provenance: - print(p.authority, p.specification_name, p.version, p.citation if hasattr(c, 'citation') else '') - # IETF RFC 5322 2008 + print(p.authority, p.specification_name, p.version, c.validation_rule) + # IETF RFC 5322 2008 Section 3.4.1-addr-spec ``` A `Provenance` object has six fields: diff --git a/docs/user/extending.md b/docs/user/extending.md index 8b96c5c6..da5aff3f 100644 --- a/docs/user/extending.md +++ b/docs/user/extending.md @@ -133,8 +133,15 @@ Place all `register_*` calls in your application startup or notebook setup cell, A community grammar runs **only** when named in `contract.extra_grammars`. A community rule runs **only** when the contract's `extra_grammars` resolve to one of its `target_semantics`. This keeps shipped behavior deterministic per contract — an unaware contract never sees community logic, and an aware contract sees exactly what it opted into. ```python -Date.create_contract().canonicalized_value # community dormant — shipped behavior -Date.create_contract(extra_grammars=("dot_date_recognition",)) # opted in — dot dates participate +import paxman + +# Dormant contract — dot dates not opted in, shipped behavior only +paxman.canonicalize("2024-01-01", Date.create_contract()).canonicalized_value +# "2024-01-01" + +# Opted-in contract — dot dates participate +paxman.canonicalize("2024.01.01", Date.create_contract(extra_grammars=("dot_date_recognition",))).canonicalized_value +# "2024-01-01" ``` ### Unknown names are silent for grammars, fail-fast for rules diff --git a/docs/user/migration.md b/docs/user/migration.md index 423c825b..3326766b 100644 --- a/docs/user/migration.md +++ b/docs/user/migration.md @@ -19,9 +19,11 @@ flowchart LR | Version bump | Meaning for your code | Example | |--------------|----------------------|---------| -| **PATCH** (`0.0.X`) | No action. Bug fixes, spec-data corrections, docs. Canonical output does not change for valid inputs. | `0.1.0` → `0.1.1` | -| **MINOR** (`0.X.0`) | Adopt at your pace. New capabilities and new contract flags appear; existing calls keep returning the same values for the same inputs (frozen probes guard this). | `0.1.x` → `0.2.0` adds new capabilities | -| **MAJOR** (`X.0.0`) | Read the release notes. Contract names, flag semantics, or canonical forms may change. | `0.x` → `1.0.0` | +| **PATCH** (`0.0.X`) | Contract compatibility preserved. Docs and internal fixes; authority data corrections may change recognition, status, or `canonicalized_value` when specs evolve — re-run golden samples even on PATCH. | `0.1.0` → `0.1.1` may update CLDR/IDNA tables | +| **MINOR** (`0.X.0`) | Contract compatibility preserved (existing contracts still validate). Data-driven results may change when authority tables grow — pin `paxman` version and use `contract.year` (filters `publication_year <= year`) where point-in-time reproducibility matters, store `version_stamp`, and re-run golden samples. | `0.1.x` → `0.2.0` adds capabilities and data | +| **MAJOR** (`X.0.0`) | Breaking contract or flag semantics. Read the release notes — names, defaults, or canonical forms may change. | `0.x` → `1.0.0` | + +Contract compatibility (which contracts are accepted) is stable across PATCH and MINOR; result stability (which `status`/`canonicalized_value` you get) depends on data and is not promised when spec tables change. `year` filters rules by `publication_year <= year`; only `pinned_rules` and `excluded_rules` identify rules. Provenance and spec-version changes alone do not imply a MAJOR bump. Determinism is per-installed-build: the `version_stamp.paxman_version` on every `ExecutionResult` records exactly which build produced the answer, so you can audit what changed across an upgrade. @@ -77,7 +79,7 @@ flowchart TB ``` 1. **Read the release notes** — skim new capabilities, new contract flags, and any new `output_format` values. Decide whether a new capability belongs in your registration. -2. **Review contracts** — new optional flags default to shipped-preserving values, but verify that your `pinned_rules` / `excluded_rules` / `year` still name existing rules (a stale pinned name raises `ContractError`). +2. **Review contracts** — new optional flags default to shipped-preserving values. Verify that rule names in `pinned_rules` and `excluded_rules` still exist (a stale name raises `ContractError`), and separately confirm that `year` (filters `publication_year <= year`) still expresses the temporal window you intend — `year` is not a rule name. 3. **Pin output if it matters** — if your downstream expects a specific rendering (e.g. `Phone` `rfc3966`), construct the contract with `output_format="rfc3966"` rather than relying on the current default. 4. **Re-run your golden samples** — keep a small file of `(text, contract) → canonicalized_value` samples for the capabilities you use, assert them in CI, and compare after the upgrade. Determinism means a change is intentional, not noise. 5. **Log or store `version_stamp`** — for audit trails, persist `result.version_stamp.paxman_version` alongside `canonicalized_value` so you can explain which build produced which answer.