diff --git a/AGENTS.md b/AGENTS.md index afaf246e..498e547c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -101,10 +101,10 @@ uv run python -m paxman email "user@example.com" # CLI smoke test Full pre-PR gate: `uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run import-linter lint && uv run pytest` ## NOTES -- `paxman/capabilities/__init__.py` exports all twenty-eight shipped capabilities (BIC, ChemicalElement, Coordinates, Country, CreditCard, Currency, Date, DOI, Domain, Email, GTIN, IBAN, IP, ISBN, ISIN, ISNI, ISSN, Language, LEI, MacAddress, Money, ORCID, Phone, SI Unit, Timezone, URL, UtcOffset, UUID); export completeness is enforced by `tests/unit/test_capability_exports.py`. -- CONTEXT.md is the domain glossary for the full shipped set (twenty-eight capabilities). It is kept in sync with the code; when adding a capability, update its Notation/table entries there too. +- `paxman/capabilities/__init__.py` exports all thirty shipped capabilities (BIC, ChemicalElement, Coordinates, Country, CreditCard, Currency, Date, DOI, Domain, Email, GTIN, IBAN, IP, ISBN, ISIN, ISNI, ISSN, Language, LEI, MacAddress, MinorPlanet, Money, ORCID, Phone, SI Unit, Timezone, UNSPSC, URL, UtcOffset, UUID); export completeness is enforced by `tests/unit/test_capability_exports.py`. +- CONTEXT.md is the domain glossary for the full shipped set (thirty capabilities). It is kept in sync with the code; when adding a capability, update its Notation/table entries there too. - No `pyrightconfig.json` — pyright config is inline `[tool.pyright]` in pyproject.toml. No `.editorconfig`. - Data modules live under `rules/data/` and `grammar/data/` — plain module-level tables separating data from logic. Generated modules (edit via snapshot + regenerate, never by hand): ISBN range message (`tools/regenerate_isbn_range_data.py`), URL IDNA UTS #46 mapping (`tools/regenerate_idna_uts46_data.py`), SIUnit prefixed-unit and grammar token tables (`tools/regenerate_si_prefix_data.py`), the Currency + Money data set (`tools/regenerate_currency_data.py`, from `paxman/shared_data/currency_snapshot.json`), BIC data (`tools/regenerate_bic_data.py`), IBAN registry data (`tools/regenerate_iban_registry_data.py`), Language tables (`tools/regenerate_language_data.py`; `tools/regenerate_iana_language_data.py` now guards removed generated modules), Unicode property ranges (`tools/regenerate_unicode_property_data.py`), Domain IDNA mapping (`tools/regenerate_domain_idna_data.py`), and Domain root-zone TLDs (`tools/regenerate_root_zone_tld_data.py`). Unmarked data files are edited directly. -- Library + CLI: `[project.scripts] paxman = "paxman.cli:main"` and `python -m paxman`; CLI supports `--list`, `--json`, stdin input. Version 0.5.0. +- Library + CLI: `[project.scripts] paxman = "paxman.cli:main"` and `python -m paxman`; CLI supports `--list`, `--json`, stdin input. Version 0.6.0. - Publishing: `.github/workflows/publish.yml` uses PyPI Trusted Publishing (OIDC) with a Git-tag ↔ `pyproject.toml` version safety check; `paxman/py.typed` ships PEP 561 conformance. - Coverage: global `fail_under = 95`; `paxman/cli.py` and `paxman/__main__.py` are omitted from coverage (smoke-tested via e2e). diff --git a/CHANGELOG.md b/CHANGELOG.md index 6faf273f..55e645b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,10 +5,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.6.0] - 2026-10-09 ### Added +- **MinorPlanet — Minor-planet designations:** new capability with one grammar (`minor_planet_recognition`: unpacked provisional incl. retrospective `A8xx`/`A9xx` and `A/` lanes with space/`_` separators, case-sensitive packed 7-char, extended `_` base-62, survey unpacked + packed, parenthesized and packed numbers, ASCII-only) and five structure-only rules backed by the Minor Planet Center DesDoc (`Section 1-unpacked-provisional-structure` — year/order/cycle, PARSER), PackedDes (`Section 3-packed-provisional` — packed/extended/survey-packed decode coherence, PARSER; `Section 4-packed-number` — 5-char/letter/tilde decode coherence, PARSER), and HowNamed (`Section 2-survey-designation` — P-L/T-1/T-2/T-3, PARSER; `Section 5-permanent-number` — parenthesized positive integer, PARSER; no registry). Canonical form is the unpacked designation (`1995 XA` / `(433)` / `2040 P-L`); offered output format `packed` (case-exact wire re-encoding, re-enters as a fixed point; the `A/` lane and non-20xx extended range fall back to the designation). Bare 5-digit packed numbers collide with ordinary numbers (ZIPs, phone fragments — caller segments these contexts); degenerate cycles (`XA0`), zero/leading-zero numbers (`(0)`) → `INVALID`; over-long years (`19955 XA`) → `MISSING`; two distinct designations in one call → `MultipleMentionsError`. + - **UNSPSC — Product/service codes:** new capability with one grammar (`unspsc_recognition`: bare 8-digit, zero-padded parents, 6-digit class alias, `UNSPSC`-labelled, `UNSPSC000.` MDM IDs, 10-digit +BFI, ASCII-only, longest-first 10/8/6) and four rules backed by the UNGM UNSPSC Code Structure article (`Section 1-hierarchy-structure` — 6/8/10 lanes, PARSER; `Section 2-level-padding` — `00`-pair lattice, PARSER), the UNDP codeset (UNGM live export 2026-09-30, 13,482 live stems plus 8 synthesized family ancestors (ancestor-walk only); `Section 3-codeset-membership` — stem plus live ancestors, always-active LOOKUP_TABLE; partial — UNDP v26.0801 XLSX gated 403) and the UNECE Classification Guidelines v2.04 (`Section 4-business-function-suffix` — informative-only, PARSER gated by `include_business_function=True`, default on; gating drops the suffix candidate, not the verdict). Canonical form is the 8-digit zero-padded wire stem; offered output formats `labeled` and `native` (both re-enter as fixed points; pair-hyphenated `segmented` display deliberately unoffered — hyphens never re-enter by design). Space/dot/hyphen-grouped, truncated 2/4-digit, and 7/9/11-digit runs → `MISSING`; mid-zero padding, unissued stems, Wikidata-cited-but-unexported codes (e.g. `11101803`) → `INVALID` under the pinned snapshot; two distinct codes in one call → `MultipleMentionsError`. - **Domain — Hostnames:** new capability with two grammars (`ascii_hostname`: LDH shapes, case-neutral, trailing-dot runs; `idn_hostname`: any script with reserved-ASCII excluded, ASCII overlap dedups by span) and five rules backed by RFC 1034 (`Section-3.1-name-syntax` — ≥2 labels, no empty label, PARSER), RFC 1035 (`Section-2.3.4-label-length` — 63 octets/label, 253 chars/name post-encode, PARSER), UTS #46 (`UTS46-statuses` — statuses with STD3 rules, D13 hyphen checks, ACE round trip, PARSER; shipped table 15.1.0), RFC 5893 (`Section-2-bidi-context` — §2 six conditions plus ContextJ join-control rejection, PARSER), and the IANA Root Zone Database v2026092300 (`root-zone-membership` — encoded-TLD membership ANDed with the other four, LOOKUP_TABLE; no corroboration → `INVALID`). Canonical form is the lowercase ASCII A-label dotted name with no trailing dot; offered output format `unicode` (per-label Punycode decode, re-enters as a fixed point). Single labels (`localhost`) and undelegated TLDs → `INVALID`; containers (`user@example.com`, URLs, `*.example.com`) → `MISSING` (boundary kills); two distinct hostnames in one call → `MultipleMentionsError`. diff --git a/CONTEXT.md b/CONTEXT.md index c8461e70..16e13813 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -776,7 +776,7 @@ for candidate in result.candidates: ### Capability Versioning - Capabilities do **not** declare their own versions — the `Capability` surface carries only `name`, grammars, rules, and the presentation seam. - - Library version is resolved in `paxman/engine/orchestrator.py` — `PAXMAN_VERSION = _resolve_version()` reads the installed `paxman` package version via `importlib.metadata` (falling back to `"0.2.1"`); source of truth is `version = "0.5.0"` in `pyproject.toml`. + - Library version is resolved in `paxman/engine/orchestrator.py` — `PAXMAN_VERSION = _resolve_version()` reads the installed `paxman` package version via `importlib.metadata` (falling back to `"0.2.1"`); source of truth is `version = "0.6.0"` in `pyproject.toml`. - Referenced in `VersionStamp.paxman_version` ### Contract Protocol diff --git a/docs/development/MILESTONE.md b/docs/development/MILESTONE.md index a905c6df..5016ac2a 100644 --- a/docs/development/MILESTONE.md +++ b/docs/development/MILESTONE.md @@ -2,7 +2,7 @@ > **Purpose:** Roadmap guidance for capabilities to bring into Paxman, ordered by priority. > This table is a planning aid — details need not be exactly accurate in terms of implementation. -> 29 capabilities are shipped (BIC, ChemicalElement, Coordinates, Country, CreditCard, Currency, Date, DOI, Domain, Email, GTIN, IBAN, IP, ISBN, ISIN, ISNI, ISSN, Language, LEI, MacAddress, Money, ORCID, Phone, SIUnit, Timezone, UNSPSC, URL, UtcOffset, UUID) and are listed in §1 only — they are excluded from the future roadmap in §2. +> 30 capabilities are shipped (BIC, ChemicalElement, Coordinates, Country, CreditCard, Currency, Date, DOI, Domain, Email, GTIN, IBAN, IP, ISBN, ISIN, ISNI, ISSN, Language, LEI, MacAddress, MinorPlanet, Money, ORCID, Phone, SIUnit, Timezone, UNSPSC, URL, UtcOffset, UUID) and are listed in §1 only — they are excluded from the future roadmap in §2. --- @@ -41,6 +41,7 @@ Verified against `paxman/capabilities/__init__.py` + `paxman/api/bootstrap.py:_S | 27 | **LEI** | ISO 17442-1:2020, GLEIF LOU prefix list | "5493000IBP32UQZ0KL24" → "5493000IBP32UQZ0KL24" | `research/2026-09-22-lei-canonicalization.md`, `plans/2026-09-22-lei-capability.md` | | 28 | **ISNI** | ISO 27729:2024 (MOD 11-2) | "ISNI 0000 0001 2103 2683" → "0000 0001 2103 2683" | `research/2026-09-26-isni-canonicalization.md`, `plans/2026-09-26-isni-capability.md` | | 29 | **UNSPSC** | UNDP UNSPSC (UNGM structure, UNGM live-export snapshot, UNECE BFI) | "UNSPSC 44103103" → "44103103" | `research/2026-09-30-unspsc-canonicalization.md`, `plans/2026-09-30-unspsc-capability.md` | +| 30 | **MinorPlanet** | MPC DesDoc, MPC PackedDes, MPC HowNamed | "J95X00A" → "1995 XA" | `research/2026-10-07-minor-planet-designation-canonicalization.md`, `plans/2026-10-07-minor-planet-capability.md` | Notes on §1: diff --git a/docs/user/v0.6.0/api-reference.md b/docs/user/v0.6.0/api-reference.md new file mode 100644 index 00000000..0f0a41c4 --- /dev/null +++ b/docs/user/v0.6.0/api-reference.md @@ -0,0 +1,366 @@ +--- +title: "API Reference" +slug: v0.6.0/api-reference +--- + +This page is the concise reference for the public Python surface you import. For the mental model behind these types, see [Concepts](concepts/). + +--- + +## Installation and imports + +```python +import paxman +from paxman.capabilities import ( + Country, + Currency, + Date, + Email, + IP, + ISBN, + Money, + Phone, + SIUnit, + URL, +) +from paxman.core.domain import Resolution +from paxman.core.errors import ( + PaxmanError, + CapabilityError, + ContractError, + MultipleMentionsError, + RecognitionError, + ValidationError, +) +``` + +Paxman has no runtime dependencies. `import paxman` is side-effect free — nothing runs until you register and call `canonicalize()`. + +--- + +## Registration + +Registration tells the engine which capabilities exist. It must complete from **one thread before the first `canonicalize()` call**; afterwards the registry freezes and reads are safe from any thread. + +```mermaid +flowchart LR + A[import paxman] --> B[register] + B --> C[first canonicalize
freezes registry] + C --> D[further canonicalize
thread-safe reads] + B -.->|after freeze| E[CapabilityError] + + style C fill:#fff8e1,stroke:#d4a017 + style E fill:#fff5f5,stroke:#cc3333 +``` + +| Function | Signature | What it does | +|----------|-----------|--------------| +| `paxman.register_all_shipped()` | `() -> tuple[str, ...]` | Registers every capability shipped in this release. Idempotent by name. Returns the names newly registered by this call. | +| `paxman.register_capability(cap)` | `(cap: Capability) -> None` | Registers one capability instance, e.g. `Email()`. Fails if the name already exists or the registry is frozen. | +| `paxman.register_grammar(name, cls)` | `(capability_name: str, grammar_cls: type[Grammar]) -> None` | Registers a community grammar (see [Extending](extending/)). Must happen before the first call. | +| `paxman.register_rule(name, cls)` | `(capability_name: str, rule_cls: type[Rule]) -> None` | Registers a community rule. Must happen before the first call. | + +```python +# Quick exploration — everything +paxman.register_all_shipped() + +# Explicit — only what you depend on +paxman.register_capability(Email()) +paxman.register_capability(Date()) +``` + +--- + +## `paxman.canonicalize()` + +The sole entry point. + +```python +def canonicalize(text: str, contract: CapabilityContract) -> ExecutionResult +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `text` | `str` | Raw input. One presumed mention per call (see [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md)). | +| `contract` | `CapabilityContract` | Created via `SomeCapability.create_contract(...)`. Carries `capability_name` so no separate name argument is needed. | + +**Returns** `ExecutionResult` — always, for every domain outcome including missing or invalid input. Domain outcomes are statuses, not exceptions. + +**Raises** + +| Exception | Cause | +|-----------|-------| +| `CapabilityError` | No capability matches `contract.capability_name` | +| `ContractError` | Malformed contract (unknown `pinned_rules`, unknown `output_format`, unknown semantics, missing required feature) | +| `MultipleMentionsError` | Two or more non-overlapping mentions resolved to different values — caller must split first | +| `RecognitionError` | Grammar raised or returned a malformed match (`rule`, `original_error`) | +| `ValidationError` | Rule raised inside `matches()`/`normalize()` (`rule`, `original_error`) | + +`MISSING` / `INVALID` / `AMBIGUOUS` do **not** raise — they are `ExecutionResult.status` values (see [Execution Result](concepts/execution-result/) and [Candidates & Ambiguity](concepts/candidates-and-ambiguity/)). + +--- + +## `paxman.scan()` + +Batch scan — one `ScanContext` substrate pass, per-capability `Mention` records. + +```python +def scan(text: str, contracts: Sequence[CapabilityContract]) -> ScanResult +``` + +| Parameter | Type | Description | +|-----------|------|-------------| +| `text` | `str` | Input text to scan (may contain many mentions). | +| `contracts` | `Sequence[CapabilityContract]` | One contract per capability to scan; each carries its `suppress_common_words` and other flags. | + +**Returns** `ScanResult` — per-capability `mentions` dict sharing one substrate. + +**Raises** + +| Exception | Cause | +|-----------|-------| +| `TypeError` | `text` is not `str` or `contracts` is not a `Sequence` | +| `CapabilityError` | A contract names an unregistered capability | + +The scan shares one `ScanContext` (word spans and lazy views) across all contracts, so querying many capabilities costs one substrate build. Suppression (`suppress_common_words`) is honored per-contract: a `Country` contract with `suppress_common_words=True` drops word-bounded hits like `to` → `TO` (Tonga) via `COMMON_WORDS` (67), while the same text scanned with `False` keeps them. For the honest F1 path on prose, `scan()` is the preferred successor to `MultipleMentionsError` on `canonicalize()` — see `docs/user/migration.md` and `docs/recipes/segmentation.md`. + +```python +import paxman +from paxman.capabilities.Country import Country +from paxman.core.errors import MultipleMentionsError + +paxman.register_all_shipped() +contract = Country.create_contract() + +# Prose with embedded values — the new honest path: +try: + result = paxman.canonicalize("Ship to United States please", contract) +except MultipleMentionsError: + # scan() shares one ScanContext substrate across all contracts in the batch + mentions = paxman.scan("Ship to United States please", [contract]) + # mentions.mentions["country"] == [ + # Mention(span=(5, 7), grammar="alpha2_recognition", notation=...), + # Mention(span=(8, 21), grammar="name_recognition", notation=...), + # ] + for m in mentions.mentions["country"]: + print(m.span, m.grammar, m.notation) +``` + +See [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md) for when to use `scan()` vs caller-owned split-then-canonicalize. + +--- + +## Contracts — `SomeCapability.create_contract()` + +Every capability exposes a **keyword-only** factory. Common parameters come first, capability-specific ones after. + +```python +contract = Email.create_contract( + # common — every capability + excluded_rules=(), # Sequence[str] + pinned_rules=None, # Sequence[str] | None — when not None, only these run + year=None, # int | None — only rules with publication_year <= year + output_format=None, # str | None — None / "default" / default / offered, else ContractError + extra_grammars=(), # tuple[str, ...] — opt-in community grammars + suppress_common_words=False, # bool — suppress word-bounded short-code hits in COMMON_WORDS (67) + # capability-specific — varies by capability + include_obfuscated=False, # example: Email only +) +``` + +The contract is a **frozen dataclass** — construct it, pass it, inspect it, but do not mutate it. See [Contracts](concepts/contracts/) for the full common-plus-specific table. + +### `output_format` policy (identical for every capability) + +- `None`, `"default"`, and the capability's `DEFAULT_OUTPUT_FORMAT` all resolve to the default. +- Any value in `OFFERED_OUTPUT_FORMATS` resolves to itself. +- Anything else raises `ContractError` immediately. Validation never consults `output_format`. + +Current defaults and offered alternatives: + +| Capability | Default (`DEFAULT_OUTPUT_FORMAT`) | Offered (`OFFERED_OUTPUT_FORMATS`) | +|------------|-----------------------------------|-------------------------------------| +| Country | `alpha2` | `alpha3`, `numeric`, `name` | +| Currency | `code` | *(none — single format)* | +| Date | `ISO` | `US` | +| DOI | `doi` | `url` | +| Email | `email` | *(none)* | +| IP | `ip` | *(none)* | +| ISBN | `isbn13` | `hyphenated` | +| Money | `code_amount` | `compact` | +| Phone | `e164` | `rfc3966`, `split` | +| SI Unit | `symbol` | *(none)* | +| URL | `url` | *(none)* | +| BIC | `bic` | `grouped`, `bic11` | +| Coordinates | `decimal` | `iso6709`, `geo_uri`, `geojson_pair`, `dms`, `dm` | +| ChemicalElement | `symbol` | `name` | +| IBAN | `electronic` | `paper` | +| ISSN | `hyphenated` | `compact`, `urn` | +| ISIN | `isin` | `grouped` | +| ISNI | `isni` | `compact`, `urn` | +| LEI | `lei` | `urn` | +| UNSPSC | `unspsc` | `labeled`, `native` | +| GTIN | `gtin14` | `native` | +| CreditCard | `pan` | `grouped` | +| Language | `bcp47` | `alpha2`, `alpha3`, `alpha3-bib`, `name` | +| MacAddress | `colon` | `hyphen`, `bare`, `cisco`, `eui64` | +| MinorPlanet | `designation` | `packed` | +| ORCID | `orcid` | `uri`, `compact` | +| Timezone | `iana` | *(none — single format)* | +| UtcOffset | `extended` | `basic` | +| UUID | `hyphenated` | `compact`, `braced`, `urn` | +| Domain | `ascii` | `unicode` | + +> The set of capabilities — and their offered formats — grows over time. Treat this table as the current release, not a closed list. + +### Capability-specific flag summary (current release) + +| Capability | Flag | Type | Default | Purpose | +|------------|------|------|---------|---------| +| Email | `include_obfuscated` | `bool` | `False` | `user at domain dot com` | +| Email | `include_localhost` | `bool` | `True` | `admin@localhost` | +| Country | `include_localized` | `bool` | `False` | CLDR multilingual names | +| Country | `include_historical` | `bool` | `False` | Deprecated names | +| Currency | `default_currency` | `str \| None` | `None` | Resolve shared bare symbol (`$`) to this alpha-3 code | +| IP | `include_ipv6` | `bool` | `True` | IPv6 | +| ISBN | `include_isbn10` | `bool` | `True` | Legacy ISBN-10 | +| ISBN | `include_range_validation` | `bool` | `False` | Registrant-range provenance | +| Language | `include_localized` | `bool` | `False` | CLDR localized display-name validation | +| Language | `include_collective` | `bool` | `False` | ISO 639-5 collective-code validation | +| Language | `include_private` | `bool` | `False` | Private-use subtag validation | +| GTIN | `include_verified` | `bool` | `False` | Verified-by-GS1 liveness lookup | +| CreditCard | `include_brand_validation` | `bool` | `False` | Brand IIN/length membership | +| Money | `dollar_sign_currency` | `str \| None` | `None` | Resolve bare `$` amount to this alpha-3 code | +| Money | `precision` | `str` | `"strict"` | `strict` / `truncate` / `round` for over-precision amounts | +| Phone | `default_country` | `str \| None` | `None` | Interpret national numbers as if in this alpha-2 country | +| SI Unit | `allow_split_word_prefixes` | `bool` | `False` | `kilo gram` → `kg` | +| SI Unit | `allow_multi_solidus` | `bool` | `False` | `kg/m/s` preserved | +| Date | `two_digit_base_year` | `int \| None` | `None` | Base for 2-digit year expansion | +| Timezone | `include_systemv` | `bool` | `False` | Validate SystemV zones (`EST5EDT` and kin) | +| common | `suppress_common_words` | `bool` | `False` | suppress word-bounded short-code hits in `COMMON_WORDS` (67) | + +Validation: `default_currency` / `dollar_sign_currency` must be uppercase alpha-3; `default_country` must be uppercase alpha-2; `precision` must be one of the three values — otherwise `ContractError` at construction time. + +--- + +## Execution result + +```python +@dataclass(frozen=True) +class ExecutionResult: + status: Resolution # MISSING | INVALID | SUCCESS | AMBIGUOUS + canonicalized_value: str | None # str on SUCCESS, None otherwise + candidates: tuple[Candidate, ...] # all validated evidence, deduplicated + contract: CapabilityContract # the contract you passed in + version_stamp: VersionStamp # .paxman_version + span: tuple[int, int] | None # [start, end) of resolved value on SUCCESS, else None + suppressed_count: int = 0 # hits dropped by suppress_common_words (ADR-0009 §16, #122) + suppressed_spans: tuple[tuple[int, int], ...] = () # [start, end) of each suppressed hit +``` + +```python +@dataclass(frozen=True) +class Candidate: + value: str + recognition_rule: str # grammar name, e.g. "standard_recognition" + validation_rule: str # rule name, e.g. "Section 3.4.1-addr-spec" + span: tuple[int, int] | None # half-open [start, end) in the input + provenance: tuple[ + Provenance, ... + ] # one or more authority citations (read-only property) + + +@dataclass(frozen=True) +class Provenance: + authority: str # "IETF", "ISO", "BIPM", "WHATWG", "CLDR", ... + specification_name: str # "RFC 5322", "ISO 3166-1", ... + kind: str # "specification" / "standard" / ... + reference_url: str + version: str | None + lifecycle: str # "active" / "deprecated" / ... + publication_year: int + + +@dataclass(frozen=True) +class VersionStamp: + paxman_version: str + recognition_revision: str # kernel recognition revision (default "0") + + +class Resolution(Enum): + MISSING = "missing" + INVALID = "invalid" + SUCCESS = "success" + AMBIGUOUS = "ambiguous" +``` + +**Reading the result** (see also [Execution Result](concepts/execution-result/)): + +```python +from paxman.core.domain import Resolution + +if result.status == Resolution.SUCCESS: + value = result.canonicalized_value # str, span is set +else: + # MISSING / INVALID / AMBIGUOUS — no single value, span is None + # inspect result.candidates and their provenance/spans + ... +``` + +`span` semantics: on `SUCCESS` it is the span of the single resolved value; on `AMBIGUOUS` use each `candidate.span`; on `MISSING`/`INVALID` it is `None` and `candidates` is empty. + +`suppressed_count` / `suppressed_spans`: populated whenever `suppress_common_words=True` suppression fires, on any status (ADR-0009 §16, A0 #122) — `0`/`()` when the flag is off or nothing was suppressed. Spans are in sorted positional order. A whole-input suppressible hit is never suppressed (A0 exemption), so `canonicalize("to", Country suppress on)` → `SUCCESS "TO"` with `0`/`()`, while `canonicalize("in/", …)` → `MISSING` with `suppressed_count == 1`, `suppressed_spans == ((0, 2),)`. + +--- + +## Errors + +All exceptions inherit from `PaxmanError`. See [Errors](concepts/errors/) for handling patterns. + +| Exception | Signal type | Typical cause | +|-----------|-------------|---------------| +| `CapabilityError` | setup | Unknown capability, duplicate name, registry already frozen | +| `ContractError` | setup | Bad `pinned_rules`, bad `output_format`, unknown semantics, missing required feature, bad `default_currency`/`default_country`/`precision` | +| `MultipleMentionsError` | usage | Two separate mentions with different values in one call — split first | +| `RecognitionError` | internal bug | Grammar raised or returned a bad span/raw_text. Fields: `rule`, `original_error` (`None` on structural failure) | +| `ValidationError` | internal bug | Rule raised inside `matches()`/`normalize()`. Fields: `rule`, `original_error` | + +Statuses vs exceptions: statuses are domain answers returned inside `ExecutionResult`; exceptions mean the call was not valid to attempt. Catch exceptions at setup and at the outer edge of a batch loop; branch on statuses in normal flow. + +--- + +## Quick lookup — choose the right capability + +| Kind of text you have | Capability to use | Factory | +|-----------------------|-------------------|---------| +| Email addresses | Email | `Email.create_contract(...)` | +| Dates | Date | `Date.create_contract(...)` | +| Country codes or names | Country | `Country.create_contract(...)` | +| Currency codes / symbols / names (no amount) | Currency | `Currency.create_contract(...)` | +| IP addresses | IP | `IP.create_contract(...)` | +| ISBNs | ISBN | `ISBN.create_contract(...)` | +| Money amounts with currency | Money | `Money.create_contract(...)` | +| Phone numbers | Phone | `Phone.create_contract(...)` | +| SI unit expressions | SI Unit | `SIUnit.create_contract(...)` | +| Absolute URLs / IRIs | URL | `URL.create_contract(...)` | +| Business identifier codes | BIC | `BIC.create_contract(...)` | +| WGS 84 coordinates | Coordinates | `Coordinates.create_contract(...)` | +| Chemical elements | ChemicalElement | `ChemicalElement.create_contract(...)` | +| Bank account numbers | IBAN | `IBAN.create_contract(...)` | +| Serial identifiers | ISSN | `ISSN.create_contract(...)` | +| Securities identification numbers | ISIN | `ISIN.create_contract(...)` | +| Researcher/organization identifiers | ISNI | `ISNI.create_contract(...)` | +| Product/service codes | UNSPSC | `UNSPSC.create_contract(...)` | +| Legal entity identifiers | LEI | `LEI.create_contract(...)` | +| Trade item identifiers | GTIN | `GTIN.create_contract(...)` | +| Payment card numbers (PANs) | CreditCard | `CreditCard.create_contract(...)` | +| Language identifiers | Language | `Language.create_contract(...)` | +| MAC addresses | MacAddress | `MacAddress.create_contract(...)` | +| Researcher identifiers | ORCID | `ORCID.create_contract(...)` | +| Time zone identifiers | Timezone | `Timezone.create_contract(...)` | +| UTC offset values | UtcOffset | `UtcOffset.create_contract(...)` | +| UUIDs | UUID | `UUID.create_contract(...)` | +| Digital object identifiers | DOI | `DOI.create_contract(...)` | +| Hostnames | Domain | `Domain.create_contract(...)` | + +New capabilities appear in minor releases — check `paxman.capabilities` for the current set. Each per-capability guide under [Capabilities](capabilities/) details its recognized forms, output formats, contract flags, and provenance. diff --git a/docs/user/v0.6.0/capabilities/bic.md b/docs/user/v0.6.0/capabilities/bic.md new file mode 100644 index 00000000..2cf65003 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/bic.md @@ -0,0 +1,153 @@ +--- +title: "BIC" +slug: v0.6.0/capabilities/bic +--- + +Canonicalizes **one BIC mention** per call — an 8- or 11-character Business Identifier Code in compact or SWIFT grouped display (with an optional `BIC`/`SWIFT` label) — to compact uppercase form. + +> **In plain language:** give it `deutdeff`, `BIC: DEUTDEFF500`, `DEUT DE FF`, or `BNPA FR PP XXX` and it hands back `DEUTDEFF`, `DEUTDEFF500`, or `BNPAFRPPXXX` when the shape is a real BIC structure per ISO 9362:2022 §5 with a known country code. The spec defines no checksum, so validation is structural (length 8 or 11, per-position charset, ISO 3166-1 country lookup plus `XK`); the location-code second character and the `XXX` head-office branch are informative and never rejected. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Compact BIC-8 (`DEUTDEFF`, lowercase folds up) and BIC-11 (`DEUTDEFF500`, `BNPAFRPPXXX`) | Wrong lengths (`DEUTDEF` 7, `DEUTDEFF5` 9, `DEUTDEFF50` 10, `DEUTDEFF5000` 12) → `MISSING` | +| SWIFT grouped display, single spaces only (`DEUT DE FF`, `DEUT DE FF 500`, `BNPA FR PP XXX`) | Double-spaced display (`DEUT DE FF`) → `MISSING` | +| Fused `BIC`/`SWIFT` label with separator (`BIC: DEUTDEFF`, `SWIFT: BNPAFRPPXXX`, `bic - NEDSZAJJ`) | Glued label (`BICDEUTDEFF`, `SWIFTDEUTDEFF500`, no separator) → `MISSING` | +| Kosovo `XK` user-assigned code (`BANKXK22`, `CBKIXKPRXXX`) | Unknown country (`DEUTXXFF`, `BNPAQQPP`) → recognized, then `INVALID` | +| Bank codes with digits (`DE1TDEFF`, 4!c since 2014) | Digit in the country position (`DEUT1EFF`, 2!a must be A–Z) → `MISSING` | +| | Glued surroundings (`XDEUTDEFF`, `DEUTDEFFY`) → `MISSING` | +| | Lowercase English trigrams at end of text (`call me at`, `call me at.`, `call me at noon`) → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"bic"` (identity — `normalize()` returns compact uppercase). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `bic` / `None` / `"default"` | Compact uppercase, branch as matched | `DEUTDEFF`, `DEUTDEFF500` | +| `grouped` | SWIFT paper form, single spaces between the 4-2-2-3 groups (encoding) | `DEUT DE FF` from `DEUTDEFF`, `DEUT DE FF 500` from `DEUTDEFF500` | +| `bic11` | Always 11 chars: BIC-8 gains `XXX` head office (lossy expansion); BIC-11 is identity | `DEUTDEFFXXX` from `DEUTDEFF` | + +Any other value raises `ContractError` — including `compact`, `paper`, and `hyphenated`, which belong to other capabilities' vocabularies, not BIC's. + +```python +from paxman.capabilities import BIC +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("deutdeff", BIC.create_contract()).canonicalized_value) +print(paxman.canonicalize("BIC: DEUTDEFF500", BIC.create_contract()).canonicalized_value) +print(paxman.canonicalize("DEUT DE FF", BIC.create_contract()).canonicalized_value) +print(paxman.canonicalize("DEUTDEFF", BIC.create_contract(output_format="grouped")).canonicalized_value) +print(paxman.canonicalize("DEUTDEFF", BIC.create_contract(output_format="bic11")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = BIC.create_contract( + output_format=None, # "bic" (default), "grouped", "bic11" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`bic_recognition`), one rule (`Section 5-bic-structure-country`). +- `year` filters by `publication_year`; e.g., `year=2021` drops the ISO 9362:2022 rule → `DEUTDEFF` becomes `INVALID`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `DEUTDEFF` | defaults | `SUCCESS` | `"DEUTDEFF"` | +| `deutdeff` | defaults | `SUCCESS` | `"DEUTDEFF"` (case folds up) | +| `BIC: DEUTDEFF` | defaults | `SUCCESS` | `"DEUTDEFF"` (label recognized, span covers `BIC DEUTDEFF`) | +| `DEUT DE FF` | defaults | `SUCCESS` | `"DEUTDEFF"` (grouped recognized) | +| `BNPA FR PP XXX` | defaults | `SUCCESS` | `"BNPAFRPPXXX"` (grouped 11 recognized) | +| `DEUTDEFF` | `output_format="grouped"` | `SUCCESS` | `"DEUT DE FF"` | +| `DEUTDEFF` | `output_format="bic11"` | `SUCCESS` | `"DEUTDEFFXXX"` (lossy `XXX` expansion) | +| `DEUTXXFF` | any | `INVALID` | recognized shape, unknown country code | +| `DEUTDEFF` | `year=2021` | `INVALID` | rule is 2022, dropped | +| `DEUTDEFF5` | any | `MISSING` | 9 chars, no shape matches | +| `DEUT1EFF` | any | `MISSING` | digit in country position, grammar charset blocks | +| `BICDEUTDEFF` | any | `MISSING` | glued label needs a separator | +| `XDEUTDEFF` | any | `MISSING` | word guard blocks left glue | +| `DEUT DE FF` | any | `MISSING` | double spaces not allowed in grouped display | +| `call me at noon` | any | `MISSING` | English-phrase filter, not a BIC | +| `DEUTDEFF / BNPAFRPP` | any | raises `MultipleMentionsError` | two distinct BICs | +| `DEUTDEFF` | `output_format="compact"` | raises `ContractError` | format not offered | + +A single grammar plus a single deterministic rule yields at most one value, so `AMBIGUOUS` is unreachable for this capability. Two distinct BICs raise `MultipleMentionsError`; two identical mentions coalesce to one `SUCCESS`. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
bic_recognition] + G --> R{Section 5
BIC structure + country} + R -->|structure + country hold| OK[SUCCESS] + R -->|no rule validates| INV[INVALID] + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import BIC +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = BIC.create_contract() + +rows = [ + "DEUTDEFF", + "deutdeff", + "DEUTDEFF500", + "BIC: DEUTDEFF", + "SWIFT: BNPAFRPPXXX", + "DEUT DE FF", + "DEUT DE FF 500", + "BNPA FR PP XXX", + "BANKXK22", + "DEUTXXFF", + "DEUT1EFF", + "DEUTDEFF5", + "BICDEUTDEFF", + "call me at noon", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:24} → {r.status.value:10} {val!r:16} ({rule})") + +try: + BIC.create_contract(output_format="compact") +except ContractError as e: + print(f"compact → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO 9362:2022** §5 BIC structure plus country lookup (length 8 or 11, per-position charset, ISO 3166-1 country plus `XK`; no checksum) — `Section 5-bic-structure-country` + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/chemical_element.md b/docs/user/v0.6.0/capabilities/chemical_element.md new file mode 100644 index 00000000..379dfe74 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/chemical_element.md @@ -0,0 +1,150 @@ +--- +title: "ChemicalElement" +slug: v0.6.0/capabilities/chemical_element +--- + +Canonicalizes **one element mention** per call — an IUPAC symbol (case-exact), an English element name (case-insensitive, including the `aluminum`/`cesium` aliases), or a labeled atomic number (`element 26`, `Z=26`) — to the proper-case IUPAC symbol. + +> **In plain language:** give it `Fe`, `fe`, `iron` (or `IRON`), `aluminum`, or `element 26` / `Z=26` and it hands back `Fe` when the shape is a real element designation per the IUPAC Red Book 2005 Ch. IR-3 (symbols and names) or the IUPAC Periodic Table 2022 (atomic numbers 1–118). The symbol branch is case-exact (`fe` folds to `Fe`, but `FE` is unclaimed by design), and atomic numbers need their label — bare `26` is never claimed. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Proper-case symbols (`Fe`, `C`, `Og`) and their all-lowercase fold (`fe` → `Fe`) | All-caps or mixed wrong-case symbols (`FE`, `fE`) → `MISSING` | +| English names, case-insensitive (`iron`, `Iron`, `IRON` → `Fe`); 120 names including the Table I aliases `aluminum` → `Al` and `cesium` → `Cs` | Retired or non-IUPAC names (`ununtrium`, `sulphur`, `ferrum`) → `MISSING` | +| Labeled atomic numbers (`element 26`, `Z=26`, `Z = 92`, `atomic number 118`); leading zeros fold (`element 026` → `26`) | Bare integers (`26`) → `MISSING` (the label is part of the pattern) | +| | Label glued without a separator (`element26`, `Z26`) → `MISSING` | +| | Isotope/formula-glued shapes (`Fe-56`, `56Fe`, `Fe2O3`, `NaCl`) → `MISSING` | +| | Unknown symbol-like tokens (`Xx`) → `MISSING` (unclaimed, not validated) | +| | Prose without an element shape (`hello world`) → `MISSING` | +| | Out-of-range labeled numbers (`element 119`, `Z = 300`) are recognized but no rule validates them → `INVALID` | + +--- + +## Canonical output + +Default `output_format` is `"symbol"` (identity — `normalize()` returns the proper-case symbol). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `symbol` / `None` / `"default"` | Proper-case IUPAC symbol | `Fe` | +| `name` | Lowercase IUPAC English name (encoding — 1:1 symbol↔name map; the lowercase rendering re-enters through the name path) | `iron` from `Fe` | + +`atomic_number` is deliberately not offered: rendering `Fe` as bare `"26"` cannot re-enter — bare integers are unclaimable by design, so `canonicalize("26")` is `MISSING` and the value would not be a fixed point. Store the `symbol` form for round-trippable output. Any other value raises `ContractError` — including `atomic_number`, `number`, `""`, and `"SYMBOL"` (case-sensitive). + +```python +from paxman.capabilities import ChemicalElement +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("Fe", ChemicalElement.create_contract()).canonicalized_value) +print(paxman.canonicalize("iron", ChemicalElement.create_contract()).canonicalized_value) +print(paxman.canonicalize("element 26", ChemicalElement.create_contract()).canonicalized_value) +print(paxman.canonicalize("Fe", ChemicalElement.create_contract(output_format="name")).canonicalized_value) +print(paxman.canonicalize("element 26", ChemicalElement.create_contract(output_format="name")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = ChemicalElement.create_contract( + output_format=None, # "symbol" (default), "name" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`chemical_element_recognition`), two rules (`Section IR-3.1-names-and-symbols`, `Section PTOE-element-registry`). +- `year` filters by `publication_year`; e.g., `year=2005` keeps the Red Book 2005 rule but drops the 2022 registry → `Fe` stays `SUCCESS` while `element 26` becomes `INVALID`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `Fe` | defaults | `SUCCESS` | `"Fe"` (symbol path) | +| `fe` | defaults | `SUCCESS` | `"Fe"` (lowercase folds) | +| `IRON` | defaults | `SUCCESS` | `"Fe"` (name path, case-insensitive) | +| `aluminum` | defaults | `SUCCESS` | `"Al"` (alias resolves, never renders) | +| `element 026` | defaults | `SUCCESS` | `"Fe"` (leading zeros fold) | +| `Z = 92` | defaults | `SUCCESS` | `"U"` (labeled atomic number) | +| `Iron (Fe)` | defaults | `SUCCESS` | `"Fe"` (co-referent mentions coalesce) | +| `26` | any | `MISSING` | bare integer, label required | +| `FE` | any | `MISSING` | all-caps unclaimed by design | +| `Xx` | any | `MISSING` | unclaimed symbol-like token | +| `Fe-56` | any | `MISSING` | isotope guard, no claim | +| `hello world` | any | `MISSING` | no element shape | +| `element 119` | any | `INVALID` | recognized label, out of range 1–118 | +| `Fe` | `year=2004` | `INVALID` | both rules (2005, 2022) dropped | +| `element 26` | `year=2005` | `INVALID` | registry rule is 2022, dropped | +| `Fe` | `output_format="atomic_number"` | raises `ContractError` | Z view not offered (no re-entry) | +| `Fe and Cu` (two distinct mentions) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
chemical_element_recognition] + G --> R1{Section IR-3.1
names and symbols} + G --> R2{Section PTOE
element registry} + R1 & R2 --> D{Dedup values} + D -->|one value| OK[SUCCESS] + D -->|none| INV[INVALID] + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import ChemicalElement +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = ChemicalElement.create_contract() + +rows = [ + "Fe", + "fe", + "IRON", + "aluminum", + "element 026", + "Z = 92", + "26", + "FE", + "element 119", + "hello world", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:16} → {r.status.value:10} {val!r:8} ({rule})") + +try: + ChemicalElement.create_contract(output_format="atomic_number") +except ContractError as e: + print(f"atomic_number → ContractError: {e}") +``` + +--- + +## Provenance + +- **Nomenclature of Inorganic Chemistry (IUPAC Recommendations 2005), Chapter IR-3** §IR-3.1 names and symbols of the elements (118 symbols; 120 names including the Table I footnoted alternatives `aluminum` and `cesium`) — `Section IR-3.1-names-and-symbols` +- **IUPAC Periodic Table of the Elements** 04 May 2022 release (118 elements, Z 1–118) — `Section PTOE-element-registry` + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/coordinates.md b/docs/user/v0.6.0/capabilities/coordinates.md new file mode 100644 index 00000000..36f02d7e --- /dev/null +++ b/docs/user/v0.6.0/capabilities/coordinates.md @@ -0,0 +1,160 @@ +--- +title: "Coordinates" +slug: v0.6.0/capabilities/coordinates +--- + +Canonicalizes **one WGS 84 coordinate mention** per call — a decimal pair, hemisphere-marked pair, DMS/DDM pair, `geo:` URI, ISO 6709 string, or GeoJSON lon-first pair (with optional altitude on the carriers) — to a signed lat-first decimal pair. + +> **In plain language:** give it `48.8566, 2.3522`, `40° 26′ 46″ N 79° 58′ 56″ W`, `geo:48.8566,2.3522`, `+48.8577+002.2950/`, or `[2.295, 48.8577]` and it hands back `48.8566, 2.3522` (and friends) when the shape is a real WGS 84 point per ISO 6709:2022 §6, RFC 5870 §3.3, or RFC 7946 §3.1.1. Latitude must land in [-90, 90], longitude in [-180, 180]; a foreign CRS label is rejected, never silently re-datumed. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Decimal pairs with `,`, `;`, `/`, or marked-whitespace separators (`48.8566, 2.3522`, `41.5;-81.0`, `+40.446 -79.982`, parenthesized `(41.5, -81.0)`) | Bare prose number runs (`pages 12 40`) and dotted quads (`192.168.1.1`) → `MISSING` | +| Hemisphere letters, front or back, any case (`N 48.8566, E 2.3522`, `41.5 N 81.0 W`) | Contradictory sign plus hemisphere (`-41.5 N, -81.0 W`) → `INVALID` | +| DMS (`40° 26′ 46″ N 79° 58′ 56″ W`, ASCII `23 26' 22" N 23 27' 30" E`) and DDM (`40° 26.767′ N 79° 58.933′ W`) | Unit overflow (`40° 70′ 0″ N …`, minutes ≥ 60) → `INVALID` | +| `geo:` URIs with optional altitude and explicit WGS 84 CRS (`geo:48.8577,2.295,350`, `geo:48.8566,2.3522;crs=wgs84`, `crs=WGS_84` also accepted) | Foreign CRS (`geo:48.8566,2.3522;crs=ed50`) → `INVALID` — no silent datum transform | +| ISO 6709 strings, degree or DM/DMS widths, with optional altitude (`+48.8577+002.2950/`, `+48.8577+002.2950+350/`) | Missing Annex H solidus (`+48.8577+002.2950`) → `INVALID` | +| GeoJSON lon-first pairs, inverted losslessly (`[2.295, 48.8577]`, `[2.295, 48.8577, 350]`) | Out-of-range envelopes (`91.0, 0.0`, `0.0, 181.0`) → `INVALID` | +| Up to 7 fraction digits, quantized to the 6 dp canonical quantum | 8+-digit fractions (`48.85660005, 2.3522`) → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"decimal"` (identity — `normalize()` returns the lat-first pair `lat, lon[, alt]`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `decimal` / `None` / `"default"` | Signed lat-first pair, 6 dp round-half-even, trailing zeros stripped, `-0` folded to `0` | `48.8566, 2.3522` | +| `iso6709` | ISO 6709 Annex H string expression, fraction padded to 4 places (encoding) | `+48.8566+002.3522/` | +| `geo_uri` | RFC 5870 `geo:` URI (encoding) | `geo:48.8566,2.3522` | +| `geojson_pair` | RFC 7946 lon-first pair (encoding) | `[2.3522, 48.8566]` | +| `dms` | Degrees-minutes-integer-seconds (documented quantization, 1″ render quantum) | `48°51′24″N 2°21′8″E` | +| `dm` | Degrees-decimal-minutes to 0.001′ (documented quantization) | `48°51.396′N 2°21.132′E` | + +Quantization is observable: `48.8566005, 2.0` canonicalizes to `48.8566, 2` while `48.8566015, 2.0` rounds half-even up to `48.856602, 2`; `-0.0, -0.0` folds to `0, 0`. Altitude, when present, rides along in every format (`geo:48.8577,2.295,350` → `+48.8577+002.2950+350/`, `geo:48.8577,2.295,350`, `[2.295, 48.8577, 350]`, `…E, 350`). `dms`/`dm` drop sub-quantum digits, so re-canonicalizing a `dms` rendering is a fixed point that drifts within half a render quantum — never stored as canonical. + +Any other value raises `ContractError` — including `utm`, which this capability has never offered (there is no silent datum transform to project onto). + +```python +from paxman.capabilities import Coordinates +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("48.8566, 2.3522", Coordinates.create_contract()).canonicalized_value) +print(paxman.canonicalize("40° 26′ 46″ N 79° 58′ 56″ W", Coordinates.create_contract()).canonicalized_value) +print(paxman.canonicalize("geo:48.8566,2.3522", Coordinates.create_contract()).canonicalized_value) +print(paxman.canonicalize("+48.8577+002.2950/", Coordinates.create_contract()).canonicalized_value) +print(paxman.canonicalize("[2.295, 48.8577]", Coordinates.create_contract()).canonicalized_value) +print(paxman.canonicalize("48.8566, 2.3522", Coordinates.create_contract(output_format="iso6709")).canonicalized_value) +print(paxman.canonicalize("48.8566, 2.3522", Coordinates.create_contract(output_format="dms")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = Coordinates.create_contract( + output_format=None, # "decimal" (default), "iso6709", "geo_uri", "geojson_pair", "dms", "dm" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`coordinates_recognition`), four rules (`Section 6-coordinate-structure`, `Section Annex-h-string-expression`, `Section 3.3-geo-uri-validity`, `Section 3.1.1-position`). +- `year` filters by `publication_year`; e.g., `year=2010` drops the ISO 6709:2022 rules → `48.8566, 2.3522` becomes `INVALID` while `geo:48.8566,2.3522` still validates via RFC 5870 (2010). + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `48.8566, 2.3522` | defaults | `SUCCESS` | `"48.8566, 2.3522"` | +| `40° 26′ 46″ N 79° 58′ 56″ W` | defaults | `SUCCESS` | `"40.446111, -79.982222"` (DMS recognized) | +| `geo:48.8566,2.3522` | defaults | `SUCCESS` | `"48.8566, 2.3522"` (RFC 5870) | +| `+48.8577+002.2950/` | defaults | `SUCCESS` | `"48.8577, 2.295"` (two ISO rules agree, one value) | +| `[2.295, 48.8577]` | defaults | `SUCCESS` | `"48.8577, 2.295"` (lon-first inverted) | +| `hello world` | any | `MISSING` | no coordinate shape | +| `48.85660005, 2.3522` | any | `MISSING` | 8-digit fraction exceeds the 7-digit recognition cap | +| `91.0, 0.0` | any | `INVALID` | latitude outside [-90, 90] | +| `-41.5 N, -81.0 W` | any | `INVALID` | sign/hemisphere conflict | +| `40° 70′ 0″ N 79° 0′ 0″ W` | any | `INVALID` | minutes ≥ 60 | +| `geo:48.8566,2.3522;crs=wgs84` | defaults | `SUCCESS` | `"48.8566, 2.3522"` (explicit WGS 84 CRS) | +| `geo:48.8566,2.3522;crs=ed50` | any | `INVALID` | foreign CRS, never re-datumed | +| `+48.8577+002.2950` | any | `INVALID` | Annex H solidus missing | +| `48.8566, 2.3522` | `year=2010` | `INVALID` | Section 6 rule is 2022, dropped | +| `geo:48.8566,2.3522` | `year=2010` | `SUCCESS` | RFC 5870 rule is 2010, kept | +| `48.8566, 2.3522` | `output_format="utm"` | raises `ContractError` | format never offered | + +One grammar (`single_value=True`) plus rules that all normalize to the same compact pair yields at most one value, so `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
coordinates_recognition] + G --> R{Section 6 / Annex H
Section 3.3 / Section 3.1.1} + R -->|a rule validates| OK[SUCCESS] + R -->|no rule validates| INV[INVALID] + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import Coordinates +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = Coordinates.create_contract() + +rows = [ + "48.8566, 2.3522", + "40° 26′ 46″ N 79° 58′ 56″ W", + "40° 26.767′ N 79° 58.933′ W", + "geo:48.8577,2.295,350", + "+48.8577+002.2950/", + "[2.295, 48.8577]", + "(41.5, -81.0)", + "48.8566005, 2.0", + "-0.0, -0.0", + "91.0, 0.0", + "-41.5 N, -81.0 W", + "hello world", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:38} → {r.status.value:10} {val!r:28} ({rule})") + +try: + Coordinates.create_contract(output_format="utm") +except ContractError as e: + print(f"utm → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO 6709:2022** §6 Coordinate structure (decimal/DMS/DDM/ISO shapes, sign/hemisphere consistency, lat [-90, 90] / lon [-180, 180]) — `Section 6-coordinate-structure` +- **ISO 6709:2022** Annex H String expression of a point (trailing solidus, WGS 84 CRS family) — `Section Annex-h-string-expression` +- **IETF RFC 5870 (2010)** §3.3 Geo URI validity (`geo:` branch, WGS 84 only) — `Section 3.3-geo-uri-validity` +- **IETF RFC 7946 (2016)** §3.1.1 Position (GeoJSON lon-first pair, altitude optional) — `Section 3.1.1-position` + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/country.md b/docs/user/v0.6.0/capabilities/country.md new file mode 100644 index 00000000..767ceeaa --- /dev/null +++ b/docs/user/v0.6.0/capabilities/country.md @@ -0,0 +1,143 @@ +--- +title: "Country" +slug: v0.6.0/capabilities/country +--- + +Canonicalizes **one country mention** per call — a code or a name — to a canonical code. + +> **In plain language:** give it `"US"`, `"United States"`, `"DE"`, or `"Alemania"` and it hands back the ISO alpha-2 code if a real spec says that name means that country. Multilingual and historical names are off by default so results are predictable; turn them on when you need them. + +--- + +## What it recognizes — and what it does not + +| Recognizes (current release — growing) | Does not recognize | +|----------------------------------------|--------------------| +| ISO 3166-1 alpha-2 (`US`, `DE`), alpha-3 (`USA`), numeric (`840`) | Ad-hoc abbreviations without a spec (`'Murica`) | +| Country names in English (`United States`, `Germany`) | Regions or subdivisions (`California`, `Bavaria`) — countries only | +| CLDR localized names (`Alemania` → `DE`) — only when `include_localized=True` | Localized names when the flag is off — recognized but `INVALID` (no authority claims them) | +| Historical/deprecated names (`Burma`) — only when `include_historical=True`, via ISO 3166-3 | Historical names when the flag is off — `INVALID` | + +--- + +## Canonical output + +Default `output_format` is `"alpha2"`. + +| `output_format` | Renders | Example for United States | +|-----------------|---------|---------------------------| +| *(default)* `alpha2` / `None` / `"default"` | ISO 3166-1 alpha-2 | `US` | +| `alpha3` | ISO 3166-1 alpha-3 | `USA` | +| `numeric` | ISO 3166-1 numeric | `840` | +| `name` | English short name | `United States` | + +Any other value raises `ContractError`. + +```python +from paxman.capabilities import Country +import paxman + +paxman.register_all_shipped() +paxman.canonicalize( + "United States", Country.create_contract() +).canonicalized_value # "US" +paxman.canonicalize( + "United States", Country.create_contract(output_format="alpha3") +).canonicalized_value # "USA" +paxman.canonicalize( + "United States", Country.create_contract(output_format="numeric") +).canonicalized_value # "840" +paxman.canonicalize( + "US", Country.create_contract(output_format="name") +).canonicalized_value # "United States" +``` + +--- + +## Contract + +```python +contract = Country.create_contract( + include_localized=False, # bool, default False — CLDR multilingual names + include_historical=False, # bool, default False — ISO 3166-3 deprecated names + output_format=None, # "alpha2" (default), "alpha3", "numeric", "name" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- `include_localized` gates the CLDR localized-name rule; when `False`, `Alemania` is recognized but **no rule validates it** → `INVALID`. +- `include_historical` gates ISO 3166-3; a historical name that validates returns the **historical entity's own former code** (e.g. `Burma` → `BU`), not a successor state's code. + +Historical names can also be viewed as excluding the current ISO 3166-1 name rules and pinning to the 3166-3 rule — the contract controls which specs are consulted (see [Contracts](../concepts/contracts/)). + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `US` | defaults | `SUCCESS` | `"US"` | +| `United States` | defaults | `SUCCESS` | name → `"US"` | +| `Alemania` | defaults | `INVALID` | recognized (name grammar) but no rule accepts it without `include_localized` | +| `Alemania` | `include_localized=True` | `SUCCESS` | → `"DE"` with CLDR provenance | +| `Burma` | defaults | `INVALID` | needs `include_historical` | +| `Burma` | `include_historical=True` | `SUCCESS` | → `"BU"` (historical entity's own code) | +| `ZZ` | any | `INVALID` or `MISSING` | no spec claims it | +| `hello world` | any | `MISSING` | no country pattern | +| `US and DE` (two different mentions) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text e.g. Alemania] --> G[Grammars:
alpha2 / alpha3 / numeric / name] + G --> R1{ISO 3166-1 name rule} + G --> R2{CLDR localized rule
only if include_localized} + G --> R3{ISO 3166-3 historical
only if include_historical} + R1 & R2 & R3 --> D{Dedup values} + D -->|one value| OK[SUCCESS] + D -->|none| INV[INVALID] + G -->|no pattern| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import Country +from paxman.core.domain import Resolution +from paxman.core.errors import CapabilityError, ContractError, MultipleMentionsError + +paxman.register_all_shipped() +contract = Country.create_contract(include_localized=True) +contract_hist = Country.create_contract(include_localized=True, include_historical=True) + +rows = ["US", "United States", "Alemania", "Burma", "not a country", "US and DE"] + +for text in rows: + for label, c in [("default+localized", contract), ("+historical", contract_hist)]: + try: + r = paxman.canonicalize(text, c) + except (MultipleMentionsError, CapabilityError, ContractError) as e: + print(f"{text!r:20} [{label}] → exception {type(e).__name__}: {e}") + continue + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + prov = r.candidates[0].provenance[0].specification_name if r.candidates else "—" + print(f"{text!r:20} [{label:18}] → {r.status.value:10} {val!r:8} ({prov})") +``` + +--- + +## Provenance + +- **ISO 3166-1** (alpha-2 / alpha-3 / numeric / English names) +- **ISO 3166-3** (historical, when enabled) +- **CLDR** (Unicode localized names, when enabled) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/credit_card.md b/docs/user/v0.6.0/capabilities/credit_card.md new file mode 100644 index 00000000..f7c25e9c --- /dev/null +++ b/docs/user/v0.6.0/capabilities/credit_card.md @@ -0,0 +1,197 @@ +--- +title: "CreditCard" +slug: v0.6.0/capabilities/credit_card +--- + +Canonicalizes **one payment card number (PAN) mention** per call — a compact 12–19-digit run, a space- or hyphen-grouped form, or a `PAN:`-labelled form — to the compact contiguous ASCII digits. + +> **In plain language:** give it `4111111111111111`, `4111 1111 1111 1111`, `4716-2210-5188-5662`, `3782 822463 10005`, or `PAN: 4111 1111 1111 1111` and it hands back `4111111111111111` when the run is 12–19 contiguous ASCII digits and the Luhn MOD-10 check digit passes per ISO/IEC 7812-1:2017 §5 + Annex B. A single flipped digit (`4111111111111112`) is `INVALID`, never a guess; masked, truncated, tokenized, and out-of-window shapes are `MISSING`. Brand prefix/length checking (Visa, Mastercard, American Express, Discover, Diners Club, JCB, China UnionPay, Maestro) is available behind `include_brand_validation=True` and off by default. + +> **PCI handling warning:** `SUCCESS` means the digits pass ISO/IEC 7812-1 structure + Luhn — it is **not** an authorization, an issuer lookup, or proof that an account exists. Real PANs are cardholder data under PCI DSS: Paxman makes no network calls and confers no compliance. Keep PANs out of logs, mask them in display per your PCI DSS obligations, and put publicly documented test PANs (like `4111111111111111`) in fixtures rather than live cardholder data. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Compact contiguous run (`4111111111111111`, `378282246310005`, `36050234196908`) | Runs outside the 12–19 window (`41111111111`, `4111111111111111111111`) → `MISSING` (floor/ceiling, never carved) | +| Spaced 4-4-4-4 grouping (`4111 1111 1111 1111`, `4242 4242 4242 4242`) | Double spaces / tabs (`4111 1111 1111 1111`) → `MISSING` (a single space or hyphen per gap only) | +| Dashed 4-4-4-4 grouping (`4716-2210-5188-5662`) and mixed space+hyphen (`4444-3333 2222 1111`) | Dot / underscore / slash grouping (`4111.1111.1111.1111`) → `MISSING` (**DEFER** — add a community grammar via `extra_grammars`) | +| Amex 4-6-5 grouping (`3782 822463 10005`, `3714 496353 98431`) | Masked / truncated (`4111****1111`, `411111...1111`) → `MISSING` (**REJECT** — PCI handling outputs are never a PAN) | +| 14-digit Diners grouping (`3622 720627 1667`) | Token (`tok_visa_abc123`) → `MISSING` (**REJECT** — different identifier domain) | +| Label with separator (`PAN: 4111 1111 1111 1111`, `card number: 4111111111111111`, `credit card 4111…`, `cc 4111…` — case-insensitive) | OCR digit look-alikes (`4l11111111111111`) → `MISSING` (**REJECT** — ASCII digits only) | +| Expiry adjacency (`4111111111111111 12/27` — the span stops before the date) | Glued label / letter-glued run (`PAN4111111111111111`, `X4111111111111111`) → `MISSING` (a separator or word boundary is required) | + +--- + +## Canonical output + +The default `output_format` is `"pan"` — the compact digits, unchanged. + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `pan` / `None` / `"default"` | Compact contiguous ASCII digits — the canonical identity | `4111111111111111` | +| `grouped` | Groups of four from the left, last group takes the remainder (encoding — the same digits re-chunked, so it re-enters exactly; grouping-agnostic, so a 15-digit Amex renders `3782 8224 6310 005`, never brand 4-6-5 — presentation never sniffs brand) | `3782 8224 6310 005` from `378282246310005` | + +Any other value raises `ContractError` — including `compact`, which is not an offered alias (the digits are already compact). + +```python +from paxman.capabilities import CreditCard +import paxman + +paxman.register_all_shipped() +print( + paxman.canonicalize( + "4111 1111 1111 1111", CreditCard.create_contract() + ).canonicalized_value +) +print( + paxman.canonicalize( + "PAN: 378282246310005", CreditCard.create_contract() + ).canonicalized_value +) +print( + paxman.canonicalize( + "4716-2210-5188-5662", CreditCard.create_contract() + ).canonicalized_value +) +print( + paxman.canonicalize( + "378282246310005", CreditCard.create_contract(output_format="grouped") + ).canonicalized_value +) +print( + paxman.canonicalize( + "4111111111111111 12/27", CreditCard.create_contract() + ).canonicalized_value +) +``` + +--- + +## Contract + +```python +contract = CreditCard.create_contract( + output_format=None, # "pan" (default), "grouped" + include_brand_validation=False, # opt-in brand prefix/length allowlist + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`pan_recognition`), two rules (`Section 5-pan-structure-luhn`, `Section 1-brand-prefix-membership`). +- `include_brand_validation` (default `False`) gates the brand rule: off, any Luhn-valid 12–19-digit run is `SUCCESS` even if no brand claims the prefix; on, the prefix **and** length must fall inside the eight-brand allowlist and the brand rule re-checks structure + Luhn locally — ADR-0012 corroboration, never a waiver of the ISO rule. +- `year` filters by `publication_year`; e.g. `year=2016` drops both rules (2017, 2026) → `4111111111111111` becomes `INVALID`, while `year=2017` keeps the ISO structure rule → still `SUCCESS`. +- Single-value: two distinct PANs in one call raise `MultipleMentionsError` — split first per `docs/recipes/segmentation.md`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `4111111111111111` | defaults | `SUCCESS` | `"4111111111111111"` | +| `4111 1111 1111 1111` | defaults | `SUCCESS` | `"4111111111111111"` (grouping is presentation-only, stripped) | +| `4716-2210-5188-5662` | defaults | `SUCCESS` | `"4716221051885662"` (dashes stripped) | +| `3782 822463 10005` | defaults | `SUCCESS` | `"378282246310005"` (4-6-5 collapses to the compact twin) | +| `3622 720627 1667` | defaults | `SUCCESS` | `"36227206271667"` (14-digit Diners grouping) | +| `4444-3333 2222 1111` | defaults | `SUCCESS` | `"4444333322221111"` (mixed separators) | +| `PAN: 4111 1111 1111 1111` | defaults | `SUCCESS` | label absorbed into the span | +| `4111111111111111 12/27` | defaults | `SUCCESS` | `"4111111111111111"`, span `(0, 16)` — the date guard keeps the expiry out of the claim | +| `9999999999999995` | defaults | `SUCCESS` | Luhn-valid, brand gate off (unknown prefix accepted) | +| `9999999999999995` | `include_brand_validation=True` | `INVALID` | prefix falls outside the allowlist | +| `4111111111111111` | `include_brand_validation=True` | `SUCCESS` | `visa` `4` / {13, 16, 19} corroborates | +| `6200000000000005` | `include_brand_validation=True` | `SUCCESS` | `china-unionpay` `62` / {16–19} corroborates | +| `6200000000000001` | any | `INVALID` | recognized UnionPay shape, Luhn fails — the brand rule never waives | +| `4111111111111112` | any | `INVALID` | recognized shape, Luhn check digit fails | +| `4111111111111111` | `output_format="grouped"` | `SUCCESS` | `"4111 1111 1111 1111"` | +| `378282246310005` | `output_format="grouped"` | `SUCCESS` | `"3782 8224 6310 005"` (4-4-4-3 re-chunk — never brand 4-6-5) | +| `4111111111111111` | `pinned_rules=("Section 5-pan-structure-luhn",)` | `SUCCESS` | ISO rule alone corroborates (vacuity clause) | +| `4111111111111111` | `year=2016` | `INVALID` | both rules are newer, dropped | +| `4111111111111111` | `year=2017` | `SUCCESS` | ISO structure rule (2017) survives | +| `4111111111111111` | `excluded_rules=("Section 5-pan-structure-luhn", "Section 1-brand-prefix-membership")` | `INVALID` | no rule validates | +| `41111111111111111` (17-run) | any | `INVALID` | the whole 17-digit run is claimed atomically, Luhn arbitrates — never carved to 16 | +| `order 12 4111111111111111` | any | `INVALID` | soft-joined digits claim the whole run (18 digits), Luhn arbitrates | +| `4111 1111 1111 1111` (double space; tabs likewise) | any | `MISSING` | double spaces and tabs are not grouping separators | +| `4111.1111.1111.1111` (also `_` and `/`) | any | `MISSING` | **DEFER** — not recognized in v1; a community grammar via `extra_grammars` is the extension path | +| `4111****1111`, `411111...1111` | any | `MISSING` | **REJECT** — masked/truncated are handling outputs, not PANs | +| `tok_visa_abc123` | any | `MISSING` | **REJECT** — token domain, not a PAN | +| `4l11111111111111` | any | `MISSING` | **REJECT** — OCR `l` is not an ASCII digit | +| `41111111111`, `4111111111111111111111` | any | `MISSING` | outside the 12–19-digit window (11 and 22 digits) | +| `PAN4111111111111111` | any | `MISSING` | glued label needs a separator | +| `X4111111111111111` | any | `MISSING` | letter-glued digit run | +| `4111111111111111` | `output_format="compact"` | raises `ContractError` | not an offered format | +| Two distinct PANs in one call | any | raises `MultipleMentionsError` | split first | + +Two deterministic rules over one single-value grammar yield at most one value, so `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
pan_recognition] + G --> R1{Section 5
structure + Luhn MOD-10} + R1 --> R2{Section 1
brand prefix (gated)} + R2 -->|corroborates or gate off| OK[SUCCESS] + R1 -->|no rule validates| INV[INVALID] + R2 -->|outside allowlist| INV + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import CreditCard +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = CreditCard.create_contract() + +rows = [ + "4111111111111111", + "4111 1111 1111 1111", + "4716-2210-5188-5662", + "3782 822463 10005", + "PAN: 4111 1111 1111 1111", + "4111111111111111 12/27", + "4111111111111112", + "6200000000000005", + "9999999999999995", + "4111 1111 1111 1111", + "4111.1111.1111.1111", + "4111****1111", + "41111111111", + "not a card", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:32} → {r.status.value:10} {val!r:24} ({rule})") + +try: + CreditCard.create_contract(output_format="compact") +except ContractError as e: + print(f"compact → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO/IEC 7812-1:2017** §5 PAN structure (IIN + individual account identifier + check digit, 12–19 digits) and Annex B Luhn MOD-10 double-add-double check digit — `Section 5-pan-structure-luhn` (`PARSER`). Catalogue at `https://www.iso.org/standard/70484.html`. The check algorithm itself is Hans Peter Luhn's modulus-10 scheme, published as US Patent 2,950,048 and adopted verbatim by Annex B. +- **Brand IIN/length tables** (eight brands — `visa`, `mastercard`, `amex`, `discover`, `diners`, `jcb`, `china-unionpay`, `maestro` — each a prefix-interval set plus allowed lengths) — `Section 1-brand-prefix-membership` (`LOOKUP_TABLE`), gated behind `include_brand_validation`. Secondary evidence, cross-checked against the payment card number reference table at `https://en.wikipedia.org/wiki/Payment_card_number`; the per-network brand specs remain authoritative. + +Deferred in v1: the "Others" brand tier (Mir, RuPay, Troy, UATP and friends) is deliberately absent from the allowlist — a structurally valid PAN of one of those brands is `SUCCESS` with the brand gate off (the default) and `INVALID` with it on. Dot/underscore/slash grouping is a **DEFER** — add it as a community grammar via `extra_grammars`; masked, truncated, tokenized, and OCR spellings are **REJECT**s and must never be repaired into a PAN. + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/currency.md b/docs/user/v0.6.0/capabilities/currency.md new file mode 100644 index 00000000..858b1b66 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/currency.md @@ -0,0 +1,142 @@ +--- +title: "Currency" +slug: v0.6.0/capabilities/currency +--- + +Canonicalizes **one currency identifier** per call — a code, a symbol, or a display name — to the uppercase ISO 4217 alpha-3 code. No amounts: `"USD 500"` is the [Money](money/) capability's domain. + +> **In plain language:** give it `"usd"`, `"€"`, `"euro"`, or `"$"` and it tells you which currency code the spec says that identifier means. A bare shared symbol like `"$"` is deliberately `INVALID` unless you say which currency you mean. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Lowercase or uppercase alpha-3 codes (`usd` → `USD`) | Codes with wrong length (`US`, `USDD`) — not matched by the grammar, therefore `MISSING` | +| CLDR display-name words (`euro` → `EUR`, `yen` → `JPY`) | Amount-glued tokens like `US$5` — not recognized at all | +| CLDR currency symbols (`€` → `EUR`, `¥` → `JPY`) | Bare shared symbols like `"$"` without disambiguation — recognized but `INVALID` unless you opt in | +| Shared bare symbols (`$`) — 29 codes share this CLDR symbol | A `default_currency` code that is not one of that symbol's own candidates — stays `INVALID` | + +> **Why shared symbols fail by default:** `"$"` is the CLDR symbol for 29 different currencies. Guessing would be wrong most of the time, so Paxman requires you to opt in. + +--- + +## Canonical output + +Single format — always an uppercase alpha-3 code. + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `code` / `None` / `"default"` | `USD`, `EUR`, … | + +Any other value raises `ContractError`. + +```python +from paxman.capabilities import Currency +import paxman + +paxman.register_all_shipped() +Currency.create_contract().output_format # "code" +paxman.canonicalize("usd", Currency.create_contract()).canonicalized_value # "USD" +paxman.canonicalize("euro", Currency.create_contract()).canonicalized_value # "EUR" +``` + +--- + +## Contract + +```python +contract = Currency.create_contract( + default_currency=None, # str | None — uppercase alpha-3, e.g. "USD" + output_format=None, # "code" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- When `default_currency` is `None` (the default), a shared bare symbol like `"$"` is recognized but never resolved → `INVALID`. +- When `default_currency` is set, the symbol resolves **only if** that code is one of the symbol's own CLDR candidates. `"$"` with `default_currency="USD"` → `USD`; `"$"` with `default_currency="MYR"` stays `INVALID` because `MYR`'s symbol is `RM`, not `"$"`. +- `default_currency` never remaps a definitive symbol (`€` is always `EUR`) or a qualified symbol (`US$` is always `USD`). + +```python +from paxman.capabilities import Currency +import paxman + +paxman.register_all_shipped() + +paxman.canonicalize("$", Currency.create_contract()).status.value # "invalid" +paxman.canonicalize( + "$", Currency.create_contract(default_currency="USD") +).canonicalized_value # "USD" +paxman.canonicalize( + "$", Currency.create_contract(default_currency="MYR") +).status.value # "invalid" — MYR is not a $ candidate +``` + +See [Contracts](../concepts/contracts/) and the [API Reference](../api-reference/#contracts--somecapabilitycreate_contract). + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `usd` | defaults | `SUCCESS` | → `USD` (case fold + ISO 4217 validation) | +| `euro` | defaults | `SUCCESS` | CLDR display name → `EUR` | +| `€` | defaults | `SUCCESS` | definitive symbol → `EUR` | +| `$` | defaults | `INVALID` | shared bare symbol, no `default_currency` | +| `$` | `default_currency="USD"` | `SUCCESS` | `USD` is a `$` candidate | +| `US$` | any | `SUCCESS` | qualified symbol → `USD` | +| `notacurrency` | any | `MISSING` | no pattern | +| `USD and EUR` (two different values) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text e.g. $] --> G[Grammars:
code / symbol / word] + G --> R{Rules:
ISO 4217 code / CLDR symbols / CLDR names} + R -->|definitive symbol
or code/word| OK[SUCCESS] + R -->|shared bare symbol
without matching default_currency| INV[INVALID] + R -->|shared + matching default_currency| OK2[SUCCESS
via default_currency] + G -->|nothing matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style OK2 fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import Currency +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c_default = Currency.create_contract() +c_usd = Currency.create_contract(default_currency="USD") + +for text in ["usd", "euro", "€", "$", "US$", "notacurrency"]: + r = paxman.canonicalize(text, c_default) + tag = r.status.value + val = r.canonicalized_value or "—" + print(f"{text!r:10} default → {tag:10} {val!r}") + if text == "$": + r2 = paxman.canonicalize(text, c_usd) + print( + f" with default_currency=USD → {r2.status.value:10} {r2.canonicalized_value!r}" + ) +``` + +--- + +## Provenance + +- **ISO 4217** — alpha-3 currency codes. +- **CLDR** — currency symbols and display names. + +Compare [Money](money/) when your input includes amounts; Currency is identifier-only. + +See also: [Money](money/), [Execution Result](../concepts/execution-result/), [Candidates & Ambiguity](../concepts/candidates-and-ambiguity/). diff --git a/docs/user/v0.6.0/capabilities/date.md b/docs/user/v0.6.0/capabilities/date.md new file mode 100644 index 00000000..5f0a6671 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/date.md @@ -0,0 +1,142 @@ +--- +title: "Date" +slug: v0.6.0/capabilities/date +--- + +Canonicalizes **one calendar date** per call to ISO `YYYY-MM-DD` — or to US form when requested. + +> **In plain language:** give it a date written any of the common ways and it hands back the ISO form if a spec says the date is real. Ambiguous numeric forms like `01/02/2026` are surfaced as `AMBIGUOUS` rather than guessed. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| `2026-01-15` — ISO 8601 `YYYY-MM-DD` | Times, durations, or datetimes (`2026-01-15T10:00`) — dates only | +| `2026/01/15` — slash-ISO `YYYY/MM/DD` | Month names (`15 January 2026`) | +| `01/02/2026` — US `MM/DD/YYYY` and European `DD/MM/YYYY` (kept as competing interpretations) | Relative expressions (`today`, `next Tuesday`) | + +All four shapes emit the same notation; which rules accept them depends on the contract and the spec calendar logic. + +--- + +## Canonical output + +Default `output_format` is `"ISO"` (`YYYY-MM-DD`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `ISO` / `None` / `"default"` | `YYYY-MM-DD` | `2026-01-15` | +| `US` | `MM/DD/YYYY` | `01/15/2026` | + +Any other value raises `ContractError`. + +```python +from paxman.capabilities import Date +import paxman + +paxman.register_all_shipped() +paxman.canonicalize( + "2026-01-15", Date.create_contract() +).canonicalized_value # "2026-01-15" +paxman.canonicalize( + "2026-01-15", Date.create_contract(output_format="US") +).canonicalized_value # "01/15/2026" +``` + +The `two_digit_base_year` field is relevant only when the input contains a 2-digit year — e.g. with `two_digit_base_year=2000`, `"01/02/26"` expands relative to 2000. Otherwise omit it. + +--- + +## Contract + +```python +contract = Date.create_contract( + two_digit_base_year=None, # int | None — base year for 2-digit expansion, e.g. 2000 + output_format=None, # "ISO" (default) or "US" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- `two_digit_base_year` shapes how `YY` is expanded; it does not affect which grammars run. +- Use `pinned_rules` when you want only one jurisdiction's calendar (e.g. pin to ISO 8601); use `year` to restrict to rules published up to a given year. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `2026-01-15` | defaults | `SUCCESS` | `"2026-01-15"` | +| `2026/01/15` | defaults | `SUCCESS` | slash-ISO → `"2026-01-15"` | +| `01/02/2026` | defaults | `AMBIGUOUS` | US → `2026-01-02` vs European → `2026-02-01` — same span, different values | +| `01/02/2026` | `pinned_rules=["Section 5.2.1.1-calendar-date"]` (ISO only) | `SUCCESS` or `INVALID` | only the pinned spec's reading remains | +| `2026-13-01` | any | `INVALID` | recognized but no calendar accepts month 13 | +| `hello` | any | `MISSING` | no date pattern at all | +| `2026-01-15, 2026-02-01` | any | raises `MultipleMentionsError` | two distinct dates — split first | + +```mermaid +flowchart TB + A[Text
01/02/2026] --> G1[ISO grammar] + A --> G2[US grammar] + A --> G3[European grammar] + A --> G4[Slash-ISO grammar] + G1 & G2 & G3 & G4 --> R[Rules:
Section 5.2.1.1-calendar-date
Derived-US-date-format
Derived-European-date-format] + R -->|one value| OK[SUCCESS] + R -->|US vs EU| AMB[AMBIGUOUS] + R -->|none| INV[INVALID] + + style OK fill:#e6ffed,stroke:#2d8a4e + style AMB fill:#fff8e1,stroke:#d4a017 + style INV fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a column with ambiguity surfaced + +```python +import paxman +from paxman.capabilities import Date +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Date.create_contract() + +rows = ["2026-01-15", "2026/01/15", "01/02/2026", "2026-13-01", "hello"] + +for text in rows: + r = paxman.canonicalize(text, contract) + if r.status == Resolution.SUCCESS: + print(f"{text!r:15} → {r.canonicalized_value}") + elif r.status == Resolution.AMBIGUOUS: + vals = sorted({c.value for c in r.candidates}) + print(f"{text!r:15} → AMBIGUOUS {vals}") + else: + print(f"{text!r:15} → {r.status.value}") +``` + +--- + +## Provenance + +Validated values cite the calendar spec whose rule accepted the notation — e.g. ISO 8601 (`Section 5.2.1.1-calendar-date`) or the derived US / European locale conventions (`Derived-US-date-format`, `Derived-European-date-format`), with section citation on `candidate.validation_rule` and `publication_year` on `candidate.provenance[0]`. + +```python +import paxman +from paxman.capabilities import Date + +paxman.register_all_shipped() +result = paxman.canonicalize("2026-01-15", Date.create_contract()) +for c in result.candidates: + p = c.provenance[0] + print( + c.value, + "via", + c.validation_rule, + f"({p.specification_name}, {p.publication_year})", + ) +``` + +See also: [Execution Result](../concepts/execution-result/), [Candidates & Ambiguity](../concepts/candidates-and-ambiguity/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/doi.md b/docs/user/v0.6.0/capabilities/doi.md new file mode 100644 index 00000000..f9b74bf8 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/doi.md @@ -0,0 +1,134 @@ +--- +title: "DOI" +slug: v0.6.0/capabilities/doi +--- + +Canonicalizes **one DOI mention** per call — bare (`10.1038/nature12345`), case variants, resolver URLs (`https://doi.org/…`), `doi:`/`DOI:` labels, or `urn:doi:`/`info:doi/` carriers — to the bare lowercase `10.registrant/suffix` name. + +> **In plain language:** give it any documented spelling of a Digital Object Identifier and it hands back the canonical bare name per ISO 26324:2025. There is no check digit and no registry: validation requires a correctly shaped prefix plus a non-empty suffix containing none of whitespace, double quotes, ampersands, or apostrophes. Case folds ASCII-only — non-Latin capitals (`Á` vs `á`) are distinct names. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Bare names, any Basic-Latin case (`10.1038/nature12345`, `10.1038/NATURE12345` — folds to lowercase) | Suffix-only strings (`nature12345`) → `MISSING` (prefix required) | +| Dotted sub-registrants (`10.13003/5jchdy`) and suffix dots/slashes (`10.7774/cevr.2016.5.1.19`) | Truncated (`10.1038/`, `10.103`) → `MISSING` (`/` + non-empty suffix required) | +| Resolver URLs (`https://doi.org/…`, `http://…`, legacy `dx.` host, `www.` host — span includes URL) | Non-`10.` handles (`20.500.1234/abc`) → `MISSING` (`10.` gate) | +| `doi:`/`DOI:` labels, `urn:doi:`/`info:doi/` carriers (span includes carrier) | shortDOI (`10/gf2p3c`) → `MISSING` (not DOI names, Handbook §6.5) | +| Literal `%2F` in suffixes (retained, never decoded; hex letters fold) | Proxy URN-colon form (`https://doi.org/urn:doi:10.123:456`) → `MISSING` (deferred) | +| Embedded mentions (`See 10.1038/nature12345.` — trailing `.`/`)`/`,` excluded) | Inner whitespace (`10.1038 / x`) → `MISSING` | +| Over-long registrants (`10.1234567890/x`) → `MISSING` (4–9 digit bound) | X-glued prefixes (`x10.1038/…`) → `MISSING` (word boundary) | + +**Case law:** ASCII `A–Z` ≡ `a–z` for equivalence only (Handbook Ch.3); the fold preserves every other code point byte-identically and performs no Unicode normalization — precomposed vs decomposed spellings never coalesce. + +--- + +## Canonical output + +Default `output_format` is `"doi"` (bare lowercase `10.registrant/suffix`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `doi` / `None` / `"default"` | Bare lowercase name | `10.1038/nature12345` | +| `url` | Resolver link | `https://doi.org/10.1038/nature12345` | + +Any other value raises `ContractError`. The offered format re-enters under the default contract (ADR-0010 fixed point). + +```python +from paxman.capabilities import DOI +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("DOI: 10.1038/NATURE12345", DOI.create_contract()).canonicalized_value) +print(paxman.canonicalize("https://doi.org/10.1038/nature12345", DOI.create_contract()).canonicalized_value) +print(paxman.canonicalize("10.1038/nature12345", DOI.create_contract(output_format="url")).canonicalized_value) +print(paxman.canonicalize("urn:doi:10.1000/182", DOI.create_contract()).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = DOI.create_contract( + output_format=None, # "doi" (default); None/"default"/"doi" all resolve to it + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- One grammar: `doi_recognition` (bare core + carrier groups, ASCII-only fold). +- One rule: `Section 4-doi-syntax` (ISO 26324:2025 structure; PARSER, always-active — no registry flag exists). +- `year` filters by `publication_year`; e.g., `year=2020` drops the 2025 rule → `10.1038/…` becomes `INVALID`. +- Deterministic by construction: same input + contract + vendored snapshot → same output. No clock, no registry lookup. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `10.1038/nature12345` | defaults | `SUCCESS` | `"10.1038/nature12345"` | +| `10.1038/NATURE12345` / mixed case | defaults | `SUCCESS` | ASCII fold restores lowercase | +| `https://doi.org/…` / `doi:` / `urn:doi:` carriers | defaults | `SUCCESS` | carriers stripped, span includes carrier | +| `10.5594/SMPTE.ST2067-21.2020` | defaults | `SUCCESS` | `"10.5594/smpte.st2067-21.2020"` (Handbook vector) | +| Unallocated-but-shaped (`10.99999/x`) | defaults | `SUCCESS` | storable, no registry (UUID precedent) | +| No `10.`/slash/suffix, shortDOI, URN-colon form | defaults | `MISSING` | nothing claimed / policy gates | +| two distinct DOIs | defaults | raises `MultipleMentionsError` | un-segmented multi-entity input (`single_value=True`) | +| `10.1038/…` | `year=2020` | `INVALID` | rule is 2025, dropped | +| `10.1038/…` | `output_format="upper"` | raises `ContractError` | never offered | + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
doi_recognition] + G --> R{Section 4
doi-syntax} + R -->|prefix/suffix structure| OK[SUCCESS] + R -->|rule dropped by year filter| INV[INVALID] + G -->|nothing claimed| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +**Sibling note:** a resolver URL also matches the URL shape — each side resolves under its own contract (DOI `SUCCESS` bare, URL `SUCCESS` absolute URI). Bare numerics (`10.1000/182`) stay clear of Phone/Date grammars. The `urn:doi:` form is URN-shaped; URN-side behavior belongs to future work. + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import DOI +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = DOI.create_contract() + +rows = [ + "10.1038/nature12345", + "DOI: 10.1038/NATURE12345", + "https://doi.org/10.7774/cevr.2016.5.1.19", + "http://dx.doi.org/10.1000/182", + "urn:doi:10.13003/5jchdy", + "10/gf2p3c", + "not a doi", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:46} → {r.status.value:10} {val!r:40} ({rule})") +``` + +--- + +## Provenance + +- **ISO 26324:2025** (specification; DOI name syntax: prefix/suffix, no checksum, no registry) — `Section 4-doi-syntax` (vendored version 2025) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [ISBN](./isbn/), [ISSN](./issn/), [ORCID](./orcid/), [URL](./url/), [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/). diff --git a/docs/user/v0.6.0/capabilities/domain.md b/docs/user/v0.6.0/capabilities/domain.md new file mode 100644 index 00000000..d896fdc1 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/domain.md @@ -0,0 +1,138 @@ +--- +title: "Domain" +slug: v0.6.0/capabilities/domain +--- + +Canonicalizes **one hostname mention** per call — mixed-case (`EXAMPLE.COM`), root-dotted (`example.com.`), Unicode U-labels (`münchen.de`), ACE A-labels (`xn--mnchen-3ya.de`), fullwidth/dot-variant spellings (`example。jp`), NFD decomposed forms, and free-text-embedded names — to the lowercase ASCII A-label form with no trailing dot. + +> **In plain language:** give it any documented spelling of an internet hostname and it hands back the canonical ASCII form per RFC 1034/1035, UTS #46, and RFC 5893, keeping only names whose TLD is delegated in the IANA Root Zone Database. Case folds (DNS names are case-insensitive); a trailing root dot is stripped; `ß` stays `ß` (non-transitional processing — `straße.de` and `STRASSE.DE` are different names). + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Bare FQDNs, any ASCII case (`example.com`, `EXAMPLE.COM` — folds to lowercase) | Single labels (`localhost`, `com`, `de`) → `INVALID` (two-label minimum; intranet scope deferred) | +| Root-dotted names (`example.com.` — dot stripped; `example.com..` → `INVALID`, empty label) | Underscore labels (`_dmarc.example.com`) → `INVALID` (STD3 rules on) | +| U-labels (`münchen.de`), NFD (decomposed umlaut — NFC before encode), fullwidth (`example。jp` → `example.jp`), mapped dots (`example。com`) | Invisible controls (embedded U+202E RLO → `INVALID`); zero-width space is *removed* (U+200B is stripped, bare `example.com` succeeds) | +| ACE labels (`XN--MNCHEN-3YA.de` — case-insensitive input, round-trip verified) | Broken ACE (`xn--.com` → `INVALID`, empty payload) | +| Hyphenated labels (`exam--ple.com`); digit-first labels (`3m3.com` — RFC 1123) | Leading/trailing hyphens (`-cdn.example.com`, `example-.com`); `--` at 3rd/4th without `xn--` (`ex--ample.com`) | +| Embedded mentions (`visit example.com today`, padded ` example.com `) | Containers (`user@example.com`, `https://example.com/`, `*.example.com`, `[::1]` → `MISSING`, boundary-killed) | +| Numeric TLD shapes (`example.123`, `192.168.0.1` — recognized, then rejected) | Undelegated TLDs → `INVALID` (registry LOOKUP miss, not a shape failure) | + +**Case law:** U-label and A-label are one entity (RFC 5890 §2.3 — `münchen.de` ≡ `xn--mnchen-3ya.de`); NFC converges decomposed spellings before encoding (W2 ≡ W5). + +--- + +## Canonical output + +Default `output_format` is `"ascii"` (lowercase A-label dotted name, no trailing dot). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `ascii` / `None` / `"default"` | A-label canonical | `xn--mnchen-3ya.de` | +| `unicode` | U-label presentation (each `xn--` label Punycode-decoded) | `münchen.de` | + +Any other value raises `ContractError`. The offered format re-enters under the default contract (ADR-0010 fixed point). + +```python +from paxman.capabilities import Domain +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("münchen.DE.", Domain.create_contract()).canonicalized_value) +print(paxman.canonicalize("XN--MNCHEN-3YA.de", Domain.create_contract()).canonicalized_value) +print(paxman.canonicalize("münchen.de", Domain.create_contract(output_format="unicode")).canonicalized_value) +print(paxman.canonicalize("straße.de", Domain.create_contract()).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = Domain.create_contract( + output_format=None, # "ascii" (default); None/"default"/"ascii" all resolve to it + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- Two grammars: `ascii_hostname` (LDH shapes, case-neutral) + `idn_hostname` (any script, reserved-ASCII excluded; ASCII overlap dedups by span). +- Five rules: `Section-3.1-name-syntax` + `Section-2.3.4-label-length` + `UTS46-statuses` + `Section-2-bidi-context` (all PARSER) and `root-zone-membership` (the single LOOKUP_TABLE — IANA TLD membership ANDed with the other four; no corroboration → `INVALID`). +- Single-label acceptance and underscore selectors are not shipped (deferred scope, no knobs). +- Deterministic by construction: same input + contract + vendored snapshots → same output. No clock, no network (the IANA snapshot is vendored). + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `example.com` / `EXAMPLE.COM` / `example.com.` | defaults | `SUCCESS` | `"example.com"` (fold + root-dot strip) | +| `münchen.de` / `münchen.DE` / `XN--MNCHEN-3YA.de` / NFD | defaults | `SUCCESS` | `"xn--mnchen-3ya.de"` (one entity) | +| `straße.de` vs `STRASSE.DE` | defaults | `SUCCESS` | `"xn--strae-oqa.de"` vs `"strasse.de"` (non-transitional) | +| `foo.unknowntld-xyz`, `example.123`, `192.168.0.1` | defaults | `INVALID` | recognized; TLD not delegated | +| `localhost`, `com`, `de` (whole input, suppression on) | defaults | `INVALID` | single label (never `MISSING` once recognized) | +| `user@example.com`, `https://example.com/`, `*.example.com` | defaults | `MISSING` | nothing claimed (boundary kills) | +| `a.com;b.com` | defaults | raises `MultipleMentionsError` | un-segmented multi-entity input (`single_value=True`) | +| `a.com;a.com` | defaults | `SUCCESS` | `"a.com"` (same value dedups, no raise) | +| `example.com` | `output_format="bogus"` | raises `ContractError` | never offered | + +```mermaid +flowchart TB + A[Text] --> G[Grammars:
ascii_hostname + idn_hostname] + G --> R{Five rules:
1034 + 1035 + UTS46 + 5893 + root-zone} + R -->|all pass, TLD delegated| OK[SUCCESS] + R -->|shape fails or TLD missing| INV[INVALID] + G -->|nothing claimed| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +**Sibling note:** a bare hostname also matches inside URL/Email text — each side resolves under its own contract. An `https://…` carrier belongs to the URL capability (Domain sees only the boundary-killed spans → `MISSING`). + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import Domain +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Domain.create_contract() + +rows = [ + "münchen.DE.", + "EXAMPLE.COM", + "example。jp", + "visit example.com today", + "foo.unknowntld-xyz", + "user@example.com", + "not a domain", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:46} → {r.status.value:10} {val!r:40} ({rule})") +``` + +--- + +## Provenance + +- **RFC 1034 §3.1** (specification; name syntax: ≥2 labels, no empty label) — `Section-3.1-name-syntax` (vendored version 1987) +- **RFC 1035 §2.3.4** (specification; 63 octets/label, 253 chars/name, measured post-encode) — `Section-2.3.4-label-length` (vendored version 1987) +- **UTS #46 v18.0.0** (specification; statuses, STD3 rules, hyphen checks, ACE round trip; shipped table 15.1.0) — `UTS46-statuses` +- **RFC 5893 §2** (specification; Bidi rule six conditions, plus ContextJ join-control rejection) — `Section-2-bidi-context` (vendored version 2010) +- **IANA Root Zone Database** (registry; tlds-alpha snapshot v2026092300, 1,438 entries) — `root-zone-membership` + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [URL](./url/), [Email](./email/), [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/). diff --git a/docs/user/v0.6.0/capabilities/email.md b/docs/user/v0.6.0/capabilities/email.md new file mode 100644 index 00000000..675c44f5 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/email.md @@ -0,0 +1,155 @@ +--- +title: "Email" +slug: v0.6.0/capabilities/email +--- + +Canonicalizes **one email address** per call — standard, obfuscated, or localhost — to a lowercased `addr-spec`. + +> **In plain language:** give it something that looks like an email address and it hands back the single correct lowercase address if a real specification says it is valid, plus a citation of that specification. If it does not look like an email, or no spec accepts it, it tells you so without raising an exception. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Standard `user@example.com` (case-insensitive) | Display-name forms like `Bob ` — pass the addr-spec part | +| Obfuscated `user at example dot com` (only when `include_obfuscated=True`) | Arbitrary prose that happens to mention a domain without address structure | +| `admin@localhost` (only when `include_localhost=True`, the default) | | + +Grammars are syntax-only; whether the match is *valid* is decided by the rules (see [Pipeline](../concepts/pipeline/)). + +--- + +## Canonical output + +Default `output_format` is `"email"` — a single format, always a lowercased `addr-spec`. + +```python +from paxman.capabilities import Email +import paxman + +paxman.register_all_shipped() +contract = Email.create_contract() +paxman.canonicalize( + "USER@Example.COM", contract +).canonicalized_value # "user@example.com" +``` + +There are no offered alternatives for Email — `None`, `"default"`, and `"email"` all resolve to the same rendering; any other value raises `ContractError`. + +--- + +## Contract + +```python +contract = Email.create_contract( + include_obfuscated=False, # bool, default False — set True to recognize "user at example dot com" + include_localhost=True, # bool, default True — recognize admin@localhost + # plus every common field: excluded_rules / pinned_rules / year / output_format / extra_grammars +) +``` + +- Toggling `include_obfuscated` or `include_localhost` changes **which grammars run**. When the relevant grammar is off, the corresponding input is `MISSING` (never seen), not `INVALID`. +- Use `excluded_rules` / `pinned_rules` / `year` to control **which rules validate** — a recognized input that no rule accepts becomes `INVALID`. +- `output_format` is ignored beyond validation; it only shapes the rendered string. + +See [Contracts](../concepts/contracts/) and the [API Reference](../api-reference/#contracts--somecapabilitycreate_contract) for the full policy. + +--- + +## Statuses + +| Input | Contract tweak | Status | Value / why | +|-------|---------------|--------|-------------| +| `user@example.com` | defaults | `SUCCESS` | `"user@example.com"` | +| `USER@Example.COM` | defaults | `SUCCESS` | lowercased to `"user@example.com"` | +| `user at example dot com` | defaults (`include_obfuscated=False`) | `MISSING` | grammar not active — nothing seen | +| `user at example dot com` | `include_obfuscated=True` | `SUCCESS` | `"user@example.com"` | +| `admin@localhost` | defaults | `SUCCESS` | localhost path via RFC 6761 | +| `admin@localhost` | `include_localhost=False` | `MISSING` | grammar not active | +| `admin@localhost` | `excluded_rules=["Section 6.3-localhost"]` | `INVALID` | recognized but no rule accepts it | +| `@@` | any | `MISSING` | no email pattern at all | +| `alice@example.com and bob@example.org` | any | raises `MultipleMentionsError` | two distinct mentions — split first (see [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md)) | + +```mermaid +flowchart TB + A[Text] --> G{Which grammars active?} + G -->|standard| S[standard_recognition] + G -->|include_obfuscated| O[obfuscated_recognition] + G -->|include_localhost| L[localhost_recognition] + S & O & L --> R{Rules: RFC 5322 §3.4.1
RFC 6761 §6.3} + R -->|one value| OK[SUCCESS] + R -->|none| INV[INVALID] + G -->|none matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style MISS fill:#fff5f5,stroke:#cc3333 + style INV fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — clean a column + +```python +import paxman +from paxman.capabilities import Email +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = Email.create_contract(include_obfuscated=True, include_localhost=True) + +rows = [ + "user@Example.COM", + "user at example dot com", + "admin@localhost", + "not an email", + "alice@example.com and bob@example.org", +] + +for text in rows: + try: + r = paxman.canonicalize(text, contract) + except Exception as e: + print(f"{text!r:35} → exception {type(e).__name__}: {e}") + continue + tag = r.status.value + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:35} → {tag:10} {val!r}") +``` + +--- + +## Provenance + +Validated values carry provenance from: + +- **IETF RFC 5322** — Internet Message Format, Section 3.4.1 `addr-spec` +- **IETF RFC 6761** — Special-Use Domain Names, Section 6.3 (localhost) + +Inspect `candidate.provenance` and `candidate.validation_rule` for the exact citation: + +```python +import paxman +from paxman.capabilities import Email + +paxman.register_all_shipped() +result = paxman.canonicalize("user@Example.COM", Email.create_contract()) +for c in result.candidates: + p = c.provenance[0] + print( + c.value, + "via", + c.validation_rule, + f"({p.authority}: {p.specification_name}, {p.publication_year})", + ) +``` + +--- + +## Span + +On `SUCCESS`, `result.span` is the exact `[start, end)` of the email in the original input — useful for highlighting in UIs or logging. On `MISSING` / `INVALID` it is `None`; on `AMBIGUOUS` (rare for Email) use each `candidate.span`. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/gtin.md b/docs/user/v0.6.0/capabilities/gtin.md new file mode 100644 index 00000000..a6305c49 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/gtin.md @@ -0,0 +1,195 @@ +--- +title: "GTIN" +slug: v0.6.0/capabilities/gtin +--- + +Canonicalizes **one GTIN mention** per call — a compact GTIN-8/12/13/14 run, a zero-padded 14-digit field, a space- or hyphen-grouped display form, an `(01)`/`AI 01`-carried element, or a `GTIN`/`UPC`/`EAN`-labelled prose mention — to the 14-digit zero-padded storage form. + +> **In plain language:** give it `614141999996`, `590-1234-12345-7`, `6 14141 99999 6`, `GTIN: 00614141999996`, or `(01) 03453120000011` and it hands back the 14-digit zero-padded form (`00614141999996`, `05901234123457`, `03453120000011`) when the length is exactly 8, 12, 13, or 14 ASCII digits, the GS1 Mod-10 check digit holds, and the leading digits fall in an allocated GS1 prefix range. A bad check digit or an unallocated prefix is `INVALID`, never a guess; shapes outside the recognized spellings are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Compact native per length — GTIN-8 `96385074`, UPC-A `614141999996`, EAN-13 `5012345670003`, GTIN-14 `10614141999993` | UPC-E 6-digit symbol (`012345`) → `MISSING` (**DEFER** — expansion to `upce_recognition` community grammar) | +| Zero-padded 14-digit equivalents (`00614141999996`, `05012345670003`, `00196618007309`) — the official GS1 XML/GDSN storage form | Runs of 6/7/9/10/11/15+ digits (`012345`, `1234567`, `6141419999`, `501234567000345`) → `MISSING` (**REJECT** — exact-length gate; truncated stems are generation-only inputs) | +| Space-grouped HRI (`6 14141 99999 6`, `1234 5670`) | Tabs / newlines / dots / slashes as separators, or double spaces (`614141 999996`) → `MISSING` (single space or hyphen between digit groups only) | +| Hyphen-grouped and mixed-separator forms (`590-1234-12345-7`, `590 1234-12345 7`, `6-14141 99999-6`) — separators stripped, never emitted | Glued label (`GTIN00614141999996`, no separator) → `MISSING` | +| Fused labels, case-insensitive (`GTIN: 00614141999996`, `upc-614141999996`, `EAN-13: 5012345670003`, `GTIN-14 10614141999993`) — span includes the label | Fullwidth digits (`614141999996`) → `MISSING` (ASCII digits only) | +| AI markers `(01)`-parens or `AI 01`-keyword (`(01)03453120000011`, `(01) 03453120000011`, `AI 01 03453120000011`, stacked `GTIN: (01) …`) — span includes the marker | A bare leading `01` (`01345678901234`) is **never** an AI marker — the digits are taken verbatim as a 14-digit run (validation then decides) | +| Prose-embedded, quoted, bracketed, trailing-period (`Batch 5012345670003 shipped.`, `"614141999996"`, `[5012345670003]`, `614141999996.`) | Bars, quiet zones, guard bars, symbol images — non-text, out of scope (**REJECT**) | +| True GTIN-14 indicator digits 1–8 (`10614141999993`, `2061414199993`) — packaging levels are distinct entities | Coupon/extended shapes are not special-cased: 13-digit `020…`/`981…` families validate like any number (`0201234567899` → `SUCCESS`), a glued `+add-on` reads its core (`036000291452+12345` → `00036000291452`), while a hyphen add-on or `CPN…` prefix → `MISSING` (no coupon grammar in v1) | +| Bookland `978`/`979`-prefixed GTIN-13 (`9780471117094`) — recognized as GTIN-13 syntactically (ISBN precedence is cross-capability, decided by which capability you call) | Two distinct GTINs in one call → raises `MultipleMentionsError` (split first — see [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md)) | + +The 8-digit UPC-E HRI form collides with GTIN-8: `01234567` is claimed as an 8-digit run and validates as GTIN-8 iff the Mod-10 check passes — it is never expanded to UPC-A in v1. + +--- + +## Canonical output + +Default `output_format` is `"gtin14"` — the 14-digit zero-padded storage form (check-preserving: `digits.rjust(14, "0")`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `gtin14` / `None` / `"default"` | 14-digit zero-padded | `00614141999996` from `614141999996` | +| `native` | The spelled length as entered (encoding — the zero-strip/zero-pad pair is reversible without side input, so re-entry recovers its pre-image exactly; identity for a true 14-digit spelling) | `614141999996` from `614141999996`; `00614141999996` spelled 14 stays `00614141999996` | + +Any other value raises `ContractError` — including `hri`: the space-grouped HRI layout is **deferred** because only the UPC-A (`6 14141 99999 6`) and EAN-8 (`1234 5670`) groupings are attested in fetched sources; EAN-13/GTIN-14 groupings stay unconfirmed, and groupings are never invented. + +```python +from paxman.capabilities import GTIN +import paxman + +paxman.register_all_shipped() +print( + paxman.canonicalize("614141999996", GTIN.create_contract()).canonicalized_value +) # → "00614141999996" +print( + paxman.canonicalize("590-1234-12345-7", GTIN.create_contract()).canonicalized_value +) # → "05901234123457" +print( + paxman.canonicalize("(01) 03453120000011", GTIN.create_contract()).canonicalized_value +) # → "03453120000011" +print( + paxman.canonicalize( + "614141999996", GTIN.create_contract(output_format="native") + ).canonicalized_value +) # → "614141999996" +print( + paxman.canonicalize("GTIN: 00614141999996", GTIN.create_contract()).canonicalized_value +) # → "00614141999996" +``` + +--- + +## Contract + +```python +contract = GTIN.create_contract( + output_format=None, # "gtin14" (default), "native" + include_verified=False, # add the Verified-by-GS1 liveness lookup + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`gtin_recognition`), three rules (`Section 1-gtin-structure-check-digit`, `Section 2-gs1-prefix`, `Section 3-verified-liveness`). +- `include_verified=True` adds the Verified-by-GS1 snapshot lookup **alongside** the always-active structure + prefix pair. An extra active authority whose lookup misses contributes no candidate — it does not veto — so results stay `SUCCESS` on the strength of the corroborated structure/prefix pair. To make liveness the only authority, pin it: `pinned_rules=("Section 3-verified-liveness",), include_verified=True` → recognized-but-unvalidated is `INVALID`. The shipped snapshot is empty (add-only refresh), so pinning it always misses by default. +- `year` filters by `publication_year` (GenSpecs 2026, Prefix 2026, Verified 2019); e.g. `year=2018` drops every rule → `INVALID`, while `year=2026` keeps the structure + prefix pair → `SUCCESS`. +- Single-value: two distinct GTINs in one call raise `MultipleMentionsError` — split first per `docs/recipes/segmentation.md`. Identical values (including padded-vs-native spellings of the same entity) coalesce to one `SUCCESS`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `96385074` | defaults | `SUCCESS` | `"00000096385074"` (GTIN-8 zero-padded) | +| `614141999996` | defaults | `SUCCESS` | `"00614141999996"` (UPC-A) | +| `5012345670003` | defaults | `SUCCESS` | `"05012345670003"` (EAN-13) | +| `6291041500213` | defaults | `SUCCESS` | `"06291041500213"` | +| `10614141999993` | defaults | `SUCCESS` | `"10614141999993"` (true GTIN-14, identity) | +| `2061414199993` | defaults | `SUCCESS` | `"02061414199993"` (indicator-2 packaging level) | +| `03453120000011` / `00196618007309` | defaults | `SUCCESS` | identity (already the 14-digit storage form) | +| `9780471117094` | defaults | `SUCCESS` | `"09780471117094"` (Bookland GTIN-13 syntactically; ISBN precedence is cross-capability) | +| `590-1234-12345-7` | defaults | `SUCCESS` | `"05901234123457"` (hyphens stripped, never emitted) | +| `6 14141 99999 6` / `1234 5670` | defaults | `SUCCESS` | `"00614141999996"` / `"00000012345670"` (HRI groupings collapse) | +| `(01) 03453120000011` / `AI 01 03453120000011` | defaults | `SUCCESS` | `"03453120000011"` (marker absorbed into the span) | +| `GTIN: 00614141999996` / `upc-614141999996` / `EAN-13: 5012345670003` | defaults | `SUCCESS` | label absorbed into the span | +| `Batch 5012345670003 shipped.` | defaults | `SUCCESS` | `"05012345670003"` (prose-embedded; trailing period excluded) | +| `00614141999996 and 614141999996` | defaults | `SUCCESS` | one value — padded-vs-native duplicates of the same entity coalesce | +| `00614141999996` | `output_format="native"` | `SUCCESS` | `"00614141999996"` (spelled 14 → identity); `614141999996` → `"614141999996"` | +| `614141999996` | `include_verified=True` | `SUCCESS` | `"00614141999996"` — the extra lookup misses the shipped-empty snapshot and adds no candidate; structure + prefix corroborate (non-veto) | +| `614141999996` | `year=2026` | `SUCCESS` | structure + prefix rules are 2026, kept | +| `614141999997` | any | `INVALID` | recognized shape, Mod-10 check digit fails | +| `9991414199996` | any | `INVALID` | check-valid, but the native prefix is unallocated | +| `01234567` | any | `INVALID` | 8-digit UPC-E HRI collision — claimed as GTIN-8, Mod-10 fails (a check-valid 8-digit run would succeed) | +| `01345678901234` | any | `INVALID` | bare `01` taken verbatim as a 14-digit run; Mod-10 fails | +| `5012345670003` | `pinned_rules=("Section 3-verified-liveness",), include_verified=True` | `INVALID` | liveness pinned as the only authority; shipped-empty snapshot rejects, nothing else can validate | +| `614141999996` | `year=2018` | `INVALID` | every rule is newer, dropped | +| `012345` (UPC-E 6-digit) | any | `MISSING` | **DEFER** — UPC-E expansion is not recognized in v1; a community grammar via `extra_grammars` is the extension path | +| `1234567` / `61414199999` / `6141419999` / `501234567000345` | any | `MISSING` | **REJECT** — 7/9/10/11/15-digit runs are not GTIN shapes (exact-length gate) | +| `GTIN00614141999996` | any | `MISSING` | glued label needs a separator | +| `614141999996` | any | `MISSING` | **REJECT** — ASCII digits only, no homoglyph folding | +| `614141 999996` (double space) | any | `MISSING` | single space or hyphen between digit groups only | +| Bars / quiet-zone text | any | `MISSING` | **REJECT** — non-text symbol overhead is out of scope | +| `614141999996` | `output_format="hri"` | raises `ContractError` | not an offered format — HRI groupings deferred (never invented) | +| `614141999996` | `output_format="grouped"` | raises `ContractError` | not an offered format | +| Two distinct GTINs in one call | any rule-active contract | raises `MultipleMentionsError` | split first — with every rule filtered out (e.g. `year=2018`, above) no candidates exist to trip the invariant, so the call resolves `INVALID` instead | + +Two deterministic rules over one single-value grammar (with a third snapshot-gated lookup that never vetoes) yield at most one value, so `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
gtin_recognition] + G --> R1{Section 1
structure + Mod-10} + R1 --> R2{Section 2
GS1 prefix allocation} + R2 -->|both hold| OK[SUCCESS] + R1 -->|no rule validates| INV[INVALID] + R2 -->|no rule validates| INV + R3{Section 3
Verified liveness
include_verified} -.->|adds a candidate
when the snapshot hits| R2 + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import GTIN +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError, MultipleMentionsError + +paxman.register_all_shipped() +contract = GTIN.create_contract() + +rows = [ + "96385074", + "614141999996", + "590-1234-12345-7", + "6 14141 99999 6", + "(01) 03453120000011", + "GTIN: 00614141999996", + "Batch 5012345670003 shipped.", + "614141999997", + "9991414199996", + "012345", + "61414199999", + "GTIN00614141999996", + "not a gtin", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:34} → {r.status.value:10} {val!r:20} ({rule})") + +try: + GTIN.create_contract(output_format="hri") +except ContractError as e: + print(f"hri → ContractError: {e}") + +try: + paxman.canonicalize("614141999996 / 5012345670003", contract) +except MultipleMentionsError: + print("two distinct → MultipleMentionsError") +``` + +--- + +## Provenance + +- **GS1 General Specifications (GenSpecs) 26.0** §1 GTIN structure — lengths 8/12/13/14, ASCII digits, and the Mod-10 check digit (weights 3/1 rightmost-anchored, `(10 - sum % 10) % 10`) — `Section 1-gtin-structure-check-digit` (`PARSER`). Specification at `https://ref.gs1.org/standards/genspecs/`. +- **GS1 Prefix allocation** — leading-digit membership over the published MO ranges plus the 4-character GTIN-8 exception blocks; native digits are tested (never the zero padding) — `Section 2-gs1-prefix` (`LOOKUP_TABLE`, always active, so every `SUCCESS` is corroborated by both the structure and the prefix authority). Reference at `https://www.gs1.org/standards/id-keys/company-prefix`. +- **Verified by GS1** — issued/live registration in the Verified snapshot (add-only, shipped empty) — `Section 3-verified-liveness` (`LOOKUP_TABLE`, enabled by `include_verified=True`; default `False`, disabled). Service at `https://www.gs1.org/services/verified-by-gs1`. + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/iban.md b/docs/user/v0.6.0/capabilities/iban.md new file mode 100644 index 00000000..71a0e056 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/iban.md @@ -0,0 +1,142 @@ +--- +title: "IBAN" +slug: v0.6.0/capabilities/iban +--- + +Canonicalizes **one IBAN mention** per call — an electronic contiguous string or paper groups-of-four (with an optional `IBAN` label) — to compact electronic form. + +> **In plain language:** give it `DE89370400440532013000`, `DE89 3704 0044 0532 0130 00`, or `IBAN: DE89 3704 0044 0532 0130 00` and it hands back `DE89370400440532013000` when the country code is in the SWIFT IBAN Registry, the per-country fixed length holds, and the MOD 97-10 checksum passes per ISO 13616-1:2020 §4-5 (via ISO/IEC 7064:2003). A bad checksum or an unregistered country is `INVALID`, never a guess; shapes outside the two spellings are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Electronic contiguous, 15–34 alphanumerics (`DE89370400440532013000`, lowercase folds up) | Hyphen-separated (`DE89-3704-0044-0532-0130-00`) → `MISSING` | +| Paper groups-of-four with single spaces (`DE89 3704 0044 0532 0130 00`, `GB29 NWBK 6016 1331 9268 19`) | Double spaces or tabs (`DE89 3704 ...`, `DE89\t3704 ...`) → `MISSING` | +| Fused uppercase `IBAN` label with separator (`IBAN: DE89 ...`, `IBAN - FR14 ...`, `IBAN DE89370400440532013000`) | Glued label (`IBANDE89370400440532013000`, no separator) → `MISSING` | +| Lowercase `iban:` prefix does not block the core (`iban:gb29nwbk60161331926819` → `GB29NWBK60161331926819`) | Irregular single-space groups (`DE89 37040 04405 32013 000`) → `MISSING` | +| Glued alnum tails are absorbed and rejected downstream (`DE89370400440532013000Y`, `...00n` → `INVALID`) | Prose without an IBAN shape (`not an iban`) → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"electronic"` (identity — `normalize()` returns the compact string). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `electronic` / `None` / `"default"` | Compact, no spaces | `DE89370400440532013000` | +| `paper` | Space-separated quartets (encoding — MOD 97-10 runs on the stripped form, so the rendering re-enters exactly) | `DE89 3704 0044 0532 0130 00` | + +Any other value raises `ContractError` — including `compact`, which is not an offered alias (normalize outside the call if you need that spelling). + +```python +from paxman.capabilities import IBAN +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("DE89 3704 0044 0532 0130 00", IBAN.create_contract()).canonicalized_value) +print(paxman.canonicalize("IBAN: GB29 NWBK 6016 1331 9268 19", IBAN.create_contract()).canonicalized_value) +print(paxman.canonicalize("NO9386011117947", IBAN.create_contract()).canonicalized_value) +print(paxman.canonicalize("DE89370400440532013000", IBAN.create_contract(output_format="paper")).canonicalized_value) +print(paxman.canonicalize("GB29NWBK60161331926819", IBAN.create_contract(output_format="paper")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = IBAN.create_contract( + output_format=None, # "electronic" (default), "paper" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`iban_recognition`), one rule (`Section 4-iban-structure-mod97`). +- `year` filters by `publication_year`; e.g., `year=2019` drops the ISO 13616-1:2020 rule → `DE89370400440532013000` becomes `INVALID`. +- Single-value: two distinct IBANs in one call raise `MultipleMentionsError` — split first per `docs/recipes/segmentation.md`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `DE89370400440532013000` | defaults | `SUCCESS` | `"DE89370400440532013000"` | +| `DE89 3704 0044 0532 0130 00` | defaults | `SUCCESS` | `"DE89370400440532013000"` (paper recognized, spaces stripped) | +| `IBAN: DE89 3704 0044 0532 0130 00` | defaults | `SUCCESS` | label absorbed into the span | +| `not an iban` | any | `MISSING` | no IBAN shape | +| `DE89-3704-0044-0532-0130-00` | any | `MISSING` | hyphens are not a paper separator | +| `IBANDE89370400440532013000` | any | `MISSING` | glued label needs a separator | +| `DE89370400440532013001` | any | `INVALID` | MOD 97-10 checksum fails | +| `DE89 3704 0044` | any | `INVALID` | recognized short paper, per-country length fails | +| `DE89370400440532013000Y` | any | `INVALID` | glued tail absorbed, length/checksum fail | +| `DE89370400440532013000` | `year=2019` | `INVALID` | rule is 2020, dropped | +| `DE89370400440532013000` | `output_format="compact"` | raises `ContractError` | not an offered format | +| Two distinct IBANs in one call | any | raises `MultipleMentionsError` | split first | + +A single grammar plus a single deterministic rule yields at most one value, so `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
iban_recognition] + G --> R{Section 4
structure + MOD 97-10} + R -->|country, length,
check digits hold| OK[SUCCESS] + R -->|no rule validates| INV[INVALID] + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import IBAN +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = IBAN.create_contract() + +rows = [ + "DE89370400440532013000", + "DE89 3704 0044 0532 0130 00", + "IBAN: GB29 NWBK 6016 1331 9268 19", + "NO9386011117947", + "DE89370400440532013001", + "DE89 3704 0044", + "DE89-3704-0044-0532-0130-00", + "IBANDE89370400440532013000", + "not an iban", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:38} → {r.status.value:10} {val!r:26} ({rule})") + +try: + IBAN.create_contract(output_format="compact") +except ContractError as e: + print(f"compact → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO 13616-1:2020** §4-5 Structure + MOD 97-10 checksum (normative reference to ISO/IEC 7064:2003; check digits 02-98; `mod97 == 1`) — `Section 4-iban-structure-mod97` +- **SWIFT IBAN Registry** Release 99 (Dec 2024, mirrors R100 Oct 2025) — 111 registered country codes and per-country fixed lengths (e.g. DE22, GB22, NO15) backing the allowlist/length checks + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/index.md b/docs/user/v0.6.0/capabilities/index.md new file mode 100644 index 00000000..40f9c716 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/index.md @@ -0,0 +1,105 @@ +--- +title: "Capabilities — Overview" +slug: v0.6.0/capabilities/index +--- + +Each page below is a self-contained guide for one kind of identifier Paxman can canonicalize. Read the one that matches your data, copy the notebook snippet, and adapt the contract flags to your needs. + +All capabilities share the same call shape — only the import, the factory, and the domain vocabulary change: + +```python +import paxman +from paxman.capabilities import X # Email, Country, ... + +paxman.register_all_shipped() # once, before first use +contract = X.create_contract(...) # domain flags here +result = paxman.canonicalize(text, contract) +``` + +For the shared concepts behind these pages see [Contracts](../concepts/contracts/), [Pipeline](../concepts/pipeline/), [Execution Result](../concepts/execution-result/), and the [API Reference](../api-reference/). + +--- + +## Choose by what you have + +| Your data looks like… | Read | +|-----------------------|------| +| `user@example.com`, `user at example dot com` | [Email](email/) | +| `2026-01-15`, `01/02/2026`, `2026/01/15` | [Date](date/) | +| `US`, `United States`, `Alemania` | [Country](country/) | +| `4111 1111 1111 1111`, `3782 822463 10005`, `PAN: 4716-2210-5188-5662` | [CreditCard](credit_card/) | +| `USD`, `$`, `euro`, `¥` (identifiers without amounts) | [Currency](currency/) | +| `192.168.1.1`, `2001:db8::1` | [IP](ip/) | +| `9780306406157`, `0306406152` | [ISBN](isbn/) | +| `0317-8471`, `0378-5955` | [ISSN](issn/) | +| `US0378331005`, `GB0002634946` | [ISIN](isin/) | +| `0000 0001 2103 2683`, `0000000121032683`, `urn:isni:…` | [ISNI](isni/) | +| `5493000IBP32UQZ0KL24`, `LEI: 213800KUD8LAJWSQ9D15` | [LEI](lei/) | +| `en`, `en-US`, `zh-Hans-CN`, `German` | [Language](language/) | +| `USD 500`, `$500`, `1.000,50 EUR` (currency **with** amount) | [Money](money/) | +| `+1 555 123 4567`, `(555) 234-5678`, `tel:+15551234567` | [Phone](phone/) | +| `kg`, `m/s²`, `megahertz`, `kPa` | [SI Unit](si-unit/) | +| `https://example.com`, `http://münchen.de` | [URL](url/) | +| `DEUTDEFF`, `DEUTDEFF500` | [BIC](bic/) | +| `00614141999996`, `GTIN: 00196618007309`, `6 14141 99999 6` | [GTIN](gtin/) | +| `48.8566, 2.3522`, `geo:48.8566,2.3522` | [Coordinates](coordinates/) | +| `Fe`, `iron`, `element 26` | [ChemicalElement](chemical_element/) | +| `DE89370400440532013000` | [IBAN](iban/) | +| `00:1A:2B:3C:4D:5E` | [MacAddress](mac_address/) | +| `0000-0002-1825-0097` | [ORCID](orcid/) | +| `America/New_York`, `US/Eastern` | [Timezone](timezone/) | +| `UTC+5`, `+05:30`, `Z` | [UtcOffset](utcoffset/) | +| `6ba7b810-9dad-11d1-80b4-00c04fd430c8`, `{…}`, `urn:uuid:…` | [UUID](uuid/) | +| `10.1038/nature12345`, `https://doi.org/10.1038/…`, `doi:10.1038/…` | [DOI](doi/) | +| `example.com`, `münchen.de`, `XN--MNCHEN-3YA.de` | [Domain](domain/) | +| `UNSPSC 44103103`, `441217`, `UNSPSC000.44103103` | [UNSPSC](unspsc/) | +| `1995 XA`, `J95X00A`, `(433)`, `2040 P-L` | [MinorPlanet](minor_planet/) | + +> The set above reflects the **current release**. New capabilities are added in minor releases — check `paxman.capabilities` or the latest release notes if you don't see what you need. + +```mermaid +flowchart LR + A[Raw text] --> B{What kind?} + B -->|email| C[Email] + B -->|date| D[Date] + B -->|place| E[Country] + B -->|money / currency| F[Currency / Money] + B -->|network| G[IP / URL] + B -->|books| H[ISBN] + B -->|phones| I[Phone] + B -->|units| J[SI Unit] + + style B fill:#fff8e1,stroke:#d4a017 +``` + +--- + +## What each page covers + +Every capability page answers the same questions in the same order: + +1. **What it canonicalizes** and what it explicitly does not. +2. **Recognized forms** — what patterns match, with what grammars. +3. **Canonical output & `output_format`** — default and offered renderings. +4. **Contract flags** — which knobs change recognition and validation. +5. **Statuses** — concrete `SUCCESS` / `MISSING` / `INVALID` / `AMBIGUOUS` examples. +6. **Notebook snippet** — runnable cleaning loop for a column. +7. **Provenance** — which specifications vouch for the answer. + +Start with the capability that matches your column; if you need more than one, register both and loop per cell (see the [Segmentation Recipe](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md) for text that mixes kinds). + +--- + +## One mention per call + +Paxman resolves **one presumed entity per `canonicalize()` call** (see [Pipeline](../concepts/pipeline/)). Text that contains two different entities with different canonical values raises `MultipleMentionsError` rather than returning a merged answer — split first, then loop. To extract *every* mention in a longer text instead, use the batch API `paxman.scan()`, which returns per-capability `Mention` records (see [API reference](../api-reference/)). + +```python +from paxman.core.errors import MultipleMentionsError + +try: + result = paxman.canonicalize("alice@example.com, bob@example.org", contract) +except MultipleMentionsError: + # split the input and canonicalize each piece — see the segmentation recipe + ... +``` diff --git a/docs/user/v0.6.0/capabilities/ip.md b/docs/user/v0.6.0/capabilities/ip.md new file mode 100644 index 00000000..6cd7a9c7 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/ip.md @@ -0,0 +1,136 @@ +--- +title: "IP" +slug: v0.6.0/capabilities/ip +--- + +Canonicalizes **one IP address** per call to its normalized textual form. + +> **In plain language:** give it `"192.168.1.1"` or `"2001:0db8:0000:0000:0000:0000:0000:0001"` and it hands back the normalized address if a spec says it is valid. IPv6 canonicalization follows RFC 5952 so the same address always looks the same. IPv6 mixed addresses with an embedded IPv4 (`::ffff:192.0.2.1`) and IPv4 leading-zero forms (`010.020.030.040` → `10.20.30.40`) are also handled. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| IPv4 dotted-decimal (`192.168.1.1`), including leading-zero (`010.020.030.040` → `10.20.30.40`) | CIDR notation (`192.168.1.0/24`) — address only | +| IPv6 — full (`2001:0db8:85a3:0000:0000:8a2e:0370:7334`), compressed (`2001:db8::1`), loopback (`::1`) | Hostnames — use [URL](url/) | +| IPv6 mixed with embedded IPv4 (`::ffff:192.0.2.1`, `64:ff9b::192.0.2.1`, `::192.0.2.1`) per RFC 4291 §2.2 / RFC 5952 §5 | Zone identifiers (`fe80::1%eth0`) — `MISSING` (grammar does not consume `%`) | + +> **Overlap note:** the trailing IPv4 inside a mixed address (`::ffff:192.0.2.1` contains `192.0.2.1`) is also emitted by the IPv4 grammar via its `\b` boundary. The engine keeps both candidates (cross-grammar containment dedup is not performed) so `canonicalize("::ffff:192.0.2.1")` yields `AMBIGUOUS ['::ffff:192.0.2.1','192.0.2.1']` — prefer the IPv6 value. See issue #113 B1. Triple-colon over-matches like `:::1` are intentionally broad and sanitized to `INVALID` (B2 deferred). + +--- + +## Canonical output + +Single format — the normalized address. IPv4 strips leading zeros per octet; IPv6 is rendered per **RFC 5952** (lowercase hex, `::` for the longest zero run, no leading zeros). Mixed addresses are normalized the same way (e.g. `::ffff:192.0.2.1` stays `::ffff:192.0.2.1`, while `64:ff9b::192.0.2.1` with a non-trivial embedded IPv4 may render the last 32 bits as hex per `ipaddress.IPv6Address`). + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `ip` / `None` / `"default"` | normalized address string | + +```python +from paxman.capabilities import IP +import paxman + +paxman.register_all_shipped() +paxman.canonicalize( + "192.168.1.1", IP.create_contract() +).canonicalized_value # "192.168.1.1" +paxman.canonicalize( + "2001:0db8:0000:0000:0000:0000:0000:0001", IP.create_contract() +).canonicalized_value # "2001:db8::1" +paxman.canonicalize( + "010.020.030.040", IP.create_contract() +).canonicalized_value # "10.20.30.40" (leading zeros stripped) +paxman.canonicalize( + "::ffff:192.0.2.1", IP.create_contract() +).candidates[0].value # "::ffff:192.0.2.1" (mixed, see overlap note) +``` + +--- + +## Contract + +```python +contract = IP.create_contract( + include_ipv6=True, # bool, default True — recognize IPv6 + output_format=None, # "ip" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- With `include_ipv6=True` (the default) both grammars run. With `False`, only `ipv4_recognition` runs — an IPv6 address is then `MISSING` (never seen), not `INVALID`. + +```python +paxman.canonicalize( + "2001:db8::1", IP.create_contract(include_ipv6=False) +).status.value # "missing" +paxman.canonicalize( + "2001:db8::1", IP.create_contract() +).canonicalized_value # "2001:db8::1" +``` + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `192.168.1.1` | any | `SUCCESS` | normalized IPv4 | +| `010.020.030.040` | any | `SUCCESS` | → `10.20.30.40` (leading zeros stripped, RFC 791 / RFC 1123) | +| `2001:db8::1` | defaults | `SUCCESS` | → `2001:db8::1` (RFC 5952) | +| `::ffff:192.0.2.1` | defaults | `AMBIGUOUS` | `['::ffff:192.0.2.1','192.0.2.1']` — overlapping IPv4 candidate (see note) | +| `2001:db8::1` | `include_ipv6=False` | `MISSING` | IPv6 grammar not active | +| `999.999.999.999` | any | `INVALID` | recognized shape but no spec accepts it | +| `:::1` | any | `INVALID` | over-broad syntax recognized, rejected by RFC 5952 | +| `hello` | any | `MISSING` | no IP pattern | +| `10.0.0.1 and 10.0.0.2` (two distinct values) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G1[ipv4_recognition] + A --> G2[ipv6_recognition
only if include_ipv6] + G1 --> R1[Rule: RFC 791] + G2 --> R2[Rule: RFC 5952] + R1 & R2 --> D{One value?} + D -->|yes| OK[SUCCESS
normalized] + D -->|none| MISS_OR_INV[MISSING or INVALID] + D -->|two values
same mention| AMB[AMBIGUOUS
rare for IP — mixed overlap] + + style OK fill:#e6ffed,stroke:#2d8a4e +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import IP +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = IP.create_contract() + +for text in [ + "192.168.1.1", + "010.020.030.040", + "2001:0db8:0000:0000:0000:0000:0000:0001", + "::ffff:192.0.2.1", + "999.999.999.999", + "hello", +]: + r = paxman.canonicalize(text, contract) + print(f"{text!r:45} → {r.status.value:10} {r.canonicalized_value!r} candidates={[c.value for c in r.candidates]!r}") +``` + +--- + +## Provenance + +- **RFC 791** — IPv4 address validation (dotted-decimal clarified by RFC 1123 §2.1). +- **RFC 4291 §2.2** — IPv6 addressing architecture, including the mixed embedded-IPv4 `LS32` form. +- **RFC 5952** — IPv6 text representation and canonical compression. + +See also: [URL](url/), [Execution Result](../concepts/execution-result/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/isbn.md b/docs/user/v0.6.0/capabilities/isbn.md new file mode 100644 index 00000000..0169bb92 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/isbn.md @@ -0,0 +1,123 @@ +--- +title: "ISBN" +slug: v0.6.0/capabilities/isbn +--- + +Canonicalizes **one ISBN identifier** per call with check-digit validation, converting legacy ISBN-10 to ISBN-13 when enabled. + +> **In plain language:** give it `"9780306406157"` or `"0306406152"` and it hands back the 13-digit ISBN if the check digit is correct. The hyphenated form is a presentation option — the underlying value is always the bare digits unless you ask for hyphens. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| ISBN-13 any 13-digit run with optional `ISBN`/`ISBN-13` label and single hyphen/space per gap (syntax); only `978`/`979` with valid check digit validates (`9780306406157`) | ISBNs with wrong check digits — recognized but `INVALID` | +| ISBN-10 (`0306406152`) — only when `include_isbn10=True` (the default) | ISBN-10 when `include_isbn10=False` → `MISSING` | +| Hyphenated/space-separated input is normalized regardless of input separators — one hyphen or space per digit gap is tolerated (`978-0-306-40615-7` → `9780306406157`); output hyphens are via `output_format` | Truncated or digit-glued runs like `1 0306406152` or `1 0-306-40615-2` → `MISSING` (digit+separator prefix); double separators like `978 0306406157` → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"isbn13"` (bare 13 digits). + +| `output_format` | Renders | Example for `9780110002224` | +|-----------------|---------|------------------------------| +| *(default)* `isbn13` / `None` / `"default"` | bare 13 digits | `9780110002224` | +| `hyphenated` | hyphenated by registrant-range (ISBN Range Message, longest-match) | `978-0-11-000222-4` | + +Hyphenation is presentation only — provenance remains the same; the hyphen positions come from the authoritative Range Message table. + +```python +from paxman.capabilities import ISBN +import paxman + +paxman.register_all_shipped() +paxman.canonicalize( + "9780110002224", ISBN.create_contract() +).canonicalized_value # "9780110002224" +paxman.canonicalize( + "9780110002224", ISBN.create_contract(output_format="hyphenated") +).canonicalized_value # "978-0-11-000222-4" +paxman.canonicalize( + "0306406152", ISBN.create_contract() +).canonicalized_value # "9780306406157" (10 → 13) +``` + +--- + +## Contract + +```python +contract = ISBN.create_contract( + include_isbn10=True, # bool, default True — recognize ISBN-10 + include_range_validation=False, # bool, default False — gate the registrant-range rule/provenance + output_format=None, # "isbn13" (default) or "hyphenated" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- With `include_isbn10=False`, only the `isbn13_recognition` grammar runs — ISBN-10 input is `MISSING` (never seen), not `INVALID`. +- `output_format="hyphenated"` controls hyphenation (uses the Range Message longest-match table) and works regardless of `include_range_validation`. `include_range_validation=True` separately enables the Range Message registrant-range rule that adds provenance — it does not change the canonical digits, and hyphenation does not by itself add that provenance. + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `9780306406157` | defaults | `SUCCESS` | bare ISBN-13, check digit valid | +| `0306406152` | defaults | `SUCCESS` | ISBN-10 → `9780306406157` | +| `0306406152` | `include_isbn10=False` | `MISSING` | grammar not active | +| `9780306406150` (bad check) | any | `INVALID` | recognized but check digit fails | +| `hello` | any | `MISSING` | no ISBN pattern | +| Two distinct ISBNs in one call | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G1[isbn13_recognition
always] + A --> G2[isbn10_recognition
only if include_isbn10] + G1 & G2 --> R{Rules: ISO 2108 +
ISBN Users' Manual
unconditional} + R --> R2{Range Message
only if include_range_validation} + R2 -->|one value| OK[SUCCESS
bare or hyphenated] + R -->|bad check| INV[INVALID] + R2 -->|bad check| INV + G1 & G2 -->|nothing matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import ISBN +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c = ISBN.create_contract() +c_hyph = ISBN.create_contract(output_format="hyphenated") + +for text in ["9780306406157", "0306406152", "9780306406150", "hello"]: + r = paxman.canonicalize(text, c) + rh = paxman.canonicalize(text, c_hyph) + a = r.canonicalized_value or "—" + b = rh.canonicalized_value or "—" + print(f"{text!r:20} → bare {a!r:15} hyphenated {b!r:20} status={r.status.value}") +``` + +--- + +## Provenance + +- **ISO 2108** — GS1 / ISBN prefix. +- **ISBN Users' Manual** — ISBN-13 / ISBN-10 check-digit logic. +- **ISBN Range Message** — registrant-range hyphenation and provenance (only with `include_range_validation=True`). + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/isin.md b/docs/user/v0.6.0/capabilities/isin.md new file mode 100644 index 00000000..81ffc9fc --- /dev/null +++ b/docs/user/v0.6.0/capabilities/isin.md @@ -0,0 +1,163 @@ +--- +title: "ISIN" +slug: v0.6.0/capabilities/isin +--- + +Canonicalizes **one ISIN mention** per call — a bare 12-character code, single-space groupings, or an `ISIN`-labelled form — to compact uppercase form. + +> **In plain language:** give it `US0378331005`, `US 037833 100 5`, or `ISIN: US0378331005` and it hands back `US0378331005` when the shape is `CC+NSIN+C` (2-letter prefix, 9-character national identifier, numeric check digit), the modulus 10 Double-Add-Double check digit holds per ISO 6166:2021 §4, and the prefix is an ISO 3166-1 alpha-2 code or an attested special prefix per the ANNA ISIN Guidelines. A bad check digit or an unattested prefix is `INVALID`, never a guess; shapes outside the recognized spellings are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Bare compact 12-char (`US0378331005`, `AU0000XVGZA3`, `GB0002634946`, `XS0931417173`; lowercase folds up) | Hyphen-separated (`US-037833-100-5`) → `MISSING` (hyphen tolerance deferred — see statuses) | +| Single-space groupings (`US 037833 100 5`, `US037833 1005`, `PL0000 503132`) | Double spaces (`US 037833 100 5`) → `MISSING` | +| `ISIN` label with separator (`ISIN: US0378331005`, `ISIN US0378331005`, `isin: us0378331005` — case-insensitive) | Glued label (`ISINUS0378331005`, no separator) → `MISSING` | +| National shapes with leading zeros (`FR0000131104`, `PL0000503132`, `IS0000000040`, `DE000BAY0017`) | Wrong lengths (`US037833100` 11, `US03783310055` 13) → `MISSING` (rejected — not ISIN shapes) | +| Attested special prefixes (`XS0931417173` international, plus `EU`/`EZ`/`XT`/`XA`–`XD`/`XF`/`XK`/`QS`/`QT` with a valid check digit) | Letter check digit (`US037833100A`) → `MISSING` (position 12 must be `0-9`) | +| Embedded in prose (`see ISIN GB0002634946 (London)`) | Glued surroundings (`XUS0378331005`, `US0378331005X`) → `MISSING` | +| | Prose without an ISIN shape (`not an isin`) → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"isin"` (identity — `normalize()` returns the compact string). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `isin` / `None` / `"default"` | Compact uppercase 12-char | `US0378331005` | +| `grouped` | Spaced display `CC NNNNNN NNN C` (encoding — validation runs on the stripped compact form, so the rendering re-enters exactly) | `US 037833 100 5` from `US0378331005` | + +Any other value raises `ContractError` — including `compact`, which is not an offered alias (normalize outside the call if you need that spelling). + +```python +from paxman.capabilities import ISIN +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("us0378331005", ISIN.create_contract()).canonicalized_value) +print(paxman.canonicalize("ISIN: GB0002634946", ISIN.create_contract()).canonicalized_value) +print(paxman.canonicalize("US 037833 100 5", ISIN.create_contract()).canonicalized_value) +print(paxman.canonicalize("US0378331005", ISIN.create_contract(output_format="grouped")).canonicalized_value) +print(paxman.canonicalize("XS0931417173", ISIN.create_contract()).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = ISIN.create_contract( + output_format=None, # "isin" (default), "grouped" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`isin_recognition`), two rules (`Section 4-isin-structure-check-digit`, `Section 5-country-and-special-prefix`). +- `year` filters by `publication_year`; e.g., `year=2020` drops both rules (2021, 2025) → `US0378331005` becomes `INVALID`, while `year=2021` keeps the ISO structure rule → still `SUCCESS`. +- Single-value: two distinct ISINs in one call raise `MultipleMentionsError` — split first per `docs/recipes/segmentation.md`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `US0378331005` | defaults | `SUCCESS` | `"US0378331005"` | +| `us0378331005` | defaults | `SUCCESS` | `"US0378331005"` (case folds up) | +| `ISIN: US0378331005` | defaults | `SUCCESS` | label absorbed into the span | +| `US 037833 100 5` | defaults | `SUCCESS` | `"US0378331005"` (grouping recognized, spaces stripped) | +| `AU0000XVGZA3` / `GB0002634946` | defaults | `SUCCESS` | country-prefix ISINs, check digit holds | +| `XS0931417173` | defaults | `SUCCESS` | special-prefix ISIN (international securities) | +| `FR0000131104` / `PL0000503132` / `IS0000000040` | defaults | `SUCCESS` | zero-padded national shapes | +| `US0378331005` | `output_format="grouped"` | `SUCCESS` | `"US 037833 100 5"` | +| `US0378331005` | `pinned_rules=("Section 4-isin-structure-check-digit",)` | `SUCCESS` | ISO rule alone corroborates (vacuity clause) | +| `US0378331003` | any | `INVALID` | recognized shape, check digit fails | +| `ZZ0378331001` | any | `INVALID` | recognized shape, prefix has no Registration Authority attestation (provisional, excluded in v1) | +| `US0378331005` | `year=2020` | `INVALID` | both rules are newer, dropped | +| `US0378331005` | `excluded_rules=(both,)` | `INVALID` | no rule validates | +| `US-037833-100-5` | any | `MISSING` | **DEFER** — hyphen tolerance is not recognized in v1; a community grammar via `extra_grammars` is the extension path | +| `US037833100` (11) / `US03783310055` (13) | any | `MISSING` | **REJECT** — short/long forms are not ISIN shapes | +| `US037833100A` | any | `MISSING` | **REJECT** — position 12 must be numeric | +| `ISINUS0378331005` | any | `MISSING` | glued label needs a separator | +| `US 037833 100 5` | any | `MISSING` | double spaces are not a grouping separator | +| `not an isin` | any | `MISSING` | no ISIN shape | +| `US0378331005` | `output_format="compact"` | raises `ContractError` | not an offered format | +| Two distinct ISINs in one call | any | raises `MultipleMentionsError` | split first | + +Two deterministic rules over one single-value grammar yield at most one value, so `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
isin_recognition] + G --> R1{Section 4
structure + check digit} + R1 --> R2{Section 5
country/special prefix} + R2 -->|both hold| OK[SUCCESS] + R1 -->|no rule validates| INV[INVALID] + R2 -->|no rule validates| INV + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import ISIN +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = ISIN.create_contract() + +rows = [ + "US0378331005", + "us0378331005", + "ISIN: GB0002634946", + "US 037833 100 5", + "AU0000XVGZA3", + "XS0931417173", + "FR0000131104", + "US0378331003", + "ZZ0378331001", + "US-037833-100-5", + "US037833100", + "ISINUS0378331005", + "not an isin", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:24} → {r.status.value:10} {val!r:16} ({rule})") + +try: + ISIN.create_contract(output_format="compact") +except ContractError as e: + print(f"compact → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO 6166:2021** §4 ISIN structure plus check digit (12 chars `CC+NSIN+C`; modulus 10 Double-Add-Double over the letter-expanded payload, `A=10 … Z=35`) — `Section 4-isin-structure-check-digit` (`PARSER`). Catalogue at `https://www.iso.org/standard/78502.html`. +- **ANNA ISIN Guidelines** V25 (Dec 2025) §5 country and special prefix allowlist (ISO 3166-1 alpha-2, 249 codes, plus 12 attested special prefixes) — `Section 5-country-and-special-prefix` (`LOOKUP_TABLE`). Reference at `https://anna-web.org/wp-content/uploads/2025/11/ISIN-Guidelines-Dec-2025_Amendment_clean.pdf`. + +Attestation strength inside the 12 special prefixes: **Guidelines-attested** (`EU`, `XS`, `XA`, `XB`, `XC`, `XD`, `XT`); **Registration-Authority-attested** (`EZ` — ISO TC68 briefing plus ANNA identifiers page plus DSB listings); **validator/user-assigned-range** (`XF`, `XK`, `QS`, `QT` — within the `XA`–`XZ` and `QM`–`QZ` user-assigned ranges, corroborated by validator snapshots). `ZZ` is provisional with no Registration Authority attestation and is excluded in v1 (recognized shape, `INVALID`). + +> **V26 note:** the ANNA ISIN Guidelines V26 (Jun 2026) supersedes V25. This capability pins V25 as its v1 authority; re-attest the prefix table against V26 before adopting it. + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/isni.md b/docs/user/v0.6.0/capabilities/isni.md new file mode 100644 index 00000000..da1aac70 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/isni.md @@ -0,0 +1,140 @@ +--- +title: "ISNI" +slug: v0.6.0/capabilities/isni +--- + +Canonicalizes **one ISNI mention** per call — a spaced display (`0000 0001 2103 2683`), a compact 16-char run, a hyphenated form, an `ISNI`-labelled mention, an `isni.org` URI, or a `urn:isni:` carrier — to the spaced display form. + +> **In plain language:** give it `ISNI 0000 0001 2103 2683`, `0000000121032683`, or `https://isni.org/isni/0000000121032683` and it hands back `0000 0001 2103 2683` when the shape is 15 digits plus a MOD 11-2 check character per ISO 27729:2024. A bad check digit is `INVALID`, never a guess; shapes outside the recognized spellings are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Spaced display (`0000 0001 2103 2683`, `ISNI 0000 0001 2103 2683`; lowercase folds up) | Double spaces / tabs between quads (`0000 0001 …`) → `MISSING` (single space only) | +| Compact 16-char (`0000000121032683`, Wikidata/VIAF dumps) | Wrong lengths (15/17-char runs) → `MISSING` (length guard) | +| Hyphenated (`0000-0001-2103-2683` — ORCID-style input normalizes to spaced) | `X` mid-run (`000X 0001 …`) → `MISSING` (check position 16 only) | +| `ISNI` label with separator (`ISNI: …`, `ISNI-…` — case-insensitive) | Glued label (`ISNI0000…`, no separator) → `MISSING` | +| `isni.org` URIs (`https://isni.org/isni/…`, `http://`, `www.` variants) | MARC `(isni)…` / VIAF `ISNI\|…` wrappers — inner compact mention still resolves as an embedded mention (no dedicated wrapper pattern in v1) | +| `urn:isni:` carrier (`urn:isni:0000000121241960`) | Fullwidth digits → `MISSING` (ASCII only, no autocorrection) | +| Lowercase `x` check (`…955x` → `…955X`) | Glued surroundings (`X0000…`, `…2683Y`) → `MISSING` (word boundary) | +| Embedded mentions (`see ISNI 0000 0001 2103 2683 (Shakespeare)`) | Prose without an ISNI shape (`not an isni`) → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"isni"` (spaced display `XXXX XXXX XXXX XXXC`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `isni` / `None` / `"default"` | Spaced display | `0000 0001 2103 2683` | +| `compact` | 16-char, no spaces | `0000000121032683` | +| `urn` | IANA carrier | `urn:isni:0000000121241960` | + +Any other value raises `ContractError` — including `hyphenated`, which is ORCID's presentation, not ISNI's. Every offered format re-enters under the default contract (ADR-0010 fixed point). + +```python +from paxman.capabilities import ISNI +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("ISNI 0000 0001 2103 2683", ISNI.create_contract()).canonicalized_value) +print(paxman.canonicalize("0000000121032683", ISNI.create_contract()).canonicalized_value) +print(paxman.canonicalize("0000-0001-2103-2683", ISNI.create_contract()).canonicalized_value) +print(paxman.canonicalize("0000000121032683", ISNI.create_contract(output_format="urn")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = ISNI.create_contract( + output_format=None, # "isni" (default); None/"default"/"isni" all resolve to it + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- One grammar: `isni_recognition` (spaced/compact/hyphen + label/host/carrier groups). +- Two rules: `Section 4-isni-structure` + `Section A-mod11-2-check-character` (ISO 27729:2024; both PARSER, always-active — no registry flag exists). +- `year` filters by `publication_year`; e.g., `year=2023` drops the 2024 rules → `0000000121032683` becomes `INVALID`. +- Deterministic by construction: same input + contract + vendored snapshot → same output. No clock, no registry lookup. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `ISNI 0000 0001 2103 2683` | defaults | `SUCCESS` | `"0000 0001 2103 2683"` | +| `0000000121032683` / hyphenated twin | defaults | `SUCCESS` | same spaced canonical (presentation-only dedup) | +| `https://isni.org/isni/…` / `urn:isni:…` | defaults | `SUCCESS` | carriers stripped, span includes carrier | +| `0000 0001 2281 955x` | defaults | `SUCCESS` | `"0000 0001 2281 955X"` (lowercase-x tolerance) | +| `000000012146438X` | defaults | `SUCCESS` | `X` check digit validates | +| `0000000121032684` | defaults | `INVALID` | recognized shape, MOD 11-2 fails | +| `ISNI0000000121032683` | defaults | `MISSING` | glued label needs a separator | +| `000000012103268` (15) / 17-char | defaults | `MISSING` | length guard | +| two distinct ISNIs | defaults | raises `MultipleMentionsError` | un-segmented multi-entity input (`single_value=True`) | +| `0000000121032683` | `year=2023` | `INVALID` | rules are 2024, dropped | +| `0000000121032683` | `output_format="hyphenated"` | raises `ContractError` | never offered | + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
isni_recognition] + G --> R{Section 4
structure + Section A
MOD 11-2} + R -->|16-char shape, check holds| OK[SUCCESS] + R -->|bad check| INV[INVALID] + R -->|rule dropped by year filter| INV + G -->|nothing claimed| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +**Sibling note:** a hyphenated 16-digit string also matches the ORCID shape — each side resolves under its own contract (ISNI `SUCCESS` spaced, ORCID `SUCCESS` hyphenated). The same holds for `ORCID:`-labeled or `urn:orcid:`-carried digits: ORCID iDs are ISNI-compatible by design, so structurally valid digits resolve as ISNI with ISO 27729 provenance (symmetric with shipped ORCID claiming `ISNI:` labels). Unassigned-but-shaped numbers read `SUCCESS` (storable, no registry — UUID precedent). + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import ISNI +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = ISNI.create_contract() + +rows = [ + "ISNI 0000 0001 2103 2683", + "0000000121032683", + "0000-0001-2103-2683", + "https://isni.org/isni/0000000121032683", + "urn:isni:0000000121241960", + "0000 0001 2281 955x", + "0000000121032684", + "ISNI0000000121032683", + "not an isni", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:46} → {r.status.value:10} {val!r:24} ({rule})") +``` + +--- + +## Provenance + +- **ISO 27729:2024** (specification; 16-char structure: 15 digits + MOD 11-2 check character) — `Section 4-isni-structure` (vendored version 2024-11) +- **ISO 27729:2024 Annex A** (specification; MOD 11-2 over the first 15 decimal digits, `X` = 10) — `Section A-mod11-2-check-character` + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [ORCID](./orcid/), [ISBN](./isbn/), [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/). diff --git a/docs/user/v0.6.0/capabilities/issn.md b/docs/user/v0.6.0/capabilities/issn.md new file mode 100644 index 00000000..b9c59d00 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/issn.md @@ -0,0 +1,128 @@ +--- +title: "ISSN" +slug: v0.6.0/capabilities/issn +--- + +Canonicalizes **one ISSN identifier** per call to its hyphenated canonical form with mod-11 check-digit validation. + +> **In plain language:** give it `"0317-8471"` or `"03178471"` or `"ISSN 0317-8471"` and it hands back `0317-8471` if the check digit is correct (`X` = 10). The hyphen is presentation — the underlying value is always hyphenated unless you ask for compact. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Bare hyphenated `0317-8471`, `0378-5955`, `1050-124X` | ISSN with wrong check digit — recognized but `INVALID` (e.g. `0378-5954` → `INVALID`) | +| Bare compact `03178471`, `1050124X` (no hyphen) | Space at hyphen position `0317 8471` → `MISSING` (strict `-?` only) | +| Label `ISSN 0317-8471`, `ISSN: 0317-8471`, `ISSN-L 0264-2875`, `ISSN-H 1365-201X` (case-insensitive, `x`→`X` folded) | Unicode dash `0317–8471` (en-dash U+2013 / em-dash) → `MISSING` (hyphen-minus U+002D only) | +| Glued `ISSN03178471` (no separator after label, `glued allow`) | `eISSN 0317-8471` / `pISSN 0378-5955` without hyphen → `MISSING` (informal prefixes not in ISSN Manual; hyphenated `e-ISSN 0317-8471` incidentally matches `ISSN 0317-8471` via substring) | +| Embedded in prose `see ISSN 0317-8471 (print)` | `1234 - 5679` (space-hyphen-space) → `MISSING` | +| `urn:issn:0317-8471` incidentally as `issn:0317-8471` (span excludes `urn:`) | `123456789` (9 digits) → `MISSING` via `BoundarySpec.WORD` | + +> **Strict hyphen:** `-?` at canonical position 4 only. Tolerant `1234 - 5679` (space around hyphen) and `1234 5679` are `MISSING` unless a `Pre` normalizer is added (memo §13#5, Oracle fix 3). **Hyphen-digit continuation** `0317-8471-2` is `MISSING` (trailing `(?![-]\d)` blocks prefix, WORD alone would have allowed it). **Label separator** is `[\s:-]*` with `glued allow` (vs IBAN `reject`); `ISSN03178471` matches, `IBANDE89...` is `MISSING` per ADR §9.7. + +--- + +## Canonical output + +Default `output_format` is `"hyphenated"` (`XXXX-XXXX`). Mirrors ISBN's presentational-only `output_format` invariant: rules never read it. + +| `output_format` | Renders | Example for `0317-8471` | +|-----------------|---------|--------------------------| +| *(default)* `hyphenated` / `None` / `"default"` | hyphenated canonical | `0317-8471` | +| `compact` | bare 8 digits | `03178471` | +| `urn` | URN namespace | `urn:issn:0317-8471` | + +`compact` strips hyphen, `urn` wraps `urn:issn:` (lowercase namespace, hyphen preserved, `X` uppercase). Formatting adds no provenance; the validating rule remains `Section 4-issn-check-digit` (`ISO 3297:2022`). + +```python +from paxman.capabilities import ISSN +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("0317-8471", ISSN.create_contract()).canonicalized_value # "0317-8471" +paxman.canonicalize("03178471", ISSN.create_contract()).canonicalized_value # "0317-8471" (bare → hyphenated) +paxman.canonicalize("ISSN 0317-8471", ISSN.create_contract()).canonicalized_value # "0317-8471" +paxman.canonicalize("0317-8471", ISSN.create_contract(output_format="compact")).canonicalized_value # "03178471" +paxman.canonicalize("0317-8471", ISSN.create_contract(output_format="urn")).canonicalized_value # "urn:issn:0317-8471" +paxman.canonicalize("1050-124x", ISSN.create_contract()).canonicalized_value # "1050-124X" (x→X) +``` + +--- + +## Contract + +```python +contract = ISSN.create_contract( + output_format=None, # "hyphenated" (default), "compact", "urn" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars / suppress_common_words +) +``` + +- No `include_*` flags (single always-active grammar, `active_grammars=None` → engine runs every `get_grammars()` grammar in order). Other caps use `include_*` to gate grammars; ISSN does not. +- `year` filters rules by `publication_year` (`ISO 3297:2022` → 2022, so `year=2021` → `INVALID` per temporal filtering, even though check digit stable since earlier editions). +- `output_format` resolves via `CapabilityContract.__post_init__` (`None`/`"default"`/`"hyphenated"` → hyphenated, `ContractError` otherwise). + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `0317-8471` | defaults | `SUCCESS` | bare hyphenated, mod-11 valid | +| `03178471` | defaults | `SUCCESS` | compact → hyphenated `0317-8471` | +| `ISSN 0317-8471` | defaults | `SUCCESS` | label with span (0,14) | +| `ISSN-L 0264-2875` | defaults | `SUCCESS` | linking ISSN, same check | +| `1050-124X` / `1050-124x` | defaults | `SUCCESS` | `X=10` (folded) | +| `0378-5954` (bad check) | any | `INVALID` | recognized but check fails (`5` expected) | +| `12X4-5679` / `X234-5679` | any | `MISSING` | `\d{4}` guard filters mid-X; rule also rejects | +| `0317-8471-2` | any | `MISSING` | hyphen-digit continuation blocked by `(?![-]\d)` | +| `call me at noon` | any | `MISSING` | no ISSN pattern | +| Two distinct ISSNs `0317-8471 0378-5955` | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G[issn_recognition
LabelMatcher ISSN/ISSN-L/ISSN-H] + G --> R[Rule: ISO 3297:2022 §4
mod-11 8→2, X=10] + R -->|one value| OK[SUCCESS
XXXX-XXXX] + R -->|bad check| INV[INVALID] + G -->|nothing matched| MISS[MISSING] + A -->|two distinct values| AMB[MULTIPLE_MENTIONS] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import ISSN +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c = ISSN.create_contract() +c_compact = ISSN.create_contract(output_format="compact") +c_urn = ISSN.create_contract(output_format="urn") + +for text in ["0317-8471", "03178471", "ISSN 0317-8471", "1050-124x", "0378-5954", "hello"]: + r = paxman.canonicalize(text, c) + rc = paxman.canonicalize(text, c_compact) + ru = paxman.canonicalize(text, c_urn) + print(f"{text!r:20} → {r.status.value:10} {r.canonicalized_value!r:15} compact={rc.canonicalized_value!r:10} urn={ru.canonicalized_value!r}") +``` + +--- + +## Provenance + +- **ISO 3297:2022** (7th edition 2022-06, ISSN International Centre, Paris) — structure (8 chars, 7 digits + `X`) and mod-11 check digit (weights 8→2, `X`=10, `11→0`). Catalogue at `https://www.iso.org/standard/84536.html`. + +Deferred per research memo: **ISSN Manual** (May 2025) display detail, **RFC 3044** / `draft-ietf-urnbis-rfc3044bis` / **IANA `urn:issn`** (URN namespace — `urn:issn:` wrapping is presentation only, no validation rule in v1), **ISSN Register** (`portal.issn.org`) linking/cluster `ISSN-L` (relational, not lexical — requires registry lookup, same lexical shape). + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). + diff --git a/docs/user/v0.6.0/capabilities/language.md b/docs/user/v0.6.0/capabilities/language.md new file mode 100644 index 00000000..2f98fac7 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/language.md @@ -0,0 +1,193 @@ +--- +title: "Language" +slug: v0.6.0/capabilities/language +--- + +Canonicalizes **one language mention** per call — a bare code, a BCP 47 tag, or a display name — to a canonical BCP 47 tag. + +> **In plain language:** give it `en`, `eng`, `en-US`, `zh-Hans-CN`, `German`, or `deutsch` and it hands back the BCP 47 canonical tag (`en`, `de`, `zh-Hans-CN`) if a real spec says that name means that language. Localized and collective/private codes are off by default so results are predictable; turn them on when you need them. + +--- + +## What it recognizes — and what it does not + +| Recognizes (current release — growing) | Does not recognize | +|----------------------------------------|--------------------| +| Bare ISO 639 codes: alpha-2 (`en`, `de`), alpha-3 Term/Bib (`eng`/`ger`), comprehensive (`cmn`), collective (`aav` when `include_collective=True`), private `qaa` when `include_private=True` | Bare 4-letter codes (`abcd`) — 4 is script, not language; 1-letter (`x`); bare 5–8 runs (`hello`, `Xenon`) → `MISSING` (no hyphenless 5–8 primary in the shipped set) | +| BCP 47 tags (`en-US`, `zh-Hans-CN`, `es-419`, `sl-nedis`, `en-a-foo`, `x-private`, `art-lojban`→`jbo`, `en-GB-oed`→`en-GB-oxendict`) with `_`→`-` folding (`fr_FR`→`fr-FR`) | Inputs with no valid tag (`-en`, `--`, single `x`) → `MISSING`; overlong/double-hyphen inputs still yield `SUCCESS` via valid prefix (`en-US-123456789`→`en-US`) | +| English display names — 60-entry curatorial subset (`German`→`de`, `Serbo Croatian`→`sh`, `Norwegian Bokmal`→`nb`, `Cherokee`→`chr`) | Names outside the 60 (`Klingonish`) → `MISSING` (grammar emits no match, not `INVALID`) | +| CLDR localized names — 24-entry subset (`allemand`→`de`, `deutsch`→`de`) — only when `include_localized=True` | Localized names when flag off — recognized but `INVALID` (no authority claims them) | +| Grandfathered / deprecated Preferred-Value (`i-cherokee`→`chr`, `iw`→`he`, `scc`→`sr` historical) | Private subtags (`qaa`, `Qaaa`, `ZZ`) without `include_private` → `INVALID` | +| Compositional descriptions — full-phrase spans only (`Singapore Chinese in traditional script`→`zh-Hant-SG`, `Chinese in simplified script`→`zh-Hans`, `Chinese in Singapore`→`zh-SG`; Wave 2: `German in Latin script`, `Armenian in Armenian script`, `German in Germany`, `Japanese in Japan` — `AMBIGUOUS` under default per note below, `SUCCESS` when suppressed) | Trailing content word (`Chinese in traditional dress` → `SUCCESS id` via bare `in` under default, `MISSING` when suppressed) | +| Parenthesized descriptions — full-phrase spans only (`Chinese (Traditional, Singapore)`→`zh-Hant-SG`, `Chinese (Simplified)`→`zh-Hans`, `Chinese (Singapore)`→`zh-SG`; Wave 2: `German (Latin)`→`de-Latn`, `Japanese (Japan)`→`ja-JP`) — `SUCCESS` under the default contract and with `suppress_common_words=True` (no `in`, so no bare-code competitor) | Unbalanced (`Chinese (Traditional`) or trailing noun inside (`Chinese (Traditional dress)`) → `MISSING` | + +> **Compositional contract note:** a compositional description is `AMBIGUOUS` under the default contract — the full-phrase reading (e.g. `zh-Hant-SG`) coexists with the bare-code `in`→`id` reading — and `SUCCESS` with `suppress_common_words=True` (the `in` claim is a suppressible common word). + +> **Subset disclaimer:** English 60 + localized 24 are hand-curated from `paxman/shared_data/language_snapshot.json` — not the full IANA Registry Description set (7,900+) or CLDR v46 root. Compositional script/region displays are a curated Wave 2 (7 scripts incl. the `armenian` collision class, 8 single-token regions; multi-token regions and `thai` deferred). Names outside are `MISSING`, not `INVALID`, so no false negative under current completeness. ISO 639 code tables are curated subsets of their authorities (alpha-2 184 comprehensive; alpha-3 T 420 + B 21; comprehensive 995; collective 84 — see `language_snapshot.json` _meta and per-rule data headers). Plan: generate full IANA Description + CLDR root via `tools/regenerate_language_data.py`. + +--- + +## Canonical output + +Default `output_format` is `"bcp47"` (identity). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `bcp47` / `None` / `"default"` | BCP 47 canonical tag (B→T, Deprecated→Preferred, case-canonical) | `en-US`, `zh-Hans-CN`, `chr` (from `i-cherokee`) | +| `alpha2` | ISO 639-1 alpha-2 when available else Term else itself, carrying extended subtags verbatim per ADR-0011 (`ger`→`de`) | `en` from `eng`, `de` from `ger`/`deu`, `en-US` from `en-US` (carry), `chr` passthrough | +| `alpha3` | ISO 639-2 Term lower of the primary subtag only for extended tags — waived projection per ADR-0011 (`de`→`deu`, `en`→`eng`) | `eng` from `en`, `deu` from `de`, `deu` from `de-CH-1901` (primary-only) | +| `alpha3-bib` | Bibliographic of the primary subtag only for extended tags — waived projection per ADR-0011 (`deu`→`ger`, `fra`→`fre`) | `ger` from `de`, `fre` from `fr`, `ger` from `de-CH-1901` (primary-only) | +| `name` | English Description title of the primary subtag only — waived projection per ADR-0011 | `German` from `de`, `English` from `eng`, `English` from `en-US` (primary-only) | + +Any other value raises `ContractError`. + +```python +from paxman.capabilities import Language +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("German", Language.create_contract()).canonicalized_value # "de" +paxman.canonicalize( + "de", Language.create_contract(output_format="alpha3") +).canonicalized_value # "deu" +paxman.canonicalize( + "de", Language.create_contract(output_format="alpha3-bib") +).canonicalized_value # "ger" +paxman.canonicalize( + "deu", Language.create_contract(output_format="alpha2") +).canonicalized_value # "de" +paxman.canonicalize( + "de", Language.create_contract(output_format="name") +).canonicalized_value # "German" +paxman.canonicalize("en-US", Language.create_contract()).canonicalized_value # "en-US" +``` + +--- + +## Contract + +```python +contract = Language.create_contract( + include_private=False, # bool, default False — qaa-qtz / Qaaa-Qabx / QM-QZ etc. + x- + include_collective=False, # bool, default False — ISO 639-5 families (aav, ber, gem…) + include_localized=False, # bool, default False — CLDR 24 localized display names + output_format=None, # "bcp47" (default), "alpha2", "alpha3", "alpha3-bib", "name" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- `include_private` gates `Section 4-private-alpha-3` and `Section-iana-registry-private`; without it `qaa` is `INVALID` (grammar claims, no rule validates). +- `include_collective` gates `Section 4-collective-code`; without it `aav` is `INVALID`. +- `include_localized` gates `Section-localized-names`; without it `allemand` is recognized but `INVALID`. +- `year` filters by `publication_year`; e.g., `year=2008` drops BCP 47 RFC 5646 (2009) → `en-US` becomes `INVALID`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `en` | defaults | `SUCCESS` | `"en"` | +| `eng` | defaults | `SUCCESS` | `"en"` (`eng→en` via alpha2 mapping) | +| `ger` | defaults | `SUCCESS` | `"de"` (B→T `ger→deu`→`de`) | +| `German` | defaults | `SUCCESS` | `de` (English name) | +| `deutsch` | `include_localized=True` | `SUCCESS` | `de` (CLDR) | +| `deutsch` | defaults | `INVALID` | recognized but no localized rule | +| `aav` | defaults | `INVALID` | collective needs flag | +| `aav` | `include_collective=True` | `SUCCESS` | `aav` | +| `qaa` | `include_private=True` | `SUCCESS` | `qaa` | +| `en-US` | `year=2008` | `INVALID` | BCP 47 rule is 2009, dropped | +| `de-nedis` | any | `INVALID` | variant prefix `nedis` requires `sl` | +| `Serbo-Croatian` | defaults | `SUCCESS` | `sh` (name rule `Section-english-name-mapping` stands alone; the BCP 47 syntax-only ghost is disqualified per ADR-0012) | +| `Klingonish` | any | `MISSING` | no language pattern | +| `en, fr` (two different mentions) | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G[Grammars:
bcp47_tag / language_code / language_name] + G --> R1{ISO 639-1 alpha-2} + G --> R2{ISO 639-2 alpha-3 T/B + deprecated} + G --> R3{ISO 639-3 comprehensive + private} + G --> R4{ISO 639-5 collective if flag} + G --> R5{BCP47 syntax + Prefix} + G --> R6{IANA registry + private} + G --> R7{English name} + G --> R8{CLDR localized if flag} + R1 & R2 & R3 & R4 & R5 & R6 & R7 & R8 --> Q{Corroboration
(PARSER needs same-rep LOOKUP)} + Q -->|uncorroborated PARSER| DROP[discarded ghost] + Q -->|corroborated| D{Dedup values} + D -->|one value| OK[SUCCESS] + D -->|none| INV[INVALID] + D -->|distinct| AMB[AMBIGUOUS] + G -->|no pattern| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 + style AMB fill:#fffbe6,stroke:#b38f00 + style DROP fill:#f5f5f5,stroke:#999999 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import Language +from paxman.core.domain import Resolution +from paxman.core.errors import CapabilityError, ContractError, MultipleMentionsError + +paxman.register_all_shipped() +contract = Language.create_contract(include_localized=True, include_private=True) +contract_coll = Language.create_contract(include_collective=True) + +rows = [ + "en", + "eng", + "ger", + "German", + "deutsch", + "zh-Hans-CN", + "en-US", + "qaa", + "aav", + "Serbo-Croatian", + "not a language", + "en, fr", +] + +for text in rows: + for label, c in [ + ("default+local+private", contract), + ("+collective", contract_coll), + ]: + try: + r = paxman.canonicalize(text, c) + except (MultipleMentionsError, CapabilityError, ContractError) as e: + print(f"{text!r:20} [{label}] → exception {type(e).__name__}: {e}") + continue + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + prov = r.candidates[0].provenance[0].specification_name if r.candidates else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print( + f"{text!r:20} [{label:22}] → {r.status.value:10} {val!r:15} ({rule} / {prov})" + ) +``` + +--- + +## Provenance + +- **ISO 639-1:2002** alpha-2 (184) + English names — `Section 4-alpha-2-code`, `Section-english-name-mapping` +- **ISO 639-2:1998** alpha-3 T/B curated subset (T 420 + B 21, B→T `ger→deu`) — `Section 4-alpha-3-code` +- **ISO 639-3:2007** comprehensive curated subset (995, SIL RA) — `Section 4-comprehensive-alpha-3`, `Section 4-private-alpha-3` (`qaa-qtz`) +- **ISO 639-5:2008** collective families curated subset (84, when `include_collective`) — `Section 4-collective-code` +- **BCP 47 RFC 5646** Section 2.1 Language-Tag ABNF (2009-09) + variant Prefix guard — `Section 2.1-syntax` +- **IANA Language Subtag Registry** Rolling File-Date 2026-08-08 — `Section-iana-registry`, `Section-iana-registry-private` (qaa-qtz/Qaaa-Qabx/QM-QZ/AA/XA-XZ/ZZ/x-) +- **CLDR Language Display Names** v46 (2025, Unicode) — `Section-localized-names` (24, when `include_localized`) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). + diff --git a/docs/user/v0.6.0/capabilities/lei.md b/docs/user/v0.6.0/capabilities/lei.md new file mode 100644 index 00000000..3987e554 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/lei.md @@ -0,0 +1,182 @@ +--- +title: "LEI" +slug: v0.6.0/capabilities/lei +--- + +Canonicalizes **one LEI mention** per call — a bare 20-character code, a single-spaced form, or an `LEI`-labelled / `urn:lei:`-carried form — to compact uppercase form. + +> **In plain language:** give it `5493000IBP32UQZ0KL24`, `5493 000IBP32UQZ0KL24`, `LEI: 5493000IBP32UQZ0KL24`, or `urn:lei:5493000IBP32UQZ0KL24` and it hands back `5493000IBP32UQZ0KL24` when the shape is `LOU+entity+check` (4-character LOU prefix, 14-character entity block, 2 numeric check digits), the whole-string MOD 97-10 remainder is 1 per ISO 17442-1:2020 §4, and the prefix is an accredited LOU block per the GLEIF LOU prefix list. A bad check digit or an unattested prefix is `INVALID`, never a guess; shapes outside the recognized spellings are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Bare compact 20-char (`213800KUD8LAJWSQ9D15`, `5493000IBP32UQZ0KL24`, `7LTWFZYICNSX8D621K86`; lowercase folds up) | Hyphen-separated (`5493-000I-BP32-UQZ0-KL24`) → `MISSING` (hyphen tolerance deferred — see statuses) | +| Single-space forms (`5493 000IBP32UQZ0KL24`, `2138 00KU D8LA JWSQ 9D15`) | Double spaces / tabs (`5493 000IBP32UQZ0KL24`) → `MISSING` | +| `LEI` label with separator (`LEI: 213800KUD8LAJWSQ9D15`, `lei 5493000IBP32UQZ0KL24`, `LEI-213800KUD8LAJWSQ9D15` — case-insensitive) | Glued label (`LEI5493000IBP32UQZ0KL24`, no separator) → `MISSING` | +| `urn:lei:` carrier (`urn:lei:213800KUD8LAJWSQ9D15`, `URN:LEI:5493000IBP32UQZ0KL24` — case-insensitive) | Wrong lengths (19 / 21 chars) → `MISSING` (rejected — not LEI shapes) | +| Non-`00` entity content (`7LTWFZYICNSX8D621K86` carries `FZ` at positions 5–6) | Fullwidth homoglyphs (`213800KUD8LAJWSQ9D1\uff15`) → `MISSING` (ASCII only) | +| Embedded in prose (`see LEI 213800KUD8LAJWSQ9D15 (London)`) | Glued surroundings (`X213800KUD8LAJWSQ9D15`, `213800KUD8LAJWSQ9D15Y`) → `MISSING` | +| URL-path-embedded (`https://search.gleif.org/#/record/5493000IBP32UQZ0KL24` — carved at the `/`/`#` boundary; URL-level semantics belong to the URL capability) | | +| | Prose without an LEI shape (`not an lei`) → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"lei"` (identity — `normalize()` returns the compact string). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `lei` / `None` / `"default"` | Compact uppercase 20-char | `5493000IBP32UQZ0KL24` | +| `urn` | `urn:lei:` carrier (encoding — the carrier branch re-recognizes the rendering to the same compact pre-image, so it re-enters exactly) | `urn:lei:5493000IBP32UQZ0KL24` from `5493000IBP32UQZ0KL24` | + +Any other value raises `ContractError` — including `compact`, which is not an offered alias (normalize outside the call if you need that spelling). + +```python +from paxman.capabilities import LEI +import paxman + +paxman.register_all_shipped() +print( + paxman.canonicalize( + "5493000ibp32uqz0kl24", LEI.create_contract() + ).canonicalized_value +) +print( + paxman.canonicalize( + "LEI: 213800KUD8LAJWSQ9D15", LEI.create_contract() + ).canonicalized_value +) +print( + paxman.canonicalize( + "5493 000IBP32UQZ0KL24", LEI.create_contract() + ).canonicalized_value +) +print( + paxman.canonicalize( + "5493000IBP32UQZ0KL24", LEI.create_contract(output_format="urn") + ).canonicalized_value +) +print( + paxman.canonicalize( + "7LTWFZYICNSX8D621K86", LEI.create_contract() + ).canonicalized_value +) +``` + +--- + +## Contract + +```python +contract = LEI.create_contract( + output_format=None, # "lei" (default), "urn" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`lei_recognition`), two rules (`Section 4-lei-structure-mod97-10`, `Section 1-lou-prefix-membership`). +- `year` filters by `publication_year`; e.g., `year=2019` drops both rules (2020, 2026) → `213800KUD8LAJWSQ9D15` becomes `INVALID`, while `year=2020` keeps the ISO structure rule → still `SUCCESS`. +- Single-value: two distinct LEIs in one call raise `MultipleMentionsError` — split first per `docs/recipes/segmentation.md`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `213800KUD8LAJWSQ9D15` | defaults | `SUCCESS` | `"213800KUD8LAJWSQ9D15"` | +| `5493000ibp32uqz0kl24` | defaults | `SUCCESS` | `"5493000IBP32UQZ0KL24"` (case folds up) | +| `LEI: 213800KUD8LAJWSQ9D15` | defaults | `SUCCESS` | label absorbed into the span | +| `urn:lei:5493000IBP32UQZ0KL24` | defaults | `SUCCESS` | carrier absorbed into the span | +| `5493 000IBP32UQZ0KL24` | defaults | `SUCCESS` | `"5493000IBP32UQZ0KL24"` (spacing recognized, spaces stripped) | +| `213800WSGIIZCXF1P572` / `506700GE1G29325QX363` | defaults | `SUCCESS` | further accredited-LOU LEIs, check digits hold | +| `7LTWFZYICNSX8D621K86` | defaults | `SUCCESS` | non-`00` entity content never rejects | +| `5493000IBP32UQZ0KL24` | `output_format="urn"` | `SUCCESS` | `"urn:lei:5493000IBP32UQZ0KL24"` | +| `https://search.gleif.org/#/record/5493000IBP32UQZ0KL24` | defaults | `SUCCESS` | `"5493000IBP32UQZ0KL24"` (carved at the `/`/`#` boundary — URL-level semantics belong to the URL capability) | +| `213800KUD8LAJWSQ9D15` | `pinned_rules=("Section 4-lei-structure-mod97-10",)` | `SUCCESS` | ISO rule alone corroborates (vacuity clause) | +| `213800KUD8LXJWSQ9D15` | any | `INVALID` | recognized shape, check digit fails (remainder 55, not 1) | +| `ZZZZ0000000000000016` | any | `INVALID` | recognized shape with valid check digits, prefix has no accredited-LOU attestation | +| `213800KUD8LAJWSQ9D15` | `year=2019` | `INVALID` | both rules are newer, dropped | +| `213800KUD8LAJWSQ9D15` | `excluded_rules=(both,)` | `INVALID` | no rule validates | +| `5493-000I-BP32-UQZ0-KL24` | any | `MISSING` | **DEFER** — hyphen tolerance is not recognized in v1; a community grammar via `extra_grammars` is the extension path (tracked in https://github.com/nexusnv/paxman-python/issues/183) | +| 19- / 21-char forms | any | `MISSING` | **REJECT** — short/long forms are not LEI shapes | +| `LEI5493000IBP32UQZ0KL24` | any | `MISSING` | glued label needs a separator | +| `5493 000IBP32UQZ0KL24` | any | `MISSING` | double spaces are not a grouping separator | +| `not an lei` | any | `MISSING` | no LEI shape | +| `5493000IBP32UQZ0KL24` | `output_format="compact"` | raises `ContractError` | not an offered format | +| Two distinct LEIs in one call | any | raises `MultipleMentionsError` | split first | + +Two deterministic rules over one single-value grammar yield at most one value, so `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
lei_recognition] + G --> R1{Section 4
structure + MOD 97-10} + R1 --> R2{Section 1
LOU prefix membership} + R2 -->|both hold| OK[SUCCESS] + R1 -->|no rule validates| INV[INVALID] + R2 -->|no rule validates| INV + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import LEI +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = LEI.create_contract() + +rows = [ + "213800KUD8LAJWSQ9D15", + "5493000ibp32uqz0kl24", + "LEI: 213800KUD8LAJWSQ9D15", + "urn:lei:5493000IBP32UQZ0KL24", + "5493 000IBP32UQZ0KL24", + "7LTWFZYICNSX8D621K86", + "213800WSGIIZCXF1P572", + "213800KUD8LXJWSQ9D15", + "ZZZZ0000000000000016", + "5493-000I-BP32-UQZ0-KL24", + "213800KUD8LAJWSQ9D1", + "LEI5493000IBP32UQZ0KL24", + "not an lei", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:32} → {r.status.value:10} {val!r:24} ({rule})") + +try: + LEI.create_contract(output_format="compact") +except ContractError as e: + print(f"compact → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO 17442-1:2020** §4 LEI structure plus check digits (20 chars `LOU4+entity14+check2`, charset `[A-Z0-9]{18}[0-9]{2}`; whole-string ISO/IEC 7064:2003 MOD 97-10, `A=10 … Z=35`, remainder 1, no rearrangement) — `Section 4-lei-structure-mod97-10` (`PARSER`). Catalogue at `https://www.iso.org/standard/78829.html`. +- **GLEIF LOU prefix list** §1 accredited-LOU prefix membership (4-char issuer blocks observed in the accredited-LOU directory and on real records in the GLEIF concatenated file; append-only snapshot — codes survive LOU retirement and transfer) — `Section 1-lou-prefix-membership` (`LOOKUP_TABLE`). Download at `https://www.gleif.org/en/lei-data/gleif-concatenated-file/download-the-concatenated-file`. + +Issued/live membership is deferred in v1: liveness is not validity — an LEI with valid structure, check digits, and an accredited prefix is `SUCCESS` whether or not it is currently issued (tracked in https://github.com/nexusnv/paxman-python/issues/184). + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/mac_address.md b/docs/user/v0.6.0/capabilities/mac_address.md new file mode 100644 index 00000000..04aab772 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/mac_address.md @@ -0,0 +1,140 @@ +--- +title: "MacAddress" +slug: v0.6.0/capabilities/mac_address +--- + +Canonicalizes **one MAC address mention** per call — an EUI-48 or EUI-64 in colon, hyphen, Cisco tri-dot, or bare hex (with an optional fused `MAC` label) — to uppercase colon form. + +> **In plain language:** give it `00-1A-2B-3C-4D-5E`, `001A.2B3C.4D5E`, `001A2B3C4D5E`, or `MAC: 00:1A:2B:3C:4D:5E` and it hands back `00:1A:2B:3C:4D:5E` when the shape is a real EUI-48/EUI-64 structure per IEEE Std 802-2024 §8.2. The spec defines no checksum, so validation is structural (hex length + shape); I/G and U/L bits are informative and never rejected. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Colon EUI-48 (`00:1A:2B:3C:4D:5E`, lowercase folds up) and EUI-64 (`00:1A:2B:3C:4D:5E:66:77`) | Truncated shapes (`00:1A:2B:3C:4D`, 7-octet `00:1A:2B:3C:4D:5E:66`) → `MISSING` | +| Hyphen EUI-48/EUI-64 (`00-1A-2B-3C-4D-5E`) | Non-hex or wrong-length runs (`001A2B3C4D5E66`, 14 hex) → `MISSING` | +| Cisco tri-dot hextets (`001A.2B3C.4D5E`) | Glued label (`MAC001A2B3C4D5E`, no separator) → `MISSING` | +| Bare hex, 12 (`001A2B3C4D5E`) or 16 (`001A2B3C4D5E6677`) digits | Prose without an address shape (`not a mac`) → `MISSING` | +| Fused `MAC` label with separator (`MAC: 00:1A:2B:3C:4D:5E`) | | + +--- + +## Canonical output + +Default `output_format` is `"colon"` (identity — `normalize()` returns colon form). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `colon` / `None` / `"default"` | Uppercase colon-separated octets | `00:1A:2B:3C:4D:5E` | +| `hyphen` | Uppercase hyphen-separated (encoding) | `00-1A-2B-3C-4D-5E` | +| `bare` | Uppercase hex, no separators (encoding) | `001A2B3C4D5E` | +| `cisco` | Uppercase tri-dot hextets (encoding) | `001A.2B3C.4D5E` | +| `eui64` | Same-entity expansion: EUI-48 gains `FF:FE` after the OUI (`...2B:FF:FE:3C...`); EUI-64 is identity | `00:1A:2B:FF:FE:3C:4D:5E` from `00:1A:2B:3C:4D:5E` | + +Any other value raises `ContractError` — including `bit_reversed`, removed in v0.4.0 per ADR-0010 (bit-reversal is an involution on the same identifier, not a canonical fixed point; store the `colon` form for round-trippable output). + +```python +from paxman.capabilities import MacAddress +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("00-1A-2B-3C-4D-5E", MacAddress.create_contract()).canonicalized_value) +print(paxman.canonicalize("001A.2B3C.4D5E", MacAddress.create_contract()).canonicalized_value) +print(paxman.canonicalize("001A2B3C4D5E6677", MacAddress.create_contract()).canonicalized_value) +print(paxman.canonicalize("00:1A:2B:3C:4D:5E", MacAddress.create_contract(output_format="hyphen")).canonicalized_value) +print(paxman.canonicalize("00:1A:2B:3C:4D:5E", MacAddress.create_contract(output_format="eui64")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = MacAddress.create_contract( + output_format=None, # "colon" (default), "hyphen", "bare", "cisco", "eui64" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`mac_address_recognition`), one rule (`Section 8.2-eui-structure`). +- `year` filters by `publication_year`; e.g., `year=2020` drops the IEEE 802-2024 rule → `00:1A:2B:3C:4D:5E` becomes `INVALID`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `00:1A:2B:3C:4D:5E` | defaults | `SUCCESS` | `"00:1A:2B:3C:4D:5E"` | +| `00-1A-2B-3C-4D-5E` | defaults | `SUCCESS` | `"00:1A:2B:3C:4D:5E"` (hyphen recognized) | +| `001A.2B3C.4D5E` | defaults | `SUCCESS` | `"00:1A:2B:3C:4D:5E"` (Cisco recognized) | +| `001A2B3C4D5E6677` | defaults | `SUCCESS` | `"00:1A:2B:3C:4D:5E:66:77"` (EUI-64) | +| `MAC: 00:1A:2B:3C:4D:5E` | defaults | `SUCCESS` | fused label, span covers the address | +| `00:1A:2B:3C:4D` | any | `MISSING` | truncated, no shape matches | +| `MAC001A2B3C4D5E` | any | `MISSING` | glued label needs a separator | +| `00:1A:2B:3C:4D:5E` | `year=2020` | `INVALID` | rule is 2024, dropped | +| `00:1A:2B:3C:4D:5E` | `output_format="bit_reversed"` | raises `ContractError` | format removed in v0.4.0 | + +A single grammar plus a single deterministic rule yields at most one value, so `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
mac_address_recognition] + G --> R{Section 8.2
EUI structure} + R -->|structure holds| OK[SUCCESS] + R -->|no rule validates| INV[INVALID] + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import MacAddress +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = MacAddress.create_contract() + +rows = [ + "00:1A:2B:3C:4D:5E", + "00-1A-2B-3C-4D-5E", + "001A.2B3C.4D5E", + "001A2B3C4D5E", + "001A2B3C4D5E6677", + "MAC: 00:1A:2B:3C:4D:5E", + "00:1A:2B:3C:4D", + "not a mac", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:28} → {r.status.value:10} {val!r:24} ({rule})") + +try: + MacAddress.create_contract(output_format="bit_reversed") +except ContractError as e: + print(f"bit_reversed → ContractError: {e}") +``` + +--- + +## Provenance + +- **IEEE Std 802-2024** §8.2 Universal addresses (EUI-48/EUI-64 structure; no checksum — proved by absence across IEEE Std 802 and RFC 7042) — `Section 8.2-eui-structure` +- **RFC 7042** IANA Considerations and IETF Protocol Usage for EUI-48/EUI-64 (EUI-48→EUI-64 `FF:FE` expansion backing the `eui64` format) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/minor_planet.md b/docs/user/v0.6.0/capabilities/minor_planet.md new file mode 100644 index 00000000..07581224 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/minor_planet.md @@ -0,0 +1,61 @@ +--- +title: "MinorPlanet" +slug: v0.6.0/capabilities/minor_planet +--- + +Canonicalizes **one minor-planet mention** per call — an unpacked provisional (`1995 XA`, `2007 TA418`), a retrospective (`A904 OA`) or `A/`-prefixed (`A/2017 U1`) designation, a packed 7-char (`J95X00A`, `K07Tf8A`) or extended `_` (`_QC0000`) form, a survey designation in either spelling (`2040 P-L` / `PLS2040`), a parenthesized number (`(433)`), or a packed number (`03202`, `a0017`, `~000z`) — to the unpacked canonical. + +> **In plain language:** give it `J95X00A`, `1995 XA`, `PLS2040`, or `(433) Eros` and it hands back `1995 XA` / `2040 P-L` / `(433)` with the MPC publication cited. Structure-only at v1 (no registry): a well-formed but never-assigned shape is `SUCCESS`, never a guess about issuance; shapes outside the lanes are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Unpacked provisionals (`1995 XA`, `2007 TA418`, `1992 QB1`) | Glued runs (`1995XA`) → `MISSING` (separator required) | +| Spaced lowercase/underscore (`2003 cp20`, `1995 xa`, `1995_XA` → upper/space canonical) | Glued lowercase (`2003cp20`) → `MISSING` (JPL search liberality is resolution, not grammar) | +| Retrospective (`A904 OA`) and `A/`-prefixed (`A/2017 U1`, 1–2 letters) | Old-style systems (`1892 A`, `1914 VV`, `SIGMA 27`) → `MISSING` (superseded 1925) | +| Packed 7-char, case-exact (`J95X00A`, `K07Tf8A`; `a0017` ≠ `A0345`) | Bad-century packs (`Q95X00A`) → `MISSING` (century is lane-defining) | +| Extended `_` packs (`_QC0000`), survey packs (`PLS2040`, `T1S3138`) | Comet/satellite forms (`C/1995 O1`, `S/2000 J 11`, `1I/2017 U1`) → `MISSING` (deferred/disjoint namespaces) | +| Surveys either spelling (`2040 P-L` / `PLS2040`) | Bare numbers (`433`) → `MISSING` (parens load-bearing) | +| Parenthesized numbers (`(433)`, `(274301)`; name outside span in `(433) Eros`) | Bare names (`Eros`) → `MISSING` (names lane deferred) | +| Packed numbers (`03202`, `A0345`, `a0017`, `~000z`, `~AZaz`) | Subscript cycles (`1995 SA₁`) → `MISSING` (ASCII-only wire) | + +Known v1 overclaims (no registry): `(555)`-shaped phone area codes match the number lane and bare 5-digit runs match the packed-number lane. Segment phone/ZIP contexts before calling; the future MPCORB snapshot disambiguates. + +--- + +## Canonical output + +Default `output_format` is `"designation"` (unpacked canonical). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `designation` / `None` / `"default"` | Unpacked canonical | `1995 XA` | +| `packed` | Case-exact wire encoding | `J95X00A` | + +Any other value raises `ContractError`. The `packed` rendering re-enters under the default contract (ADR-0010 fixed point); the `A/` lane defines no packed mapping and falls back to the designation. + +```python +from paxman.capabilities import MinorPlanet +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("J95X00A", MinorPlanet.create_contract()).canonicalized_value) +print(paxman.canonicalize("(433) Eros", MinorPlanet.create_contract()).canonicalized_value) +print(paxman.canonicalize("1995 XA", MinorPlanet.create_contract(output_format="packed")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = MinorPlanet.create_contract( + output_format=None, # "designation" (default); "packed" offered + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +No capability-specific flags at v1. `year` filters rules by `publication_year` (all three publications are 2026 living documents). diff --git a/docs/user/v0.6.0/capabilities/money.md b/docs/user/v0.6.0/capabilities/money.md new file mode 100644 index 00000000..8e6376db --- /dev/null +++ b/docs/user/v0.6.0/capabilities/money.md @@ -0,0 +1,166 @@ +--- +title: "Money" +slug: v0.6.0/capabilities/money +--- + +Canonicalizes **one money amount paired with a currency** per call to `CODE amount` padded to ISO 4217 minor units. + +> **In plain language:** give it `"USD500"` or `"1.000,50 EUR"` and it hands back `"USD 500.00"` / `"EUR 1000.50"` if the specs say the amount and currency go together. Currency without an amount — or an amount without a currency — is not Money's domain (see [Currency](currency/)). + +--- + +## What it recognizes — and what it does not + +| Recognizes (currency + amount adjacent) | Does not recognize | +|-----------------------------------------|--------------------| +| `USD500`, `USD 500.00`, `EUR 1.000,50` | Currency identifier alone (`USD`) — use [Currency](currency/) | +| `€500`, `CN¥1000`, `US$ 500` (qualified symbols) | Bare shared symbol without disambiguation (`$500` or `¥1000` with default contract — `¥` is shared by CNY/JPY) — recognized but `INVALID` | +| `1.000,50 EUR` — European comma-decimal: last separator is the decimal point | Amount-glued tokens without a clean boundary — `MISSING` | +| `$500` with `dollar_sign_currency="USD"` — opt-in for shared symbols | Bare `$` without `dollar_sign_currency` — `INVALID` | + +> Shared symbol handling follows Currency's `default_currency` idea but for amounts: Money's `dollar_sign_currency` resolves only to one of the symbol's own CLDR candidates (`$500` + `USD` → `USD 500.00`, while `$500` + `MYR` stays `INVALID` — `MYR` is not a `$` candidate), exactly like Currency's `default_currency`. + +--- + +## Canonical output + +Default `output_format` is `"code_amount"` (space between code and amount). + +| `output_format` | Renders | Example for USD 500 | +|-----------------|---------|---------------------| +| *(default)* `code_amount` / `None` / `"default"` | `CODE + space + amount` | `USD 500.00` | +| `compact` | removes the single space | `USD500.00` | + +The amount is padded/normalized to the currency's ISO 4217 minor units (e.g. 2 decimals for USD/EUR). How over-precision is handled depends on `precision` (see contract). + +```python +from paxman.capabilities import Money +import paxman + +paxman.register_all_shipped() +# Money.create_contract() is a contract — use paxman.canonicalize() to get a result +paxman.canonicalize( + "USD500", Money.create_contract() +).canonicalized_value # "USD 500.00" +paxman.canonicalize( + "USD500", Money.create_contract(output_format="compact") +).canonicalized_value # "USD500.00" +paxman.canonicalize( + "1.000,50 EUR", Money.create_contract() +).canonicalized_value # "EUR 1000.50" +``` + +--- + +## Contract + +```python +contract = Money.create_contract( + dollar_sign_currency=None, # str | None — uppercase alpha-3, e.g. "USD" + precision="strict", # "strict" (default) | "truncate" | "round" — over-precision policy + output_format=None, # "code_amount" (default) or "compact" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- **`dollar_sign_currency`**: when `None` (default), a multi-candidate bare symbol amount like `$500` is recognized but never resolved → `INVALID`. When set to one of the symbol's own CLDR candidates, bare `$`/`¥`/`£` resolve to that code (e.g. `$` + `USD` → `USD 500.00`; `$` + `MYR` stays `INVALID` since `MYR` is not a `$` candidate — same guard as Currency's `default_currency`). `€500` never needs this — `€` is definitive for `EUR` and ignores `dollar_sign_currency`; qualified symbols like `US$`/`CA$`/`RM` are also definitive. An unknown code like `ZZZ` stays `INVALID` via the minor-unit guard. +- **`precision`**: how to treat more fractional digits than the currency's minor units allow — `strict` rejects → `INVALID`, `truncate` drops excess digits, `round` half-to-even. +- `output_format` never affects validation — only the space. + +```python +paxman.canonicalize("$500", Money.create_contract()).status.value # "invalid" +paxman.canonicalize( + "$500", Money.create_contract(dollar_sign_currency="USD") +).canonicalized_value # "USD 500.00" +# Bare "$" resolves only to one of its own CLDR candidates (parity with +# Currency's default_currency; non-candidate MYR stays INVALID) +paxman.canonicalize( + "$500", Money.create_contract(dollar_sign_currency="MYR") +).status.value # "invalid" +# Unknown code never resolves — INVALID via MINOR_UNITS guard +paxman.canonicalize( + "$500", Money.create_contract(dollar_sign_currency="ZZZ") +).status.value # "invalid" +paxman.canonicalize( + "USD 1.999", Money.create_contract(precision="strict") +).status.value # "invalid" — too many decimals for USD +paxman.canonicalize( + "USD 1.999", Money.create_contract(precision="round") +).canonicalized_value # "USD 2.00" +``` + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `USD500` | defaults | `SUCCESS` | → `USD 500.00` | +| `€500` | defaults | `SUCCESS` | → `EUR 500.00` (definitive symbol) | +| `$500` | defaults | `INVALID` | shared symbol, no `dollar_sign_currency` | +| `$500` | `dollar_sign_currency="USD"` | `SUCCESS` | → `USD 500.00` (`USD` is a `$` candidate) | +| `$500` | `dollar_sign_currency="MYR"` | `INVALID` | `MYR` is not a `$` candidate | +| `1.000,50 EUR` | defaults | `SUCCESS` | European comma-decimal → `EUR 1000.50` | +| `USD 1.999` | `precision="strict"` | `INVALID` | over-precision → rejected | +| `hello` | any | `MISSING` | no Money pattern | +| Two different amounts | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text e.g. $500] --> G[Grammars:
code / symbol / word + amount] + G --> R{Rules: ISO 4217 +
CLDR symbols/names
+ minor units} + R -->|definitive symbol
or code/word| OK[SUCCESS
CODE amount] + R -->|shared symbol
+ dollar_sign_currency| INV[INVALID default
or SUCCESS when requested] + R -->|over-precision + strict| INV2[INVALID] + G -->|no pattern| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style INV2 fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — clean a column with mixed formats + +```python +import paxman +from paxman.capabilities import Money +from paxman.core.domain import Resolution +from paxman.core.errors import CapabilityError, ContractError, MultipleMentionsError + +paxman.register_all_shipped() +c_strict = Money.create_contract() +c_usd = Money.create_contract(dollar_sign_currency="USD") +c_round = Money.create_contract(precision="round") + +rows = ["USD500", "€500", "$500", "1.000,50 EUR", "USD 1.999", "hello"] + +for text in rows: + for label, c in [("strict", c_strict), ("USD bare-$", c_usd), ("round", c_round)]: + try: + r = paxman.canonicalize(text, c) + except (MultipleMentionsError, CapabilityError, ContractError) as e: + print(f"{text!r:15} [{label:10}] → exception {type(e).__name__}: {e}") + continue + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:15} [{label:10}] → {r.status.value:10} {val!r}") +``` + +--- + +## Provenance + +- **ISO 4217:2015** — currency codes and minor units (List One, as amended through #180, snapshot 2026-01-01 via SIX; provenance `PUBLICATION` year 2015, `https://www.iso.org/iso-4217-currency-codes.html`). `CURRENCY_CODES` holds 165 codes with numeric minor units (13 N.A. codes excluded); `MINOR_UNITS` maps exponent (0 for JPY/KRW, 2 for most, 3 for BHD, 4 for CLF/UYW). +- **Unicode CLDR v47 (2025-03-13)** — currency symbols and English display names (`https://cldr.unicode.org/`, `https://cldr.unicode.org/downloads/cldr-47`). Word recognition is case-insensitive (any casing of `Euro`/`euro`/`EURO` resolves to `EUR`); symbols are case-exact (`lei` vs `Lei`). Newer CLDR v48/48.1 (2025-10 and 2026-01) exists — regeneration planned via `tools/regenerate_currency_data.py`. +- Amount parsing: last separator wins, single separator always decimal (`1,00` → `1`, `1.234` → `1.234`); grouping with multiple separators folds base-1000 (`1,00.50` → `1000.50`); narrow NBSP (`U+202F`) is the only space-grouping form — ASCII `1 234.56` is not grouped (see Limitations). + +## Limitations + +- **Single separator is always decimal — no thousand grouping.** `USD 1,000` → `SUCCESS USD 1.00`, not `USD 1000.00`. A lone `,`/`.` is read as the decimal point per the locked `parse_amount` table, so thousand-grouped input without a decimal part mis-canonicalizes. Treating a single separator with exactly 3 trailing digits as grouping would need a spec ruling (see follow-up); until then, pre-normalize thousand-grouped amounts before calling. +- **ASCII space never groups.** `USD 1 234.56` does not parse as grouped thousands; only narrow NBSP (`U+202F`) groups with spaces. + +Compare [Currency](currency/) for identifier-only canonicalization (no amount). + +See also: [Execution Result](../concepts/execution-result/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/orcid.md b/docs/user/v0.6.0/capabilities/orcid.md new file mode 100644 index 00000000..c9b2a169 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/orcid.md @@ -0,0 +1,151 @@ +--- +title: "ORCID" +slug: v0.6.0/capabilities/orcid +--- + +Canonicalizes **one ORCID mention** per call — a hyphenated `XXXX-XXXX-XXXX-XXXC` identifier, its bare-digit compact form, or an `orcid.org` URI (with an optional `ORCID`/`ISNI` label) — to hyphenated form. + +> **In plain language:** give it `0000-0002-1825-0097`, `0000000218250097`, `https://orcid.org/0000-0002-1825-0097`, or `ORCID: 0000-0002-1825-0097` and it hands back `0000-0002-1825-0097` when the 16-character structure holds and the MOD 11-2 check character passes per ISO 27729:2024 §4 + Annex A. A wrong check character is `INVALID`, never a guess; shapes outside the hyphenated/compact/URI spellings are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Hyphenated 4-4-4-4 (`0000-0002-1825-0097`; lowercase `x` folds to `X`) | Space-grouped (`0000 0002 1825 0097`) → `MISSING` | +| Compact bare 16 characters (`0000000218250097`, incl. re-entry of `compact` output) | Truncated or overlong runs (`0000-0002-1825-009`, `...-00977`) → `MISSING` | +| URI forms (`https://orcid.org/...`, `http://...`, `orcid.org/...`, `www.orcid.org/...`) | Glued label (`ORCID0000-0002-1825-0097`, no separator) → `MISSING` | +| Fused `ORCID`/`ISNI` label with separator (`ORCID: 0000-0002-1825-0097`, `orcid - ...`) | Digit-glued runs (`X0000-0002-1825-0097`, `...0097Y`) → `MISSING` | +| Embedded in prose, quoted, or bracketed (`see ... for author`, `"..."`, `[...]`) | Fullwidth digits → `MISSING` (payload is ASCII-only) | +| | Prose without an ORCID shape (`not an orcid`) → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"orcid"` (identity — `normalize()` returns hyphenated form). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `orcid` / `None` / `"default"` | Hyphenated `XXXX-XXXX-XXXX-XXXC` | `0000-0002-1825-0097` | +| `compact` | Bare 16 characters, no hyphens (encoding) | `0000000218250097` | +| `uri` | Canonical `https://` URI (encoding — `http://` input still renders `https://`) | `https://orcid.org/0000-0002-1825-0097` | + +Any other value raises `ContractError` — including `isni` (the grammar accepts an `ISNI:` label, but `isni` is not an output format). + +```python +from paxman.capabilities import ORCID +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("0000000218250097", ORCID.create_contract()).canonicalized_value) +print(paxman.canonicalize("https://orcid.org/0000-0002-1825-0097", ORCID.create_contract()).canonicalized_value) +print(paxman.canonicalize("ORCID: 0000-0002-1694-233x", ORCID.create_contract()).canonicalized_value) +print(paxman.canonicalize("0000-0002-1825-0097", ORCID.create_contract(output_format="compact")).canonicalized_value) +print(paxman.canonicalize("0000-0002-1825-0097", ORCID.create_contract(output_format="uri")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = ORCID.create_contract( + output_format=None, # "orcid" (default), "compact", "uri" + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- No grammar toggles: one grammar (`orcid_recognition`), two rules (`Section 4-orcid-structure`, `Section A-mod11-2-check-character`). +- Both rules validate the full structure-plus-checksum conjunction and normalize identically, so a `SUCCESS` carries dual provenance on one value. +- `year` filters by `publication_year`; e.g., `year=2020` drops both ISO 27729:2024 rules → `0000-0002-1825-0097` becomes `INVALID`. +- Single-value: two distinct ORCIDs in one call raise `MultipleMentionsError` — split first per `docs/recipes/segmentation.md`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `0000-0002-1825-0097` | defaults | `SUCCESS` | `"0000-0002-1825-0097"` | +| `0000000218250097` | defaults | `SUCCESS` | `"0000-0002-1825-0097"` (compact recognized) | +| `https://orcid.org/0000-0002-1825-0097` | defaults | `SUCCESS` | URI host absorbed, hyphenated value | +| `ORCID: 0000-0002-1825-0097` | defaults | `SUCCESS` | label absorbed into the span | +| `0000-0002-1694-233x` | defaults | `SUCCESS` | `"0000-0002-1694-233X"` (lowercase `x` folds up) | +| `0000000218250097` (re-entry of `compact` output) | defaults | `SUCCESS` | fixed point: renders `0000-0002-1825-0097` again | +| `https://orcid.org/0000-0002-1825-0097` (re-entry of `uri` output) | defaults | `SUCCESS` | fixed point: renders `0000-0002-1825-0097` again | +| `not an orcid` | any | `MISSING` | no ORCID shape | +| `0000 0002 1825 0097` | any | `MISSING` | spaces are not separators | +| `ORCID0000-0002-1825-0097` | any | `MISSING` | glued label needs a separator | +| `0000-0002-1825-0098` | any | `INVALID` | structure holds, MOD 11-2 check fails | +| `0000-0002-1825-0097` | `year=2020` | `INVALID` | rules are 2024, dropped | +| `0000-0002-1825-0097` | `output_format="isni"` | raises `ContractError` | not an offered format | +| Two distinct ORCIDs in one call | any | raises `MultipleMentionsError` | split first | + +Both rules normalize identically, so the two candidates always agree on one value and `AMBIGUOUS` is unreachable for this capability. + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
orcid_recognition] + G --> R{Section 4 structure
+ Annex A MOD 11-2} + R -->|structure and check hold| OK[SUCCESS
hyphenated, compact, or uri] + R -->|no rule validates| INV[INVALID] + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import ORCID +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = ORCID.create_contract() + +rows = [ + "0000-0002-1825-0097", + "0000000218250097", + "https://orcid.org/0000-0002-1825-0097", + "ORCID: 0000-0002-1694-233x", + "0000-0002-1825-0098", + "0000 0002 1825 0097", + "ORCID0000-0002-1825-0097", + "not an orcid", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:44} → {r.status.value:10} {val!r:42} ({rule})") + +# compact re-entry fixed point (v0.4.0): compact output re-enters to the hyphenated form +compact_out = paxman.canonicalize("0000-0002-1825-0097", ORCID.create_contract(output_format="compact")).canonicalized_value +back = paxman.canonicalize(compact_out, contract) +print(f"{compact_out!r} re-enters → {back.status.value} {back.canonicalized_value!r}") + +try: + ORCID.create_contract(output_format="isni") +except ContractError as e: + print(f"isni → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO 27729:2024** §4 ISNI/ORCID structure (16 characters: 15 digits + MOD 11-2 check character, with Annex A check) — `Section 4-orcid-structure` +- **ISO 27729:2024** Annex A MOD 11-2 check character over the first 15 decimal digits (`X` = 10, with §4 structure) — `Section A-mod11-2-check-character` + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/phone.md b/docs/user/v0.6.0/capabilities/phone.md new file mode 100644 index 00000000..427fcfa8 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/phone.md @@ -0,0 +1,137 @@ +--- +title: "Phone" +slug: v0.6.0/capabilities/phone +--- + +Canonicalizes **one phone number** per call to E.164 (or to tel-URI / split form when requested). + +> **In plain language:** give it `"+1 555 123 4567"` and it hands back `"+15551234567"` if the numbering plan says the number is valid. National-shaped numbers like `"(555) 234-5678"` carry no country code, so they need `default_country="US"` (→ `"+15552345678"`); without it they are `INVALID`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| E.164 international (`+1 5551234567`) | Text that only mentions a country without digits | +| `00`-prefix international (`0044 20 ...`) | Extensions without a dialable number | +| `tel:` URI (`tel:+1-555-123-4567`) | Plain prose — `MISSING` | +| NANP / national numbers (`(555) 234-5678`) — only when a `default_country` is provided that is in the NANP | National-shaped input without `default_country` — recognized but `INVALID` | + +--- + +## Canonical output + +Default `output_format` is `"e164"`. + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `e164` / `None` / `"default"` | `+` + country code + national significant number | `+15551234567` | +| `rfc3966` | `tel:` URI | `tel:+15551234567` | +| `split` | `+` + country code + space + national significant number (uniform for every country code) | `+1 5551234567` | + +`national` (bare national significant number) was de-offered per ADR-0011 — it dropped the country code and could not re-enter under the default contract. `Phone.create_contract(output_format="national")` raises `ContractError` with a migration message naming `split`. + +```python +from paxman.capabilities import Phone +import paxman + +paxman.register_all_shipped() +paxman.canonicalize( + "+1 555 123 4567", Phone.create_contract() +).canonicalized_value # "+15551234567" +paxman.canonicalize( + "+15551234567", Phone.create_contract(output_format="rfc3966") +).canonicalized_value # "tel:+15551234567" +paxman.canonicalize( + "+15551234567", Phone.create_contract(output_format="split") +).canonicalized_value # "+1 5551234567" + +# National-shaped input needs default_country +paxman.canonicalize( + "(555) 234-5678", Phone.create_contract(default_country="US") +).canonicalized_value # "+15552345678" +``` + +--- + +## Contract + +```python +contract = Phone.create_contract( + default_country=None, # str | None — uppercase alpha-2, e.g. "US" + output_format=None, # "e164" (default), "rfc3966", "split" + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- When `default_country` is `None`, national-shaped input is recognized but never validated → `INVALID`. International, `00`-prefix, and `tel:` forms validate without it because the country code is in the number itself. +- `default_country` is input-only: it resolves national-shaped input and plays no role in output rendering. `output_format="split"` re-enters param-free under the default contract (uniform `+CC NSN` for every country code; the space is presentation-only). `rfc3966` is the only extension-preserving format. +- `default_country` must be uppercase alpha-2 when present; otherwise `ContractError` at construction. + +--- + +## Statuses + +| Input | Contract | Status | Why | +|-------|----------|--------|-----| +| `+1 555 123 4567` | any | `SUCCESS` | → `+15551234567` | +| `(555) 234-5678` | defaults (`default_country=None`) | `INVALID` | recognized but needs a default country to validate | +| `(555) 234-5678` | `default_country="US"` | `SUCCESS` | → `+15552345678` | +| `hello` | any | `MISSING` | no phone pattern | +| Two distinct numbers | any | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G1[E.164 grammar] + A --> G2[00-prefix grammar] + A --> G3[tel-URI grammar] + A --> G4[National grammar] + G1 & G2 & G3 --> R1[E.164 / RFC 3966 rules
embed country code] + G4 --> R2{NANP / E.164 country rules
need default_country?} + R1 --> OK[SUCCESS] + R2 -->|with default_country| OK + R2 -->|without| INV[INVALID] + G1 & G2 & G3 & G4 -->|nothing matched| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import Phone +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c_intl = Phone.create_contract() +c_us = Phone.create_contract(default_country="US") +c_rfc = Phone.create_contract(output_format="rfc3966", default_country="US") + +rows = ["+1 555 123 4567", "(555) 234-5678", "0044 20 7946 0958", "hello"] + +for text in rows: + for label, c in [("intl", c_intl), ("US", c_us), ("rfc3966+US", c_rfc)]: + try: + r = paxman.canonicalize(text, c) + except Exception as e: + print(f"{text!r:25} [{label}] → exception {type(e).__name__}") + continue + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:25} [{label:12}] → {r.status.value:10} {val!r}") +``` + +--- + +## Provenance + +- **ITU-T E.164** — international numbering plan. +- **RFC 3966** — `tel:` URI. +- **NANP** — North American Numbering Plan (when relevant). + +See also: [Execution Result](../concepts/execution-result/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/si-unit.md b/docs/user/v0.6.0/capabilities/si-unit.md new file mode 100644 index 00000000..39de75f3 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/si-unit.md @@ -0,0 +1,127 @@ +--- +title: "SI Unit" +slug: v0.6.0/capabilities/si-unit +--- + +Canonicalizes **one SI unit expression** per call — a symbol, a name, or a product/quotient compound — to its canonical symbol. + +> **In plain language:** give it `"Kilogram"`, `"megahertz"`, or `"m/s²"` and it hands back `"kg"`, `"MHz"`, or `"m/s2"` if the SI Brochure or ISO 80000 says that expression is a valid unit. Quantities like `"25°C"` are not its domain (the number is not an SI unit). + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Symbols (`kg`, `Pa`, `kΩ`, `m/s²`) | Quantities / magnitudes (`25°C`, `5 kg`) — `MISSING`; identity-only, no amounts | +| Names (`Kilogram`, `megahertz`) — case-insensitive, hyphen-tolerant | Name-compounds (`metre per second`) — words recognized separately → `AMBIGUOUS` | +| Prefixed symbols and names (`MHz`, `kPa`) | Symbol-prefix spacing (`k g`) — `INVALID`, never accepted | +| Product/quotient compounds (`m/s²`, `kg·m/s²`, `kPa`) | Multi-solidus without flag (`kg/m/s`) — `INVALID` unless `allow_multi_solidus=True` | +| Spoken word-prefix forms (`kilo gram`) — only when `allow_split_word_prefixes=True` | `kilo gram` without that flag — `INVALID` | + +--- + +## Canonical output + +Single format — the canonical symbol form. + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `symbol` / `None` / `"default"` | canonical symbol (`kg`, `MHz`, `m/s2`) | + +```python +from paxman.capabilities import SIUnit +import paxman + +paxman.register_all_shipped() +paxman.canonicalize("Kilogram", SIUnit.create_contract()).canonicalized_value # "kg" +paxman.canonicalize("megahertz", SIUnit.create_contract()).canonicalized_value # "MHz" +paxman.canonicalize("m/s²", SIUnit.create_contract()).canonicalized_value # "m/s2" +``` + +--- + +## Contract + +```python +contract = SIUnit.create_contract( + allow_split_word_prefixes=False, # bool, default False — "kilo gram" → "kg" when True + allow_multi_solidus=False, # bool, default False — preserve "kg/m/s" when True + output_format=None, # "symbol" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- With `allow_split_word_prefixes=False` (default), `kilo gram` is recognized as `INVALID` — it is not a valid SI identity. Symbol-prefix spacing like `k g` is always `INVALID`; there is no flag for it. +- With `allow_multi_solidus=False` (default) per ISO 80000-1 §6.6.2, more than one top-level solidus is `INVALID`. Set `True` to preserve the legacy accept-multi-solidus behavior. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `Kilogram` | defaults | `SUCCESS` | → `kg` | +| `m/s²` | defaults | `SUCCESS` | compound → `m/s2` | +| `metre per second` | defaults | `AMBIGUOUS` | words recognized separately, competing groupings | +| `25°C` | any | `MISSING` | quantity, not an identity-only unit | +| `kilo gram` | defaults | `INVALID` | needs `allow_split_word_prefixes` | +| `kilo gram` | `allow_split_word_prefixes=True` | `SUCCESS` | → `kg` | +| `k g` | any | `INVALID` | symbol-prefix spacing always rejected | +| `kg/m/s` | defaults | `INVALID` | multi-solidus without flag | +| `kg/m/s` | `allow_multi_solidus=True` | `SUCCESS` | preserved | + +```mermaid +flowchart TB + A[Text e.g. kilo gram] --> G1[symbol grammar] + A --> G2[name grammar] + A --> G3[compound grammar] + G1 & G2 & G3 --> R{Rules: BIPM SI Brochure 9th ed.
ISO 80000-1 compounds
split-prefix rescue} + R -->|one value| OK[SUCCESS
symbol form] + R -->|name-compound
words separate| AMB[AMBIGUOUS] + R -->|needs allow_split_word_prefixes
or allow_multi_solidus| INV[INVALID] + G1 & G2 & G3 -->|nothing| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style AMB fill:#fff8e1,stroke:#d4a017 + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import SIUnit +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +c = SIUnit.create_contract() +c_spoken = SIUnit.create_contract(allow_split_word_prefixes=True) +c_multi = SIUnit.create_contract(allow_multi_solidus=True) + +for text in [ + "Kilogram", + "m/s²", + "metre per second", + "kilo gram", + "k g", + "kg/m/s", + "25°C", +]: + for label, cc in [("default", c), ("spoken", c_spoken), ("multi", c_multi)]: + r = paxman.canonicalize(text, cc) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + print(f"{text!r:18} [{label:8}] → {r.status.value:10} {val!r}") +``` + +--- + +## Provenance + +- **BIPM SI Brochure (9th ed.)** — base units, derived units, non-SI units accepted for use, prefixes, names. +- **ISO 80000-1** — compound expressions. + +See also: [Execution Result](../concepts/execution-result/), [Candidates & Ambiguity](../concepts/candidates-and-ambiguity/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/timezone.md b/docs/user/v0.6.0/capabilities/timezone.md new file mode 100644 index 00000000..b7ebeb7b --- /dev/null +++ b/docs/user/v0.6.0/capabilities/timezone.md @@ -0,0 +1,168 @@ +--- +title: "Timezone" +slug: v0.6.0/capabilities/timezone +--- + +Canonicalizes **one IANA time zone mention** per call — a zone key (`America/New_York`), a legacy Link (`US/Eastern`, `Canada/Eastern`), a fixed zone (`UTC`, `Etc/GMT+5`, `Etc/GMT-8`), or a POSIX SystemV name (`EST5EDT`, gated) — to the case-exact IANA key. + +> **In plain language:** give it `America/New_York`, `US/Eastern`, `america/new_york`, or `Etc/GMT+5` and it hands back the canonical IANA key (`US/Eastern` → `America/New_York`) per the vendored IANA Time Zone Database 2026d snapshot. Bare abbreviations (`EST`, `IST`, `CST`) are recognized but refused — never silently resolved — because one abbreviation names many zones and no timestamp is available to disambiguate. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Canonical keys (`America/New_York`, case folds up: `america/new_york`, `AMERICA/NEW_YORK`) | Unknown keys (`America/Narnia`) → `MISSING` (lexicon over the curated subset claims nothing unlisted) | +| Legacy Links (`US/Eastern`, `Canada/Eastern`, `US/PACIFIC`, `Asia/Calcutta`, `Australia/ACT` — resolve to canonical) | Windows names (`Eastern Standard Time`, territory-sensitive CLDR mapping, deferred) → `MISSING` | +| Fixed zones (`UTC`, `GMT`, `Etc/UTC`, `Etc/GMT`, `Etc/GMT+5`, `Etc/GMT-8` — sign kept as authored, never reinterpreted) | Bare abbreviations, carved (`EST`, `MST`, `HST`, `CET`) → `INVALID` (refused, never resolved) | +| POSIX SystemV names (`EST5EDT`, `CST6CDT`, `MST7MDT`, `PST8PDT` — claimed for shape; `INVALID` unless `include_systemv=True`) | Bare abbreviations, ambiguous (`IST`, `CST`, `PST`) → `INVALID` (refused, never a silent pick) | +| Embedded mentions (`visit US/Eastern tomorrow` — span covers the key only) | Unlisted abbreviations and prose (`XYZ`, `JST`, `hello world`, `Zulu`) → `MISSING` | +| | Glued runs and paths (`XUS/Eastern`, `US/EasternX`, `/usr/share/zoneinfo/America/New_York`) → `MISSING` (slash-aware boundary) | + +**Abbreviation law:** identifier equivalence is not lexical abbreviation equivalence. Short-caps `backward` Links (`EST` → `America/Panama`, `MST` → `America/Phoenix`, `HST` → `Pacific/Honolulu`, `CET` → `Europe/Brussels`) are carved out of the name lexicon into the abbreviation family, so the bare token reads as an abbreviation and refuses. The seeker writes the canonical key. + +--- + +## Canonical output + +Default `output_format` is `"iana"` (identity — the canonical value IS the default format). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `iana` / `None` / `"default"` | Case-exact IANA zone key | `America/New_York` | + +Any other value raises `ContractError` — including `abbreviation` and `link`, never offered per ADR-0011 (rendering an abbreviation is a lossy projection; choosing which alias to render is arbitrary policy, not canonicalization). + +```python +from paxman.capabilities import Timezone +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("America/New_York", Timezone.create_contract()).canonicalized_value) +print(paxman.canonicalize("US/Eastern", Timezone.create_contract()).canonicalized_value) +print(paxman.canonicalize("america/new_york", Timezone.create_contract()).canonicalized_value) +print(paxman.canonicalize("Asia/Calcutta", Timezone.create_contract()).canonicalized_value) +print(paxman.canonicalize("EST5EDT", Timezone.create_contract(include_systemv=True)).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = Timezone.create_contract( + output_format=None, # "iana" (default); None/"default"/"iana" all resolve to "iana" + include_systemv=False, # True admits the POSIX SystemV Zones (EST5EDT and kin) + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- Two grammars: `timezone_name_recognition` (case-folded lexicon over identifier keys, Links, fixed zones, SystemV names) and `timezone_abbreviation_recognition` (UPPER-exact lexicon over carved + ambiguous abbreviations, `WORD` guards). +- Four rules: `Section zone-key-membership`, `Section link-resolution`, `Section systemv-zones` (gated on `include_systemv` via `requires_features`; dropped rule → `INVALID`), `Section abbreviation-refusal` (always refuses). +- `year` filters by `publication_year`; e.g., `year=2020` drops the tzdb-2026 rules → `America/New_York` becomes `INVALID`. +- `suppress_common_words` is a no-op here: no Timezone matcher is suppressible. +- Deterministic by construction: same input + contract + vendored 2026d snapshot → same output. No clock, no DST-transition evaluation, no environment tzdata — membership and Link resolution only. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `America/New_York` | defaults | `SUCCESS` | `"America/New_York"` | +| `america/new_york` | defaults | `SUCCESS` | `"America/New_York"` (fold restores canonical case) | +| `US/Eastern` | defaults | `SUCCESS` | `"America/New_York"` (Link resolution) | +| `Canada/Eastern` | defaults | `SUCCESS` | `"America/Toronto"` (Link resolution, Wave-2) | +| `US/PACIFIC` | defaults | `SUCCESS` | `"America/Los_Angeles"` (Link resolution, Wave-2) | +| `Asia/Calcutta` | defaults | `SUCCESS` | `"Asia/Kolkata"` (Link resolution) | +| `UTC` / `GMT` | defaults | `SUCCESS` | `"UTC"` / `"GMT"` (fixed zones; bare `GMT` admitted per #161) | +| `Etc/GMT+5` | defaults | `SUCCESS` | `"Etc/GMT+5"` (fixed zone; POSIX sign kept as authored, never flipped) | +| `Etc/GMT-8` | defaults | `SUCCESS` | `"Etc/GMT-8"` (fixed zone; POSIX sign kept as authored, Wave-2) | +| `EST5EDT` | defaults | `INVALID` | claimed for shape, rule gated off | +| `EST5EDT` | `include_systemv=True` | `SUCCESS` | `"EST5EDT"` (fixed-rule zone, valid key) | +| `EST` / `MST` / `HST` / `CET` | any | `INVALID` | carved Links read as abbreviations, refused | +| `IST` / `CST` / `PST` | any | `INVALID` | ambiguous abbreviations, refused — never a silent pick | +| `arrive CET tomorrow` | defaults | `INVALID` | bare abbreviation refused everywhere | +| `XYZ` / `JST` / `hello world` / `Zulu` | any | `MISSING` | nothing claimed (JST out of the v1 curated subset by design) | +| `America/Narnia` | any | `MISSING` | unlisted key is unclaimable under the lexicon-over-subset design | +| `Eastern Standard Time` | any | `MISSING` | Windows namespace deferred entirely | +| `XUS/Eastern` / zoneinfo paths | any | `MISSING` | slash-aware boundary forbids mid-path extraction | +| `US/Eastern then America/Chicago` | any | raises `MultipleMentionsError` | two distinct zones fail fast (`single_value=True`) | +| `EST then CET` | defaults | `INVALID` | refused mentions do not compete, so no `MultipleMentionsError` | +| `America/New_York` | `year=2020` | `INVALID` | rules are 2026, dropped | +| `America/New_York` | `output_format="abbreviation"` | raises `ContractError` | lossy projection, never offered | + +```mermaid +flowchart TB + A[Text] --> G1[Grammar:
timezone_name_recognition] + A --> G2[Grammar:
timezone_abbreviation_recognition] + G1 --> R1{Section zone-key
membership} + R1 -->|member| OK[SUCCESS] + R1 -->|Link| R2{Section link
resolution} + R2 -->|alias| OK + R1 -->|SystemV + flag| R3{Section systemv
zones} + R3 -->|gated on| OK + R1 -->|no rule validates| INV[INVALID] + G2 --> REF{Section abbreviation
refusal} + REF -->|always refuses| INV + G1 & G2 -->|nothing claimed| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import Timezone +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = Timezone.create_contract() + +rows = [ + "America/New_York", + "US/Eastern", + "Canada/Eastern", + "US/PACIFIC", + "america/new_york", + "Asia/Calcutta", + "Etc/GMT+5", + "Etc/GMT-8", + "EST5EDT", + "EST", + "IST", + "XYZ", + "America/Narnia", + "Eastern Standard Time", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:28} → {r.status.value:10} {val!r:24} ({rule})") + +print(paxman.canonicalize("EST5EDT", Timezone.create_contract(include_systemv=True)).canonicalized_value) +try: + Timezone.create_contract(output_format="abbreviation") +except ContractError as e: + print(f"abbreviation → ContractError: {e}") +``` + +--- + +## Provenance + +- **IANA Time Zone Database** 2026d (registry; zone-key membership, Link targets, SystemV Zones) — `Section zone-key-membership` (zone1970.tab col 3 + backward Links + etcetera, vendored File-Date), `Section link-resolution` (backward Link TARGET LINK-NAME lines, vendored File-Date 2026d), `Section systemv-zones` (backward SystemV Zones EST5EDT/CST6CDT/MST7MDT/PST8PDT, vendored File-Date 2026d) +- **IANA Time Zone Database** 2026d via theory.html (registry; abbreviation ambiguity plus backward short-caps carve set) — `Section abbreviation-refusal` (theory.html abbreviation ambiguity: IST/CST/PST + backward short-caps Links carve set, vendored File-Date 2026d) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [UtcOffset](./utcoffset/), [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/unspsc.md b/docs/user/v0.6.0/capabilities/unspsc.md new file mode 100644 index 00000000..20a3f320 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/unspsc.md @@ -0,0 +1,143 @@ +--- +title: "UNSPSC" +slug: v0.6.0/capabilities/unspsc +--- + +Canonicalizes **one UNSPSC mention** per call — a bare 8-digit code (`44103103`), a zero-padded parent (`43000000`, `43210000`, `43211500`), a 6-digit class alias (`441217`), a `UNSPSC`-labelled mention, a `UNSPSC000.` MDM system ID, or a 10-digit code with a business-function suffix — to the 8-digit wire stem. + +> **In plain language:** give it `UNSPSC 44103103`, `44103103`, `UNSPSC000.44103103`, or `441217` and it hands back `44103103` / `44121700` when the stem is a live row of the pinned UNGM-export snapshot. A well-formed but unissued stem is `INVALID`, never a guess; shapes outside the recognized spellings are `MISSING`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Bare 8-digit commodity (`44103103` printer/facsimile toner, `43211503` notebook computers, `10101501` cats, `25101703` ambulances) | 7/9/11-digit runs (`4410310`, `441031031`) → `MISSING` (length guard) | +| Zero-padded parents (`43000000` segment, `43210000` family, `43211500` class — distinct codes, not prefixes) | Truncated prose prefixes (`43`, `4321`) → `MISSING` (real codes are `43000000`, `43210000`) | +| 6-digit class alias (`441217` → `44121700`, SAP/California lane) | Dotted/hyphenated grouping (`43.21.15.03`, `43-21-15-03`, `44103103-14`) → `MISSING` (digits-only wire) | +| `UNSPSC` label (`UNSPSC 44103103`, `UNSPSC: 44103103`, `UNSPSC #43211507`, lowercase, `Code` word) | Space-grouped pairs (`43 21 15 03`, prose display only) → `MISSING` | +| MDM system IDs (`UNSPSC000.44103103` — Stibo STEP lane) | Float tail as code (`44103103.0` matches `44103103`; `.0` stays outside the span) | +| 10-digit + business-function suffix (`4410310314` — suffix informative only) | Bare suffixes (`14`, `retail` without the 8-digit stem) → `MISSING` | +| CSV lanes (`unspsc,43211509` — comma is a field separator) | Glued surroundings (`X44103103`, `44103103Y`) → `MISSING` (word boundary) | +| Embedded mentions (`Printers (44103103) approved`) | Prose without an 8-digit shape → `MISSING` | + +--- + +## Canonical output + +Default `output_format` is `"unspsc"` (8-digit zero-padded wire stem). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `unspsc` / `None` / `"default"` | 8-digit stem | `44103103` | +| `labeled` | `UNSPSC` prefix | `UNSPSC 44103103` | +| `native` | Spelling-preserving (6-digit stays 6-digit, 10-digit keeps its suffix) | `441217` / `4410310314` | + +Any other value raises `ContractError`. Every offered format re-enters under the default contract (ADR-0010 fixed point). The pair-hyphenated display (`43-21-15-03`) is deliberately unoffered — hyphens never re-enter by design. + +```python +from paxman.capabilities import UNSPSC +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("UNSPSC 44103103", UNSPSC.create_contract()).canonicalized_value) +print(paxman.canonicalize("441217", UNSPSC.create_contract()).canonicalized_value) +print(paxman.canonicalize("UNSPSC000.44103103", UNSPSC.create_contract()).canonicalized_value) +print(paxman.canonicalize("44103103", UNSPSC.create_contract(output_format="labeled")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = UNSPSC.create_contract( + output_format=None, # "unspsc" (default); None/"default"/"unspsc" all resolve to it + include_business_function=True, # gate the BFI-suffix rule (dropped rule removes the suffix candidate, not the verdict) + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- One grammar: `unspsc_recognition` (6/8/10 lanes + fused label, longest-first). +- Four rules: `Section 1-hierarchy-structure` + `Section 2-level-padding` (UNGM structure; both PARSER, always-active), `Section 3-codeset-membership` (UNDP codeset snapshot; LOOKUP_TABLE, always-active), `Section 4-business-function-suffix` (UNECE guidelines; PARSER, gated by `include_business_function`). +- Deterministic by snapshot: same input + contract + library snapshot → same output. The snapshot pins the UNGM live export of 2026-09-30 (13,482 live stems); a code moved or inactivated after that export resolves `INVALID` under the pinned snapshot, not silently `SUCCESS`. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `44103103` / `UNSPSC 44103103` / `UNSPSC000.44103103` | defaults | `SUCCESS` | `"44103103"` (presentation-only dedup) | +| `441217` | defaults | `SUCCESS` | `"44121700"` (alias pads `+"00"`) | +| `4410310314` | defaults | `SUCCESS` | `"44103103"` (suffix trace-only) | +| `43000000` / `43210000` / `43211500` | defaults | `SUCCESS` | parents are codes in their own right | +| `43001503` / `00101501` | defaults | `INVALID` | mid-zero padding breaks the lattice | +| `44103199` / `99999999` | defaults | `INVALID` | well-formed but unissued | +| `11101803` (Wikidata-cited platinum) | defaults | `INVALID` | absent from the pinned snapshot by design — not a claim about the full UNDP codeset | +| `57110000` (synthesized ancestor) | defaults | `INVALID` | serves the ancestor walk only, never stem membership | +| `4410310314` | `include_business_function=False` | `SUCCESS` | `"44103103"` (suffix candidate dropped; stem still validates — ISBN range-validation precedent) | +| `43 21 15 03` / `43.21.15.03` / `43` | defaults | `MISSING` | no grammar claims these shapes | +| two distinct codes | defaults | raises `MultipleMentionsError` | un-segmented multi-entity input (`single_value=True`) | +| `44103103` | `output_format="segmented"` | raises `ContractError` | never offered | + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
unspsc_recognition] + G --> R{Sections 1+2
shape + padding} + R -->|well-formed| M{Section 3
membership + ancestors} + M -->|live stem| OK[SUCCESS] + M -->|unissued| INV[INVALID] + R -->|mid-zero| INV + G -->|nothing claimed| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +**Sibling note:** an 8-digit string may also match the GTIN-8 shape — each side resolves under its own contract (UNSPSC `SUCCESS` is codeset membership, GTIN `SUCCESS` is a Mod-10 check digit). There is no cross-capability arbitration; route by authority. + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import UNSPSC +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = UNSPSC.create_contract() + +rows = [ + "UNSPSC 44103103", + "44103103", + "UNSPSC000.44103103", + "441217", + "4410310314", + "43000000", + "43 21 15 03", + "44103199", + "not a code", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:28} → {r.status.value:10} {val!r:14} ({rule})") +``` + +--- + +## Provenance + +- **United Nations Development Programme, UNSPSC Code Structure (UNGM Help Center)** (specification, 2025-07-08; four-level positional definition) — `Section 1-hierarchy-structure`, `Section 2-level-padding` +- **United Nations Development Programme, UNSPSC Codeset (UNGM live export 2026-09-30)** (registry, 13,482 live stems; partial — UNDP v26.0801 XLSX gated) — `Section 3-codeset-membership` +- **United Nations Economic Commission for Europe, Classification Guidelines (Business Function Identifiers)** (specification, v2.04; suffix informative-only, no value table cited) — `Section 4-business-function-suffix` + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [GTIN](./gtin/), [ISBN](./isbn/), [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/). diff --git a/docs/user/v0.6.0/capabilities/url.md b/docs/user/v0.6.0/capabilities/url.md new file mode 100644 index 00000000..06123ec2 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/url.md @@ -0,0 +1,124 @@ +--- +title: "URL" +slug: v0.6.0/capabilities/url +--- + +Canonicalizes **one absolute URI / IRI** per call per the **WHATWG URL Standard** (plus UTS #46 IDNA for internationalized hosts), preserving percent-encoding byte-for-byte — except encoded dot segments (`%2e`, `%2e%2e`), which resolve like `.`/`..`. + +> **In plain language:** give it `"HTTPS://Example.COM:443/path/../other"` and it hands back `"https://example.com/other"` — scheme and host lowercased, default port removed, dot segments resolved. For opaque schemes like `mailto:`, the scheme is lowercased (`MAILTO:USER@Example.COM` → `mailto:USER@Example.COM`), tab/LF/CR are stripped before parsing, and non-ASCII is UTF-8 percent-encoded (`mailto:u@exämple.com` → `mailto:u@ex%C3%A4mple.com`). C0 controls and DEL are not preserved: recognition stops before them, so `mailto:a` + C0 + `b@c` canonicalizes to just `mailto:a`. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Absolute URIs/IRIs with a scheme (`https://example.com`, `mailto:user@example.com`) | Relative references (`/path/../other`, `?q=1`, `#frag`) — `MISSING` | +| Internationalized hosts (`http://münchen.de`) — IDNA via UTS #46 | Plain domains without a scheme — not an *absolute URI* | +| Any absolute URI including opaque (non-special) schemes | | + +--- + +## Canonical output + +Single format — the WHATWG URL serialization (identity formatter). Characteristic normalizations: + +- Scheme and host lowercased. +- Default port removed (`https://example.com:443` → `https://example.com`). +- Dot segments resolved (`/path/../other` → `/other`). +- Internationalized hosts mapped via UTS #46 (`münchen.de` → `xn--mnchen-3ya.de`). +- Percent-encoding preserved byte-for-byte, except encoded dot segments (`%2e`/`%2E` → `.`, `%2e%2e` → `..`), which are removed during dot-segment resolution (`/a/%2e/b` → `/a/b`). +- Opaque schemes (`mailto:`, `data:`, etc.): scheme lowercased, tab/LF/CR stripped before parsing, printable ASCII preserved byte-for-byte, non-ASCII UTF-8 percent-encoded (`ä` → `%C3%A4`). C0 controls and DEL cut recognition short (`data:text/plain,` + C0 + `A` → `data:text/plain,`) or yield `MISSING`. + +| `output_format` | Renders | +|-----------------|---------| +| *(only)* `url` / `None` / `"default"` | WHATWG serialization | + +```python +from paxman.capabilities import URL +import paxman + +paxman.register_all_shipped() +paxman.canonicalize( + "HTTPS://Example.COM:443/path/../other", URL.create_contract() +).canonicalized_value # "https://example.com/other" +paxman.canonicalize( + "mailto:user@example.com", URL.create_contract() +).canonicalized_value # "mailto:user@example.com" (opaque → verbatim) +paxman.canonicalize( + "http://münchen.de", URL.create_contract() +).canonicalized_value # "http://xn--mnchen-3ya.de/" +``` + +--- + +## Contract + +```python +contract = URL.create_contract( + output_format=None, # "url" (only format) + # plus every common field: excluded_rules / pinned_rules / year / extra_grammars +) +``` + +URL has no capability-specific flags in the current release — every recognized URL is validated by the single WHATWG URL Standard rule. + +--- + +## Statuses + +| Input | Status | Why | +|-------|--------|-----| +| `https://example.com/other` | `SUCCESS` | WHATWG-serialized | +| `HTTPS://Example.COM:443/path/../other` | `SUCCESS` | → `https://example.com/other` (lowercased, port removed, dot resolved) | +| `http://münchen.de` | `SUCCESS` | IDN → `http://xn--mnchen-3ya.de/` | +| `not a url` | `MISSING` | no absolute-URI pattern | +| `//example.com/path` (relative) | `MISSING` | no scheme, not absolute | +| Two distinct URLs in one call | raises `MultipleMentionsError` | split first | + +```mermaid +flowchart TB + A[Text] --> G[absolute-uri grammar] + G --> R{WHATWG URL Standard rule} + R -->|absolute URI| OK[SUCCESS
WHATWG serialization] + G -->|no absolute pattern| MISS[MISSING] + A -->|two distinct URLs| ERR[MultipleMentionsError
split first] + + style OK fill:#e6ffed,stroke:#2d8a4e + style MISS fill:#fff5f5,stroke:#cc3333 + style ERR fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet + +```python +import paxman +from paxman.capabilities import URL + +paxman.register_all_shipped() +contract = URL.create_contract() + +rows = [ + "HTTPS://Example.COM:443/path/../other", + "mailto:user@example.com", + "http://münchen.de", + "not a url", + "/relative/path", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value or "—" + print(f"{text!r:45} → {r.status.value:10} {val!r} span={r.span}") +``` + +--- + +## Provenance + +- **WHATWG URL Standard** — absolute-URI / IRI validation and serialization. +- **UTS #46 / IDNA** — mapping of internationalized hosts. + +See also: [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/utcoffset.md b/docs/user/v0.6.0/capabilities/utcoffset.md new file mode 100644 index 00000000..0009e31a --- /dev/null +++ b/docs/user/v0.6.0/capabilities/utcoffset.md @@ -0,0 +1,154 @@ +--- +title: "UtcOffset" +slug: v0.6.0/capabilities/utcoffset +--- + +Canonicalizes **one UTC offset mention** per call — a human-notation numeric offset (`UTC+5`, `+0530`, `+05:30`, `Z`) — to canonical extended `+HH:MM`. + +> **In plain language:** give it `UTC+5`, `GMT-05:30`, `+0530`, or `Z` and it hands back `+05:00`, `-05:30`, `+05:30`, or `+00:00` per ISO 8601-1:2019 offset representations (RFC 3339 timestamp-compatible subset for `±HH:MM` and `Z`). The range is the real-world total-minutes bound `-12:00..+14:00` (Baker Island to Kiritimati); `-00:00` is refused because it states ignorance of the offset, not zero. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Prefixed offsets (`UTC+5`, `utc+5`, `GMT+5`, `GMT-05:30`, `UTC+05:30` — human semantics, never POSIX sign-flipped) | Out-of-range prefixed shapes (`UTC+15:00`) → `MISSING` (grammar `\|HH\| ≤ 14` guard never claims them) | +| Bare offsets (`+05`, `+0530`, `+05:30`, `-08:00` — reduced, basic, extended forms) | Bare `+15:00` → `MISSING` (same grammar guard) | +| `Z` / `z` designator (→ `+00:00`) | Claimed-but-impossible totals (`+14:30`, `-12:30` — inside the shape guard, past the signed total-minutes bound) → `INVALID` | +| Embedded mentions (`meeting at +05:30 tomorrow` — span covers the offset only) | Unknown offset (`-00:00`, any accepted form) → `INVALID` (ignorance, not zero) | +| | Zone keys (`Etc/GMT+5` — slash-aware lookbehind forbids mid-key extraction; belongs to Timezone) → `MISSING` | +| | Military prose (`Zulu`) → `MISSING` (not the `Z` designator) | + +**POSIX-sign callout:** `GMT+5` renders `+05:00` — documented human notation. POSIX would read `GMT+5` as UTC−05:00; the grammar is explicitly human-semantics and never sign-flips. Whole-hour `Etc/GMT±X` keys stay in Timezone with their sign kept as authored. + +--- + +## Canonical output + +Default `output_format` is `"extended"` (identity — `normalize()` returns `+HH:MM`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `extended` / `None` / `"default"` | Signed zero-padded `+HH:MM` | `+05:30` | +| `basic` | Presentation-only re-encoding, colon stripped (re-enters: parses back to the same extended canonical) | `+0530` | + +Any other value raises `ContractError` — including `abbreviation`, removed-by-design outputs that would project away the numeric value. + +```python +from paxman.capabilities import UtcOffset +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("UTC+5", UtcOffset.create_contract()).canonicalized_value) +print(paxman.canonicalize("+0530", UtcOffset.create_contract()).canonicalized_value) +print(paxman.canonicalize("Z", UtcOffset.create_contract()).canonicalized_value) +print(paxman.canonicalize("GMT-05:30", UtcOffset.create_contract()).canonicalized_value) +print(paxman.canonicalize("+05:30", UtcOffset.create_contract(output_format="basic")).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = UtcOffset.create_contract( + output_format=None, # "extended" (default), "basic" (presentation-only re-encoding) + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- One grammar (`utc_offset_recognition`: human-notation regex with slash-aware lookbehind and right word guard), one rule (`Section offset-structure`). +- `year` filters by `publication_year`; e.g., `year=2010` drops the ISO 8601-1:2019 rule → `UTC+5` becomes `INVALID` (`year=2020` keeps it — the rule is 2019). +- `suppress_common_words` is a no-op here: the offset matcher is not suppressible. +- Deterministic by construction: same input + contract + library snapshot → same output. No clock, no zone lookup — shape plus total-minutes arithmetic only. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `UTC+5` | defaults | `SUCCESS` | `"+05:00"` (zero-pad, human semantics) | +| `GMT-05:30` | defaults | `SUCCESS` | `"-05:30"` | +| `+0530` | defaults | `SUCCESS` | `"+05:30"` (basic form recognized) | +| `+05:30` | defaults | `SUCCESS` | `"+05:30"` | +| `Z` / `z` | defaults | `SUCCESS` | `"+00:00"` (zero designator) | +| `+14:00` / `-12:00` | defaults | `SUCCESS` | range endpoints hold (Kiritimati / Baker Island) | +| `+0530` | `output_format="basic"` | `SUCCESS` | `"+0530"` (re-encoding; re-enters to the same extended canonical) | +| `-00:00` | any | `INVALID` | unknown-offset states ignorance — a different entity from zero | +| `+14:30` / `-12:30` | any | `INVALID` | claimed shape past the signed total-minutes bound | +| `+15:00` / `UTC+15:00` | any | `MISSING` | grammar range guard never claims them | +| `+05:60` / `+05:6` / `UTC+5:60` | any | `MISSING` | malformed minutes never claimed — right edge forbids `:`/`+`/`-` continuations, so no truncated `+05:00` `SUCCESS` (#162) | +| `Etc/GMT+5` | any | `MISSING` | slash-glued guard rejects mid-key extraction | +| `Zulu` | any | `MISSING` | military prose, not the designator | +| `+05:30 and -08:00` | any | raises `MultipleMentionsError` | two distinct offsets fail fast (`single_value=True`) | +| `UTC+5` | `year=2010` | `INVALID` | rule is 2019, dropped | +| `UTC+5` | `output_format="abbreviation"` | raises `ContractError` | only `basic` is offered | + +A single grammar plus a single deterministic rule yields at most one value per mention, so `AMBIGUOUS` is unreachable for this capability (two *distinct* mentions raise instead). + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
utc_offset_recognition] + G --> R{Section offset
structure + range} + R -->|shape + total in -12:00..+14:00| OK[SUCCESS] + R -->|claimed but refused| INV[INVALID] + G -->|no shape| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import UtcOffset +from paxman.core.domain import Resolution +from paxman.core.errors import ContractError + +paxman.register_all_shipped() +contract = UtcOffset.create_contract() + +rows = [ + "UTC+5", + "GMT-05:30", + "+0530", + "+05:30", + "Z", + "+14:00", + "-12:00", + "-00:00", + "+14:30", + "+15:00", + "UTC+15:00", + "Etc/GMT+5", + "Zulu", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:28} → {r.status.value:10} {val!r:24} ({rule})") + +print(paxman.canonicalize("+05:30", UtcOffset.create_contract(output_format="basic")).canonicalized_value) +try: + UtcOffset.create_contract(output_format="abbreviation") +except ContractError as e: + print(f"abbreviation → ContractError: {e}") +``` + +--- + +## Provenance + +- **ISO 8601-1** 2019 (specification; offset representations and real-world range, normalized to `+HH:MM`) — `Section offset-structure` (ISO 8601-1:2019 Section 3 offset representations; RFC 3339 Section 5.6 time-numoffset +-HH:MM, Z) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [Timezone](./timezone/), [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/), [Segmentation](https://github.com/nexusnv/paxman-python/blob/main/docs/recipes/segmentation.md). diff --git a/docs/user/v0.6.0/capabilities/uuid.md b/docs/user/v0.6.0/capabilities/uuid.md new file mode 100644 index 00000000..61b738a5 --- /dev/null +++ b/docs/user/v0.6.0/capabilities/uuid.md @@ -0,0 +1,138 @@ +--- +title: "UUID" +slug: v0.6.0/capabilities/uuid +--- + +Canonicalizes **one UUID mention** per call — hyphenated (`6ba7b810-9dad-11d1-80b4-00c04fd430c8`), bare 32-hex, `{braced}`, or `urn:uuid:`-prefixed — to the lowercase hyphenated form. + +> **In plain language:** give it any documented spelling of a 128-bit UUID and it hands back the canonical lowercase `8-4-4-4-12` form per IETF RFC 9562. There is no check digit and no registry: structure (length + hex charset) is the whole validation. Nil (`000…0`) and Max (`fff…f`) sentinels are valid values. + +--- + +## What it recognizes — and what it does not + +| Recognizes | Does not recognize | +|------------|--------------------| +| Hyphenated, any case (`6ba7b810-…`, `6BA7B810-…` — folds to lowercase) | Wrong length (31/33-hex, 37-char dashed) → `MISSING` (length guard) | +| Bare 32-hex (`6ba7b8109dad…`, DB/API dumps) | Non-hex letters (`…g…`) → `MISSING` (ASCII hex only, no autocorrection) | +| Braced (`{6ba7b810-…}`, Python/Microsoft output) | Integer dumps (`32980073…`) → `MISSING` (Money/Phone territory) | +| URN-prefixed (`urn:uuid:6ba7b810-…`, any case) | Label-prefixed prose (`UUID: …`, `GUID: …`) → `MISSING` (deferred to `extra_grammars`) | +| Nil / Max sentinels | Glued runs (`x6ba7b810-…`) → `MISSING` (word boundary) | +| Embedded mentions (`request 6ba7b810-… done` — span covers the id, carrier included) | Space/dot-grouped spellings → `MISSING` (unattested) | + +**Version/variant law:** the version nibble and variant bits are informative, never gates — RFC 9562 constrains minting (§5) separately from storage (§4). A `0` or `9`–`F` version digit still reads `SUCCESS`. + +--- + +## Canonical output + +Default `output_format` is `"hyphenated"` (lowercase `8-4-4-4-12`). + +| `output_format` | Renders | Example | +|-----------------|---------|---------| +| *(default)* `hyphenated` / `None` / `"default"` | Lowercase hyphenated | `6ba7b810-9dad-11d1-80b4-00c04fd430c8` | +| `compact` | 32-char lowercase hex | `6ba7b8109dad11d180b400c04fd430c8` | +| `braced` | Hyphenated in `{}` | `{6ba7b810-9dad-11d1-80b4-00c04fd430c8}` | +| `urn` | RFC 9562 URN | `urn:uuid:6ba7b810-9dad-11d1-80b4-00c04fd430c8` | + +Any other value raises `ContractError`. Every offered format re-enters under the default contract (ADR-0010 fixed point). + +```python +from paxman.capabilities import UUID +import paxman + +paxman.register_all_shipped() +print(paxman.canonicalize("6BA7B810-9DAD-11D1-80B4-00C04FD430C8", UUID.create_contract()).canonicalized_value) +print(paxman.canonicalize("{6ba7b810-9dad-11d1-80b4-00c04fd430c8}", UUID.create_contract()).canonicalized_value) +print(paxman.canonicalize("urn:uuid:6ba7b810-9dad-11d1-80b4-00c04fd430c8", UUID.create_contract()).canonicalized_value) +print(paxman.canonicalize("6ba7b8109dad11d180b400c04fd430c8", UUID.create_contract()).canonicalized_value) +``` + +--- + +## Contract + +```python +contract = UUID.create_contract( + output_format=None, # "hyphenated" (default); None/"default"/"hyphenated" all resolve to it + # plus every common field: suppress_common_words / excluded_rules / pinned_rules / year / extra_grammars +) +``` + +- One grammar: `uuid_recognition` (regex over hyphenated/bare/braced/URN carriers, case-folded). +- One rule: `Section 4-uuid-format` (RFC 9562 §4 structure; PARSER, always-active — no registry flag exists). +- `year` filters by `publication_year`; e.g., `year=2020` drops the 2024 rule → `6ba7b810-…` becomes `INVALID`. +- Deterministic by construction: same input + contract + vendored snapshot → same output. No clock, no registry lookup. + +--- + +## Statuses + +| Input | Contract | Status | Value / why | +|-------|----------|--------|-------------| +| `6ba7b810-9dad-11d1-80b4-00c04fd430c8` | defaults | `SUCCESS` | `"6ba7b810-9dad-11d1-80b4-00c04fd430c8"` | +| `6BA7B810-…` / mixed case | defaults | `SUCCESS` | fold restores lowercase | +| `6ba7b8109dad…` (32-hex) | defaults | `SUCCESS` | bare branch regroups | +| `{6ba7b810-…}` / `urn:uuid:6ba7b810-…` | defaults | `SUCCESS` | carriers stripped, span includes carrier | +| `00000000-0000-0000-0000-000000000000` | defaults | `SUCCESS` | Nil sentinel is a value | +| `ffffffff-ffff-ffff-ffff-ffffffffffff` | defaults | `SUCCESS` | Max sentinel is a value | +| version `0`/`9` nibble | defaults | `SUCCESS` | informative, never gates | +| 31/33-hex, bad charset, prose | defaults | `MISSING` | nothing claimed | +| two distinct ids | defaults | raises `MultipleMentionsError` | un-segmented multi-entity input (`single_value=True`) | +| `6ba7b810-…` | `year=2020` | `INVALID` | rule is 2024, dropped | +| `6ba7b810-…` | `output_format="upper"` | raises `ContractError` | never offered | + +```mermaid +flowchart TB + A[Text] --> G[Grammar:
uuid_recognition] + G --> R{Section 4
uuid-format} + R -->|32-hex structure| OK[SUCCESS] + R -->|bad length/charset| MISS[MISSING] + R -->|rule dropped by year filter| INV[INVALID] + G -->|nothing claimed| MISS[MISSING] + + style OK fill:#e6ffed,stroke:#2d8a4e + style INV fill:#fff5f5,stroke:#cc3333 + style MISS fill:#fff5f5,stroke:#cc3333 +``` + +**Sibling note:** a bare 32-hex run starting with two letters + two digits also matches the IBAN shape — each side resolves under its own contract (UUID `SUCCESS`, IBAN `INVALID` unless MOD 97 passes). The `urn:uuid:` form is URI-shaped; URL-side behavior belongs to the URL guide. + +--- + +## Notebook snippet — normalize a mixed column + +```python +import paxman +from paxman.capabilities import UUID +from paxman.core.domain import Resolution + +paxman.register_all_shipped() +contract = UUID.create_contract() + +rows = [ + "6ba7b810-9dad-11d1-80b4-00c04fd430c8", + "6BA7B810-9DAD-11D1-80B4-00C04FD430C8", + "6ba7b8109dad11d180b400c04fd430c8", + "{6ba7b810-9dad-11d1-80b4-00c04fd430c8}", + "urn:uuid:6ba7b810-9dad-11d1-80b4-00c04fd430c8", + "00000000-0000-0000-0000-000000000000", + "not a uuid", +] + +for text in rows: + r = paxman.canonicalize(text, contract) + val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" + rule = r.candidates[0].validation_rule if r.candidates else "—" + print(f"{text!r:46} → {r.status.value:10} {val!r:40} ({rule})") +``` + +--- + +## Provenance + +- **IETF RFC 9562** (specification; 128-bit format, hex-and-dash ABNF, version/variant layouts, Nil/Max sentinels) — `Section 4-uuid-format` (vendored File-Date May 2024) + +Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. + +See also: [ORCID](./orcid/), [MacAddress](./mac_address/), [Execution Result](../concepts/execution-result/), [Provenance](../concepts/provenance/). diff --git a/docs/user/v0.6.0/citations.md b/docs/user/v0.6.0/citations.md new file mode 100644 index 00000000..9e517e03 --- /dev/null +++ b/docs/user/v0.6.0/citations.md @@ -0,0 +1,365 @@ +--- +title: "Citations" +slug: v0.6.0/citations +--- + +Every validated value in Paxman carries **provenance** — the authority, specification, version, and section that vouches for it. This page is the curated, authority-grouped index of **all cited provenance** across the 30 shipped capabilities. Use it to audit results, write methods sections, or compare Paxman against another system. + +> **How to read provenance in code:** `result.candidates[n].provenance[0]` is a `Provenance` (`authority`, `specification_name`, `version`, `publication_year`, `reference_url`, `kind`, `lifecycle`) and `candidate.validation_rule` is the section citation (e.g. `Section 3.4.1-addr-spec`). See [Provenance](concepts/provenance/) for the object shape. + +```python +for c in result.candidates: + p = c.provenance[0] + print(f"{c.validation_rule} — {p.authority} {p.specification_name} ({p.version}), {p.reference_url}") +``` + +All entries below are derived from the `PUBLICATION` constants and `citation`/`name` fields on each shipped rule (`paxman/capabilities//rules/*.py`). No network lookup is involved — the data is baked into the installed library build and versioned via `version_stamp`. + +--- + +## At a glance — citations by authority + +| Authority | Specifications | Capabilities | Rules | +|-----------|---------------|--------------|-------| +| **BIPM** | SI Brochure: The International System of Units (SI), 9th ed. (2019) | SI Unit | 6 | +| **IETF** | RFC 5322, RFC 6761, RFC 791, RFC 5952, RFC 3966, BCP 47 RFC 5646, RFC 5870, RFC 7946, RFC 9562, RFC 1034, RFC 1035, RFC 5893 | Email, IP, Phone, Language, Coordinates, UUID, Domain | 12 | +| **IANA** | IANA Language Subtag Registry (Rolling File-Date 2026-08-08); IANA Time Zone Database 2026d; IANA Root Zone Database (tlds-alpha v2026092300) | Language, Timezone, Domain | 8 | +| **ISO** | ISO 9362:2022, ISO 27729:2024, ISO 3166-1:2020, ISO 3166-3, ISO 4217, ISO 8601:2019, ISO 8601-1:2019, ISO 2108:2017, ISO 13616-1:2020, ISO 639-1/2/5, ISO 80000-1:2022, ISO 6709:2022, ISO 26324:2025, ISO 6166:2021, ISO 17442-1:2020, ISO/IEC 7812-1:2017 | BIC, ORCID, ISNI, Country, Currency, Money, Date, ISBN, IBAN, Language, SI Unit, Coordinates, UtcOffset, DOI, ISIN, LEI, CreditCard | 28 | +| **ISSN International Centre** | ISO 3297:2022 | ISSN | 1 | +| **ANNA** | ANNA ISIN Guidelines V25 (Dec 2025) | ISIN | 1 | +| **GLEIF** | GLEIF LOU prefix list (accredited-LOU directory + concatenated-file census) | LEI | 1 | +| **GS1** | GS1 General Specifications 26.0, GS1 Prefix allocation, Verified by GS1 | GTIN | 3 | +| **Brand networks** | Brand IIN/length tables (secondary) | CreditCard | 1 | +| **ITU-T** | E.164 (2010) | Phone | 2 | +| **NANPA** | North American Numbering Plan (NANP) (2024) | Phone | 2 | +| **Unicode Consortium (CLDR)** | CLDR v45, Unicode CLDR v47, CLDR Language Display Names v46 | Country, Currency, Money, Language | 6 | +| **Unicode Consortium** | UTS #46 v18.0.0 (shipped table 15.1.0) | Domain | 1 | +| **SIL International (ISO 639-3 RA)** | ISO 639-3:2007 | Language | 2 | +| **International ISBN Agency** | ISBN Users' Manual (2012), ISBN Range Message (2026-08-05) | ISBN | 2 | +| **WHATWG** | URL Standard (Living Standard) | URL | 1 | +| **IEEE** | IEEE Std 802-2024 | MacAddress | 1 | +| **IUPAC** | IUPAC Periodic Table 04 May 2022, IUPAC Red Book 2005 Ch. IR-3 | ChemicalElement | 2 | +| **Minor Planet Center** | New- And Old-Style Minor Planet Designations (DesDoc); Packed Provisional and Permanent Designations (PackedDes); How Are Minor Planets Named? (HowNamed) | MinorPlanet | 5 | +| **UNDP** | UNSPSC Code Structure (UNGM Help Center); UNSPSC Codeset (UNGM live export 2026-09-30) | UNSPSC | 3 | +| **UNECE** | Classification Guidelines (Business Function Identifiers) v2.04 | UNSPSC | 1 | +| **Derived convention** | US locale — MM/DD/YYYY, European locale — DD/MM/YYYY | Date | 2 | + +> **Note on "Derived convention":** The two Date rules for `MM/DD/YYYY` and `DD/MM/YYYY` are locale conventions rather than an external standards publication. They are included here for completeness and carry `kind="convention"` with no `reference_url`. + +--- + +## BIPM — Bureau International des Poids et Mesures + +Specification: **SI Brochure: The International System of Units (SI)**, 9th edition (2019) — `https://www.bipm.org/en/publications/si-brochure` — `kind="specification"`, `lifecycle="active"`, `publication_year=2019`. + +| Capability | Rule (`validation_rule`) | Citation | Publication | +|------------|--------------------------|----------|-------------| +| SI Unit | `Section 2.3.1-base-units` | BIPM SI Brochure (9th ed., 2019), Table 1 | SI Brochure | +| SI Unit | `Section 2.3.2-derived-units` | BIPM SI Brochure (9th ed., 2019), Tables 3–4 | SI Brochure | +| SI Unit | `Section 4.1-non-si-units` | BIPM SI Brochure (9th ed., 2019), Tables 8–9 | SI Brochure | +| SI Unit | `Section 3.2-prefixes` | BIPM SI Brochure (9th ed., 2019), Table 5 and §3.2 | SI Brochure | +| SI Unit | `Section-names` | BIPM SI Brochure (9th ed., 2019), Tables 1, 3–4, 8–9 (unit names) | SI Brochure | +| SI Unit | `Section 3.2-split-word-prefixes` | BIPM SI Brochure (9th ed., 2019), §3.2 (prefixes attached to unit names) | SI Brochure | + +--- + +## IETF — Internet Engineering Task Force + +| Capability | Specification | Version | Rule | Citation | Year | Reference | +|------------|---------------|---------|------|----------|------|-----------| +| Email | RFC 5322 | 2008 | `Section 3.4.1-addr-spec` | Section 3.4.1 (addr-spec) | 2008 | https://tools.ietf.org/html/rfc5322 | +| Email | RFC 6761 | 2012 | `Section 6.3-localhost` | Section 6.3 (localhost) | 2012 | https://tools.ietf.org/html/rfc6761 | +| IP | RFC 791 | 1981 | `Section 3.2-ipv4-address` | Section 3.2 (internet addressing) — dotted-decimal per RFC 1123 §2.1 | 1981 | https://datatracker.ietf.org/doc/html/rfc791 | +| IP | RFC 5952 | 2010 | `Section 4-ipv6-text-representation` | Section 4 (IPv6 text representation) + RFC 4291 §2.2 (mixed `LS32` `::ffff:192.0.2.1`) | 2010 | https://datatracker.ietf.org/doc/html/rfc5952 | +| Phone | RFC 3966 | 2004 | `Section 3-tel-uri` | Section 3 (tel URI) / Section 3.1 (global numbers) | 2004 | https://tools.ietf.org/html/rfc3966 | +| Language | BCP 47 RFC 5646 | 2009-09 | `Section 2.1-syntax` | Section 2.1 (Language-Tag ABNF, well-formed only) | 2009 | https://www.rfc-editor.org/rfc/rfc5646.txt | +| Coordinates | RFC 5870 | 2010 | `Section 3.3-geo-uri-validity` | Section 3.3 (Geo URI validity) | 2010 | https://www.rfc-editor.org/rfc/rfc5870.txt | +| Coordinates | RFC 7946 | 2016 | `Section 3.1.1-position` | Section 3.1.1 (Position) | 2016 | https://www.rfc-editor.org/rfc/rfc7946.txt | +| UUID | RFC 9562 | 2024 | `Section 4-uuid-format` | Section 4 (128-bit format; hex-and-dash ABNF, no checksum, no registry) | 2024 | https://www.rfc-editor.org/rfc/rfc9562 | +| Domain | RFC 1034 | 1987 | `Section-3.1-name-syntax` | Section 3.1 (null label reserved for root: no empty labels) + Domain policy (≥2-label minimum; single-label scope deferred) | 1987 | https://www.rfc-editor.org/rfc/rfc1034 | +| Domain | RFC 1035 | 1987 | `Section-2.3.4-label-length` | Section 2.3.4 (63 octets/label, 253 chars/name post-encode) | 1987 | https://www.rfc-editor.org/rfc/rfc1035 | +| Domain | RFC 5893 | 2010 | `Section-2-bidi-context` | Section 2 (Bidi rule six conditions); ContextJ join-control rejection per RFC 5892 contextual rules | 2010 | https://www.rfc-editor.org/rfc/rfc5893 | + +All IETF entries are `kind="specification"`, `lifecycle="active"`. + +--- + +## IANA — Internet Assigned Numbers Authority + +Specification: **IANA Language Subtag Registry**, Rolling File-Date 2026-08-08 — `https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry` — `kind="registry"`, `lifecycle="active"`, `publication_year=2026`. + +| Capability | Rule | Citation | +|------------|------|----------| +| Language | `Section-iana-registry` | IANA Registry Type language/script/region/variant Deprecated Preferred Prefix Suppress-Script | +| Language | `Section-iana-registry-private` | IANA Registry private-use qaa-qtz/Qaaa-Qabx/QM-QZ/AA/XA-XZ/ZZ/x- | +| Language | `Section-iana-registry-description` | IANA Registry compositional description phrases (` in