Repository navigation
docs(user): user-facing documentation (index, getting-started, concepts, capabilities, api-reference, extending, migration) #33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,241 @@ | ||
| # API Reference | ||
|
|
||
| This page is the concise reference for the public Python surface you import. For the mental model behind these types, see [Concepts](concepts/index.md). | ||
|
|
||
| --- | ||
|
|
||
| ## Installation and imports | ||
|
|
||
| ```python | ||
| import paxman | ||
| from paxman.capabilities import Country, Currency, Date, Email, IP, ISBN, Money, Phone, SIUnit, URL | ||
| from paxman.core.domain import Resolution | ||
| from paxman.core.errors import ( | ||
| PaxmanError, CapabilityError, ContractError, | ||
| MultipleMentionsError, RecognitionError, ValidationError, | ||
| ) | ||
| ``` | ||
|
|
||
| Paxman has no runtime dependencies. `import paxman` is side-effect free — nothing runs until you register and call `canonicalize()`. | ||
|
|
||
| --- | ||
|
|
||
| ## Registration | ||
|
|
||
| Registration tells the engine which capabilities exist. It must complete from **one thread before the first `canonicalize()` call**; afterwards the registry freezes and reads are safe from any thread. | ||
|
|
||
| ```mermaid | ||
| flowchart LR | ||
| A[import paxman] --> B[register] | ||
| B --> C[first canonicalize<br>freezes registry] | ||
| C --> D[further canonicalize<br>thread-safe reads] | ||
| B -.->|after freeze| E[CapabilityError] | ||
|
|
||
| style C fill:#fff8e1,stroke:#d4a017 | ||
| style E fill:#fff5f5,stroke:#cc3333 | ||
| ``` | ||
|
|
||
| | Function | Signature | What it does | | ||
| |----------|-----------|--------------| | ||
| | `paxman.register_all_shipped()` | `() -> None` | Registers every capability shipped in this release. Idempotent by name. | | ||
| | `paxman.register_capability(cap)` | `(cap: Capability) -> None` | Registers one capability instance, e.g. `Email()`. Fails if the name already exists or the registry is frozen. | | ||
| | `paxman.register_grammar(name, cls)` | `(capability_name: str, grammar_cls: type[Grammar]) -> None` | Registers a community grammar (see [Extending](extending.md)). Must happen before the first call. | | ||
| | `paxman.register_rule(name, cls)` | `(capability_name: str, rule_cls: type[Rule]) -> None` | Registers a community rule. Must happen before the first call. | | ||
|
|
||
| ```python | ||
| # Quick exploration — everything | ||
| paxman.register_all_shipped() | ||
|
|
||
| # Explicit — only what you depend on | ||
| paxman.register_capability(Email()) | ||
| paxman.register_capability(Date()) | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## `paxman.canonicalize()` | ||
|
|
||
| The sole entry point. | ||
|
|
||
| ```python | ||
| def canonicalize(text: str, contract: CapabilityContract) -> ExecutionResult | ||
| ``` | ||
|
|
||
| | Parameter | Type | Description | | ||
| |-----------|------|-------------| | ||
| | `text` | `str` | Raw input. One presumed mention per call (see [Segmentation](../recipes/segmentation.md)). | | ||
| | `contract` | `CapabilityContract` | Created via `SomeCapability.create_contract(...)`. Carries `capability_name` so no separate name argument is needed. | | ||
|
|
||
| **Returns** `ExecutionResult` — always, for every domain outcome including missing or invalid input. Domain outcomes are statuses, not exceptions. | ||
|
|
||
| **Raises** | ||
|
|
||
| | Exception | Cause | | ||
| |-----------|-------| | ||
| | `CapabilityError` | No capability matches `contract.capability_name`, duplicate name, or registry frozen | | ||
| | `ContractError` | Malformed contract (unknown `pinned_rules`, unknown `output_format`, unknown semantics, missing required feature) | | ||
| | `MultipleMentionsError` | Two or more non-overlapping mentions resolved to different values — caller must split first | | ||
| | `RecognitionError` | Grammar raised or returned a malformed match (`rule`, `original_error`) | | ||
| | `ValidationError` | Rule raised inside `matches()`/`normalize()` (`rule`, `original_error`) | | ||
|
|
||
| `MISSING` / `INVALID` / `AMBIGUOUS` do **not** raise — they are `ExecutionResult.status` values (see [Execution Result](concepts/execution-result.md) and [Candidates & Ambiguity](concepts/candidates-and-ambiguity.md)). | ||
|
|
||
| --- | ||
|
|
||
| ## Contracts — `SomeCapability.create_contract()` | ||
|
|
||
| Every capability exposes a **keyword-only** factory. Common parameters come first, capability-specific ones after. | ||
|
|
||
| ```python | ||
| contract = Email.create_contract( | ||
| # common — every capability | ||
| excluded_rules=(), # Sequence[str] | ||
| pinned_rules=None, # Sequence[str] | None — when not None, only these run | ||
| year=None, # int | None — only rules with publication_year <= year | ||
| output_format=None, # str | None — None / "default" / default / offered, else ContractError | ||
| extra_grammars=(), # tuple[str, ...] — opt-in community grammars | ||
| # capability-specific — varies by capability | ||
| include_obfuscated=False, # example: Email only | ||
| ) | ||
| ``` | ||
|
|
||
| The contract is a **frozen dataclass** — construct it, pass it, inspect it, but do not mutate it. See [Contracts](concepts/contracts.md) for the full common-plus-specific table. | ||
|
|
||
| ### `output_format` policy (identical for every capability) | ||
|
|
||
| - `None`, `"default"`, and the capability's `DEFAULT_OUTPUT_FORMAT` all resolve to the default. | ||
| - Any value in `OFFERED_OUTPUT_FORMATS` resolves to itself. | ||
| - Anything else raises `ContractError` immediately. Validation never consults `output_format`. | ||
|
|
||
| Current defaults and offered alternatives: | ||
|
|
||
| | Capability | Default (`DEFAULT_OUTPUT_FORMAT`) | Offered (`OFFERED_OUTPUT_FORMATS`) | | ||
| |------------|-----------------------------------|-------------------------------------| | ||
| | Country | `alpha2` | `alpha3`, `numeric`, `name` | | ||
| | Currency | `code` | *(none — single format)* | | ||
| | Date | `ISO` | `US` | | ||
| | Email | `email` | *(none)* | | ||
| | IP | `ip` | *(none)* | | ||
| | ISBN | `isbn13` | `hyphenated` | | ||
| | Money | `code_amount` | `compact` | | ||
| | Phone | `e164` | `rfc3966`, `national` | | ||
| | SI Unit | `symbol` | *(none)* | | ||
| | URL | `url` | *(none)* | | ||
|
|
||
| > The set of capabilities — and their offered formats — grows over time. Treat this table as the current release, not a closed list. | ||
|
|
||
| ### Capability-specific flag summary (current release) | ||
|
|
||
| | Capability | Flag | Type | Default | Purpose | | ||
| |------------|------|------|---------|---------| | ||
| | Email | `include_obfuscated` | `bool` | `False` | `user at domain dot com` | | ||
| | Email | `include_localhost` | `bool` | `True` | `admin@localhost` | | ||
| | Country | `include_localized` | `bool` | `False` | CLDR multilingual names | | ||
| | Country | `include_historical` | `bool` | `False` | Deprecated names | | ||
| | Currency | `default_currency` | `str \| None` | `None` | Resolve shared bare symbol (`$`) to this alpha-3 code | | ||
| | IP | `include_ipv6` | `bool` | `True` | IPv6 | | ||
| | ISBN | `include_isbn10` | `bool` | `True` | Legacy ISBN-10 | | ||
| | ISBN | `include_range_validation` | `bool` | `False` | Registrant-range provenance | | ||
| | Money | `dollar_sign_currency` | `str \| None` | `None` | Resolve bare `$` amount to this alpha-3 code | | ||
| | Money | `precision` | `str` | `"strict"` | `strict` / `truncate` / `round` for over-precision amounts | | ||
| | Phone | `default_country` | `str \| None` | `None` | Interpret national numbers as if in this alpha-2 country | | ||
| | SI Unit | `allow_split_word_prefixes` | `bool` | `False` | `kilo gram` → `kg` | | ||
| | SI Unit | `allow_multi_solidus` | `bool` | `False` | `kg/m/s` preserved | | ||
| | Date | `two_digit_base_year` | `int \| None` | `None` | Base for 2-digit year expansion | | ||
|
|
||
| Validation: `default_currency` / `dollar_sign_currency` must be uppercase alpha-3; `default_country` must be uppercase alpha-2; `precision` must be one of the three values — otherwise `ContractError` at construction time. | ||
|
|
||
| --- | ||
|
|
||
| ## Execution result | ||
|
|
||
| ```python | ||
| @dataclass(frozen=True) | ||
| class ExecutionResult: | ||
| status: Resolution # MISSING | INVALID | SUCCESS | AMBIGUOUS | ||
| canonicalized_value: str | None # str on SUCCESS, None otherwise | ||
| candidates: tuple[Candidate, ...] # all validated evidence, deduplicated | ||
| contract: CapabilityContract # the contract you passed in | ||
| version_stamp: VersionStamp # .paxman_version | ||
| span: tuple[int, int] | None # [start, end) of resolved value on SUCCESS, else None | ||
| ``` | ||
|
|
||
| ```python | ||
| @dataclass(frozen=True) | ||
| class Candidate: | ||
| value: str | ||
| recognition_rule: str # grammar name, e.g. "standard_recognition" | ||
| validation_rule: str # rule name, e.g. "Section 3.4.1-addr-spec" | ||
| span: tuple[int, int] | None # half-open [start, end) in the input | ||
| provenance: tuple[Provenance, ...] # one or more authority citations (read-only property) | ||
|
|
||
| @dataclass(frozen=True) | ||
| class Provenance: | ||
| authority: str # "IETF", "ISO", "BIPM", "WHATWG", "CLDR", ... | ||
| specification_name: str # "RFC 5322", "ISO 3166-1", ... | ||
| kind: str # "specification" / "standard" / ... | ||
| reference_url: str | ||
| version: str | None | ||
| lifecycle: str # "active" / "deprecated" / ... | ||
| publication_year: int | ||
|
|
||
| @dataclass(frozen=True) | ||
| class VersionStamp: | ||
| paxman_version: str | ||
|
|
||
| class Resolution(Enum): | ||
| MISSING = "missing" | ||
| INVALID = "invalid" | ||
| SUCCESS = "success" | ||
| AMBIGUOUS = "ambiguous" | ||
| ``` | ||
|
|
||
| **Reading the result** (see also [Execution Result](concepts/execution-result.md)): | ||
|
|
||
| ```python | ||
| from paxman.core.domain import Resolution | ||
|
|
||
| if result.status == Resolution.SUCCESS: | ||
| value = result.canonicalized_value # str, span is set | ||
| else: | ||
| # MISSING / INVALID / AMBIGUOUS — no single value, span is None | ||
| # inspect result.candidates and their provenance/spans | ||
| ... | ||
| ``` | ||
|
|
||
| `span` semantics: on `SUCCESS` it is the span of the single resolved value; on `AMBIGUOUS` use each `candidate.span`; on `MISSING`/`INVALID` it is `None` and `candidates` is empty. | ||
|
|
||
| --- | ||
|
|
||
| ## Errors | ||
|
|
||
| All exceptions inherit from `PaxmanError`. See [Errors](concepts/errors.md) for handling patterns. | ||
|
|
||
| | Exception | Signal type | Typical cause | | ||
| |-----------|-------------|---------------| | ||
| | `CapabilityError` | setup | Unknown capability, duplicate name, registry already frozen | | ||
| | `ContractError` | setup | Bad `pinned_rules`, bad `output_format`, unknown semantics, missing required feature, bad `default_currency`/`default_country`/`precision` | | ||
| | `MultipleMentionsError` | usage | Two separate mentions with different values in one call — split first | | ||
| | `RecognitionError` | internal bug | Grammar raised or returned a bad span/raw_text. Fields: `rule`, `original_error` (`None` on structural failure) | | ||
| | `ValidationError` | internal bug | Rule raised inside `matches()`/`normalize()`. Fields: `rule`, `original_error` | | ||
|
|
||
| Statuses vs exceptions: statuses are domain answers returned inside `ExecutionResult`; exceptions mean the call was not valid to attempt. Catch exceptions at setup and at the outer edge of a batch loop; branch on statuses in normal flow. | ||
|
|
||
| --- | ||
|
|
||
| ## Quick lookup — choose the right capability | ||
|
|
||
| | Kind of text you have | Capability to use | Factory | | ||
| |-----------------------|-------------------|---------| | ||
| | Email addresses | Email | `Email.create_contract(...)` | | ||
| | Dates | Date | `Date.create_contract(...)` | | ||
| | Country codes or names | Country | `Country.create_contract(...)` | | ||
| | Currency codes / symbols / names (no amount) | Currency | `Currency.create_contract(...)` | | ||
| | IP addresses | IP | `IP.create_contract(...)` | | ||
| | ISBNs | ISBN | `ISBN.create_contract(...)` | | ||
| | Money amounts with currency | Money | `Money.create_contract(...)` | | ||
| | Phone numbers | Phone | `Phone.create_contract(...)` | | ||
| | SI unit expressions | SI Unit | `SIUnit.create_contract(...)` | | ||
| | Absolute URLs / IRIs | URL | `URL.create_contract(...)` | | ||
|
|
||
| New capabilities appear in minor releases — check `paxman.capabilities` for the current set. Each per-capability guide under [Capabilities](capabilities/index.md) details its recognized forms, output formats, contract flags, and provenance. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| # Country | ||
|
|
||
| Canonicalizes **one country mention** per call — a code or a name — to a canonical code. | ||
|
|
||
| > **In plain language:** give it `"US"`, `"United States"`, `"DE"`, or `"Alemania"` and it hands back the ISO alpha-2 code if a real spec says that name means that country. Multilingual and historical names are off by default so results are predictable; turn them on when you need them. | ||
|
|
||
| --- | ||
|
|
||
| ## What it recognizes — and what it does not | ||
|
|
||
| | Recognizes (current release — growing) | Does not recognize | | ||
| |----------------------------------------|--------------------| | ||
| | ISO 3166-1 alpha-2 (`US`, `DE`), alpha-3 (`USA`), numeric (`840`) | Ad-hoc abbreviations without a spec (`'Murica`) | | ||
| | Country names in English (`United States`, `Germany`) | Regions or subdivisions (`California`, `Bavaria`) — countries only | | ||
| | CLDR localized names (`Alemania` → `DE`) — only when `include_localized=True` | Localized names when the flag is off — recognized but `INVALID` (no authority claims them) | | ||
| | Historical/deprecated names (`Burma`) — only when `include_historical=True`, via ISO 3166-3 | Historical names when the flag is off — `INVALID` | | ||
|
|
||
| --- | ||
|
|
||
| ## Canonical output | ||
|
|
||
| Default `output_format` is `"alpha2"`. | ||
|
|
||
| | `output_format` | Renders | Example for United States | | ||
| |-----------------|---------|---------------------------| | ||
| | *(default)* `alpha2` / `None` / `"default"` | ISO 3166-1 alpha-2 | `US` | | ||
| | `alpha3` | ISO 3166-1 alpha-3 | `USA` | | ||
| | `numeric` | ISO 3166-1 numeric | `840` | | ||
| | `name` | English short name | `United States` | | ||
|
|
||
| Any other value raises `ContractError`. | ||
|
|
||
| ```python | ||
| from paxman.capabilities import Country | ||
| import paxman | ||
|
|
||
| paxman.register_all_shipped() | ||
| paxman.canonicalize("United States", Country.create_contract()).canonicalized_value # "US" | ||
| paxman.canonicalize("United States", Country.create_contract(output_format="alpha3")).canonicalized_value # "USA" | ||
| paxman.canonicalize("United States", Country.create_contract(output_format="numeric")).canonicalized_value # "840" | ||
| paxman.canonicalize("US", Country.create_contract(output_format="name")).canonicalized_value # "United States" | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Contract | ||
|
|
||
| ```python | ||
| contract = Country.create_contract( | ||
| include_localized=False, # bool, default False — CLDR multilingual names | ||
| include_historical=False, # bool, default False — ISO 3166-3 deprecated names | ||
| output_format=None, # "alpha2" (default), "alpha3", "numeric", "name" | ||
| # plus every common field: excluded_rules / pinned_rules / year / extra_grammars | ||
| ) | ||
| ``` | ||
|
|
||
| - `include_localized` gates the CLDR localized-name rule; when `False`, `Alemania` is recognized but **no rule validates it** → `INVALID`. | ||
| - `include_historical` gates ISO 3166-3; a historical name that validates returns the **historical entity's own former code** (e.g. `Burma` → `BU`), not a successor state's code. | ||
|
|
||
| Historical names can also be viewed as excluding the current ISO 3166-1 name rules and pinning to the 3166-3 rule — the contract controls which specs are consulted (see [Contracts](../concepts/contracts.md)). | ||
|
|
||
| --- | ||
|
|
||
| ## Statuses | ||
|
|
||
| | Input | Contract | Status | Value / why | | ||
| |-------|----------|--------|-------------| | ||
| | `US` | defaults | `SUCCESS` | `"US"` | | ||
| | `United States` | defaults | `SUCCESS` | name → `"US"` | | ||
| | `Alemania` | defaults | `INVALID` | recognized (name grammar) but no rule accepts it without `include_localized` | | ||
| | `Alemania` | `include_localized=True` | `SUCCESS` | → `"DE"` with CLDR provenance | | ||
| | `Burma` | defaults | `INVALID` | needs `include_historical` | | ||
| | `Burma` | `include_historical=True` | `SUCCESS` | → `"BU"` (historical entity's own code) | | ||
| | `ZZ` | any | `INVALID` or `MISSING` | no spec claims it | | ||
| | `hello world` | any | `MISSING` | no country pattern | | ||
| | `US and DE` (two different mentions) | any | raises `MultipleMentionsError` | split first | | ||
|
|
||
| ```mermaid | ||
| flowchart TB | ||
| A[Text e.g. Alemania] --> G[Grammars:<br>alpha2 / alpha3 / numeric / name] | ||
| G --> R1{ISO 3166-1 name rule} | ||
| G --> R2{CLDR localized rule<br>only if include_localized} | ||
| G --> R3{ISO 3166-3 historical<br>only if include_historical} | ||
| R1 & R2 & R3 --> D{Dedup values} | ||
| D -->|one value| OK[SUCCESS] | ||
| D -->|none| INV[INVALID] | ||
| G -->|no pattern| MISS[MISSING] | ||
|
|
||
| style OK fill:#e6ffed,stroke:#2d8a4e | ||
| style INV fill:#fff5f5,stroke:#cc3333 | ||
| style MISS fill:#fff5f5,stroke:#cc3333 | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Notebook snippet — normalize a mixed column | ||
|
|
||
| ```python | ||
| import paxman | ||
| from paxman.capabilities import Country | ||
| from paxman.core.domain import Resolution | ||
| from paxman.core.errors import CapabilityError, ContractError, MultipleMentionsError | ||
|
|
||
| paxman.register_all_shipped() | ||
| contract = Country.create_contract(include_localized=True) | ||
| contract_hist = Country.create_contract(include_localized=True, include_historical=True) | ||
|
|
||
| rows = ["US", "United States", "Alemania", "Burma", "not a country", "US and DE"] | ||
|
|
||
| for text in rows: | ||
| for label, c in [("default+localized", contract), ("+historical", contract_hist)]: | ||
| try: | ||
| r = paxman.canonicalize(text, c) | ||
| except (MultipleMentionsError, CapabilityError, ContractError) as e: | ||
| print(f"{text!r:20} [{label}] → exception {type(e).__name__}: {e}") | ||
| continue | ||
| val = r.canonicalized_value if r.status == Resolution.SUCCESS else "—" | ||
| prov = r.candidates[0].provenance[0].specification_name if r.candidates else "—" | ||
| print(f"{text!r:20} [{label:18}] → {r.status.value:10} {val!r:8} ({prov})") | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Provenance | ||
|
|
||
| - **ISO 3166-1** (alpha-2 / alpha-3 / numeric / English names) | ||
| - **ISO 3166-3** (historical, when enabled) | ||
| - **CLDR** (Unicode localized names, when enabled) | ||
|
|
||
| Each candidate's `validation_rule` carries the section, and `candidate.provenance[0].publication_year` the year. | ||
|
|
||
| See also: [Execution Result](../concepts/execution-result.md), [Provenance](../concepts/provenance.md), [Segmentation](../../recipes/segmentation.md). | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.