Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
ddb157f
feat(si_unit): add SIUnitNotation and package skeleton
azaharizaman Aug 12, 2026
3b10ebe
feat(si_unit): add SIUnitContract
azaharizaman Aug 12, 2026
2418198
feat(si_unit): add authority data tables
azaharizaman Aug 12, 2026
4cbd6a1
test(si_unit): strengthen authority-table locks
azaharizaman Aug 12, 2026
8ca1851
feat(si_unit): add prefixed-unit data generator
azaharizaman Aug 12, 2026
0fefe49
test(si_unit): guard generator constants and prefixed-name uniqueness
azaharizaman Aug 12, 2026
e5a9129
feat(si_unit): add BIPM rule sections
azaharizaman Aug 12, 2026
5f6ac0d
feat(si_unit): add ISO 80000-1 compound rule
azaharizaman Aug 12, 2026
2d03c3b
feat(si_unit): add recognition grammars
azaharizaman Aug 12, 2026
8e620e8
test(si_unit): add cross-layer data-consistency tests
azaharizaman Aug 12, 2026
c9c0f80
feat(si_unit): wire SIUnitCapability with create_contract
azaharizaman Aug 12, 2026
ec505dd
feat(si_unit): register SIUnit capability and extend export/surface g…
azaharizaman Aug 12, 2026
7c9bd33
style(si_unit): ruff-format D7 consistency guard rewrite
azaharizaman Aug 12, 2026
a86e36e
test(si_unit): lock SIUnit pipeline semantics and determinism
azaharizaman Aug 12, 2026
8f7d9ac
test(si_unit): add property invariants and e2e coverage
azaharizaman Aug 12, 2026
ca6a97b
docs(si_unit): document SIUnit capability and update capability counts
azaharizaman Aug 12, 2026
e22029a
docs(si_unit): correct SIUnit examples and generated-data scope
azaharizaman Aug 12, 2026
e4b5478
docs: updated plan file to follow implementation
azaharizaman Aug 12, 2026
fa2fc62
fix(si_unit): updated the docstring
azaharizaman Aug 12, 2026
12c23bf
fix(si_unit): address oracle review findings
azaharizaman Aug 12, 2026
6b6ffaf
feat(si_unit): add CLI tools for SI Unit canonicalization and output
azaharizaman Aug 12, 2026
a796e83
feat(si_unit): multi-solidus guard and split-prefix handling (ADR-000…
azaharizaman Aug 17, 2026
502c893
docs(si_unit): sync docs/contract to committed SI Unit split-prefix l…
azaharizaman Aug 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@
**Branch:** feature/CURRENCY-capability

## OVERVIEW
Paxman is a Python 3.11+ canonicalization library: takes ambiguous human input, returns what authoritative specs say it means, with full provenance. Deterministic, provenance-first. 9 capabilities (Country, Currency, Date, Email, IP, ISBN, Money, Phone, URL). Toolchain: uv + hatchling, ruff, strict pyright, import-linter, pytest at 95% coverage.
Paxman is a Python 3.11+ canonicalization library: takes ambiguous human input, returns what authoritative specs say it means, with full provenance. Deterministic, provenance-first. 10 capabilities (Country, Currency, Date, Email, IP, ISBN, Money, Phone, SI Unit, URL). Toolchain: uv + hatchling, ruff, strict pyright, import-linter, pytest at 95% coverage.

## STRUCTURE
```text
paxman/
├── api/ # canonicalize() — sole public entry
├── engine/ # run_capability() pipeline orchestrator
├── core/ # domain objects, Contract protocol, registry, errors
└── capabilities/ # 9 self-contained capability packages
└── capabilities/ # 10 self-contained capability packages
tests/ # unit / capabilities/<cap> / integration / property / e2e
tools/ # regenerate_isbn_range_data.py (only script)
tools/ # regenerate_isbn_range_data.py, regenerate_si_prefix_data.py, regenerate_idna_uts46_data.py
docs/ # adr/, report/, research/, superpowers/plans+specs
```

Expand All @@ -31,7 +31,7 @@ docs/ # adr/, report/, research/, superpowers/plans+specs
| Recognition (per cap) | `paxman/capabilities/<Name>/grammar/` |
| Validation (per cap) | `paxman/capabilities/<Name>/rules/` |
| Presentation seam | `paxman/capabilities/<Name>/capability.py` → `format_value()` |
| Regenerate ISBN data | `tools/regenerate_isbn_range_data.py` |
| Regenerate generated data | `tools/regenerate_isbn_range_data.py` (ISBN range), `tools/regenerate_si_prefix_data.py` (SIUnit prefixed units), `tools/regenerate_idna_uts46_data.py` (URL IDNA mapping) |
| Merge-blocking commands | `.github/workflows/ci.yml` (authoritative) |
| Past implementation plans | `docs/superpowers/plans/` |

Expand Down Expand Up @@ -76,17 +76,19 @@ uv run ruff format --check paxman/ tests/ # format check
uv run pyright # strict typecheck
uv run import-linter lint # layer boundaries
uv run pytest # all tests
uv run pytest -m unit|capability|integration|e2e # by marker (also: property, country, isbn, money)
uv run pytest -m "unit or capability or integration or e2e" # by marker (also: property, country, currency, isbn, money, url, si_unit)
uv run pytest --cov=paxman --cov-report=term-missing --tb=short -q
uv run coverage report --include="paxman/{core,capabilities,engine,api}/*" --fail-under=95
uv run python tools/regenerate_isbn_range_data.py # regenerate ISBN data module
uv run coverage report --include="paxman/core/*,paxman/capabilities/*,paxman/engine/*,paxman/api/*" --fail-under=95
uv run python tools/regenerate_isbn_range_data.py # regenerate ISBN range message module
uv run python tools/regenerate_si_prefix_data.py # regenerate SIUnit prefixed-unit modules
uv run python tools/regenerate_idna_uts46_data.py # regenerate URL IDNA UTS #46 mapping
```
Full pre-PR gate: `ruff check . && ruff format --check . && pyright && import-linter lint && pytest`
Full pre-PR gate: `uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run import-linter lint && uv run pytest`

## NOTES
- `paxman/capabilities/__init__.py` exports all nine shipped capabilities (Country, Currency, Date, Email, IP, ISBN, Money, Phone, URL); export completeness is enforced by `tests/unit/test_capability_exports.py`. A 10th — SI Unit (BIPM SI Brochure) — is in development on `feature/si-unit-capability`; update counts and the export/enumeration lists when it lands.
- CONTEXT.md is the domain glossary for the full shipped set (nine capabilities). It is kept in sync with the code; when adding a capability, update its Notation/table entries there too.
- `paxman/capabilities/__init__.py` exports all ten shipped capabilities (Country, Currency, Date, Email, IP, ISBN, Money, Phone, SI Unit, URL); export completeness is enforced by `tests/unit/test_capability_exports.py`.
- CONTEXT.md is the domain glossary for the full shipped set (ten capabilities). It is kept in sync with the code; when adding a capability, update its Notation/table entries there too.
- No `pyrightconfig.json` — pyright config is inline `[tool.pyright]` in pyproject.toml. No `.editorconfig`.
- Data modules live under `rules/data/` (Country, Currency, ISBN, Money, Phone, URL) and `grammar/data/` (Country, Currency, Money) — plain module-level tables separating data from logic, maintained in place. Only ISBN's range message is generated: XML snapshot → `range_message.py` via `tools/regenerate_isbn_range_data.py`; unmarked data files are edited directly.
- Data modules live under `rules/data/` (Country, Currency, ISBN, Money, Phone, SI Unit, URL) and `grammar/data/` (Country, Currency, Money, SI Unit) — plain module-level tables separating data from logic, maintained in place. Only the ISBN range message, the URL IDNA UTS #46 mapping, and the SIUnit prefixed-unit and grammar token tables are generated (each via its `tools/regenerate_*_data.py` script); unmarked data files are edited directly.
- Library only — no CLI, no `__main__.py`, no `[project.scripts]`. Version 0.2.0.
- Coverage: global `fail_under = 95` + per-package 95% gates in CI.
26 changes: 18 additions & 8 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ Capability-defined intermediate representation that Grammars must produce:
- **Currency:** `CurrencyNotation(text, shape)` — `shape` is `"code"` / `"qualified_symbol"` / `"symbol"` / `"word"`; codes are grammar-folded to uppercase, words to lowercase, symbols keep exact casing
- **Money:** `MoneyNotation(currency_part, amount_part, currency_shape, amount_shape)` — verbatim currency + amount tokens with grammar-assigned shape discriminators
- **ISBN:** `ISBNNotation(shape, digits)` — `shape` is `"isbn10"` / `"isbn13"`, `digits` is the digit string (`X` only as final char of an isbn10 shape)
- **SIUnit:** `SIUnitNotation(text, shape)` — `shape` is `"symbol"` / `"name"` / `"compound"` / `"split_word_prefix"` / `"split_symbol_prefix"`; `text` is the unit expression as written (symbols keep exact casing, names are grammar-folded to lowercase, compounds keep the written form)
- **IP / Phone / URL:** capability-defined shapes for address / number / URI components

**Note:** Capabilities define Notation using frozen dataclasses for type safety and immutability. The `as_list()` method bridges the typed notation to the generic `list[str]` interface.
Expand All @@ -65,7 +66,7 @@ class EmailNotation:

## The Capabilities

Paxman ships nine built-in capabilities, each wired to an authoritative specification:
Paxman ships ten built-in capabilities, each wired to an authoritative specification:

| Capability | Domain | Authorities |
|------------|--------|-------------|
Expand All @@ -77,11 +78,12 @@ Paxman ships nine built-in capabilities, each wired to an authoritative specific
| **ISBN** | ISBNs | ISO 2108, ISBN Users' Manual, ISBN Range Message |
| **Money** | Money amounts | ISO 4217, CLDR |
| **Phone** | Phone numbers | ITU-T E.164, RFC 3966, NANP |
| **SI Unit** | SI unit expressions | BIPM SI Brochure, ISO 80000-1 |
| **URL** | URLs | WHATWG URL Standard |

Capability classes are exported from `paxman/capabilities/__init__.py` as acronym aliases (`EmailCapability as Email`, etc.); the export list is enforced by `tests/unit/test_capability_exports.py`.

**In development:** the **SI Unit** capability (BIPM SI Brochure, 9th edition) is the planned 10th capability (MILESTONE row 23; branch `feature/si-unit-capability`). It will be the first capability whose canonical value is case-meaningful (`K` kelvin vs `k` kilo). This document will be updated again once it ships.
**Note:** the **SI Unit** capability is the first whose canonical value is case-meaningful (`K` kelvin vs `k` kilo). It canonicalizes unit expressions only (symbols, names, product/quotient compounds) to the canonical symbol form; no quantities, no magnitudes, no name-compounds.

---

Expand Down Expand Up @@ -194,7 +196,7 @@ class SectionCode(Rule[CurrencyNotation]):
return self.TABLE[notation.text]
```

Authority-backed lookup tables live in `rules/data/` (e.g., `iso4217_list_one.py`, `cldr_currencies.py`), separated from rule logic; lexicon keys serving grammars live in `grammar/data/`. Only the ISBN range message is generated from a source snapshot (via `tools/regenerate_isbn_range_data.py`) — everything else is maintained in place.
Authority-backed lookup tables live in `rules/data/` (e.g., `iso4217_list_one.py`, `cldr_currencies.py`), separated from rule logic; lexicon keys serving grammars live in `grammar/data/`. Seven data modules across three generators are produced via tools: the ISBN range message (`tools/regenerate_isbn_range_data.py`), the URL IDNA UTS #46 mapping (`tools/regenerate_idna_uts46_data.py`), and the SIUnit prefixed-unit and grammar token tables (`tools/regenerate_si_prefix_data.py`) — everything else is maintained in place.

### Parser Example
```python
Expand Down Expand Up @@ -619,7 +621,7 @@ for candidate in result.candidates:
### Capability Versioning
- Each capability has its own version in `capability.py`
- Capability version is independent of engine version
- All nine built-in capabilities currently ship `version = "1.0.0"`
- All ten built-in capabilities currently ship `version = "1.0.0"`
- Example:
```python
# capabilities/Email/capability.py
Expand Down Expand Up @@ -775,6 +777,14 @@ paxman/
│ ├── grammar/ # e164, tel_uri, international_00, national_recognition (+ common.py LEGACY)
│ ├── rules/ # e164_ed2010, rfc_3966_ed2004, nanp_ed2024
│ └── rules/data/ # e164_country_codes, nanp_tables
├── SIUnit/ # grammar/ (5) + rules/ (3) + grammar/data/ + rules/data/ — BIPM SI Brochure, ISO 80000-1
│ ├── capability.py # SIUnitCapability
│ ├── contract.py # SIUnitContract
│ ├── notation.py # SIUnitNotation (text, shape)
│ ├── grammar/ # symbol, name, compound_recognition, split_word_recognition, split_symbol_recognition
│ ├── grammar/data/ # unit_symbol_tokens, unit_name_tokens, compound_tokens (+ GENERATED via tools/regenerate_si_prefix_data.py)
│ ├── rules/ # bipm_si_brochure_ed2019, iso_80000_ed2022, split_prefixes
│ └── rules/data/ # si_base_units, si_derived_units, si_nonsi_units, si_prefixes, unit_names (+ GENERATED prefixed_units, prefixed_unit_names)
└── URL/ # grammar/ (1) + rules/ (1) + rules/data/ — WHATWG URL Standard
├── capability.py # URLCapability
├── contract.py # URLContract
Expand All @@ -785,8 +795,6 @@ paxman/
└── rules/data/ # idna_uts46_mapping
```

**In development (not yet in tree):** the SI Unit capability (branch `feature/si-unit-capability`, research at `docs/research/2026-08-09-si-unit-canonicalization.md`). Structure will follow the Currency LOOKUP_TABLE template with BIPM SI Brochure data under `rules/data/`.

### Package Responsibilities

| Package | Responsibility |
Expand All @@ -813,7 +821,7 @@ tests/
│ ├── test_capability_contract.py# CapabilityContract (output_format policy, defaults)
│ ├── test_capability.py # Capability ABC
│ ├── test_capability_surface.py # Surface homogeneity across capabilities
│ ├── test_capability_exports.py # __init__ export completeness (9 capabilities)
│ ├── test_capability_exports.py # __init__ export completeness (10 capabilities)
│ ├── test_version_stamp.py # VersionStamp
│ ├── test_discovery.py # Registry register/freeze/reset
│ ├── test_errors.py # Exception hierarchy
Expand All @@ -830,6 +838,7 @@ tests/
│ ├── isbn/ # + test_contract, test_notation, test_data
│ ├── money/ # + test_contract, test_notation, test_data, test_parsing
│ ├── phone/ # + test_data
│ ├── si_unit/ # test_grammar, test_rules, test_capability, test_contract, test_notation, test_data, test_data_consistency
│ └── url/ # + test_contract, test_notation, test_data, test_parsing, test_rule
├── integration/ # -m integration pipeline, ambiguity, temporal,
│ │ # feature gating, format_value seam, per-capability pipelines
Expand Down Expand Up @@ -874,7 +883,7 @@ def test_ambiguity_detection(): ...
def test_canonicalize_email_success(): ...
```

Per-capability markers are registered for `country`, `currency`, `isbn`, `money`, and `url` — run one capability's suite directly with `uv run pytest tests/capabilities/<cap>` (or `-m <cap>`). Capability dirs are lowercase (`isbn`, not `ISBN`).
Per-capability markers are registered for `country`, `currency`, `isbn`, `money`, `si_unit`, and `url` — run one capability's suite directly with `uv run pytest tests/capabilities/<cap>` (or `-m <cap>`). Capability dirs are lowercase (`isbn`, not `ISBN`).

---

Expand Down Expand Up @@ -922,6 +931,7 @@ markers = [
"isbn: isbn capability tests",
"money: money capability tests",
"url: url capability tests",
"si_unit: si unit capability tests",
]
testpaths = ["tests"]
```
Expand Down
47 changes: 45 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ If multiple specifications disagree on the canonical value, the status is `AMBIG

## Capabilities

Paxman ships with nine built-in capabilities:
Paxman ships with ten built-in capabilities:

| Capability | Domain | Grammars | Rules | Description |
|------------|--------|----------|-------|-------------|
Expand All @@ -62,6 +62,7 @@ Paxman ships with nine built-in capabilities:
| **ISBN** | ISBNs | 2 (isbn13, isbn10) | 4 | ISO 2108, ISBN Users' Manual, ISBN Range Message |
| **Money** | Money amounts | 3 (code, symbol, word) | 3 | ISO 4217, CLDR |
| **Phone** | Phone numbers | 4 (E.164, tel-URI, 00-prefix, national) | 5 | ITU-T E.164, RFC 3966, NANP |
| **SI Unit** | SI unit expressions | 5 (symbol, name, compound, split_word_prefix, split_symbol_prefix) | 7 | BIPM SI Brochure, ISO 80000-1 |
| **URL** | URLs | 1 (absolute-uri) | 1 | WHATWG URL Standard |

### Email Capability
Expand Down Expand Up @@ -331,6 +332,46 @@ result = paxman.canonicalize("http://münchen.de", contract)
# → "http://xn--mnchen-3ya.de/"
```

### SI Unit Capability

Recognizes SI unit expressions: symbols, names, and product/quotient compounds, canonicalizing to the canonical symbol form with BIPM SI Brochure (9th ed.) and ISO 80000-1 provenance. Identity-only: no quantities, no magnitudes, no name-compounds ("metre per second" does not resolve as a compound — its words are recognized separately, yielding AMBIGUOUS; "25°C" is MISSING).

```python
from paxman.capabilities import SIUnit

register_capability(SIUnit())

# Unit name resolves to its canonical symbol
contract = SIUnit.create_contract()
result = paxman.canonicalize("Kilogram", contract)
# → "kg"

# Prefixed name resolves to the prefixed symbol
contract = SIUnit.create_contract()
result = paxman.canonicalize("megahertz", contract)
# → "MHz"

# Compound expression canonicalizes to the symbol form
contract = SIUnit.create_contract()
result = paxman.canonicalize("m/s²", contract)
# → "m/s2"

# Spoken word-prefix form, merged only when opted in
contract = SIUnit.create_contract(allow_split_word_prefixes=True)
result = paxman.canonicalize("kilo gram", contract)
# → "kg"

# Symbol-prefix spacing is always rejected (no flag)
contract = SIUnit.create_contract()
result = paxman.canonicalize("k g", contract)
# → Status: INVALID

# Multi-solidus preserved only when opted in
contract = SIUnit.create_contract(allow_multi_solidus=True)
result = paxman.canonicalize("kg/m/s", contract)
# → "kg/m/s"
```

---

## Contract Configuration
Expand Down Expand Up @@ -365,6 +406,8 @@ Every capability provides a `create_contract()` factory method with common and c
| Money | `output_format` | `str` | Output format (`"code_amount"` default, `"compact"`) |
| Phone | `default_country` | `str` | ISO 3166-1 alpha-2 country code to resolve national numbers (e.g., `"US"`) |
| Phone | `output_format` | `str` | Output format (`"e164"` default, `"rfc3966"`, `"national"`) |
| SIUnit | `allow_split_word_prefixes` | `bool` | Merge a word prefix split from its unit by whitespace (e.g. `"kilo gram"` → `"kg"`) when True; default False rejects the spoken form (→ INVALID) |
| SIUnit | `allow_multi_solidus` | `bool` | Preserve the legacy accept-multi-solidus behavior (e.g. `"kg/m/s"`) when True; default False rejects more than one top-level solidus (→ INVALID) per ISO 80000-1 §6.6.2 |

### Rule Pinning and Exclusion

Expand Down Expand Up @@ -399,7 +442,7 @@ result = paxman.canonicalize("2026-01-15", contract)

## Community Extensions

Paxman ships with nine built-in capabilities, but a capability is closed for modification yet open for extension: you can add recognition and validation without touching the library. Register a `Grammar` subclass and the `Rule` subclass that validates it, then opt a contract into them by naming the grammar in `extra_grammars`:
Paxman ships with ten built-in capabilities, but a capability is closed for modification yet open for extension: you can add recognition and validation without touching the library. Register a `Grammar` subclass and the `Rule` subclass that validates it, then opt a contract into them by naming the grammar in `extra_grammars`:

```python
import re
Expand Down
Loading
Loading