diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f3c4c629..4d8beab4 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -37,7 +37,9 @@ all cross-match policy: same span (e.g. US vs European date reading of `01/02/2026`) are both preserved and ambiguity stays observable. - **Ordering:** recognitions are emitted in the total order - `(start, end, active_grammars index, grammar name)`, i.e. document order. + `(start, end, active-set index, grammar name)`, i.e. document order — where the + active set is `contract.active_grammars` or, when the contract returns `None`, + every shipped grammar in `get_grammars()` order. - **Candidate dedup** (`value, recognition_rule, validation_rule`) runs after validation as a stability net. @@ -128,7 +130,7 @@ Contracts pass configuration parameters to validation rules, enabling rules to a **Base Contract Parameters:** - **`output_format`**: Controls the canonical value format (e.g., `"ISO"` for `YYYY-MM-DD`, `"US"` for `MM/DD/YYYY`). `CapabilityContract.__post_init__` resolves `None`, `"default"`, and each capability's default format to a concrete string; the capability's `format_value()` seam applies the format to the rule-produced default canonical value. Validation rules never inspect `output_format` — they always normalize to the default canonical form. See "The Formatting Seam" below. - **`pinned_rules`**: Pins to specific validation rules by name. When set, ONLY those rules run — `excluded_rules` is ignored. Takes precedence over `excluded_rules`. -- **`extra_grammars`**: Names community grammars (opt-in) to run alongside the capability's shipped `active_grammars`, in order. Unknown names are silently skipped; shipped names listed here are deduplicated. Registration happens through `paxman.register_grammar` / `paxman.register_rule` (see "Community Extensions" below). +- **`extra_grammars`**: Names community grammars (opt-in) to run alongside the capability's shipped active set — `contract.active_grammars`, or every shipped grammar when the contract returns `None` — in order. Unknown names are silently skipped; shipped names listed here are deduplicated. Registration happens through `paxman.register_grammar` / `paxman.register_rule` (see "Community Extensions" below). **Date-Specific Parameters:** - **`two_digit_base_year`**: Specifies the base year for interpreting two-digit years (e.g., `2000` means `"26"` becomes `2026`). Only available on Date contracts, not part of the base Contract protocol. Used by US and European grammars to resolve ambiguous year values. @@ -167,7 +169,7 @@ Every stage is a pure function of its inputs — no clocks, no randomness, no en Capabilities are closed for modification but open for extension. Community contributors register additional grammars (and the rules that validate them) against an existing capability through `paxman.core.extensions` — never by editing the capability package. The registries freeze with the capability registry at the first pipeline run. -A contract opts a registered grammar in by naming it in `extra_grammars`, a base `CapabilityContract` field surfaced on every `create_contract` factory. The engine composes the shipped active set with the opted-in extras, deduplicating names while preserving order — shipped `active_grammars` slots first, extras after (unknown extra names are silently skipped). Opt-in preserves determinism: a contract that names no extras composes to exactly the shipped set, so non-opt-in behavior is byte-identical. +A contract opts a registered grammar in by naming it in `extra_grammars`, a base `CapabilityContract` field surfaced on every `create_contract` factory. The engine composes the shipped active set with the opted-in extras, deduplicating names while preserving order — shipped slots first, extras after (unknown extra names are silently skipped). The shipped slots are `contract.active_grammars` when the contract implements it (the gated capabilities), or every shipped grammar in `get_grammars()` order when it returns `None` (the base default). Opt-in preserves determinism: a contract that names no extras composes to exactly the shipped set, so non-opt-in behavior is byte-identical. Community rules follow the same opt-in discipline: a registered rule runs only when the contract names one of its `target_grammars` in `extra_grammars`. An un-opted community rule — even one targeting a shipped grammar — never affects results, so a default contract resolves with shipped rules only. diff --git a/CONTEXT.md b/CONTEXT.md index c15d6fbc..4da7da5f 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -92,7 +92,7 @@ Syntactic extraction rules that: - Scan raw text for patterns - Produce **span-bearing `RecognitionMatch` objects** (notation + half-open `[start, end)` span + matched `raw_text`) — never bare notations - Live in `capabilities//grammar/` -- Are **filtered by the orchestrator** based on the contract's `active_grammars` +- Are **selected by the orchestrator** from the contract's `active_grammars`, or from every shipped `get_grammars()` entry when the contract returns `None` (the base default) - Do NOT validate, de-duplicate, or order — the engine owns containment dedup and document ordering - Only recognize (syntax, shape, lexicon keys) — never map tokens to canonical values, never import rule-layer data @@ -283,17 +283,18 @@ class StandardEmailGrammar(Grammar[EmailNotation]): ### Date Capability Details -The Date capability has **3 grammars** and **3 validation rules**: +The Date capability has **4 grammars** and **3 validation rules**: #### Grammars (Recognition) | Grammar | Delimiter | N1 (first) | N2 (second) | N3 (third) | Notes | |---------|-----------|------------|-------------|------------|-------| | ISO | `-` | year | month | day | 4-digit year only | +| Slash-ISO | `/` | year | month | day | 4-digit year; shares ISO position mapping | | US | `/` | month | day | year | Supports 2-digit years | | European | `/` | day | month | year | Supports 2-digit years | -**Note:** European and US grammars both use `/` as delimiter. The ambiguity arises from different position mappings, not delimiters. +**Note:** European, US, and slash-ISO grammars use `/` as delimiter. The ambiguity arises from different position mappings, not delimiters; a leading 4-digit year is unambiguous (slash-ISO only), while a leading 1–2-digit field is ambiguous between US and European. #### Validation Rules @@ -345,7 +346,7 @@ Presentation is a single seam, not a rule concern: ### Feature Gating — two loci, two statuses Input-shape and authority features gate at different points and produce different statuses: -- **Input-shape features** (`include_*`) toggle grammars via the contract's `active_grammars` property. A disabled grammar never recognizes → its inputs are **`MISSING`**. +- **Input-shape features** (`include_*`) toggle grammars via the contract's `active_grammars` property — implemented only by the gated capabilities (Email, IP, ISBN); other contracts inherit the base `None` default, which runs every shipped grammar. A disabled grammar never recognizes → its inputs are **`MISSING`**. - **Authority features** gate rules via `requires_features` (declared on the rule). The rule is dropped from the run → recognized but unvalidated input is **`INVALID`**. - Never gate inside `matches()` / `recognize()`; never cast a contract to read `include_*` flags inside a rule (`typing.cast` is only for validity-affecting parameters). @@ -646,8 +647,8 @@ class Contract(Protocol): ... @property - def active_grammars(self) -> Sequence[str]: - """List of grammar names to activate.""" + def active_grammars(self) -> Sequence[str] | None: + """Grammar names to activate; None runs every shipped grammar.""" ... @property @@ -675,7 +676,7 @@ class Contract(Protocol): ... ``` -Contracts subclass `CapabilityContract` (never `Contract` directly), are `@dataclass(frozen=True)` **without** `slots=True`, and set `DEFAULT_OUTPUT_FORMAT` / `OFFERED_OUTPUT_FORMATS`, `capability_name` via `field`, and `active_grammars`. +Contracts subclass `CapabilityContract` (never `Contract` directly), are `@dataclass(frozen=True)` **without** `slots=True`, and set `DEFAULT_OUTPUT_FORMAT` / `OFFERED_OUTPUT_FORMATS` and `capability_name` via `field`. `active_grammars` is optional: only feature-gated capabilities (Email, IP, ISBN) implement it; others inherit the base `None` default (run every shipped grammar). --- diff --git a/HOW_TO_ADD_NEW_CAPABILITY.md b/HOW_TO_ADD_NEW_CAPABILITY.md index 4337e263..278770c9 100644 --- a/HOW_TO_ADD_NEW_CAPABILITY.md +++ b/HOW_TO_ADD_NEW_CAPABILITY.md @@ -412,7 +412,7 @@ Concretely, a `CapabilityContract` subclass: - Overrides `DEFAULT_OUTPUT_FORMAT` (a concrete string) and `OFFERED_OUTPUT_FORMATS` (a `frozenset[str]` of *alternative* formats) as class variables. The default format is **not** included in `OFFERED_OUTPUT_FORMATS`. - Sets `capability_name` via `field(default="", init=False)`. - Declares `output_format` nowhere — the base field `output_format: str | None = None` is inherited. It is **never** a non-optional `str`. The base `__post_init__` resolves `None`, `"default"`, and the default format string to the concrete default, validates offered alternatives, and raises `ContractError` for anything else. -- Implements the abstract `active_grammars` property. +- Implements `active_grammars` **only when recognition is feature-gated** (the Email/IP/ISBN pattern). Otherwise the property is omitted entirely: the base returns `None` and the engine runs every shipped grammar in `get_grammars()` order. - Adds its own `__post_init__` validation by calling `super().__post_init__()` first. Use `@dataclass(frozen=True)` exactly like the base — do NOT add `slots=True` (incompatible with the base's `super()` pattern). `CapabilityContract` satisfies the `Contract` protocol structurally, so your subclass does too. @@ -455,7 +455,7 @@ class SectionYourRule(Rule[YourDomainNotation]): **Feature gating has two loci, and they produce different `Resolution` statuses:** -- **Input-shape features toggle grammars via `active_grammars`.** A flag like `include_obfuscated` decides whether the `obfuscated_recognition` grammar runs at all. A disabled grammar recognizes nothing, so input readable only by that grammar yields `MISSING`. +- **Input-shape features toggle grammars via `active_grammars`** (implemented only by the gated capabilities — Email, IP, ISBN; other contracts inherit the base `None` default, which runs every shipped grammar). A flag like `include_obfuscated` decides whether the `obfuscated_recognition` grammar runs at all. A disabled grammar recognizes nothing, so input readable only by that grammar yields `MISSING`. - **Authority features use `requires_features`.** A flag like `include_localized` gates the CLDR rule that validates localized names, not the grammar. Recognition still runs and produces a notation, but the engine drops the gated rule, so the recognized-but-unvalidated input yields `INVALID`. **Hard rule: never gate inside `matches()`.** Do not read `include_*` feature-toggle flags, and do not `cast(Contract, ...)` to reach them, inside `matches()`. `matches()` must never consult `output_format` either; validity comes from the notation, the specification, and any legitimate validity-affecting parameters (e.g. `default_country`, `two_digit_base_year`). The engine owns feature routing: declare the dependency in `requires_features` and let the filter decide whether the rule runs. @@ -473,7 +473,7 @@ Define the Contract in `paxman/capabilities/YourDomain/contract.py` (separate fi 3. Override `DEFAULT_OUTPUT_FORMAT` (a concrete string) and `OFFERED_OUTPUT_FORMATS` (a `frozenset[str]` of alternative formats, excluding the default) as class variables 4. Set `capability_name` via `field(default="yourdomain", init=False)` (users never set this) 5. Add configuration fields for toggling grammars (e.g., `include_obfuscated: bool = False`) -6. Implement `active_grammars` as a `@property` that builds the grammar list from configuration flags +6. Implement `active_grammars` as a `@property` that builds the grammar list from configuration flags — only if recognition is feature-gated; otherwise omit it (base default: run every shipped grammar) `excluded_rules`, `pinned_rules`, `year`, and `output_format` are declared once on `CapabilityContract` — you don't redeclare them. @@ -516,11 +516,11 @@ two_digit_base_year: int | None = None # Date: base year for 2-digit year parsi - Pass parameters to rules (strings, ints, options) - Control output behavior (`output_format`) -### Implementing `active_grammars` +### Implementing `active_grammars` (optional) -Two approaches exist for implementing `active_grammars`: +`active_grammars` is **optional**: the base `CapabilityContract.active_grammars` returns `None`, and the engine falls back to running every shipped grammar returned by `get_grammars()`, in order. Implement it only when recognition is feature-gated — an `include_*` flag decides whether a grammar runs at all: -1. **Conditional** (Email, IP): Build the list from boolean flags. Grammars are included only when their flag is `True`. +1. **Conditional** (Email, IP, ISBN): Build the list from boolean flags. Grammars are included only when their flag is `True`. ```python @property @@ -533,15 +533,7 @@ def active_grammars(self) -> list[str]: return grammars ``` -2. **Always-all** (Date, Country): Return all grammar names unconditionally. All grammars always run, and rules handle filtering via `notation.shape` or other discriminators. - -```python -@property -def active_grammars(self) -> list[str]: - return ["iso8601_recognition", "us_recognition", "european_recognition"] -``` - -Choose conditional when grammars are expensive or mutually exclusive. Choose always-all when grammars are cheap and rules need to see all representations. +Do **not** implement a static "always-all" override returning every grammar name (the former Date/Country pattern). The base `None` default already runs every shipped grammar, so a static override adds maintenance with zero behavior change — and it silently excludes any future grammar added to `get_grammars()` unless someone remembers to extend the list. Choose conditional when grammars are expensive or mutually exclusive; otherwise omit the property entirely and let the fallback run all shipped grammars. ### Implementing `output_format` (always optional, homogeneous across capabilities) @@ -608,7 +600,7 @@ def format_value( **The Contract must satisfy the `Contract` protocol** (inheriting `CapabilityContract` does this structurally): - `capability_name: str` — the capability this contract configures -- `active_grammars: Sequence[str]` — list of grammar names to activate +- `active_grammars: Sequence[str] | None` — grammar names to activate; `None` (base default) runs every shipped grammar in `get_grammars()` order - `excluded_rules: Sequence[str]` — list of rule names to exclude - `pinned_rules: Sequence[str] | None` — pin to specific rules (takes precedence over `excluded_rules` when set) - `year: int | None` — year for temporal filtering diff --git a/HOW_TO_ADD_NEW_GRAMMAR.md b/HOW_TO_ADD_NEW_GRAMMAR.md new file mode 100644 index 00000000..e4e3dad4 --- /dev/null +++ b/HOW_TO_ADD_NEW_GRAMMAR.md @@ -0,0 +1,292 @@ +# How to Add a New Grammar to an Existing Capability + +This guide walks you through making an existing Paxman capability smarter by teaching it to recognize a new representation. It is written for contributors who are new to the project and assumes no prior knowledge of the internal architecture. + +Where [HOW_TO_ADD_NEW_CAPABILITY.md](HOW_TO_ADD_NEW_CAPABILITY.md) covers building a capability from scratch, this guide covers the smaller, more common task: **adding one recognition format to a shipped capability** (e.g., teaching the Date capability to recognize `YYYY/MM/DD`). + +By the end of this guide, you will have added a grammar that changes real behavior — an input that previously resolved `MISSING` now resolves `SUCCESS` — with tests proving the difference. + +> **Two ways to extend recognition.** If you are a Paxman contributor shipping the grammar *inside* the library, this guide is for you. If you are a downstream user who wants to extend a capability without touching the library, use the community seam instead: write the same `Grammar` subclass, register it with `paxman.register_grammar()` / `paxman.register_rule()`, and opt a contract into it via `extra_grammars`. See README's **Community Extensions** section for that path. The grammar class itself looks identical in both paths; only the wiring differs. + +--- + +## Prerequisites + +Before starting, understand these concepts (all defined in depth in HOW_TO_ADD_NEW_CAPABILITY.md): + +- **Grammar** — a recognition unit that scans raw text and extracts span-bearing notations. Syntax only: it never validates, dedups, orders, or maps tokens to canonical values. +- **Rule** — a validation unit that checks a notation against an authoritative specification and produces the canonical value with provenance. Semantics. +- **Notation** — the intermediate token grammars produce and rules consume. +- **`active_grammars`** — the *optional* contract property naming which grammars run. A contract that does not implement it runs every shipped grammar returned by `get_grammars()`; only the gated capabilities (Email, IP, ISBN) implement it to name a subset. +- **`target_grammars`** — the rule metadata declaring which grammar(s) a rule validates. A recognition only routes to rules that name its producing grammar. + +**The one sentence that matters:** a new grammar changes behavior only when it is (1) returned by `get_grammars()`, and (2) targeted by at least one rule via `target_grammars` — plus (3) named in `active_grammars`, but **only for the gated capabilities (Email, IP, ISBN) that implement it**. For the other six capabilities, the contract has no `active_grammars` and the engine runs every shipped grammar, so `get_grammars()` alone activates the new grammar. Miss any condition and the grammar silently never runs — so a shipped grammar ships with a test that proves the difference (Step 5). + +--- + +## Step 1: Choose the Representation and the Strategy + +Before writing code, answer these questions: + +1. **What new representation are you recognizing?** Write down examples of real human input, including edge cases (`2024/1/5`, not just `2024/01/01`). +2. **Does an existing notation already fit it?** If the representation decomposes into the same fields as an existing grammar (e.g., Date's `DateNotation(N1, N2, N3)`), reuse it. Only extend the notation when the new format genuinely carries different components. +3. **Does an existing rule already assign the same meaning?** If the new format means the same thing and normalizes the same way as an already-validated format (e.g., `2024/01/01` *is* ISO 8601's calendar date), you may only need to extend an existing rule's `target_grammars` (Step 4, option A). +4. **Which recognition strategy fits the representation?** + +Every grammar follows one of two core strategies (see HOW_TO_ADD_NEW_CAPABILITY.md Step 4 for the extended set): + +| Your representation… | Strategy | Example | +|---|---|---| +| Has a distinctive syntactic shape (delimiters, fixed widths, character classes) | **Regex** — compile at module scope, iterate with `re.finditer()`, map groups to notation fields | `YYYY/MM/DD` dates, `IPv6` addresses, `ISBN-10` | +| Is a finite vocabulary of free-form tokens (names, spelled-out forms) | **Lexicon** — normalize input, test a key-only table under `grammar/data/`, emit the trimmed token | Country `name_recognition` | + +The boundary is a hard rule: **grammars own syntax, rules own meaning**. A grammar never maps a token to a canonical value and never imports rule-layer authority data (enforced by the semantic purity gate). If your representation needs a synonym or a canonical mapping, the rule owns it. + +--- + +## Step 2: Write the Grammar File (TDD: test first) + +Test-first, as everywhere in this project: write the failing grammar tests before the grammar exists. + +### 2a: Grammar tests (red) + +Add a test class for your grammar to `tests/capabilities//test_grammar.py`, following the existing test classes in that file: + +```python +@pytest.mark.capability +class TestSlashISODateGrammar: + """Tests for slash-ISO date grammar (YYYY/MM/DD).""" + + def test_recognizes_valid_input(self) -> None: + grammar = SlashISODateGrammar() + result = grammar.recognize("2026/07/26") + assert len(result) == 1 + assert result[0].notation.as_list() == ["2026", "07", "26"] + + def test_does_not_match_us_or_european_order(self) -> None: + """A 2-digit-first slash date is not a slash-ISO date.""" + grammar = SlashISODateGrammar() + assert grammar.recognize("07/26/2026") == [] + assert grammar.recognize("26/07/2026") == [] + + def test_emits_spans(self) -> None: + result = SlashISODateGrammar().recognize("x 2026/07/26 y") + assert len(result) == 1 + assert result[0].start == 2 + assert result[0].end == 12 + assert result[0].raw_text == "2026/07/26" + assert result[0].notation.as_list() == ["2026", "07", "26"] + + def test_grammar_name(self) -> None: + grammar = SlashISODateGrammar() + assert grammar.name == "slash_iso_recognition" +``` + +Cover, at minimum: the happy path, the notation field mapping, variant input (different widths/separators), multiple matches, empty input, an incompatible format the grammar must *not* claim, and span correctness (`start`/`end`/`raw_text`). + +### 2b: The grammar file (green) + +Create `paxman/capabilities//grammar/_recognition.py`: + +1. Import `Grammar` and `RecognitionMatch` from `paxman.core.domain`, and the capability's notation. +2. Compile the regex once at module scope — never inside `recognize()` (it runs for every input). +3. Define a class extending `Grammar[Notation]` with a `name` of the form `{format}_recognition` (snake_case, unique within the capability). +4. Implement `recognize(text) -> list[RecognitionMatch[Notation]]` — one `RecognitionMatch` per `finditer()` hit, carrying the notation, the half-open `[start, end)` span, and `raw_text`. + +```python +"""Slash-ISO date grammar — recognizes YYYY/MM/DD format. + +Notation mapping: N1=year, N2=month, N3=day +""" + +from __future__ import annotations + +import re + +from paxman.capabilities.Date.notation import DateNotation +from paxman.core.domain import Grammar, RecognitionMatch + +_SLASH_ISO_PATTERN = re.compile(r"(\d{4})/(\d{1,2})/(\d{1,2})") + + +class SlashISODateGrammar(Grammar[DateNotation]): + """Slash-delimited ISO date recognition: YYYY/MM/DD.""" + + name = "slash_iso_recognition" + + def recognize(self, text: str) -> list[RecognitionMatch[DateNotation]]: + """Extract YYYY/MM/DD date patterns from text.""" + return [ + RecognitionMatch( + notation=DateNotation(N1=year, N2=month, N3=day), + start=match.start(), + end=match.end(), + raw_text=match.group(0), + ) + for match in _SLASH_ISO_PATTERN.finditer(text) + for year, month, day in [match.groups()] + ] +``` + +**The `recognize()` contract is enforced by the engine.** Every match must satisfy `0 <= start <= end <= len(text)` and `raw_text == text[start:end]`; a grammar returning a malformed match raises `RecognitionError` naming the grammar (see `_recognize` in `paxman/engine/orchestrator.py`). The engine owns within-grammar containment dedup and total recognition ordering — the grammar only extracts and emits spans. + +**Guard boundaries against sibling grammars.** When two grammars could claim the same span, use lookarounds so each claims only its own representation. The slash-ISO pattern is naturally disjoint from the US/European grammars (a leading 4-digit year vs. a leading 1–2-digit month/day) — verify that with a negative test like `test_does_not_match_us_or_european_order`. + +--- + +## Step 3: Wire the Grammar into the Capability + +A grammar must be returned by **`get_grammars()`** — and, only for the gated capabilities, named in **`active_grammars`** — or it silently never runs: + +1. **`get_grammars()`** in `paxman/capabilities//capability.py` — supplies the grammar *instances* the engine can compose. This is the one wiring step every capability needs: + +```python +def get_grammars(self) -> list[Grammar[DateNotation]]: + return [ + ISO8601DateGrammar(), + USDateGrammar(), + EuropeanDateGrammar(), + SlashISODateGrammar(), # new + ] +``` + +2. **`active_grammars`** in `paxman/capabilities//contract.py` — **only for the gated capabilities** (Email, IP, ISBN) that implement it to select grammars behind feature flags. For Date — and Country, Currency, Money, Phone, and URL — there is no `active_grammars` to update: the base contract returns `None` and the engine falls back to running every shipped grammar in `get_grammars()` order. **No contract edit is needed for these six.** + +If the capability does implement `active_grammars`, append the new name **at the end** of the list (gated behind its `include_*` flag if the capability has one): + +```python +@property +def active_grammars(self) -> list[str]: + grammars = ["standard_recognition"] + if self.include_obfuscated: + grammars.append("obfuscated_recognition") + return grammars +``` + +Recognition order (and the same-span tiebreak) follows the runnable set — `get_grammars()` order for the six all-active capabilities, the `active_grammars` list for the three gated ones — so appending at the end keeps every existing grammar's behavior byte-identical. Update the capability test asserting the grammar count (`test_get_grammars_returns_all`) and add one asserting the new name is wired. + +> **The missing-half bug (gated capabilities only).** For Email, IP, and ISBN, the engine builds the runnable set from `contract.active_grammars`, not from `get_grammars()` — a grammar returned by `get_grammars()` but missing from `active_grammars` is dead code that will pass unit tests on the class and fail silently in the pipeline. The integration test in Step 5 is what catches this. The all-active capabilities have no such hole: the engine falls back to `get_grammars()` itself. + +--- + +## Step 4: Make a Rule Validate It + +A recognition only becomes a candidate when a rule names its grammar in `target_grammars`. Two options: + +### Option A — Extend an existing rule (same meaning, same normalization) + +When the new representation means the same thing and normalizes to the same canonical form as a format an existing rule already validates (the slash-ISO date *is* ISO 8601's calendar date), extend that rule's `target_grammars`. This is the minimal change: `matches()` and `normalize()` already handle the notation. + +```python +class Section431CalendarDate(Rule[DateNotation]): + """ISO 8601 Section 4.3.1 — Calendar date. + + Validates both the dash-delimited ISO grammar and the slash-delimited + variant (which shares the same position mapping and canonical form). + """ + + name = "Section 4.3.1-calendar-date" + strategy = RuleStrategy.PARSER + provenance = PUBLICATION + citation = "Section 4.3.1 (calendar date)" + target_grammars = frozenset( + {"iso8601_recognition", "slash_iso_recognition"} # extended + ) + requires_features = frozenset() +``` + +### Option B — Add a new rule (new meaning, different normalization, or different authority) + +When the new format needs its own validation or provenance, add a rule file — one file per publication, one class per spec section (see HOW_TO_ADD_NEW_CAPABILITY.md Step 5 for the full rule template). `Rule.__init_subclass__` enforces the six metadata attributes (`name`, `strategy`, `provenance`, `citation`, `target_grammars`, `requires_features`) at class-definition time, and `target_grammars` must be a non-empty `frozenset[str]`. + +Whichever option you choose, the engine **fails fast** if you get it wrong: `_validate_affinity` raises `ContractError` when a rule's `target_grammars` names a grammar that is not in the composed set (shipped + opted-in community), so a dangling name can never silently disable a rule. + +--- + +## Step 5: Prove It Makes a Difference (integration test) + +The whole point of the grammar is observable behavior change. Write an integration test asserting the full pipeline resolves input that previously did not — this is the test that proves Steps 3 and 4 were both completed. + +Add to `tests/integration/test__capability.py` (which registers the capability and drives `paxman.canonicalize()`): + +```python +def test_slash_iso_date_resolves(self) -> None: + """YYYY/MM/DD input resolves via the slash-ISO grammar. + + Before this grammar shipped, "2024/01/01" was not recognized by any + Date grammar (US/European require a leading month/day, ISO requires + dashes) and resolved MISSING; the slash-ISO grammar makes it SUCCESS. + """ + contract = Date.create_contract() + result = paxman.canonicalize("2024/01/01", contract) + assert result.status == Resolution.SUCCESS + assert result.canonicalized_value == "2024-01-01" + assert result.candidates[0].recognition_rule == "slash_iso_recognition" + + +def test_slash_iso_invalid_month_invalid(self) -> None: + """A slash-ISO shape with an impossible month is INVALID, not resolved.""" + contract = Date.create_contract() + result = paxman.canonicalize("2024/13/01", contract) + assert result.status == Resolution.INVALID + + +def test_slash_iso_does_not_disturb_us_ambiguity(self) -> None: + """US/European slash formats still resolve exactly as before.""" + contract = Date.create_contract() + result = paxman.canonicalize("07/26/2026", contract) + assert result.status == Resolution.SUCCESS + assert result.canonicalized_value == "2026-07-26" +``` + +Three assertions to always include: + +- **Difference** — the new format now resolves (assert `recognition_rule` names your grammar, proving *your* grammar produced the candidate). +- **Semantics still hold** — impossible values under the new shape are `INVALID`, not resolved. +- **No regression** — sibling formats resolve exactly as before (your grammar claims only its own representation). + +--- + +## Step 6: Update the Documentation + +A shipped grammar is part of the capability's public surface. Update: + +- **`README.md`** — the capabilities table's **Grammars** count for the capability (e.g., `3 (ISO, US, European)` → `4 (ISO, US, European, slash-ISO)`) and the capability section's format list and examples. +- **`CONTEXT.md`** — the "Capability Details" grammar table (delimiter, component mapping, notes) for the affected capability. + +--- + +## Step 7: Quality Gates + +Before merging, run the full pre-PR gate (all must pass): + +```bash +uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run import-linter lint && uv run pytest +``` + +Project conventions that apply to new grammar code (violations fail review): + +- **No type suppression** — no `# type: ignore` / `# noqa` / `# pyright: ignore` in source; fix the root cause. +- **No cross-capability imports** — a capability imports only from `paxman.core` (enforced by import-linter). +- **No rule-layer imports in grammars** — grammars never import from `rules/` or `rules/data/` (semantic purity gate). +- **Syntax only** — no validation, dedup, ordering, or token→canonical mapping in the grammar; rules own all of that. +- **Coverage ≥ 95%** per package — new grammar and wiring must be fully covered by the Step 2/5 tests. + +--- + +## Worked Example: What This Guide Just Did + +This guide's every snippet is drawn from a real change: the **slash-ISO date grammar** (`YYYY/MM/DD`) added to the Date capability. The complete change, in order: + +| Step | File | Change | +|------|------|--------| +| 2a | `tests/capabilities/date/test_grammar.py` | `TestSlashISODateGrammar` (failing first) | +| 2b | `paxman/capabilities/Date/grammar/slash_iso_recognition.py` | New `SlashISODateGrammar` (`name = "slash_iso_recognition"`) | +| 3 | `paxman/capabilities/Date/capability.py` | `SlashISODateGrammar()` appended to `get_grammars()` | +| 3 | `paxman/capabilities/Date/contract.py` | No change — Date is all-active; the engine runs every `get_grammars()` entry | +| 4 | `paxman/capabilities/Date/rules/iso_8601_ed2019.py` | `Section431CalendarDate.target_grammars` extended to include the new grammar | +| 5 | `tests/capabilities/date/test_capability.py` | Grammar count 3 → 4; new name wired | +| 5 | `tests/integration/test_date_capability.py` | `"2024/01/01"` → `SUCCESS "2024-01-01"` (was `MISSING`) | +| 6 | `README.md`, `CONTEXT.md` | Counts, format list, grammar table row | + +**Proof it works:** `paxman.canonicalize("2024/01/01", Date.create_contract())` now returns `SUCCESS` with canonical value `2024-01-01` and `recognition_rule == "slash_iso_recognition"`; before the change the same input resolved `MISSING`. diff --git a/README.md b/README.md index d39ff2da..250daf44 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ Paxman ships with nine built-in capabilities: | Capability | Domain | Grammars | Rules | Description | |------------|--------|----------|-------|-------------| | **Email** | Email addresses | 3 (standard, obfuscated, localhost) | 2 | RFC 5322, RFC 6761 | -| **Date** | Dates | 3 (ISO, US, European) | 3 | ISO 8601, US federal, EN 50160 | +| **Date** | Dates | 4 (ISO, US, European, slash-ISO) | 3 | ISO 8601, US federal, EN 50160 | | **Country** | Country codes/names | 4 (alpha-2, alpha-3, numeric, name) | 6 | ISO 3166, CLDR | | **Currency** | Currency identifiers | 3 (code, symbol, word) | 3 | ISO 4217, CLDR | | **IP** | IP addresses | 2 (IPv4, IPv6) | 2 | RFC 791, RFC 5952 | @@ -90,7 +90,7 @@ result = paxman.canonicalize("admin@localhost", contract) ### Date Capability -Recognizes dates in ISO 8601 (`YYYY-MM-DD`), US (`MM/DD/YYYY`), and European (`DD/MM/YYYY`) formats. +Recognizes dates in ISO 8601 (`YYYY-MM-DD`), slash-ISO (`YYYY/MM/DD`), US (`MM/DD/YYYY`), and European (`DD/MM/YYYY`) formats. ```python from paxman.capabilities import Date @@ -102,6 +102,11 @@ contract = Date.create_contract() result = paxman.canonicalize("2026-01-15", contract) # → "2026-01-15" +# Slash-ISO format (4-digit year first) +contract = Date.create_contract() +result = paxman.canonicalize("2026/01/15", contract) +# → "2026-01-15" + # US/European format (potentially ambiguous) contract = Date.create_contract() result = paxman.canonicalize("01/02/2026", contract) diff --git a/capability_homogeneity_audit.md b/capability_homogeneity_audit.md index 6bd71426..4170d2f0 100644 --- a/capability_homogeneity_audit.md +++ b/capability_homogeneity_audit.md @@ -135,12 +135,21 @@ duplication and the Phone `_canonical` triplication. **Resolved 2026-08-04** by the centralize-output-format work — see the addendum below; the finding above is preserved as historical record. -### F5. `active_grammars` toggleable (Email/IP) vs static (Date/Country/Phone) — `ACCEPTED` +### F5. `active_grammars` toggleable (Email/IP) vs static (Date/Country/Phone) — `RESOLVED` Already adjudicated in plan `2026-08-02-capability-surface-homogeneity.md` and legitimized by `HOW_TO_ADD_NEW_CAPABILITY.md`. The property *shape* (`Sequence[str]`, consumed by `_recognize`) is unanimous. Low severity — not a defect. +**Resolved 2026-08-11** by making `active_grammars` optional: the base +`CapabilityContract.active_grammars` returns `None` and the engine falls back to +running every shipped grammar in `get_grammars()` order. The static all-active +overrides were deleted (Country, Currency, Date, Money, Phone, URL); only the +feature-gated capabilities (Email, IP, ISBN) keep an override. A contract that +returns `None` composes to the identical runnable set the deleted static lists +declared, so behavior is byte-identical — and a future grammar added to +`get_grammars()` activates without a contract edit. + --- ## Tier 2 — Grammar layer divergences @@ -214,7 +223,7 @@ Ranked defects (from the rule-comparison agent): | 9 | Grammar canonicalization-at-recognition spectrum | Defect-lite | | 10 | Grammar dedup / ordering inconsistent | Smell | | 11 | Dead/mislabeled data (`"EQUatorial GUINEA"`; wrong `reference_url`) | Data defect | -| 12 | F5 static vs toggleable `active_grammars` | Accepted | +| 12 | F5 static vs toggleable `active_grammars` | Resolved (2026-08-11) | | 13 | Phone `strip_separators`; Country rule-name prefix | Accepted | --- diff --git a/paxman/capabilities/AGENTS.md b/paxman/capabilities/AGENTS.md index c715ee1b..c2e7cdef 100644 --- a/paxman/capabilities/AGENTS.md +++ b/paxman/capabilities/AGENTS.md @@ -26,7 +26,7 @@ paxman/capabilities/ | Build/extend a capability | root `HOW_TO_ADD_NEW_CAPABILITY.md` (62KB spec — read first) | | Wire grammars + rules | `/capability.py` → `get_grammars()`, `get_rules()`, static `create_contract()` | | Presentation | `/capability.py` → `format_value()` — the ONLY presentation seam | -| Feature flags / active grammars | `/contract.py` → `include_*` fields + `active_grammars` property | +| Feature flags / active grammars | `/contract.py` → `include_*` fields + `active_grammars` property (optional: only Email/IP/ISBN override it; `None` = run every shipped grammar) | | Token shape | `/notation.py` | | Recognition | `/grammar/` (+ `grammar/data/` for lexicon key tables) | | Validation | `/rules/` (+ `rules/data/` for authority tables) | @@ -38,10 +38,10 @@ paxman/capabilities/ Every capability must conform to the same structural surface. `CapabilityContract` and `Rule.__init_subclass__` make most of it structural rather than documentary: - **Notation** — `@dataclass(frozen=True, slots=True)`; one `str` field per component; the sole type parameter of the capability's `Grammar[NotationT]` / `Rule[NotationT]`. -- **Contract** — `@dataclass(frozen=True)` extending `CapabilityContract`, NO `slots=True` (incompatible with the base `super()` pattern). Sets `DEFAULT_OUTPUT_FORMAT` / `OFFERED_OUTPUT_FORMATS` class vars; `capability_name` via `field(init=False)`; inherits `output_format` (always optional; base `__post_init__` resolves it and validates offered alternatives); implements `active_grammars`. +- **Contract** — `@dataclass(frozen=True)` extending `CapabilityContract`, NO `slots=True` (incompatible with the base `super()` pattern). Sets `DEFAULT_OUTPUT_FORMAT` / `OFFERED_OUTPUT_FORMATS` class vars; `capability_name` via `field(init=False)`; inherits `output_format` (always optional; base `__post_init__` resolves it and validates offered alternatives); `active_grammars` is optional — only feature-gated capabilities (Email, IP, ISBN) override it, and the base `None` default runs every shipped grammar. - **Grammar** — one file = one recognizer; `name` = `{format}_recognition` (snake_case, unique per capability); emits span-bearing `RecognitionMatch` (half-open `[start, end)`, `raw_text`); syntax-only — extraction and sanitization, never validation, dedup, ordering, or token→canonical mapping. Two sanctioned strategies: Regex (shape) and Lexicon (key-only tables, kept in `grammar/data/` apart from recognition logic); see HOW_TO for the extended set. - **Rule** — one file = one publication (module-level `PUBLICATION` provenance constant); class = one spec section; declares `name` (`Section {X.Y.Z}-{description}`), `strategy`, `provenance`, `citation`, `target_grammars` (non-empty), `requires_features` — all six enforced at import time. Rule classes sharing one publication live in the same file; authority-backed lookup tables live in `rules/data/`, separated from rule logic. -- **Feature gating — two loci, two statuses** — input-shape features toggle grammars via `active_grammars` (disabled grammar → `MISSING`); authority features gate rules via `requires_features` (dropped rule → `INVALID`). Never gate inside `matches()`; never cast to read `include_*` flags. `typing.cast` is only for validity-affecting parameters. +- **Feature gating — two loci, two statuses** — input-shape features toggle grammars via `active_grammars` (disabled grammar → `MISSING`), implemented only by the gated capabilities (Email, IP, ISBN) — other contracts inherit the `None` default, which runs every shipped grammar; authority features gate rules via `requires_features` (dropped rule → `INVALID`). Never gate inside `matches()`; never cast to read `include_*` flags. `typing.cast` is only for validity-affecting parameters. - **Presentation-only invariant** — rules never reference `output_format` (CI-scanned); `normalize()` always returns the default canonical form; `format_value()` is the only presentation seam, overridden only when `OFFERED_OUTPUT_FORMATS` is non-empty; formatting adds no provenance; offered formats must preserve the capability's ambiguity contract. - **`create_contract()`** — static, keyword-only; fixed common block first (`excluded_rules`, `pinned_rules`, `year`, `output_format`), capability-specific params after. diff --git a/paxman/capabilities/Country/contract.py b/paxman/capabilities/Country/contract.py index 9a9b4de4..ad320fa1 100644 --- a/paxman/capabilities/Country/contract.py +++ b/paxman/capabilities/Country/contract.py @@ -35,20 +35,6 @@ class CountryContract(CapabilityContract): include_localized: bool = False include_historical: bool = False - @property - def active_grammars(self) -> list[str]: - """All grammars active by default. - - Returns: - List of grammar names to activate. - """ - return [ - "alpha2_recognition", - "alpha3_recognition", - "numeric_recognition", - "name_recognition", - ] - VALID_OUTPUT_FORMATS: Final[frozenset[str]] = frozenset( {CountryContract.DEFAULT_OUTPUT_FORMAT, *CountryContract.OFFERED_OUTPUT_FORMATS} diff --git a/paxman/capabilities/Currency/contract.py b/paxman/capabilities/Currency/contract.py index 5e98f7ea..29ff03fe 100644 --- a/paxman/capabilities/Currency/contract.py +++ b/paxman/capabilities/Currency/contract.py @@ -81,15 +81,3 @@ def __post_init__(self) -> None: """ super().__post_init__() _validate_alpha3(self.default_currency) - - @property - def active_grammars(self) -> tuple[str, ...]: - """All grammars active by default. - - All three recognition grammars are always active; Currency has no - input-shape feature flags. - - Returns: - The three recognition grammar names. - """ - return ("code_recognition", "symbol_recognition", "word_recognition") diff --git a/paxman/capabilities/Date/capability.py b/paxman/capabilities/Date/capability.py index b918220d..fa0650c3 100644 --- a/paxman/capabilities/Date/capability.py +++ b/paxman/capabilities/Date/capability.py @@ -12,6 +12,9 @@ from paxman.capabilities.Date.grammar.iso8601_recognition import ( ISO8601DateGrammar, ) +from paxman.capabilities.Date.grammar.slash_iso_recognition import ( + SlashISODateGrammar, +) from paxman.capabilities.Date.grammar.us_recognition import USDateGrammar from paxman.capabilities.Date.notation import DateNotation from paxman.capabilities.Date.rules.en_50160_ed2010 import Section4DateFormat @@ -36,6 +39,7 @@ def get_grammars(self) -> list[Grammar[DateNotation]]: ISO8601DateGrammar(), USDateGrammar(), EuropeanDateGrammar(), + SlashISODateGrammar(), ] def get_rules(self) -> list[Rule[DateNotation]]: diff --git a/paxman/capabilities/Date/contract.py b/paxman/capabilities/Date/contract.py index 1c5cf508..da834a31 100644 --- a/paxman/capabilities/Date/contract.py +++ b/paxman/capabilities/Date/contract.py @@ -23,11 +23,3 @@ class DateContract(CapabilityContract): capability_name: str = field(default="date", init=False) two_digit_base_year: int | None = None - - @property - def active_grammars(self) -> list[str]: - return [ - "iso8601_recognition", - "us_recognition", - "european_recognition", - ] diff --git a/paxman/capabilities/Date/grammar/slash_iso_recognition.py b/paxman/capabilities/Date/grammar/slash_iso_recognition.py new file mode 100644 index 00000000..3c9e407b --- /dev/null +++ b/paxman/capabilities/Date/grammar/slash_iso_recognition.py @@ -0,0 +1,41 @@ +"""Slash-ISO date grammar — recognizes YYYY/MM/DD format. + +Notation mapping: N1=year, N2=month, N3=day +""" + +from __future__ import annotations + +import re + +from paxman.capabilities.Date.notation import DateNotation +from paxman.core.domain import Grammar, RecognitionMatch + +_SLASH_ISO_PATTERN = re.compile(r"(\d{4})/(\d{1,2})/(\d{1,2})") + + +class SlashISODateGrammar(Grammar[DateNotation]): + """Slash-delimited ISO date recognition: YYYY/MM/DD. + + Shares the ISO 8601 position mapping (N1=year, N2=month, N3=day) with a + ``/`` delimiter instead of ``-``; single-digit month/day components are + accepted and zero-padded by the validating rule. The leading 4-digit year + keeps the pattern disjoint from the US and European grammars, which both + require a leading month/day. + + Notation mapping: N1=year, N2=month, N3=day + """ + + name = "slash_iso_recognition" + + def recognize(self, text: str) -> list[RecognitionMatch[DateNotation]]: + """Extract YYYY/MM/DD date patterns from text.""" + return [ + RecognitionMatch( + notation=DateNotation(N1=year, N2=month, N3=day), + start=match.start(), + end=match.end(), + raw_text=match.group(0), + ) + for match in _SLASH_ISO_PATTERN.finditer(text) + for year, month, day in [match.groups()] + ] diff --git a/paxman/capabilities/Date/rules/iso_8601_ed2019.py b/paxman/capabilities/Date/rules/iso_8601_ed2019.py index 0d23da22..7f49a6b1 100644 --- a/paxman/capabilities/Date/rules/iso_8601_ed2019.py +++ b/paxman/capabilities/Date/rules/iso_8601_ed2019.py @@ -22,6 +22,9 @@ class Section431CalendarDate(Rule[DateNotation]): """ISO 8601 Section 4.3.1 — Calendar date. + Validates both the dash-delimited ISO grammar and the slash-delimited + variant (which shares the same position mapping and canonical form). + Notation mapping (ISO 8601 grammar): N1 = year, N2 = month, N3 = day """ @@ -30,7 +33,7 @@ class Section431CalendarDate(Rule[DateNotation]): strategy = RuleStrategy.PARSER provenance = PUBLICATION citation = "Section 4.3.1 (calendar date)" - target_grammars = frozenset({"iso8601_recognition"}) + target_grammars = frozenset({"iso8601_recognition", "slash_iso_recognition"}) requires_features = frozenset() def matches(self, notation: DateNotation, contract: Contract) -> bool: diff --git a/paxman/capabilities/Money/contract.py b/paxman/capabilities/Money/contract.py index d20fbf49..7b89e50c 100644 --- a/paxman/capabilities/Money/contract.py +++ b/paxman/capabilities/Money/contract.py @@ -95,15 +95,3 @@ def __post_init__(self) -> None: f"got {self.precision!r}" ) _validate_alpha3(self.dollar_sign_currency) - - @property - def active_grammars(self) -> tuple[str, ...]: - """All grammars active by default. - - All three recognition grammars are always active; Money has no - input-shape feature flags. - - Returns: - The three recognition grammar names. - """ - return ("code_recognition", "symbol_recognition", "word_recognition") diff --git a/paxman/capabilities/Phone/contract.py b/paxman/capabilities/Phone/contract.py index 9455252c..e322d599 100644 --- a/paxman/capabilities/Phone/contract.py +++ b/paxman/capabilities/Phone/contract.py @@ -90,20 +90,3 @@ def __post_init__(self) -> None: """ super().__post_init__() _validate_alpha2(self.default_country) - - @property - def active_grammars(self) -> list[str]: - """All grammars active by default. - - All grammars are cheap regex scans; rules filter by shape and by - contract parameters (e.g., national rules gate on default_country). - - Returns: - List of grammar names to activate. - """ - return [ - "e164_recognition", - "tel_uri_recognition", - "international_00_recognition", - "national_recognition", - ] diff --git a/paxman/capabilities/URL/contract.py b/paxman/capabilities/URL/contract.py index db23e305..5d59fb91 100644 --- a/paxman/capabilities/URL/contract.py +++ b/paxman/capabilities/URL/contract.py @@ -22,12 +22,3 @@ class URLCapabilityContract(CapabilityContract): OFFERED_OUTPUT_FORMATS: ClassVar[frozenset[str]] = frozenset() capability_name: str = field(default="url", init=False) - - @property - def active_grammars(self) -> tuple[str, ...]: - """All grammars active by default. - - Returns: - The single recognition grammar name. - """ - return ("absolute_uri_recognition",) diff --git a/paxman/core/AGENTS.md b/paxman/core/AGENTS.md index 29feb363..87239351 100644 --- a/paxman/core/AGENTS.md +++ b/paxman/core/AGENTS.md @@ -24,7 +24,7 @@ Core owns the domain vocabulary (pipeline value objects + `Rule`/`Grammar` ABCs) - **Value objects** (`domain.py`): `@dataclass(frozen=True, slots=True)`. Spans are half-open with `len(raw_text) == end - start`, enforced in `__post_init__`. `GrammarRule` enforces lowercase names; `RecognizedRep.__hash__` handles unhashable list notations; `Candidate._provenance` is `init=False` and tuple-ized in `__init__`. - **`Rule` subclasses** must declare `name`, `strategy`, `provenance`, `citation`, `target_grammars` (non-empty `frozenset[str]`), `requires_features` (`frozenset[str]`) as class attrs. `Rule.__init_subclass__` raises `TypeError` at import time for missing/mistyped metadata — keep it a hard import-time failure. `matches()`/`normalize()` never raise. - **`Grammar` subclasses** must declare `name`; `recognize()` returns span-bearing `RecognitionMatch` only, never bare notation. -- **Contracts:** subclass `CapabilityContract`, never `Contract` directly. Set `DEFAULT_OUTPUT_FORMAT`/`OFFERED_OUTPUT_FORMATS`, set `capability_name` via `field(default=..., init=False)`, implement `active_grammars`. +- **Contracts:** subclass `CapabilityContract`, never `Contract` directly. Set `DEFAULT_OUTPUT_FORMAT`/`OFFERED_OUTPUT_FORMATS`, set `capability_name` via `field(default=..., init=False)`. `active_grammars` is optional: it returns `None` by default (the engine then runs every shipped `get_grammars()` entry); only feature-gated capabilities (Email, IP, ISBN) override it. - **`output_format`** is always optional: `None`/`"default"`/`DEFAULT_OUTPUT_FORMAT` resolve to the default, offered formats resolve to themselves, anything else raises `ContractError`. Resolved once in `CapabilityContract.__post_init__`; subclasses with their own `__post_init__` call `super().__post_init__()` first. Note: `resolve_output_format` is imported lazily there to break the `capability_contract` ↔ `contract` import cycle. - **`pinned_rules` wins over `excluded_rules`** (non-`None` pins; empty tuple pins to nothing); `year` filtering still applies after pinning. - **Registry** (`discovery.py`): module-level `_registry` dict + `_frozen` flag. `register_capability()` takes `Any` and isinstance-checks `Capability`; rejects dupes and post-freeze adds with `CapabilityError`. `freeze_registry()` is called by the engine at pipeline start; `reset_registry()` is TESTING ONLY (autouse fixtures). diff --git a/paxman/core/capability_contract.py b/paxman/core/capability_contract.py index 625602ae..7050f690 100644 --- a/paxman/core/capability_contract.py +++ b/paxman/core/capability_contract.py @@ -10,7 +10,7 @@ from __future__ import annotations -from abc import ABC, abstractmethod +from abc import ABC from collections.abc import Sequence from dataclasses import dataclass, field from typing import ClassVar @@ -24,18 +24,21 @@ class CapabilityContract(ABC): - ``output_format`` is **always optional** and validated/normalized by a single base ``__post_init__`` via :func:`resolve_output_format`. - - ``active_grammars`` is capability-specific and must be implemented by - each subclass. + - ``active_grammars`` is optional: the base-class default (``None``) + activates every shipped grammar in ``get_grammars()`` declaration + order, so adding a grammar to a capability is a single-edit change. Subclasses MUST: - Override the ``DEFAULT_OUTPUT_FORMAT`` and ``OFFERED_OUTPUT_FORMATS`` class variables. - Set ``capability_name`` via ``field(default="", init=False)``. - - Implement the abstract ``active_grammars`` property. - Call ``super().__post_init__()`` first if they add their own ``__post_init__`` validation. + Subclasses MAY override ``active_grammars`` to narrow the surface — + typically to gate grammars behind ``include_*`` feature flags. + The class satisfies the :class:`Contract` protocol structurally. """ @@ -81,7 +84,12 @@ def __post_init__(self) -> None: ) @property - @abstractmethod - def active_grammars(self) -> Sequence[str]: - """Grammar names to activate.""" - ... + def active_grammars(self) -> Sequence[str] | None: + """Grammar names to activate. + + ``None`` — the base-class default — tells the engine to activate + every shipped grammar in the capability's ``get_grammars()`` + declaration order. Override to narrow the surface (e.g. gate + grammars behind ``include_*`` feature flags). + """ + return None diff --git a/paxman/core/contract.py b/paxman/core/contract.py index 08f18526..784fe433 100644 --- a/paxman/core/contract.py +++ b/paxman/core/contract.py @@ -17,8 +17,13 @@ def capability_name(self) -> str: ... @property - def active_grammars(self) -> Sequence[str]: - """Grammar names to activate.""" + def active_grammars(self) -> Sequence[str] | None: + """Grammar names to activate. + + ``None`` (the base-class default) means "all shipped grammars, in + ``get_grammars()`` declaration order" — the engine falls back to the + capability's shipped grammar list. + """ ... @property diff --git a/paxman/engine/orchestrator.py b/paxman/engine/orchestrator.py index 2c89b35a..ffa81383 100644 --- a/paxman/engine/orchestrator.py +++ b/paxman/engine/orchestrator.py @@ -56,15 +56,21 @@ def run_capability(text: str, contract: Contract) -> ExecutionResult: freeze_registry() capability = get_capability(contract.capability_name) + shipped_grammars = capability.get_grammars() all_grammars = [ - *capability.get_grammars(), + *shipped_grammars, *get_extended_grammars(capability.name), ] _assert_unique_names("grammar", all_grammars) all_rules = [*capability.get_rules(), *_activated_rules(capability, contract)] _assert_unique_names("rule", all_rules) _validate_affinity(all_grammars, all_rules) - recognitions = _recognize(text, all_grammars, contract) + recognitions = _recognize( + text, + all_grammars, + [g.name for g in shipped_grammars], + contract, + ) had_recognitions = len(recognitions) > 0 rules = _filter_rules(all_rules, contract) @@ -97,7 +103,10 @@ def _assert_unique_names(kind: str, items: Sequence[Grammar[Any] | Rule[Any]]) - def _recognize( - text: str, all_grammars: Sequence[Grammar[Any]], contract: Contract + text: str, + all_grammars: Sequence[Grammar[Any]], + shipped_names: Sequence[str], + contract: Contract, ) -> list[RecognizedRep[Any]]: """Run active grammars, dedup contained matches per grammar, and order. @@ -108,12 +117,20 @@ def _recognize( strictly within a single grammar's output (never across grammars, so cross-grammar ambiguity stays observable), and recognitions are emitted in the total order (start, end, active set index, grammar name) where the - index follows the composed active set: ``contract.active_grammars`` first, - then any opt-in ``contract.extra_grammars`` names (unknown extra names are - silently skipped, D4). + index follows the composed active set: ``contract.active_grammars`` + first, then any opt-in ``contract.extra_grammars`` names (unknown extra + names are silently skipped, D4). + + A contract whose ``active_grammars`` is ``None`` (the base-class default) + falls back to ``shipped_names`` — every shipped grammar in + ``get_grammars()`` declaration order — so adding a shipped grammar to a + capability activates it with no contract edit. Community grammars stay + opt-in via ``extra_grammars`` in both cases. """ supported_names = {g.name for g in all_grammars} extra_grammars = getattr(contract, "extra_grammars", ()) + declared = contract.active_grammars + active_source = shipped_names if declared is None else declared # Deduplicate contract names, keeping first occurrence: each supported # grammar runs at most once and grammar_index stays aligned with # active_grammars (a duplicate contract entry must not double-run it). @@ -121,9 +138,7 @@ def _recognize( # order after the shipped slots. active_names = list( dict.fromkeys( - n - for n in [*contract.active_grammars, *extra_grammars] - if n in supported_names + n for n in [*active_source, *extra_grammars] if n in supported_names ) ) grammar_index = {name: i for i, name in enumerate(active_names)} diff --git a/tests/capabilities/country/test_capability.py b/tests/capabilities/country/test_capability.py index a17187f2..6250e0cf 100644 --- a/tests/capabilities/country/test_capability.py +++ b/tests/capabilities/country/test_capability.py @@ -96,15 +96,20 @@ def test_default_include_historical(self) -> None: contract = CountryContract() assert contract.include_historical is False - def test_active_grammars_returns_all(self) -> None: - """Verify all 4 grammars are active by default.""" + def test_active_grammars_defaults_to_all_shipped(self) -> None: + """No override: the engine runs every shipped grammar (fallback). + + The contract declares no active_grammars (None — the base default), + and the capability ships exactly the four recognition grammars. + """ contract = CountryContract() - grammars = contract.active_grammars - assert len(grammars) == 4 - assert "alpha2_recognition" in grammars - assert "alpha3_recognition" in grammars - assert "numeric_recognition" in grammars - assert "name_recognition" in grammars + assert contract.active_grammars is None + assert [g.name for g in CountryCapability().get_grammars()] == [ + "alpha2_recognition", + "alpha3_recognition", + "numeric_recognition", + "name_recognition", + ] def test_is_frozen(self) -> None: """Verify immutability.""" diff --git a/tests/capabilities/currency/test_contract.py b/tests/capabilities/currency/test_contract.py index 1e61a849..713cfe60 100644 --- a/tests/capabilities/currency/test_contract.py +++ b/tests/capabilities/currency/test_contract.py @@ -7,6 +7,7 @@ import pytest +from paxman.capabilities.Currency.capability import CurrencyCapability from paxman.capabilities.Currency.contract import CurrencyContract from paxman.core.errors import ContractError @@ -17,12 +18,14 @@ def test_capability_name() -> None: assert CurrencyContract().capability_name == "currency" -def test_active_grammars() -> None: - assert CurrencyContract().active_grammars == ( +def test_active_grammars_defaults_to_all_shipped() -> None: + """No override: the engine runs every shipped grammar (fallback).""" + assert CurrencyContract().active_grammars is None + assert [g.name for g in CurrencyCapability().get_grammars()] == [ "code_recognition", "symbol_recognition", "word_recognition", - ) + ] def test_default_output_format_resolution() -> None: diff --git a/tests/capabilities/date/test_capability.py b/tests/capabilities/date/test_capability.py index 35a6eb74..5880100d 100644 --- a/tests/capabilities/date/test_capability.py +++ b/tests/capabilities/date/test_capability.py @@ -63,7 +63,13 @@ def test_version(self) -> None: def test_get_grammars_returns_all(self) -> None: cap = DateCapability() - assert len(cap.get_grammars()) == 3 + assert len(cap.get_grammars()) == 4 + + def test_get_grammars_includes_slash_iso(self) -> None: + """The slash-ISO grammar is wired into the capability.""" + cap = DateCapability() + names = {g.name for g in cap.get_grammars()} + assert "slash_iso_recognition" in names def test_get_rules_returns_all(self) -> None: cap = DateCapability() diff --git a/tests/capabilities/date/test_grammar.py b/tests/capabilities/date/test_grammar.py index eb73f086..0dae85ce 100644 --- a/tests/capabilities/date/test_grammar.py +++ b/tests/capabilities/date/test_grammar.py @@ -10,6 +10,9 @@ from paxman.capabilities.Date.grammar.iso8601_recognition import ( ISO8601DateGrammar, ) +from paxman.capabilities.Date.grammar.slash_iso_recognition import ( + SlashISODateGrammar, +) from paxman.capabilities.Date.grammar.us_recognition import USDateGrammar @@ -134,3 +137,57 @@ def test_emits_spans(self) -> None: @property def grammar(self) -> EuropeanDateGrammar: return EuropeanDateGrammar() + + +@pytest.mark.capability +class TestSlashISODateGrammar: + """Tests for slash-ISO date grammar (YYYY/MM/DD).""" + + def test_recognizes_valid_input(self) -> None: + grammar = SlashISODateGrammar() + result = grammar.recognize("2026/07/26") + assert len(result) == 1 + assert result[0].notation.as_list() == ["2026", "07", "26"] + + def test_recognizes_single_digit_components(self) -> None: + grammar = SlashISODateGrammar() + result = grammar.recognize("2026/7/6") + assert len(result) == 1 + assert result[0].notation.as_list() == ["2026", "7", "6"] + + def test_recognizes_multiple(self) -> None: + grammar = SlashISODateGrammar() + result = grammar.recognize("Dates: 2026/07/26 and 2025/12/31") + assert len(result) == 2 + + def test_returns_empty_for_empty_input(self) -> None: + grammar = SlashISODateGrammar() + result = grammar.recognize("") + assert result == [] + + def test_returns_empty_for_no_match(self) -> None: + grammar = SlashISODateGrammar() + result = grammar.recognize("No dates here") + assert result == [] + + def test_does_not_match_us_or_european_order(self) -> None: + """A 2-digit-first slash date is not a slash-ISO date.""" + grammar = SlashISODateGrammar() + assert grammar.recognize("07/26/2026") == [] + assert grammar.recognize("26/07/2026") == [] + + def test_grammar_name(self) -> None: + grammar = SlashISODateGrammar() + assert grammar.name == "slash_iso_recognition" + + def test_emits_spans(self) -> None: + result = self.grammar.recognize("x 2026/07/26 y") + assert len(result) == 1 + assert result[0].start == 2 + assert result[0].end == 12 + assert result[0].raw_text == "2026/07/26" + assert result[0].notation.as_list() == ["2026", "07", "26"] + + @property + def grammar(self) -> SlashISODateGrammar: + return SlashISODateGrammar() diff --git a/tests/capabilities/money/test_contract.py b/tests/capabilities/money/test_contract.py index 769f5d44..42c50fbe 100644 --- a/tests/capabilities/money/test_contract.py +++ b/tests/capabilities/money/test_contract.py @@ -5,6 +5,7 @@ import pytest +from paxman.capabilities.Money.capability import MoneyCapability from paxman.capabilities.Money.contract import MoneyContract from paxman.core.errors import ContractError @@ -33,13 +34,14 @@ def test_dollar_sign_currency_default(self) -> None: """dollar_sign_currency defaults to None (bare $ stays INVALID).""" assert MoneyContract().dollar_sign_currency is None - def test_active_grammars(self) -> None: - """All three recognition grammars are active by default.""" - assert MoneyContract().active_grammars == ( + def test_active_grammars_defaults_to_all_shipped(self) -> None: + """No override: the engine runs every shipped grammar (fallback).""" + assert MoneyContract().active_grammars is None + assert [g.name for g in MoneyCapability().get_grammars()] == [ "code_recognition", "symbol_recognition", "word_recognition", - ) + ] def test_frozen(self) -> None: """Assigning a field raises FrozenInstanceError.""" diff --git a/tests/capabilities/phone/test_capability.py b/tests/capabilities/phone/test_capability.py index 3874686c..4fb4572a 100644 --- a/tests/capabilities/phone/test_capability.py +++ b/tests/capabilities/phone/test_capability.py @@ -113,15 +113,16 @@ def test_custom_output_format(self) -> None: contract = PhoneContract(output_format="rfc3966") assert contract.output_format == "rfc3966" - def test_active_grammars_returns_all(self) -> None: - """Verify all 4 grammars are active by default.""" + def test_active_grammars_defaults_to_all_shipped(self) -> None: + """No override: the engine runs every shipped grammar (fallback).""" contract = PhoneContract() - grammars = contract.active_grammars - assert len(grammars) == 4 - assert "e164_recognition" in grammars - assert "tel_uri_recognition" in grammars - assert "international_00_recognition" in grammars - assert "national_recognition" in grammars + assert contract.active_grammars is None + assert [g.name for g in PhoneCapability().get_grammars()] == [ + "e164_recognition", + "tel_uri_recognition", + "international_00_recognition", + "national_recognition", + ] def test_is_frozen(self) -> None: """Verify immutability.""" diff --git a/tests/capabilities/url/test_capability.py b/tests/capabilities/url/test_capability.py index e0e0fdc8..b0d75862 100644 --- a/tests/capabilities/url/test_capability.py +++ b/tests/capabilities/url/test_capability.py @@ -81,7 +81,7 @@ def test_create_contract_defaults(self) -> None: output_format="url", ) assert contract.capability_name == "url" - assert contract.active_grammars == ("absolute_uri_recognition",) + assert contract.active_grammars is None def test_create_contract_excludes_rule(self) -> None: """Excluded rules propagate into the contract.""" diff --git a/tests/capabilities/url/test_contract.py b/tests/capabilities/url/test_contract.py index 9f6976bb..c80960ab 100644 --- a/tests/capabilities/url/test_contract.py +++ b/tests/capabilities/url/test_contract.py @@ -16,14 +16,14 @@ class TestURLCapabilityContractDefaults: """Default field values.""" def test_defaults(self) -> None: - """Every standard field takes its base default; grammar is the only one.""" + """Every standard field takes its base default; active_grammars is None.""" contract = URLCapabilityContract() assert contract.capability_name == "url" assert contract.excluded_rules == () assert contract.pinned_rules is None assert contract.year is None assert contract.output_format == "url" - assert contract.active_grammars == ("absolute_uri_recognition",) + assert contract.active_grammars is None def test_frozen(self) -> None: """Assigning a field raises FrozenInstanceError.""" diff --git a/tests/integration/test_date_capability.py b/tests/integration/test_date_capability.py index 4a792d0f..3792d16a 100644 --- a/tests/integration/test_date_capability.py +++ b/tests/integration/test_date_capability.py @@ -114,3 +114,36 @@ def test_pinned_iso_rule_with_us_output_format(self) -> None: assert result.status == Resolution.SUCCESS assert result.canonicalized_value == "01/15/2026" assert result.candidates[0].value == "01/15/2026" + + def test_slash_iso_date_resolves(self) -> None: + """YYYY/MM/DD input resolves via the slash-ISO grammar. + + Before this grammar shipped, "2024/01/01" was not recognized by any + Date grammar (US/European require a leading month/day, ISO requires + dashes) and resolved MISSING; the slash-ISO grammar makes it SUCCESS. + """ + contract = Date.create_contract() + result = paxman.canonicalize("2024/01/01", contract) + assert result.status == Resolution.SUCCESS + assert result.canonicalized_value == "2024-01-01" + assert result.candidates[0].recognition_rule == "slash_iso_recognition" + + def test_slash_iso_single_digit_components(self) -> None: + """Single-digit month/day are zero-padded in the canonical value.""" + contract = Date.create_contract() + result = paxman.canonicalize("2024/1/5", contract) + assert result.status == Resolution.SUCCESS + assert result.canonicalized_value == "2024-01-05" + + def test_slash_iso_invalid_month_invalid(self) -> None: + """A slash-ISO shape with an impossible month is INVALID, not resolved.""" + contract = Date.create_contract() + result = paxman.canonicalize("2024/13/01", contract) + assert result.status == Resolution.INVALID + + def test_slash_iso_does_not_disturb_us_ambiguity(self) -> None: + """US/European slash formats still resolve exactly as before.""" + contract = Date.create_contract() + result = paxman.canonicalize("07/26/2026", contract) + assert result.status == Resolution.SUCCESS + assert result.canonicalized_value == "2026-07-26" diff --git a/tests/integration/test_recognition_seam.py b/tests/integration/test_recognition_seam.py index d4f09e3e..045984df 100644 --- a/tests/integration/test_recognition_seam.py +++ b/tests/integration/test_recognition_seam.py @@ -32,6 +32,7 @@ Rule, RuleStrategy, ) +from paxman.core.extensions import register_grammar, register_rule from paxman.engine.orchestrator import run_capability @@ -144,19 +145,33 @@ def get_rules(self) -> list[Rule[_ProbeNotation]]: class _ProbeContract: - """Minimal contract; default active_grammars order is long=0, short=1.""" + """Minimal contract. - def __init__(self, active_grammars: list[str] | None = None) -> None: - self._active_grammars = active_grammars or ["probe_long", "probe_short"] + ``active_grammars=None`` (the default) exercises the engine fallback: + the capability's shipped ``get_grammars()`` names run in declaration + order. Passing an explicit list overrides the fallback. + """ + + def __init__( + self, + active_grammars: list[str] | None = None, + extra_grammars: tuple[str, ...] = (), + ) -> None: + self._active_grammars = active_grammars + self._extra_grammars = extra_grammars @property def capability_name(self) -> str: return "probe" @property - def active_grammars(self) -> list[str]: + def active_grammars(self) -> list[str] | None: return self._active_grammars + @property + def extra_grammars(self) -> tuple[str, ...]: + return self._extra_grammars + @property def excluded_rules(self) -> list[str]: return [] @@ -321,3 +336,94 @@ def get_grammars(self) -> list[Grammar[_ProbeNotation]]: assert calls == ["probe_long"] assert result.status == Resolution.SUCCESS assert result.canonicalized_value == "L:AAAA" + + +class TestDefaultGrammarSurface: + """The engine fallback when a contract omits ``active_grammars``. + + Option B: ``active_grammars`` becomes optional. A contract returning + ``None`` (the base-class default) activates every **shipped** grammar in + ``get_grammars()`` declaration order — so adding a new shipped grammar + activates it with no contract edit. Community grammars stay opt-in via + ``extra_grammars``. + """ + + @pytest.mark.integration + def test_contract_without_active_grammars_runs_all_shipped(self) -> None: + """The None fallback activates every shipped grammar, in order.""" + register_capability(_ProbeCapability()) + result = run_capability("AA AAAA", _ProbeContract()) + + # Both shipped grammars ran via the fallback; same-span tiebreak + # follows get_grammars() order (long=0, short=1). + assert [c.value for c in result.candidates] == [ + "L:AA", + "S:AA", + "L:AAAA", + ] + + @pytest.mark.integration + def test_fallback_never_activates_community_grammars(self) -> None: + """The fallback covers shipped grammars only; community stays opt-in. + + This distinguishes "fallback to shipped get_grammars()" from a naive + "fallback to every supported grammar": a registered community grammar + must NOT run under the None fallback — only ``extra_grammars`` opts + it in (D4). + """ + calls: list[str] = [] + + class _CommunityGrammar(Grammar[_ProbeNotation]): + name = "probe_community" + + def recognize(self, text: str) -> list[RecognitionMatch[_ProbeNotation]]: + calls.append(self.name) + return [ + RecognitionMatch( + notation=_ProbeNotation("COMMUNITY"), + start=0, + end=len(text), + raw_text=text, + ) + ] + + class _CommunityRule(Rule[_ProbeNotation]): + name = "community_rule" + strategy = RuleStrategy.REGEX + provenance = Provenance( + authority="test", + specification_name="test", + kind="test", + reference_url="https://test", + version=None, + lifecycle="active", + publication_year=2024, + ) + citation = "test" + target_grammars = frozenset({"probe_community"}) + requires_features = frozenset() + + def matches(self, notation: _ProbeNotation, contract: Contract) -> bool: + return True + + def normalize(self, notation: _ProbeNotation, contract: Contract) -> str: + return "COMMUNITY" + + register_grammar("probe", _CommunityGrammar) + register_rule("probe", _CommunityRule) + register_capability(_ProbeCapability()) + + # "XXXX" matches no shipped grammar, so any recognition here could + # only come from the community grammar — the fallback must not run it. + result = run_capability("XXXX", _ProbeContract()) + assert calls == [] + assert result.status == Resolution.MISSING + assert result.candidates == () + + # Same contract opted in via extra_grammars: community grammar runs. + result = run_capability( + "XXXX", _ProbeContract(extra_grammars=("probe_community",)) + ) + assert calls == ["probe_community"] + assert result.status == Resolution.SUCCESS + assert result.canonicalized_value == "COMMUNITY" diff --git a/tests/unit/test_capability_contract.py b/tests/unit/test_capability_contract.py index 0e9b80b0..9782fc97 100644 --- a/tests/unit/test_capability_contract.py +++ b/tests/unit/test_capability_contract.py @@ -42,7 +42,7 @@ def active_grammars(self) -> Sequence[str]: @dataclass(frozen=True) class _MissingActiveGrammars(CapabilityContract): - """Subclass that forgets the abstract ``active_grammars`` property.""" + """Subclass that omits the optional ``active_grammars`` property.""" DEFAULT_OUTPUT_FORMAT: ClassVar[str] = "test" OFFERED_OUTPUT_FORMATS: ClassVar[frozenset[str]] = frozenset({"alt"}) @@ -113,10 +113,15 @@ def test_base_class_exposes_no_as_dict_method(self) -> None: assert not hasattr(_TestContract(), "as_dict") @pytest.mark.unit - def test_active_grammars_is_abstract(self) -> None: - """CapabilityContract cannot be instantiated without active_grammars.""" - with pytest.raises(TypeError): - _MissingActiveGrammars() + def test_active_grammars_defaults_to_none(self) -> None: + """active_grammars is optional; the default (None) means 'all shipped'. + + A contract that omits the override is instantiable, and the engine + falls back to the capability's shipped grammar list — adding a + grammar to ``get_grammars()`` activates it with no contract edit. + """ + contract = _MissingActiveGrammars() + assert contract.active_grammars is None @pytest.mark.unit def test_instance_satisfies_contract_protocol(self) -> None: