Skip to content
8 changes: 5 additions & 3 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,9 @@ all cross-match policy:
same span (e.g. US vs European date reading of `01/02/2026`) are both
preserved and ambiguity stays observable.
- **Ordering:** recognitions are emitted in the total order
`(start, end, active_grammars index, grammar name)`, i.e. document order.
`(start, end, active-set index, grammar name)`, i.e. document order — where the
active set is `contract.active_grammars` or, when the contract returns `None`,
every shipped grammar in `get_grammars()` order.
- **Candidate dedup** (`value, recognition_rule, validation_rule`) runs
after validation as a stability net.

Expand Down Expand Up @@ -128,7 +130,7 @@ Contracts pass configuration parameters to validation rules, enabling rules to a
**Base Contract Parameters:**
- **`output_format`**: Controls the canonical value format (e.g., `"ISO"` for `YYYY-MM-DD`, `"US"` for `MM/DD/YYYY`). `CapabilityContract.__post_init__` resolves `None`, `"default"`, and each capability's default format to a concrete string; the capability's `format_value()` seam applies the format to the rule-produced default canonical value. Validation rules never inspect `output_format` — they always normalize to the default canonical form. See "The Formatting Seam" below.
- **`pinned_rules`**: Pins to specific validation rules by name. When set, ONLY those rules run — `excluded_rules` is ignored. Takes precedence over `excluded_rules`.
- **`extra_grammars`**: Names community grammars (opt-in) to run alongside the capability's shipped `active_grammars`, in order. Unknown names are silently skipped; shipped names listed here are deduplicated. Registration happens through `paxman.register_grammar` / `paxman.register_rule` (see "Community Extensions" below).
- **`extra_grammars`**: Names community grammars (opt-in) to run alongside the capability's shipped active set — `contract.active_grammars`, or every shipped grammar when the contract returns `None` — in order. Unknown names are silently skipped; shipped names listed here are deduplicated. Registration happens through `paxman.register_grammar` / `paxman.register_rule` (see "Community Extensions" below).

**Date-Specific Parameters:**
- **`two_digit_base_year`**: Specifies the base year for interpreting two-digit years (e.g., `2000` means `"26"` becomes `2026`). Only available on Date contracts, not part of the base Contract protocol. Used by US and European grammars to resolve ambiguous year values.
Expand Down Expand Up @@ -167,7 +169,7 @@ Every stage is a pure function of its inputs — no clocks, no randomness, no en

Capabilities are closed for modification but open for extension. Community contributors register additional grammars (and the rules that validate them) against an existing capability through `paxman.core.extensions` — never by editing the capability package. The registries freeze with the capability registry at the first pipeline run.

A contract opts a registered grammar in by naming it in `extra_grammars`, a base `CapabilityContract` field surfaced on every `create_contract` factory. The engine composes the shipped active set with the opted-in extras, deduplicating names while preserving order — shipped `active_grammars` slots first, extras after (unknown extra names are silently skipped). Opt-in preserves determinism: a contract that names no extras composes to exactly the shipped set, so non-opt-in behavior is byte-identical.
A contract opts a registered grammar in by naming it in `extra_grammars`, a base `CapabilityContract` field surfaced on every `create_contract` factory. The engine composes the shipped active set with the opted-in extras, deduplicating names while preserving order — shipped slots first, extras after (unknown extra names are silently skipped). The shipped slots are `contract.active_grammars` when the contract implements it (the gated capabilities), or every shipped grammar in `get_grammars()` order when it returns `None` (the base default). Opt-in preserves determinism: a contract that names no extras composes to exactly the shipped set, so non-opt-in behavior is byte-identical.

Community rules follow the same opt-in discipline: a registered rule runs only when the contract names one of its `target_grammars` in `extra_grammars`. An un-opted community rule — even one targeting a shipped grammar — never affects results, so a default contract resolves with shipped rules only.

Expand Down
15 changes: 8 additions & 7 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ Syntactic extraction rules that:
- Scan raw text for patterns
- Produce **span-bearing `RecognitionMatch` objects** (notation + half-open `[start, end)` span + matched `raw_text`) — never bare notations
- Live in `capabilities/<CapabilityName>/grammar/`
- Are **filtered by the orchestrator** based on the contract's `active_grammars`
- Are **selected by the orchestrator** from the contract's `active_grammars`, or from every shipped `get_grammars()` entry when the contract returns `None` (the base default)
- Do NOT validate, de-duplicate, or order — the engine owns containment dedup and document ordering
- Only recognize (syntax, shape, lexicon keys) — never map tokens to canonical values, never import rule-layer data

Expand Down Expand Up @@ -283,17 +283,18 @@ class StandardEmailGrammar(Grammar[EmailNotation]):

### Date Capability Details

The Date capability has **3 grammars** and **3 validation rules**:
The Date capability has **4 grammars** and **3 validation rules**:

#### Grammars (Recognition)

| Grammar | Delimiter | N1 (first) | N2 (second) | N3 (third) | Notes |
|---------|-----------|------------|-------------|------------|-------|
| ISO | `-` | year | month | day | 4-digit year only |
| Slash-ISO | `/` | year | month | day | 4-digit year; shares ISO position mapping |
| US | `/` | month | day | year | Supports 2-digit years |
| European | `/` | day | month | year | Supports 2-digit years |

**Note:** European and US grammars both use `/` as delimiter. The ambiguity arises from different position mappings, not delimiters.
**Note:** European, US, and slash-ISO grammars use `/` as delimiter. The ambiguity arises from different position mappings, not delimiters; a leading 4-digit year is unambiguous (slash-ISO only), while a leading 1–2-digit field is ambiguous between US and European.

#### Validation Rules

Expand Down Expand Up @@ -345,7 +346,7 @@ Presentation is a single seam, not a rule concern:
### Feature Gating — two loci, two statuses

Input-shape and authority features gate at different points and produce different statuses:
- **Input-shape features** (`include_*`) toggle grammars via the contract's `active_grammars` property. A disabled grammar never recognizes → its inputs are **`MISSING`**.
- **Input-shape features** (`include_*`) toggle grammars via the contract's `active_grammars` property — implemented only by the gated capabilities (Email, IP, ISBN); other contracts inherit the base `None` default, which runs every shipped grammar. A disabled grammar never recognizes → its inputs are **`MISSING`**.
- **Authority features** gate rules via `requires_features` (declared on the rule). The rule is dropped from the run → recognized but unvalidated input is **`INVALID`**.
- Never gate inside `matches()` / `recognize()`; never cast a contract to read `include_*` flags inside a rule (`typing.cast` is only for validity-affecting parameters).

Expand Down Expand Up @@ -646,8 +647,8 @@ class Contract(Protocol):
...

@property
def active_grammars(self) -> Sequence[str]:
"""List of grammar names to activate."""
def active_grammars(self) -> Sequence[str] | None:
"""Grammar names to activate; None runs every shipped grammar."""
...

@property
Expand Down Expand Up @@ -675,7 +676,7 @@ class Contract(Protocol):
...
```

Contracts subclass `CapabilityContract` (never `Contract` directly), are `@dataclass(frozen=True)` **without** `slots=True`, and set `DEFAULT_OUTPUT_FORMAT` / `OFFERED_OUTPUT_FORMATS`, `capability_name` via `field`, and `active_grammars`.
Contracts subclass `CapabilityContract` (never `Contract` directly), are `@dataclass(frozen=True)` **without** `slots=True`, and set `DEFAULT_OUTPUT_FORMAT` / `OFFERED_OUTPUT_FORMATS` and `capability_name` via `field`. `active_grammars` is optional: only feature-gated capabilities (Email, IP, ISBN) implement it; others inherit the base `None` default (run every shipped grammar).

---

Expand Down
24 changes: 8 additions & 16 deletions HOW_TO_ADD_NEW_CAPABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ Concretely, a `CapabilityContract` subclass:
- Overrides `DEFAULT_OUTPUT_FORMAT` (a concrete string) and `OFFERED_OUTPUT_FORMATS` (a `frozenset[str]` of *alternative* formats) as class variables. The default format is **not** included in `OFFERED_OUTPUT_FORMATS`.
- Sets `capability_name` via `field(default="<name>", init=False)`.
- Declares `output_format` nowhere — the base field `output_format: str | None = None` is inherited. It is **never** a non-optional `str`. The base `__post_init__` resolves `None`, `"default"`, and the default format string to the concrete default, validates offered alternatives, and raises `ContractError` for anything else.
- Implements the abstract `active_grammars` property.
- Implements `active_grammars` **only when recognition is feature-gated** (the Email/IP/ISBN pattern). Otherwise the property is omitted entirely: the base returns `None` and the engine runs every shipped grammar in `get_grammars()` order.
- Adds its own `__post_init__` validation by calling `super().__post_init__()` first. Use `@dataclass(frozen=True)` exactly like the base — do NOT add `slots=True` (incompatible with the base's `super()` pattern).

`CapabilityContract` satisfies the `Contract` protocol structurally, so your subclass does too.
Expand Down Expand Up @@ -455,7 +455,7 @@ class SectionYourRule(Rule[YourDomainNotation]):

**Feature gating has two loci, and they produce different `Resolution` statuses:**

- **Input-shape features toggle grammars via `active_grammars`.** A flag like `include_obfuscated` decides whether the `obfuscated_recognition` grammar runs at all. A disabled grammar recognizes nothing, so input readable only by that grammar yields `MISSING`.
- **Input-shape features toggle grammars via `active_grammars`** (implemented only by the gated capabilities — Email, IP, ISBN; other contracts inherit the base `None` default, which runs every shipped grammar). A flag like `include_obfuscated` decides whether the `obfuscated_recognition` grammar runs at all. A disabled grammar recognizes nothing, so input readable only by that grammar yields `MISSING`.
- **Authority features use `requires_features`.** A flag like `include_localized` gates the CLDR rule that validates localized names, not the grammar. Recognition still runs and produces a notation, but the engine drops the gated rule, so the recognized-but-unvalidated input yields `INVALID`.

**Hard rule: never gate inside `matches()`.** Do not read `include_*` feature-toggle flags, and do not `cast(Contract, ...)` to reach them, inside `matches()`. `matches()` must never consult `output_format` either; validity comes from the notation, the specification, and any legitimate validity-affecting parameters (e.g. `default_country`, `two_digit_base_year`). The engine owns feature routing: declare the dependency in `requires_features` and let the filter decide whether the rule runs.
Expand All @@ -473,7 +473,7 @@ Define the Contract in `paxman/capabilities/YourDomain/contract.py` (separate fi
3. Override `DEFAULT_OUTPUT_FORMAT` (a concrete string) and `OFFERED_OUTPUT_FORMATS` (a `frozenset[str]` of alternative formats, excluding the default) as class variables
4. Set `capability_name` via `field(default="yourdomain", init=False)` (users never set this)
5. Add configuration fields for toggling grammars (e.g., `include_obfuscated: bool = False`)
6. Implement `active_grammars` as a `@property` that builds the grammar list from configuration flags
6. Implement `active_grammars` as a `@property` that builds the grammar list from configuration flags — only if recognition is feature-gated; otherwise omit it (base default: run every shipped grammar)

`excluded_rules`, `pinned_rules`, `year`, and `output_format` are declared once on `CapabilityContract` — you don't redeclare them.

Expand Down Expand Up @@ -516,11 +516,11 @@ two_digit_base_year: int | None = None # Date: base year for 2-digit year parsi
- Pass parameters to rules (strings, ints, options)
- Control output behavior (`output_format`)

### Implementing `active_grammars`
### Implementing `active_grammars` (optional)

Two approaches exist for implementing `active_grammars`:
`active_grammars` is **optional**: the base `CapabilityContract.active_grammars` returns `None`, and the engine falls back to running every shipped grammar returned by `get_grammars()`, in order. Implement it only when recognition is feature-gated — an `include_*` flag decides whether a grammar runs at all:

1. **Conditional** (Email, IP): Build the list from boolean flags. Grammars are included only when their flag is `True`.
1. **Conditional** (Email, IP, ISBN): Build the list from boolean flags. Grammars are included only when their flag is `True`.

```python
@property
Expand All @@ -533,15 +533,7 @@ def active_grammars(self) -> list[str]:
return grammars
```

2. **Always-all** (Date, Country): Return all grammar names unconditionally. All grammars always run, and rules handle filtering via `notation.shape` or other discriminators.

```python
@property
def active_grammars(self) -> list[str]:
return ["iso8601_recognition", "us_recognition", "european_recognition"]
```

Choose conditional when grammars are expensive or mutually exclusive. Choose always-all when grammars are cheap and rules need to see all representations.
Do **not** implement a static "always-all" override returning every grammar name (the former Date/Country pattern). The base `None` default already runs every shipped grammar, so a static override adds maintenance with zero behavior change — and it silently excludes any future grammar added to `get_grammars()` unless someone remembers to extend the list. Choose conditional when grammars are expensive or mutually exclusive; otherwise omit the property entirely and let the fallback run all shipped grammars.

### Implementing `output_format` (always optional, homogeneous across capabilities)

Expand Down Expand Up @@ -608,7 +600,7 @@ def format_value(
**The Contract must satisfy the `Contract` protocol** (inheriting `CapabilityContract` does this structurally):

- `capability_name: str` — the capability this contract configures
- `active_grammars: Sequence[str]` — list of grammar names to activate
- `active_grammars: Sequence[str] | None` — grammar names to activate; `None` (base default) runs every shipped grammar in `get_grammars()` order
- `excluded_rules: Sequence[str]` — list of rule names to exclude
- `pinned_rules: Sequence[str] | None` — pin to specific rules (takes precedence over `excluded_rules` when set)
- `year: int | None` — year for temporal filtering
Expand Down
Loading
Loading