Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ docs/ # adr/, report/, research/, superpowers/plans+specs

## ANTI-PATTERNS (THIS PROJECT)
- **No `# type: ignore` / `# noqa` / `# pyright: ignore` in `paxman/` source** — fix root cause or use scoped ruff `per-file-ignores` (sanctioned pattern in pyproject). Tests may use `# type: ignore[misc]` for immutability checks.
- Never guess/infer/suggest — same input + contract = byte-identical output.
- Deterministic by construction: given the same input, the same contract, and the same library snapshot (fixed library version, registry contents, and rule-data tables), the pipeline always yields the same canonical output — no world-knowledge, no clock, no environment-dependent ordering, no fuzzy logic, no network inference across recognition, validation, and canonicalization.
- No cross-capability imports; capabilities import only from `paxman.core`; `paxman.core` imports nothing from `paxman.*`.
- Grammars must not map tokens to canonical values or import rule-layer data.
- Rules never contain the token `output_format` (code, comments, or docstrings).
Expand Down
6 changes: 3 additions & 3 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Paxman is a canonicalization authority resolver — a library that takes ambiguo

### Determinism

Paxman never guesses. Given the same input, the same contract configuration, and the same library snapshot (fixed library version, registry contents, and rule-data tables), the output is always byte-identical. This property holds by construction: every pipeline stage is a pure, deterministic function of its inputs — grammar recognition, rule validation, capability formatting, candidate deduplication, and status determination include no clocks, no randomness, and no environment-dependent ordering. Identical inputs always produce identical outputs, enabling auditability and reproducibility. Across library snapshots, provenance or rule-routing changes may alter the resulting metadata; the byte-identical guarantee is scoped to a fixed snapshot.
Paxman never guesses. Given the same input, the same contract configuration, and the same library snapshot (fixed library version, registry contents, and rule-data tables), the pipeline always yields the same canonical output. This property holds by construction: every pipeline stage — grammar recognition, rule validation, capability formatting, candidate deduplication, and status determination — is a pure function of its inputs with no world-knowledge, no clock, no environment-dependent ordering, no fuzzy logic, and no network inference. Identical inputs therefore always produce identical outputs, enabling auditability and reproducibility. Across library snapshots, provenance or rule-routing changes may alter the resulting metadata; the determinism guarantee is scoped to a fixed snapshot.

### Provenance-First

Expand Down Expand Up @@ -163,13 +163,13 @@ Determinism is a structural property of the layered pipeline, not a post-hoc art
- **Validation layer.** Rules accept recognized representations and produce candidates, each carrying a canonical value and provenance. A rule's output depends only on the representation and the contract.
- **Result layer.** The engine deduplicates identical candidates and folds the distinct candidate values into one of the resolution statuses: `SUCCESS` when all candidates agree on a single canonical value, `AMBIGUOUS` when they disagree, `INVALID` when nothing validated, and `MISSING` when no grammar recognized anything.

Every stage is a pure function of its inputs — no clocks, no randomness, no environment-dependent ordering — so the same input, contract configuration, and library snapshot (fixed library version, registry contents, and rule-data tables) always produce the same output, byte-for-byte. The `VersionStamp` on each execution result records the library version for provenance; the byte-identical guarantee itself rests on this determinism-by-construction. Changes to provenance or rule routing across library snapshots can alter result metadata; the byte-identical guarantee is scoped to a fixed snapshot.
Every stage is a pure function of its inputs — no clocks, no randomness, no environment-dependent ordering, no world-knowledge, no fuzzy logic, no network inference — so the same input, contract configuration, and library snapshot (fixed library version, registry contents, and rule-data tables) always produce the same canonical output. The `VersionStamp` on each execution result records the library version for provenance; the determinism guarantee rests on this determinism-by-construction. Changes to provenance or rule routing across library snapshots can alter result metadata; the determinism guarantee is scoped to a fixed snapshot.

### Community Extensions

Capabilities are closed for modification but open for extension. Community contributors register additional grammars (and the rules that validate them) against an existing capability through `paxman.core.extensions` — never by editing the capability package. The registries freeze with the capability registry at the first pipeline run.

A contract opts a registered grammar in by naming it in `extra_grammars`, a base `CapabilityContract` field surfaced on every `create_contract` factory. The engine composes the shipped active set with the opted-in extras, deduplicating names while preserving order — shipped slots first, extras after (unknown extra names are silently skipped). The shipped slots are `contract.active_grammars` when the contract implements it (the gated capabilities), or every shipped grammar in `get_grammars()` order when it returns `None` (the base default). Opt-in preserves determinism: a contract that names no extras composes to exactly the shipped set, so non-opt-in behavior is byte-identical.
A contract opts a registered grammar in by naming it in `extra_grammars`, a base `CapabilityContract` field surfaced on every `create_contract` factory. The engine composes the shipped active set with the opted-in extras, deduplicating names while preserving order — shipped slots first, extras after (unknown extra names are silently skipped). The shipped slots are `contract.active_grammars` when the contract implements it (the gated capabilities), or every shipped grammar in `get_grammars()` order when it returns `None` (the base default). Opt-in preserves determinism: a contract that names no extras composes to exactly the shipped set, so non-opt-in behavior is identical (deterministic).

Community rules follow the same opt-in discipline: a registered rule runs only when the contract's `extra_grammars` resolve to one of its `target_semantics` ids. An un-opted community rule — even one targeting a shipped grammar's semantics — never affects results, so a default contract resolves with shipped rules only.

Expand Down
4 changes: 2 additions & 2 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
A **canonicalization authority resolver** — a library that takes ambiguous human input and returns what authoritative specifications say that input means, with full provenance. Paxman is both a **syntactic recognizer** (finds values in text) and a **semantic interpreter** (validates against authoritative specifications).

**Invariants:**
- **Deterministic:** Never guess, never infer, never suggest
- **No fabrication:** Never guess, never infer, never suggest
- **Provenance-first:** Always cite authority-defined specifications, registries, policies
- **Replay-safe:** Same input + same contract = byte-identical output
- **Deterministic:** Given the same input, the same contract, and the same library snapshot (fixed library version, registry contents, and rule-data tables), the pipeline yields the same canonical output — no world-knowledge, no clock, no environment-dependent ordering, no fuzzy logic, no network inference across recognition, validation, and canonicalization.

### Capability
A domain module (e.g., Email) that:
Expand Down
2 changes: 1 addition & 1 deletion HOW_TO_ADD_NEW_CAPABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -1053,7 +1053,7 @@ A shipped capability is closed for modification but open for extension: you can
Semantics to rely on:

- The extension registries freeze with the capability registry — registration after the first pipeline run raises `CapabilityError`.
- Opt-in only: an un-named registered grammar never affects results, keeping shipped behavior byte-identical for non-opt-in contracts.
- Opt-in only: an un-named registered grammar never affects results, so shipped behavior stays identical (deterministic) for non-opt-in contracts.
- Community rules are opt-in too: a registered rule runs only when the contract's `extra_grammars` resolve to one of its `target_semantics`; an un-opted rule — even one targeting a shipped grammar's semantics — never affects results.
- Unknown `extra_grammars` names are silently skipped; shipped names listed in `extra_grammars` are deduplicated.
- Composition is guarded: a grammar name colliding with a shipped name, or an opted-in community rule naming a missing grammar, fails fast at pipeline start.
Expand Down
2 changes: 1 addition & 1 deletion HOW_TO_ADD_NEW_GRAMMAR.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ def active_grammars(self) -> list[str]:
return grammars
```

Recognition order (and the same-span tiebreak) follows the runnable set — `get_grammars()` order for the six all-active capabilities, the `active_grammars` list for the three gated ones — so appending at the end keeps every existing grammar's behavior byte-identical. Update the capability test asserting the grammar count (`test_get_grammars_returns_all`) and add one asserting the new name is wired.
Recognition order (and the same-span tiebreak) follows the runnable set — `get_grammars()` order for the six all-active capabilities, the `active_grammars` list for the three gated ones — so appending at the end keeps every existing grammar's behavior identical (deterministic). Update the capability test asserting the grammar count (`test_get_grammars_returns_all`) and add one asserting the new name is wired.

> **The missing-half bug (gated capabilities only).** For Email, IP, and ISBN, the engine builds the runnable set from `contract.active_grammars`, not from `get_grammars()` — a grammar returned by `get_grammars()` but missing from `active_grammars` is dead code that will pass unit tests on the class and fail silently in the pipeline. The integration test in Step 5 is what catches this. The all-active capabilities have no such hole: the engine falls back to `get_grammars()` itself.

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -484,7 +484,7 @@ Rules of the seam:

- **Register before the first `canonicalize()` call** — the extension registries freeze with the capability registry.
- **Opt-in only** — a registered grammar runs only when named in `extra_grammars` (available on every `create_contract` factory), and a registered rule runs only when the contract's `extra_grammars` resolve to one of its `target_semantics` ids; un-named grammars and un-opted rules never affect results.
- **Unknown `extra_grammars` names are silently skipped** for grammar activation, so a contract naming an uninstalled grammar still runs byte-identically; the unknown name is kept as-is as the semantics key for rule activation. A name that is not a grammar name but matches a known semantics id therefore activates that semantics's rules without opting in a grammar — those rules fire only on recognitions carrying that semantics (fail-fast `ContractError` applies only to ids no grammar claims).
- **Unknown `extra_grammars` names are silently skipped** for grammar activation, so a contract naming an uninstalled grammar still runs identically (deterministically); the unknown name is kept as-is as the semantics key for rule activation. A name that is not a grammar name but matches a known semantics id therefore activates that semantics's rules without opting in a grammar — those rules fire only on recognitions carrying that semantics (fail-fast `ContractError` applies only to ids no grammar claims).
- **Names must be unique in the composed set** — a community grammar colliding with a shipped name fails fast with `CapabilityError`.
- **Community rules declare `target_semantics`** and activate only when the contract's `extra_grammars` resolve to one of those ids; a rule opted in via an id that no grammar claims fails fast with `ContractError`, while a rule that is not opted in stays inert regardless of any dangling targets.

Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Paxman incorporates several design principles that reduce attack surface:

- **Strict Type Checking**: The project enforces strict mode static type checking across all layers, catching type-related errors at development time rather than runtime.

- **Deterministic Computation**: Given the same input and contract configuration, output is always byte-identical. This prevents timing-based side channels and ensures reproducibility.
- **Deterministic Computation**: Given the same input, the same contract configuration, and the same library snapshot (fixed library version, registry contents, and rule-data tables), the pipeline always yields the same canonical output — no world-knowledge, no clock, no environment-dependent ordering, no fuzzy logic, no network inference across recognition, validation, and canonicalization. This ensures reproducibility; it is not a constant-time implementation and provides no protection against timing-based side channels.

- **Input Validation**: The system distinguishes between MISSING (no recognition), INVALID (recognized but unvalidated), and SUCCESS/AMBIGUOUS states, ensuring untrusted input is never silently accepted without authoritative validation.

Expand Down
2 changes: 1 addition & 1 deletion TESTING_STRATEGY.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Testing Strategy

Paxman's test suite exists to protect one core promise: given the same input and the same contract, the output is always byte-identical. Every test, from the lowest-level unit check to the full end-to-end run, reinforces that guarantee. The suite is built on `pytest` and organized into four layers that mirror the architectural boundaries of the system itself.
Paxman's test suite exists to protect one core promise: given the same input, the same contract, and the same library snapshot (fixed library version, registry contents, and rule-data tables), the pipeline always yields the same canonical output. Every test, from the lowest-level unit check to the full end-to-end run, reinforces that determinism — achieved by construction through no world-knowledge, no clock, no environment-dependent ordering, no fuzzy logic, and no network inference across recognition, validation, and canonicalization. The suite is built on `pytest` and organized into four layers that mirror the architectural boundaries of the system itself.

---

Expand Down
44 changes: 44 additions & 0 deletions docs/development/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# DEVELOPMENT DOCUMENTS — NON-SHIPPING

**Scope:** This `docs/development/` directory and all of its subfolders contain development-related documents only.

## Status

These documents are **not part of the shipped release** of Paxman. They will not be included in published distributions (sdist/wheel) or any release artifact.

## Maintenance Disclaimer

- These documents may **drift out of sync** with the actual implementation as Paxman evolves.
- It is **not the maintainers' responsibility** to keep these documents accurate or up to date.
- These documents may be **removed at any time** without notice or a migration path.

## Hard Guidance for Code and Docs

Any code file, inline documentation, docstring, or other documentation in the repository
(including files outside this directory) **must NOT** reference these documents — neither by
filename nor by quoting or paraphrasing any part of their contents.

Treat the contents of `docs/development/` as ephemeral working notes that may disappear or
become stale. Do not create dependencies on them.

## Not Authoritative

These documents do **not** define the project's overall architecture. For architecture-related
documentation, refer to `ARCHITECTURE.md`.

These documents do **not** record decisions that persist beyond the scope of the file or the
development stage in which the document is considered active. For decisions persisted across the
project's lifetime, refer to the ADRs in `docs/adr/`.

Once an implementation has landed, the documents in this folder and its subfolders are considered
**outdated**.

## PR Review Policy

PR reviewers **must skip** the documents in this folder and its subfolders when writing review
comments, **except** in the following cases:

- The implementation for the particular file in question has **not yet landed**.
- The reviewer is **specifically requested** to review one or more of these files.
- The reviewer **cites** these documents solely to clarify a point they are making (not as
authoritative or binding guidance).
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ notation *before* calling `matches()`, so routing is explicit and capability-agn
- Annotate all 18 production `Rule` subclasses with their true acceptance domain.
- TDD tests proving ambiguity is preserved and replay hashes are stable.

> **Note — replay-hash removed; do not use as a sample.** The `replay_hash` (replay-hash / byte-identical canonicalization) mechanism referenced in the paragraph above was **removed** from Paxman (see `docs/adr/0002-remove-replay-hash.md`). The text above reflects the pre-removal design and MUST NOT be copied, adapted, or used as a template for future work.

**Non-goals (explicitly out of scope — separate findings in the audit):**
- **F2** `include_*` feature gating (grammar-gate vs rule-gate unification).
- **F3** grammar-time canonicalization / single-source authority tables.
Expand All @@ -50,6 +52,8 @@ Each rule keeps its existing value-level `matches()`/`normalize()` logic as defe
The declared set must equal the rule's *effective* acceptance domain (derived from the
audit's cross-capability map), so behavior and the replay hash are byte-identical.

> **Note — replay-hash removed; do not use as a sample.** The `replay_hash` (replay-hash / byte-identical canonicalization) mechanism referenced in the paragraph above was **removed** from Paxman (see `docs/adr/0002-remove-replay-hash.md`). The text above reflects the pre-removal design and MUST NOT be copied, adapted, or used as a template for future work.

---

## 3. Findings recap (audit F1 → target)
Expand Down Expand Up @@ -158,6 +162,8 @@ rule today). **This is the contract the implementation must satisfy — do not s
- **Why:** the filter realizes F1; dedup keeps the replay hash stable if any future
author over-declares `target_grammars` (audit's replay-safety condition).

> **Note — replay-hash removed; do not use as a sample.** The `replay_hash` (replay-hash / byte-identical canonicalization) mechanism referenced in the paragraph above was **removed** from Paxman (see `docs/adr/0002-remove-replay-hash.md`). The text above reflects the pre-removal design and MUST NOT be copied, adapted, or used as a template for future work.

### Step 3 — Engine affinity invariant (`paxman/engine/orchestrator.py`)
**Files:** `paxman/engine/orchestrator.py` (new helper; call in `run_capability` after
`_filter_rules`, line 57).
Expand Down Expand Up @@ -239,6 +245,8 @@ then confirm green.
9. `_validate_affinity` fires: a stub capability whose rule declares a non-existent
grammar name → `run_capability` raises `ContractError`.

> **Note — replay-hash removed; do not use as a sample.** The `replay_hash` (replay-hash / byte-identical canonicalization) mechanism referenced in the paragraph above was **removed** from Paxman (see `docs/adr/0002-remove-replay-hash.md`). The text above reflects the pre-removal design and MUST NOT be copied, adapted, or used as a template for future work.

---

## 7. Verification (quality gates)
Expand Down Expand Up @@ -278,6 +286,8 @@ integration/e2e suites unchanged in outcome.
pattern). Step 3's invariant plus the `HOW_TO` presentational-only section document
the discipline; no runtime shape coupling is introduced.

> **Note — replay-hash removed; do not use as a sample.** The `replay_hash` (replay-hash / byte-identical canonicalization) mechanism referenced in the paragraph above was **removed** from Paxman (see `docs/adr/0002-remove-replay-hash.md`). The text above reflects the pre-removal design and MUST NOT be copied, adapted, or used as a template for future work.

---

## 9. Out-of-scope follow-ups (do NOT bundle into this plan)
Expand Down
Loading
Loading