DRIVERS-3667 Drivers specification standards

AGENTS.md

@@ -34,7 +34,7 @@ files.
 
 ```bash
 npm install -g js-yaml   # Required once
-make -C source           # Convert all YAML test files to JSON
+make -C source           # Convert most YAML test files to JSON
 ```
 
 ### Schema Latest Update (after adding a new unified test format schema version)
@@ -43,91 +43,22 @@ make -C source           # Convert all YAML test files to JSON
 make update-schema-latest -C source
 ```
 
-### Test Generation Scripts
-
-Some specs include Python scripts that generate test files. These often live alongside their respective specs rather
-than in a single directory. Examples:
-
-```bash
-python source/server-discovery-and-monitoring/tests/errors/generate-error-tests.py   # SDAM error tests
-python source/client-side-encryption/etc/generate-corpus.py                          # Client-side encryption corpus tests
-python source/etc/generate-handshakeError-tests.py                                   # Handshake error tests
-```
-
-A general pattern: look for `generate-*.py` scripts in a spec's `etc/` or `tests/` subdirectory.
-
 ## Architecture
 
-### Specification Documents (`source/`)
-
-Each subdirectory under `source/` is a specification area (e.g., `crud/`, `transactions/`, `auth/`). Specs are written
-in GitHub Flavored Markdown at 120-character line width, following the
-[MongoDB Documentation Style Guidelines](https://www.mongodb.com/docs/meta/style-guide/). Each spec typically contains:
-
-- `*.md` — the specification itself, with a `## Changelog` section at the bottom
-- `tests/` — YAML test files (human-editable) and auto-generated JSON (never edit manually)
-
-### Changelog Format
-
-Each spec file contains its own `## Changelog` section (not a separate file). Entries are dated and prepended at the
-top:
-
-```markdown
-## Changelog
-
-- 2026-05-13: Describe the change made.
-
-- 2024-03-01: Previous change.
-```
-
 ### Unified Test Format
 
 The unified test format (`source/unified-test-format/`) defines a YAML/JSON schema for cross-driver tests. Key rules:
 
-- **Always use the lowest possible schema version** that satisfies a test's requirements — do not default to the latest
-- YAML files are the source of truth; JSON files are auto-generated and committed
 - Schema versions live in `source/unified-test-format/schema-*.json`; `schema-latest.json` must match the highest
     version (run `make update-schema-latest -C source` to update it)
 - CI validates test files against the declared schema version using `ajv-cli`. Using any fields from a newer schema
     version than declared will fail validation.
 
-### Test File Conventions
-
-- YAML test files are human-editable and located in `tests/` subdirectories
-- JSON files are generated automatically by CI from YAML — never edit `.json` test files manually
-- Use `runOnRequirements` to restrict tests to specific server versions/topologies
-- Topologies to consider: standalone, replica set, sharded cluster, load balanced
-
 ### Documentation Build (`mkdocs.yml`)
 
 MkDocs with `pymdown-extensions` and `mkdocs-github-admonitions-plugin`. The navigation is largely auto-generated via
 `scripts/generate_index.py`. The build must pass in `--strict` mode (no warnings).
 
-## Test Authoring Rules
-
-### Immutability of Existing Tests
-
-**Do not modify existing tests** unless they are testing incorrect behavior. Default to creating new tests or new test
-files instead of altering existing ones.
-
-- Test files can only be deleted once **no driver runs them anymore**.
-- For spec changes that remove functionality: use `runOnRequirements` (unified tests) or have drivers skip the test
-    (non-unified tests like SDAM).
-- Outdated prose tests must not be removed — mark them as such (e.g., strikethrough or *Removed*).
-- Retiring an EOL server version (removing now-dead version-gated tests, prose, and pseudocode after the minimum
-    supported server version is raised) is a repo-wide sweep with many non-obvious traps. Use the
-    `retiring-server-versions` skill for the full checklist before starting one.
-
-### Prose Test Numbering
-
-Always use relative numbered bullets (`1.`) for prose tests. **New tests must be appended at the end** of the list,
-since drivers may reference existing tests by number.
-
-### Test Isolation
-
-Only test functionality directly related to the new spec requirements. Omit irrelevant fields in command expectations.
-This makes tests more resilient against future spec updates.
-
 ## Linting & Formatting
 
 | Tool                  | Purpose                                     |
@@ -138,25 +69,3 @@ This makes tests more resilient against future spec updates.
 | `shellcheck`          | Shell script linting                        |
 
 All checks run via `pre-commit`. CI enforces them in `.github/workflows/lint.yml`.
-
-## PR Requirements
-
-Per the PR template and project workflow:
-
-- Title must include a DRIVERS ticket (e.g., `DRIVERS-1234`)
-- Update the `## Changelog` section in each modified spec file
-- Test changes in at least one language driver
-- Include links to the driver implementation PRs in the PR description (e.g.,
-    `Python implementation: https://github.com/mongodb/mongo-python-driver/pull/…`)
-- Tests must pass against all supported server versions and topologies
-
-## Specification Writing Guidelines
-
-From `source/driver-mantras.md`:
-
-- Prefer **MUST** over **SHOULD** — wishy-washy specs produce incompatible drivers
-- Be topology-agnostic wherever possible
-- Minimize configuration options ("No Knobs" principle)
-- Check wire protocol version, not server version
-- Follow semantic versioning for behavior changes
-- Design for the next release, not hypothetical future ones ("Defy augury")

CONTRIBUTING.md

@@ -0,0 +1,162 @@
+# Specification standards
+
+This document describes the shared standards for all specifications contained within this repository. These standards
+apply to all specification prose, pseudocode, tests, and comments, both human and machine-authored. Use these guidelines
+both when writing and reviewing specification changes.
+
+This is a living document: when an issue or difference of opinion occurs several times across reviews, the final
+resolution SHOULD be added as a new guideline here.
+
+## Repository structure
+
+- All specifications MUST go in the `source/` directory under their own subdirectory (e.g `source/auth/`).
+
+## Style and formatting
+
+- All prose MUST use proper English grammar: write in complete sentences, start with a capital letter, use correct
+    punctuation, and end with a period. [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) terms defined in a
+    specification's `META` section are exempt.
+- Authors MUST avoid metaphors, similes, analogies, and other figurative language.
+- Authors MUST use numbered lists for enumerating steps in a process.
+- Authors MUST use bulleted lists for enumerating related but unordered lists of items.
+- All specifications MUST use [GitHub Flavored Markdown](https://github.github.com/gfm/) and follow the
+    [MongoDB Documentation Style Guidelines](https://www.mongodb.com/docs/meta/style-guide/) with a 120-line character
+    limit.
+- Authors MUST abide by the automated linters described in [README.md](./README.md).
+
+## RFC 2119 keywords
+
+- Authors MUST use "MUST" wherever alignment of drivers across languages is required.
+- Authors MUST use "SHOULD" only when valid exceptions to a requirement exist and can be documented.
+- When in doubt, authors MUST use "MUST" instead of "SHOULD".
+- Authors MUST follow [RFC 8174](https://www.rfc-editor.org/info/rfc8174/) and use all-caps when invoking RFC 2119
+    keywords.
+
+## Pseudocode
+
+- Authors SHOULD add pseudocode following a numbered list of MUST steps that defines an algorithm or a description of a
+    schema.
+- Comments MUST follow the standards above for prose style.
+- Authors MUST use Python syntax for algorithms and Typescript syntax for BSON and wire documents.
+- Pseudocode MUST be purely explanatory, not normative. It cannot substitute for a prose description of a required
+    behavior.
+
+## Tests
+
+- Tests MUST cover every behavior required in the specification.
+- Tests MUST only verify functionality directly related to their specification. Omit irrelevant fields in expected
+    output.
+- Authors SHOULD use unified tests over prose tests whenever possible.
+    - Authors SHOULD prefer unified tests over new test formats.
+    - Authors SHOULD expand unified test capabilities over prose tests where expansion would permit the testing of new
+        behaviors.
+- Authors MUST number prose tests starting with `1.`.
+- Authors MUST add new tests to the end of list of prose tests.
+- Authors MUST NOT modify existing tests unless to fix correctness issues. Create new tests instead of modifying
+    existing ones.
+- Authors MUST specify all environmental and topological requirements for each test.
+- Authors MUST NOT assume a specific driver architecture when creating tests unless that architecture is explicitly
+    required by the specification.
+- Tests MUST be in a separate `tests/` directory within specification directory, not in the specification itself.
+    - Prose tests MUST be in a `tests/README.md` file.
+    - Unified tests MUST be in a `tests/unified` subdirectory.
+- Authors MUST use `runOnRequirements` (for unified tests) or instructions on skipping (for other tests) to ensure tests
+    are only executed when supported.
+- Tests not run by any driver MUST be deleted.
+- Authors MUST re-number existing prose tests to prevent numbering gaps when deleting unused prose tests.
+- Unified tests MUST use the lowest possible schema version that satisfies their requirements.
+- Tests MUST verify or explicitly exclude behavior for all four supported topologies: standalone, replica set, sharded
+    cluster, and load balanced.
+- Authors MUST make manual changes to `.yml` test files and then generate the `.json` versions using
+    [these instructions](./README.md#converting-to-json).
+- Authors MUST NOT make manual changes to `.json` test files generated from a `.yml` source.
+
+## Changelog
+
+- All changes MUST have a changelog entry in the modified specification.
+- Changelog entries MUST be concise summaries of their described changes.
+- Changelog entries MUST be in reverse chronological order, with the latest change at the top.
+- Each entry MUST follow a standard format: `YYYY-MM-DD: Description.`, and MUST be separated from its neighbors by a
+    blank line.
+
+## Links
+
+- Cross-spec links MUST use relative paths instead of absolute ones.
+- Terms defined in another specification MUST be linked instead of re-defining.
+
+## Deprecation
+
+- Deprecated features MUST be marked as deprecated and removed entirely from the specification once no driver and server
+    pair supports them.
+- When retiring an EOL server version, authors MUST remove all version-gated tests, prose, and pseudocode specific to
+    the EOL version. Use the `retiring-server-versions` skill as a starting point.
+
+## Document Structure
+
+- Specifications MUST use the following sections, in this order, when present. Abstract, META, Specification, and
+    Changelog are required.
+
+```
+## Abstract 
+
+A brief description of the specification's intent and why it requires its own specification.
+
+## META 
+
+The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and
+"OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).
+
+## Terms 
+
+Definitions of all technical terms used in the specification.
+
+## Specification 
+
+The bulk of the document. Contains all actual requirements and the use of keywords defined in the META section.
+
+## Implementation Notes
+
+Guidance for driver authors specific to actual implementation. This can include allowances for language differences, examples of non-obvious complexity in code, and the like.
+
+## Design Rationale
+
+Motivations for the design choices made in the specification. Answer the "why" of choices, not the "how".
+If there are notable rejected designs, include brief explanations for their rejection in a "Rejected Alternatives" subsection.
+
+## Backwards Compatibility 
+
+Implications of the specification for older driver and server versions that do not support its changes.
+
+## Reference Implementation 
+
+The drivers responsible for producing the initial reference implementation of the specification.
+
+## Future Work 
+
+Future additions to the specification that did not qualify for the initial version.
+
+## Questions and Answers 
+
+Answers to questions that have or will frequently come up for driver authors working to implement or understand the specification.
+
+## Test Plan 
+
+A link to the separate test document associated with the specification.
+
+## Changelog
+
+A dated, bulleted list of changes made to the specification.
+```
+
+## LLM usage
+
+- Authors MUST take responsibility for all changes made under their name, regardless of how they were created.
+
+## PR Requirements
+
+- PR titles MUST include a DRIVERS ticket (e.g., `DRIVERS-1234`).
+- PRs that modify driver or test behavior MUST also fulfill these requirements:
+    - Authors MUST test changes in at least one language driver.
+    - Authors MUST include links to the driver implementation PRs in the PR description (e.g.,
+        `Python implementation: https://github.com/mongodb/mongo-python-driver/pull/…`).
+    - Tests MUST pass against all supported server versions and topologies.

README.md

@@ -5,16 +5,13 @@
 This repository holds in progress and completed specification for features of MongoDB, Drivers, and associated products.
 Also contained is a rudimentary system for producing these documents.
 
-## Driver Mantras
-
-See [Documentation](./source/driver-mantras.md).
+## Contributing
 
-## Writing Documents
+See [CONTRIBUTING.md](./CONTRIBUTING.md).
 
-Write documents using [GitHub Flavored Markdown](https://github.github.com/gfm/), following the
-[MongoDB Documentation Style Guidelines](https://www.mongodb.com/docs/meta/style-guide/).
+## Driver Mantras
 
-Store all source documents in the `source/` directory.
+See [Documentation](./source/driver-mantras.md).
 
 ## Linting
 
@@ -36,43 +33,6 @@ To run a manual hook like `shellcheck` manually, run:
 pre-commit run --all-files --hook-stage manual shellcheck
 ```
 
-## Prose Test Numbering
-
-When numbering prose tests, always use relative numbered bullets (`1.`). New tests must be appended at the end of the
-test list, since drivers may refer to existing tests by number.
-
-Outdated tests must not be removed completely, but may be marked as such (e.g. by striking through or replacing the
-entire test with a note (e.g. *Removed*).
-
-## Automated Test Best Practices
-
-### Immutability of Existing Tests
-
-**Do not modify existing tests**, unless they are testing incorrect behavior. Default to creating new tests or test
-files instead of altering existing ones.
-
-Test files can only be deleted once no driver runs them anymore. In the meantime, for cases where a spec change removes
-functionality:
-
-- **Unified Tests:** Use `runOnRequirements` to ensure tests are only executed by drivers supporting the required
-    functionality.
-- **Non-Unified Tests (e.g., SDAM):** Drivers should skip tests that no longer apply to them.
-
-### Test Isolation
-
-When creating a new test, only test functionality directly related to the new spec requirements. Omit irrelevant fields
-in command expectations.
-
-This makes tests more resilient against spec updates and avoids needing to change tests down the line.
-
-### Schema Version Usage
-
-Use the **lowest possible schema version** for each test.
-
-Do NOT default to using the latest unified test format schema version, as the drivers may not all implement it. Use the
-oldest schema version that supports all functionality used in the test, even if it requires creating a new test file
-with a lower schema version.
-
 ## Building Documents
 
 We use [mkdocs](https://www.mkdocs.org/) to render the documentation. To see a live view of the documentation, in a
@@ -94,7 +54,17 @@ using `yaml2json` anymore, but instead the [js-yaml](https://www.npmjs.com/packa
 converter, so that JSON is formatted consistently.
 
 Run `npm install -g js-yaml`, then run `make` in the `source` directory at the top level of this repository to convert
-all YAML test files to JSON.
+most YAML test files to JSON.
+
+Some specs include Python scripts that generate test files. These often live alongside their respective specs:
+
+```bash
+python source/server-discovery-and-monitoring/tests/errors/generate-error-tests.py   # SDAM error tests
+python source/client-side-encryption/etc/generate-corpus.py                          # Client-side encryption corpus tests
+python source/etc/generate-handshakeError-tests.py                                   # Handshake error tests
+```
+
+In general, these follow the `generate-*.py` naming convention in a spec's `etc/` or `tests/` subdirectory.
 
 ## Licensing