diff --git a/AGENTS.md b/AGENTS.md index f9d9e944e6..9a73ce7d1a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,7 +34,7 @@ files. ```bash npm install -g js-yaml # Required once -make -C source # Convert all YAML test files to JSON +make -C source # Convert most YAML test files to JSON ``` ### Schema Latest Update (after adding a new unified test format schema version) @@ -43,91 +43,22 @@ make -C source # Convert all YAML test files to JSON make update-schema-latest -C source ``` -### Test Generation Scripts - -Some specs include Python scripts that generate test files. These often live alongside their respective specs rather -than in a single directory. Examples: - -```bash -python source/server-discovery-and-monitoring/tests/errors/generate-error-tests.py # SDAM error tests -python source/client-side-encryption/etc/generate-corpus.py # Client-side encryption corpus tests -python source/etc/generate-handshakeError-tests.py # Handshake error tests -``` - -A general pattern: look for `generate-*.py` scripts in a spec's `etc/` or `tests/` subdirectory. - ## Architecture -### Specification Documents (`source/`) - -Each subdirectory under `source/` is a specification area (e.g., `crud/`, `transactions/`, `auth/`). Specs are written -in GitHub Flavored Markdown at 120-character line width, following the -[MongoDB Documentation Style Guidelines](https://www.mongodb.com/docs/meta/style-guide/). Each spec typically contains: - -- `*.md` — the specification itself, with a `## Changelog` section at the bottom -- `tests/` — YAML test files (human-editable) and auto-generated JSON (never edit manually) - -### Changelog Format - -Each spec file contains its own `## Changelog` section (not a separate file). Entries are dated and prepended at the -top: - -```markdown -## Changelog - -- 2026-05-13: Describe the change made. - -- 2024-03-01: Previous change. -``` - ### Unified Test Format The unified test format (`source/unified-test-format/`) defines a YAML/JSON schema for cross-driver tests. Key rules: -- **Always use the lowest possible schema version** that satisfies a test's requirements — do not default to the latest -- YAML files are the source of truth; JSON files are auto-generated and committed - Schema versions live in `source/unified-test-format/schema-*.json`; `schema-latest.json` must match the highest version (run `make update-schema-latest -C source` to update it) - CI validates test files against the declared schema version using `ajv-cli`. Using any fields from a newer schema version than declared will fail validation. -### Test File Conventions - -- YAML test files are human-editable and located in `tests/` subdirectories -- JSON files are generated automatically by CI from YAML — never edit `.json` test files manually -- Use `runOnRequirements` to restrict tests to specific server versions/topologies -- Topologies to consider: standalone, replica set, sharded cluster, load balanced - ### Documentation Build (`mkdocs.yml`) MkDocs with `pymdown-extensions` and `mkdocs-github-admonitions-plugin`. The navigation is largely auto-generated via `scripts/generate_index.py`. The build must pass in `--strict` mode (no warnings). -## Test Authoring Rules - -### Immutability of Existing Tests - -**Do not modify existing tests** unless they are testing incorrect behavior. Default to creating new tests or new test -files instead of altering existing ones. - -- Test files can only be deleted once **no driver runs them anymore**. -- For spec changes that remove functionality: use `runOnRequirements` (unified tests) or have drivers skip the test - (non-unified tests like SDAM). -- Outdated prose tests must not be removed — mark them as such (e.g., strikethrough or *Removed*). -- Retiring an EOL server version (removing now-dead version-gated tests, prose, and pseudocode after the minimum - supported server version is raised) is a repo-wide sweep with many non-obvious traps. Use the - `retiring-server-versions` skill for the full checklist before starting one. - -### Prose Test Numbering - -Always use relative numbered bullets (`1.`) for prose tests. **New tests must be appended at the end** of the list, -since drivers may reference existing tests by number. - -### Test Isolation - -Only test functionality directly related to the new spec requirements. Omit irrelevant fields in command expectations. -This makes tests more resilient against future spec updates. - ## Linting & Formatting | Tool | Purpose | @@ -138,25 +69,3 @@ This makes tests more resilient against future spec updates. | `shellcheck` | Shell script linting | All checks run via `pre-commit`. CI enforces them in `.github/workflows/lint.yml`. - -## PR Requirements - -Per the PR template and project workflow: - -- Title must include a DRIVERS ticket (e.g., `DRIVERS-1234`) -- Update the `## Changelog` section in each modified spec file -- Test changes in at least one language driver -- Include links to the driver implementation PRs in the PR description (e.g., - `Python implementation: https://github.com/mongodb/mongo-python-driver/pull/…`) -- Tests must pass against all supported server versions and topologies - -## Specification Writing Guidelines - -From `source/driver-mantras.md`: - -- Prefer **MUST** over **SHOULD** — wishy-washy specs produce incompatible drivers -- Be topology-agnostic wherever possible -- Minimize configuration options ("No Knobs" principle) -- Check wire protocol version, not server version -- Follow semantic versioning for behavior changes -- Design for the next release, not hypothetical future ones ("Defy augury") diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000000..2de886778a --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,162 @@ +# Specification standards + +This document describes the shared standards for all specifications contained within this repository. These standards +apply to all specification prose, pseudocode, tests, and comments, both human and machine-authored. Use these guidelines +both when writing and reviewing specification changes. + +This is a living document: when an issue or difference of opinion occurs several times across reviews, the final +resolution SHOULD be added as a new guideline here. + +## Repository structure + +- All specifications MUST go in the `source/` directory under their own subdirectory (e.g `source/auth/`). + +## Style and formatting + +- All prose MUST use proper English grammar: write in complete sentences, start with a capital letter, use correct + punctuation, and end with a period. [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) terms defined in a + specification's `META` section are exempt. +- Authors MUST avoid metaphors, similes, analogies, and other figurative language. +- Authors MUST use numbered lists for enumerating steps in a process. +- Authors MUST use bulleted lists for enumerating related but unordered lists of items. +- All specifications MUST use [GitHub Flavored Markdown](https://github.github.com/gfm/) and follow the + [MongoDB Documentation Style Guidelines](https://www.mongodb.com/docs/meta/style-guide/) with a 120-line character + limit. +- Authors MUST abide by the automated linters described in [README.md](./README.md). + +## RFC 2119 keywords + +- Authors MUST use "MUST" wherever alignment of drivers across languages is required. +- Authors MUST use "SHOULD" only when valid exceptions to a requirement exist and can be documented. +- When in doubt, authors MUST use "MUST" instead of "SHOULD". +- Authors MUST follow [RFC 8174](https://www.rfc-editor.org/info/rfc8174/) and use all-caps when invoking RFC 2119 + keywords. + +## Pseudocode + +- Authors SHOULD add pseudocode following a numbered list of MUST steps that defines an algorithm or a description of a + schema. +- Comments MUST follow the standards above for prose style. +- Authors MUST use Python syntax for algorithms and Typescript syntax for BSON and wire documents. +- Pseudocode MUST be purely explanatory, not normative. It cannot substitute for a prose description of a required + behavior. + +## Tests + +- Tests MUST cover every behavior required in the specification. +- Tests MUST only verify functionality directly related to their specification. Omit irrelevant fields in expected + output. +- Authors SHOULD use unified tests over prose tests whenever possible. + - Authors SHOULD prefer unified tests over new test formats. + - Authors SHOULD expand unified test capabilities over prose tests where expansion would permit the testing of new + behaviors. +- Authors MUST number prose tests starting with `1.`. +- Authors MUST add new tests to the end of list of prose tests. +- Authors MUST NOT modify existing tests unless to fix correctness issues. Create new tests instead of modifying + existing ones. +- Authors MUST specify all environmental and topological requirements for each test. +- Authors MUST NOT assume a specific driver architecture when creating tests unless that architecture is explicitly + required by the specification. +- Tests MUST be in a separate `tests/` directory within specification directory, not in the specification itself. + - Prose tests MUST be in a `tests/README.md` file. + - Unified tests MUST be in a `tests/unified` subdirectory. +- Authors MUST use `runOnRequirements` (for unified tests) or instructions on skipping (for other tests) to ensure tests + are only executed when supported. +- Tests not run by any driver MUST be deleted. +- Authors MUST re-number existing prose tests to prevent numbering gaps when deleting unused prose tests. +- Unified tests MUST use the lowest possible schema version that satisfies their requirements. +- Tests MUST verify or explicitly exclude behavior for all four supported topologies: standalone, replica set, sharded + cluster, and load balanced. +- Authors MUST make manual changes to `.yml` test files and then generate the `.json` versions using + [these instructions](./README.md#converting-to-json). +- Authors MUST NOT make manual changes to `.json` test files generated from a `.yml` source. + +## Changelog + +- All changes MUST have a changelog entry in the modified specification. +- Changelog entries MUST be concise summaries of their described changes. +- Changelog entries MUST be in reverse chronological order, with the latest change at the top. +- Each entry MUST follow a standard format: `YYYY-MM-DD: Description.`, and MUST be separated from its neighbors by a + blank line. + +## Links + +- Cross-spec links MUST use relative paths instead of absolute ones. +- Terms defined in another specification MUST be linked instead of re-defining. + +## Deprecation + +- Deprecated features MUST be marked as deprecated and removed entirely from the specification once no driver and server + pair supports them. +- When retiring an EOL server version, authors MUST remove all version-gated tests, prose, and pseudocode specific to + the EOL version. Use the `retiring-server-versions` skill as a starting point. + +## Document Structure + +- Specifications MUST use the following sections, in this order, when present. Abstract, META, Specification, and + Changelog are required. + +``` +## Abstract + +A brief description of the specification's intent and why it requires its own specification. + +## META + +The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and +"OPTIONAL" in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). + +## Terms + +Definitions of all technical terms used in the specification. + +## Specification + +The bulk of the document. Contains all actual requirements and the use of keywords defined in the META section. + +## Implementation Notes + +Guidance for driver authors specific to actual implementation. This can include allowances for language differences, examples of non-obvious complexity in code, and the like. + +## Design Rationale + +Motivations for the design choices made in the specification. Answer the "why" of choices, not the "how". +If there are notable rejected designs, include brief explanations for their rejection in a "Rejected Alternatives" subsection. + +## Backwards Compatibility + +Implications of the specification for older driver and server versions that do not support its changes. + +## Reference Implementation + +The drivers responsible for producing the initial reference implementation of the specification. + +## Future Work + +Future additions to the specification that did not qualify for the initial version. + +## Questions and Answers + +Answers to questions that have or will frequently come up for driver authors working to implement or understand the specification. + +## Test Plan + +A link to the separate test document associated with the specification. + +## Changelog + +A dated, bulleted list of changes made to the specification. +``` + +## LLM usage + +- Authors MUST take responsibility for all changes made under their name, regardless of how they were created. + +## PR Requirements + +- PR titles MUST include a DRIVERS ticket (e.g., `DRIVERS-1234`). +- PRs that modify driver or test behavior MUST also fulfill these requirements: + - Authors MUST test changes in at least one language driver. + - Authors MUST include links to the driver implementation PRs in the PR description (e.g., + `Python implementation: https://github.com/mongodb/mongo-python-driver/pull/…`). + - Tests MUST pass against all supported server versions and topologies. diff --git a/README.md b/README.md index a355b3ce52..1418571b2d 100644 --- a/README.md +++ b/README.md @@ -5,16 +5,13 @@ This repository holds in progress and completed specification for features of MongoDB, Drivers, and associated products. Also contained is a rudimentary system for producing these documents. -## Driver Mantras - -See [Documentation](./source/driver-mantras.md). +## Contributing -## Writing Documents +See [CONTRIBUTING.md](./CONTRIBUTING.md). -Write documents using [GitHub Flavored Markdown](https://github.github.com/gfm/), following the -[MongoDB Documentation Style Guidelines](https://www.mongodb.com/docs/meta/style-guide/). +## Driver Mantras -Store all source documents in the `source/` directory. +See [Documentation](./source/driver-mantras.md). ## Linting @@ -36,43 +33,6 @@ To run a manual hook like `shellcheck` manually, run: pre-commit run --all-files --hook-stage manual shellcheck ``` -## Prose Test Numbering - -When numbering prose tests, always use relative numbered bullets (`1.`). New tests must be appended at the end of the -test list, since drivers may refer to existing tests by number. - -Outdated tests must not be removed completely, but may be marked as such (e.g. by striking through or replacing the -entire test with a note (e.g. *Removed*). - -## Automated Test Best Practices - -### Immutability of Existing Tests - -**Do not modify existing tests**, unless they are testing incorrect behavior. Default to creating new tests or test -files instead of altering existing ones. - -Test files can only be deleted once no driver runs them anymore. In the meantime, for cases where a spec change removes -functionality: - -- **Unified Tests:** Use `runOnRequirements` to ensure tests are only executed by drivers supporting the required - functionality. -- **Non-Unified Tests (e.g., SDAM):** Drivers should skip tests that no longer apply to them. - -### Test Isolation - -When creating a new test, only test functionality directly related to the new spec requirements. Omit irrelevant fields -in command expectations. - -This makes tests more resilient against spec updates and avoids needing to change tests down the line. - -### Schema Version Usage - -Use the **lowest possible schema version** for each test. - -Do NOT default to using the latest unified test format schema version, as the drivers may not all implement it. Use the -oldest schema version that supports all functionality used in the test, even if it requires creating a new test file -with a lower schema version. - ## Building Documents We use [mkdocs](https://www.mkdocs.org/) to render the documentation. To see a live view of the documentation, in a @@ -94,7 +54,17 @@ using `yaml2json` anymore, but instead the [js-yaml](https://www.npmjs.com/packa converter, so that JSON is formatted consistently. Run `npm install -g js-yaml`, then run `make` in the `source` directory at the top level of this repository to convert -all YAML test files to JSON. +most YAML test files to JSON. + +Some specs include Python scripts that generate test files. These often live alongside their respective specs: + +```bash +python source/server-discovery-and-monitoring/tests/errors/generate-error-tests.py # SDAM error tests +python source/client-side-encryption/etc/generate-corpus.py # Client-side encryption corpus tests +python source/etc/generate-handshakeError-tests.py # Handshake error tests +``` + +In general, these follow the `generate-*.py` naming convention in a spec's `etc/` or `tests/` subdirectory. ## Licensing