Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
286 lines (231 loc) · 11.1 KB

CONTRIBUTING.md

File metadata and controls

286 lines (231 loc) · 11.1 KB
Raw

Contributing to lm15-python

This document covers the maintainer-facing machinery: the fixture/conformance system, the doc-drift checker, the provider adapter development guide, and the day-to-day commands.

Repository layout

lm15-python/
├── lm15/
│   ├── types.py              # canonical dataclasses: Request, Response, Parts, tools, endpoints
│   ├── providers/            # OpenAI, Anthropic, Gemini adapters (+ compat, subscription, async)
│   ├── compat.py             # typed API-dialect compatibility policies
│   ├── models.py             # optional ModelInfo metadata + ModelRegistry
│   ├── profiles.py           # ProviderProfile/EndpointProfile resolution helpers
│   ├── result.py             # stream materialization + lazy Result helper
│   ├── serde.py              # canonical JSON dictionaries
│   ├── errors.py             # normalized lm15 error hierarchy
│   ├── live.py               # websocket/live-session helpers
│   ├── sse.py                # server-sent event parser
│   └── transports/           # stdlib HTTP/1.1 sync + async transports
├── conformance/              # fixture suite and reports
├── tests/                    # unit + conformance tests wired into pytest
├── benchmarks/               # transport benchmarks
└── pyproject.toml

The contract pin (CONTRACT_PIN)

CI runs the language-neutral lm15-contract harness against this repo at the exact contract commit recorded in CONTRACT_PIN (a full SHA of lm15-dev/lm15-contract main). Discipline: when a change here requires a contract change (new fixture, spec row, serde kind, protocol op), land the contract commit first — with the evidence AUTHORITY.md demands — then bump CONTRACT_PIN to that SHA in the same lm15-python commit as the code change, so both repos stay green at every pinned pair of revisions. Never bump the pin just to make CI green: the diff between the old and new pin is part of your review.

How the fixture/conformance system works

The conformance suite checks that lm15's canonical model stays aligned with real provider APIs.

logical lm15 case
  conformance/cross_sdk/test_cases.json
        │
        ▼
lm15-python provider adapter builds HTTP request
        │
        ▼
expected provider fixture
  conformance/provider_requests/cases/<provider>/<feature>.json
        │
        ├── check_request_fixtures.py compares request shape
        ├── validate_live.py can send the request to the real API
        ├── check_response_fixtures.py parses saved response bodies/SSE
        ├── check_error_fixtures.py normalizes provider error bodies
        ├── check_endpoint_fixtures.py checks files/batch/image/audio/live
        ├── check_serde_fixtures.py checks canonical JSON round trips
        └── check_doc_drift.py checks provider docs against features.yaml

Run everything:

python3 conformance/run_all.py --strict

Run one check:

python3 conformance/check_doc_drift.py --strict
python3 conformance/check_response_fixtures.py --strict

Run or preview one live provider fixture, if the relevant API key is set:

python3 conformance/provider_requests/validate_live.py --dry-run --task openai.basic_text
python3 conformance/provider_requests/validate_live.py --task openai.basic_text

Generated reports are written under conformance/reports/ and are ignored by git.

Adding or completing a fixture

  1. Add or update the logical case in conformance/cross_sdk/test_cases.json.

  2. Add the expected provider HTTP request in conformance/provider_requests/cases/<provider>/<feature>.json.

  3. Add the feature to conformance/provider_requests/features.yaml so doc drift can tell whether provider documentation is represented.

  4. If response parsing should be checked, add an expect_lm15 block to the provider case and save a real response body under conformance/provider_requests/results/bodies/<provider>.<feature>/.

  5. If the provider has a special error shape, add an error case under conformance/errors/cases/<provider>.json.

  6. Run:

    python3 conformance/run_all.py --strict
pytest -q

Example expect_lm15 block:

{
  "expect_lm15": {
    "parts": {
      "text": {"min": 1},
      "citation": {"min": 1}
    },
    "finish_reason": "stop",
    "usage": {"required": true}
  }
}

Doc-drift fixture check

conformance/check_doc_drift.py parses the scraped provider reference pages in the sibling lm15-contract/scrapes/<provider>/pages/ directory (refreshed by each provider's update.sh; LM15_API_REFERENCES overrides the location) and compares top-level request parameters with conformance/provider_requests/features.yaml. There is no snapshot in this repo: a second copy drifted once already.

Some always-on lm15 request fields do not need separate feature entries:

IGNORE_PARAMS = {"model", "messages", "contents", "input"}

Provider docs often use camelCase/PascalCase while lm15 feature names use snake_case, so the drift check normalizes names before deciding that a param is unmapped.

If check_doc_drift.py --strict reports an unmapped param, either:

  • add a real feature entry to features.yaml, or
  • add the param to IGNORE_PARAMS only if it is a core field that should never have a separate fixture.

Provider adapter development guide

Provider classes live in lm15/providers/ and inherit BaseProviderLM. A provider adapter is responsible for:

  • build_request(request, stream) — map canonical Request to an HTTP request.
  • parse_response(request, response) — map provider JSON to canonical Response.
  • parse_stream_events(...) — map SSE chunks to StreamEvents (the single stream-parse path per provider).
  • normalize_error(status, body) — map provider errors to lm15.errors.
  • Optional endpoint methods: file_upload, batch_submit, image_generate, audio_generate, live.

Keep provider-only options in Config.extensions rather than adding universal fields unless the same concept is supported across providers.

What makes a contribution acceptable

Every pull request is judged against these requirements; they are the source of truth for both contributors and reviewers:

  1. Tests. Behavior changes come with tests — unit tests in tests/, and fixture/conformance updates when the wire behavior changes (see the fixture workflow above). "It works locally" is not evidence.

  2. The contract comes first. Changes to wire behavior land in lm15-contract before they land here, and CONTRACT_PIN moves in the same commit as the code (see the pin discipline above).

  3. Stdlib only. No new runtime dependencies. See the dependency policy below for the (high) bar to change that.

  4. CI must be green. The full pytest suite, the strict conformance suite, and the pinned contract harness all run on every push and pull request.

  5. Sign your work (DCO). Every commit must carry a Signed-off-by: line asserting the Developer Certificate of Origin — that you have the legal right to contribute the change. Enable the repo hook once per clone and every commit is signed off automatically:

    git config core.hooksPath .githooks

    (Manual alternative: git commit -s.)

  6. No secrets. Never commit API keys, tokens, or fixture data captured with real credentials. Secret scanning with push protection is enabled and will reject such pushes.

  7. Adapt, record, refuse — in that order (MAP-13). When a setting cannot go to a wire as asked, ask what the caller MEANT and whether this provider can deliver it — never "does this wire have a field of this name". Same meaning, other spelling: translate. Cannot, and nothing depends on it: adapt (drop / clamp / substitute / client-side / satisfied / defaulted) and record it in Response.adaptations. Refuse only under MAP-13's four conditions, naming the one that applies. A new refusal in a pull request must cite its condition; a refusal that rests on a compat-preset value with no live receipt is rejected (lm15-contract/docs/mapping-rules.md MAP-13, lm15-dev/THEORY.md §3.9, §3.17).

Building from source

The package is pure Python with no build-time dependencies beyond setuptools:

git clone https://github.com/lm15-dev/lm15-python.git
cd lm15-python
python -m venv .venv && . .venv/bin/activate
pip install -e .              # editable install
pip install pytest pytest-asyncio pyyaml   # test tooling
pytest -q                     # verify the checkout

To build distributable artifacts (sdist + wheel):

pip install build
python -m build

Optional extra: pip install -e '.[live]' pulls in websockets for live/realtime sessions. Nothing else is required.

Dependency policy

How lm15 selects, obtains, and tracks dependencies:

  • Selection: default no. The runtime is Python stdlib only — this is a design guarantee (see the README footprint section), not an accident. A new runtime dependency requires a maintainer decision that the functionality is impossible or unreasonable to provide with the stdlib, and that the candidate is actively maintained, widely used, and itself light on transitive dependencies. The only current example is the optional websockets extra for live sessions — it is opt-in and the core never imports it.
  • Obtainment. Dependencies are declared in pyproject.toml and installed from PyPI through standard tooling (pip/uv); CI installs them the same way. Nothing is vendored, and no artifacts are fetched outside the package manager.
  • Tracking. pyproject.toml is the single source of truth for runtime requirements. Development and CI tooling (pytest, mkdocs) is listed in the workflows that use it. GitHub dependency graph and secret scanning watch the repository.

Cross-language conformance: lm15-contract

The sibling lm15-contract repository is where cross-language conformance lives — it is the authority, and this package is one implementation of it. Start with its AUTHORITY.md (what is normative and what wins on conflict), spec/ (the canonical rules: serde, mapping, errors), and harness/ (the runner that executes the shared corpus of cases, goldens, bodies, and error fixtures against each language implementation). Behavior changes here must be reflected there first, then re-verified through the harness.

Useful commands

# First check out lm15-contract as a sibling at CONTRACT_PIN.
# Tests read its cloud signing/token vectors during collection.

# Unit and conformance tests through pytest
pytest -q

# Full offline conformance suite
python3 conformance/run_all.py --strict

# Request fixture comparison only
python3 conformance/check_request_fixtures.py --strict

# Response/SSE fixture parser check only
python3 conformance/check_response_fixtures.py --strict

# Canonical JSON round trips only
python3 conformance/check_serde_fixtures.py --strict

# Provider-doc coverage only
python3 conformance/check_doc_drift.py --strict