Widen the ruff rule set and fix the findings

[JordanCoin]

Stacked on #64 — review and merge that first; this targets its branch, so the diff here is only the lint work.

Follow-up to the pin in #64. That pin restored deterministic lint (a ruff release had widened its own defaults and reddened CI on unchanged code), but it also left ~300 findings unreported, and some had real signal. Rather than leave the narrow pin as the de-facto answer, this enables the rules that matter and fixes what they found.

Enabled: I, UP, B, SIM, DTZ, C4, PIE, RUF on top of E4/E7/E9/F.

Real defects found

A wrong-timezone deadline bug (in the preceding commit, from enabling DTZ). openfoia request send did req.sent_at = datetime.now() — local time into a column every reader treats as UTC. days_pending() and is_overdue() compare it against UTC, so for a user in UTC+9 or UTC-8 the statutory FOIA deadline was off by up to a day. For a tool whose job is tracking those deadlines, that matters.

Fixing it needed care rather than a blanket conversion: the ORM's DateTime columns are timezone-naive, so swapping in an aware datetime.now(timezone.utc) would raise TypeError: can't subtract offset-naive and offset-aware datetimes against every existing row. So models.utcnow() returns naive UTC — removing the deprecated datetime.utcnow() (CI targets 3.13) while preserving the storage convention exactly. 21 call sites migrated, including an aliased dt.utcnow() a naive search missed. Timezone-aware columns are a separate, deliberate migration.

B023 — the OCR loop's worker closure didn't bind image. Benign today because it's awaited within the iteration, but parallelizing that loop (an obvious future optimization) would have silently OCR'd the last page N times.

B904 — 17 raise sites inside except blocks lost the exception chain. All are control-flow re-raises (typer.Exit, SystemExit, HTTPException) after a friendly message is printed, so they now say from None explicitly rather than dropping the chain by accident.

F401 — three optional-dependency availability probes were indistinguishable from dead imports; annotated as deliberate.

Judgement calls, documented in pyproject.toml rather than applied blindly

• UP042 ignored. class X(str, Enum) → StrEnum changes what str(member) returns ("federal" vs "AgencyLevel.FEDERAL"). These enums are persisted through SQLAlchemy and serialized into API responses and exports — that's a data-format change wearing a lint fix's clothing.
• B008 ignored for server.py/cli.py — = Depends(...) and = typer.Option(...) in signature defaults are FastAPI's and Typer's idiom, so it fires on every route and command.
• S101/SLF001 ignored under tests/ — asserts and private access are the point there.
• BLE001 (~82) deliberately still off, with a note explaining why. It's the highest-signal rule for this codebase — a swallowed exception is exactly what made a failed records lookup report as a clean result — but fixing it means a per-site audit of error handling, which deserves its own reviewable change rather than a bulk sweep.

Mechanical

Import ordering, modern typing syntax, contextlib.suppress, ClassVar on mutable class attributes, and two collapsible ifs — both verified to have no else branch, so the elif-chain semantics are unchanged.

Also: the U+2028/U+2029 literals in the XSS escaper and its test are now written as source escapes instead of raw invisible characters. Same behaviour, but reviewers can actually see them, and it removes the RUF001 finding honestly rather than by suppression.

Verification

181 tests pass, ruff check and ruff format --check clean. Also exercised the real app rather than tests alone: openfoia init and openfoia agency list work, and openfoia init --password "it's-a-test" produces a genuinely encrypted database (that apostrophe is the key-escaping regression from #64).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KWzqFx1d2uB6cjQV81mgJh

---

Generated by Claude Code

[Copilot]

Pull request overview

This PR expands the Ruff lint rule set (beyond the historical E4/E7/E9/F subset) and applies repo-wide mechanical fixes plus a few targeted correctness fixes uncovered by the wider rules—most notably standardizing timestamp writes/reads around a naive-UTC helper to avoid timezone skew while staying compatible with existing timezone-naive ORM columns.

Changes:

• Enable additional Ruff rule families (I, UP, B, SIM, DTZ, C4, PIE, RUF) and document intentional ignores/per-file-ignores in pyproject.toml.
• Introduce openfoia.models.utcnow() (naive UTC) and migrate timestamp call sites away from deprecated datetime.utcnow() / unsafe datetime.now() assignments for persisted UTC columns.
• Apply mechanical lint-driven refactors (import ordering, contextlib.suppress, typing modernizations) and update/extend tests (including new datetime semantics regressions).

Reviewed changes

Copilot reviewed 35 out of 35 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_security_injection.py	Make U+2028/U+2029 assertions explicit via \u escapes.
tests/test_security_gateways.py	Adjust pytest.raises(..., match=...) to use a compiled regex with flags.
tests/test_extraction.py	Import ordering/blank-line normalization for Ruff.
tests/test_datetime_semantics.py	Add regression tests to pin “naive UTC” storage convention and ban deprecated/unsafe datetime patterns.
tests/benchmark_extraction.py	Import ordering and unused-variable rename in benchmark script.
pyproject.toml	Widen/select Ruff rule families; add targeted ignores/per-file-ignores with rationale.
openfoia/tor_browse.py	Replace deprecated UTC calls; simplify interrupt handling via contextlib.suppress.
openfoia/templates.py	Modernize typing; document intended local-time letter dates; DTZ suppressions for non-persisted timestamps.
openfoia/server.py	Import ordering; replace deprecated UTC usage; suppress exception chaining where intentional; minor refactors using contextlib.suppress.
openfoia/security.py	Replace deprecated UTC usage; use contextlib.suppress for best-effort filesystem cleanup.
openfoia/records/sec_edgar.py	Remove stray blank line / formatting cleanup.
openfoia/records/muckrock.py	Type class constant as ClassVar[...] for lint/type clarity.
openfoia/records/init.py	Import ordering for adapter auto-registration.
openfoia/pipeline/web.py	Simplify parser control flow; swap to datetime.now(UTC); use contextlib.suppress for best-effort parsing.
openfoia/pipeline/ocr.py	Fix closure late-binding hazard; annotate class constants; import ordering tweaks.
openfoia/pipeline/metadata.py	Simplify conditional logic while preserving semantics.
openfoia/pipeline/ingest.py	Replace deprecated UTC calls in stored metadata timestamps.
openfoia/pipeline/extract.py	Import ordering; typing modernizations; clarify optional-dep probe comments; simplify conditionals.
openfoia/pipeline/init.py	Import ordering and __all__ ordering.
openfoia/models.py	Add utcnow() helper (naive UTC); update defaults/comparisons away from deprecated datetime.utcnow(); typing cleanups.
openfoia/migrations/versions/001_initial_schema.py	Modernize typing imports/annotations.
openfoia/graph_template.py	Replace invisible U+2028/U+2029 literals with explicit escapes for reviewability.
openfoia/gateways/mail.py	Clarify optional-dep probe; standardize to datetime.now(UTC); minor unused-variable cleanup.
openfoia/gateways/fax.py	Standardize to datetime.now(UTC) and import ordering.
openfoia/gateways/email.py	Replace deprecated UTC usage with utcnow() helper; import ordering.
openfoia/gateways/init.py	Import ordering / __all__ ordering.
openfoia/ftm.py	Import ordering; clarify optional-dep probe comment.
openfoia/ftm_import.py	Import ordering.
openfoia/db.py	Typing import modernizations; use contextlib.suppress for chmod best-effort.
openfoia/crossref.py	Remove redundant pass in exception class.
openfoia/config.py	Use contextlib.suppress for best-effort permission tightening.
openfoia/cli.py	Replace unsafe persisted timestamp assignment with utcnow(); import ordering; from None for intentional re-raises; minor simplifications.
openfoia/campaign.py	Replace deprecated UTC usage with utcnow() helper; keep rendered dates consistent.
openfoia/agent.py	Replace deprecated/aliased UTC usage with utcnow() helper; refactor try/except to contextlib.suppress; import ordering.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[JordanCoin]

Valid — fixed in c9a9383. mypy confirms it:

openfoia/templates.py:39: error: Incompatible types in assignment
  (expression has type "None", variable has type "list[str]")  [assignment]

Went with list[str] | None = None rather than default_factory=list. The field is currently unused, so both are safe, but the surrounding dataclass fields (date_range_start, date_range_end, exclusions) already use | None = None, and switching to a factory would change the contract to "always a list" for a field nothing reads yet. Matching the existing style is the smaller claim.

Also fixed the other mypy error in the same file while I was there (list_templates() returning a bare list[dict] → list[dict[str, str]]). That file is mypy-clean now; mypy runs in CI as continue-on-error, which is why neither surfaced as a failure.

---

Generated by Claude Code