SQLAlchemy v2: production fixes on top of #150

[djgalloway]

Builds on #150 (SQLAlchemy 2 port by @zmc). This branch merges main into that work and fixes the problems that surface when it runs against a full copy of the production Sepia database. Opening it separately rather than pushing onto #150 so the port and the fixes can be reviewed independently; happy to fold it in if preferred.

Every fix has a regression test in paddles/tests/controllers/test_teuthology_flows.py. tox is green (201 passed, 5 skipped) and flake8 is clean.

How it was tested

Cloned the production paddles-db (41 GB, 138k runs, 8.6M jobs, 1,156 nodes) into a throwaway namespace, built this branch's image in-cluster, applied the migration to the copy, and drove it with the exact HTTP flows teuthology (report.py, lock/ops.py, lock/query.py, dispatcher/supervisor.py) and pulpito use, comparing every read against production.

Deployment / driver

• paddles.db.get_engine() rewrites postgresql:// (and postgresql+psycopg2://) URLs to postgresql+psycopg://. Under SQLAlchemy 2 a bare postgresql:// URL selects psycopg2, which this PR removes, so the deployed secret and Crunchy's generated pgbouncer-uri would otherwise fail at startup and in the DB-connectivity probes. alembic/env.py normalizes the same way and its PGPASS hook uses psycopg 3.

Migration b6c4013b876e

• Chained after 3eb94eceb2cc (Job.package_source, Add Job.package_source #151) instead of cf54532418e7, so alembic upgrade head works on a database that already carries package_source.
• Added a data-cleanup step before the NOT NULL / UNIQUE constraints. Production data violates all of them: 3 nodes with up IS NULL, 40 legacy nodes with machine_type IS NULL, 78 job rows with job_id IS NULL (they carry no other data and are unreachable through the API), and 112 duplicate (run_id, job_id) pairs. Without the cleanup the migration aborts on its first statement and, under container_start.sh's set -e, every new pod crashloops.
• Named the unique constraint uq_jobs_run_id_job_id so downgrade() can drop it; the previous downgrade() could not compile (DROP CONSTRAINT on an unnamed constraint).
• The ALTER TABLEs hold ACCESS EXCLUSIVE on jobs for a full-table scan (~13 s for 8.5M rows). The migration docstring says to deploy it deliberately, not through an unattended ImageStream rollout.

Job / Run behavior

• Job.updated and the parent Run.updated advance on every write again, including a status-less PUT — which the supervisor watchdog sends purely to bump updated, and which stale-lock detection and pulpito runtimes read.
• job_nodes is populated on update as well as insert. teuthology sends targets by PUT after locking, not at POST, so linking only on insert left /nodes/<name>/jobs/, node job_stats, the reimage-failure detector and testnode-monitor with no new data.
• Run.status is a stored, indexed column again, recomputed via Run.set_status() / refresh_status() when a job's status changes or a job is deleted. /runs/status/<status>/ (pulpito's running-runs filter) went from ~25 s of correlated-subquery full scans back to sub-second.
• success is stored alongside status (teuthology's set_status() always sends both).
• Job.update() sets only real columns and reconciles status/success; unknown or protected keys (id, posted, ...) no longer 500, and an invalid status is a 400, not a 500 (the validator's %-format on a tuple was raising TypeError).
• POST accepts a job whose body carries "queue": null next to a job_id, and the generated-id path ignores non-numeric ids.

Controllers / hooks

• Trailing-slash redirects keep their Location header: the error hook lets 3xx through, and the session hook no longer rolls back on a redirect or other <400 response. teuthology's get_run() / get_jobs(run, id) build slash-less URLs and rely on following the redirect; previously they got a Location-less 302 and a {"message": "Found"} body.
• /jobs/?posted_after=&posted_before= compares against parsed datetimes, not raw strings (500 under psycopg 3's typed parameters).
• The run filter-index endpoints (/runs/branch/, /runs/machine_type/, ...) dedupe and drop NULLs again; a single NULL value was 500ing the whole list.
• CorsHook and SentryHook are wired back into the app.

Timestamps

• Added paddles.util.utcnow() (naive UTC) and use it wherever a timestamp is written, so stored values and JSON output don't depend on the server's TimeZone and don't drift the way tz-aware writes into timestamp without time zone columns can.

Maintenance commands

• Ported every pecan command (expire_jobs, set_status, dedupe, delete, reparse, node_jobs, node_stats, queue_stats, set_targets) onto a small SessionCommand base that opens a real SQLAlchemy 2 session, replacing the removed models.start() / commit() / Query API that made them fail to import. Dropped import_nodes, which targeted a lock server that no longer exists.

---

🤖 Prepared with Claude Fable 5.1.