UTMIST | #54 [task]: Add API readiness probes and bot health checks

[kr1shap]

What this changes

Adds unauthenticated /health/ready probes to team-tracking and documentation-system that check each service's own Postgres connection and return a service-specific 503 when unavailable. The Discord bot now serves /health/ready on Railway's PORT (default 3002 locally), returning 503 until its gateway is ready or after a disconnect. Railway checks the readiness routes; the APIs' DB-independent /health liveness routes remain available.

Why

Closes #54. Part of the observability epic #46. Previously the two APIs could report healthy while their databases were down, and Railway had no Discord gateway readiness signal for the bot.

Zone

services/team-tracking · services/documentation-system · discord-bot · docs

This PR deliberately spans zones because #54 requires both DB-backed APIs and the Discord bot to participate in the same Railway readiness rollout. The shared deployment runbook also needs to describe the new paths and bot binding.

How to verify

Completed locally without Docker:

cd services/team-tracking
uv run pytest --ignore=tests/test_postgres_adapter.py -q
uv run ruff check .
uv run ruff format --check .

cd ../documentation-system
uv run pytest --ignore=tests/test_postgres_adapter.py -q
uv run ruff check .
uv run ruff format --check .

cd ../../discord-bot
npm ci
npm test
npm run lint
npm run format:check

The offline API tests use fake healthy and unreachable engines and assert /health/ready returns 200/503 while /health stays 200. The bot test uses a fake client to assert 503 -> 200 -> 503 across disconnect, readiness, and disconnect again.

DOCKER NEEDED BELOW

cd services/team-tracking
docker compose up -d postgres
uv run alembic upgrade head
uv run pytest

cd ../documentation-system
docker compose up -d postgres
uv run alembic upgrade head
RUN_PG_TESTS=1 uv run pytest -q

Then request both APIs' /health and /health/ready without an API key. With their databases up, both should return 200. With a database stopped, /health should remain 200 while /health/ready returns 503 with that service's name. Restore the database afterward. In Discord mode, GET http://localhost:3002/health/ready should return 200 once the gateway is ready.

Checklist

• Branched off staging and targeting staging.
• Ran the service's full suite locally, including Postgres.
• uv run ruff check . and uv run ruff format --check . clean for both Python services.
• Read the affected services' docs/CONTRIBUTING.md guides and completed their Docker-free checks.
• Updated API, service deployment, bot README and .env.example, and Railway deployment docs.

Deployment notes

• Railway now probes /health/ready for team-tracking, documentation-system, and discord-bot. A new deployment waits for its own DB or Discord gateway before becoming healthy.
• Bot health binds to Railway's injected PORT; local Discord mode uses port 3002 when PORT is unset. The web playground continues using WEB_PORT=3001.
• No migration, new required secret, API-key rotation, or Discord command registration is needed.

[qiuethan]

Approving. is_ready is Protocol-first and in both adapters, the unauthenticated 503 body is static (no DSN or DB error text), /health stays DB-independent, docs/railway.json/.env.example all updated. Fast suites + ruff green locally for both services, bot test/lint/format green. Not run: test_postgres_adapter.py (no local Postgres).

Worth a follow-up, non-blocking:

• discord-bot/src/index.js:40 + discord-bot/railway.json: the health listener only starts when ENABLE_DISCORD=true, but the Railway healthcheck is unconditional. A web-only deploy (ENABLE_DISCORD=false ENABLE_WEB=true) sits 300s and fails healthcheck with no hint why. Either bind unconditionally and return 503 with no client, or add one sentence to discord-bot/README.md and docs/RAILWAY-DEPLOYMENT.md.
• services/*/src/storage/postgres.py is_ready: the failure is swallowed silently, so an operator sees 503 with no log line. Log type(exc).__name__ only (never str(exc), psycopg messages include host/user). Same shape in both services.

Nits: is_ready sits above __init__ in both in_memory.py files but below it in postgres.py; services/team-tracking/docs/DEPLOYMENT.md post-deploy checklist replaced the runnable railway run … curl command with prose, keep the command with the new path.