RelayForge

Run the live recovery demo · Method and hosting limits

The public demo runs the actual Python worker against a fixed sample receiver in a temporary SQLite database: two failed attempts, successful recovery and one receiver record after replay. It resets after each run; the PostgreSQL queue and full API run locally or in CI.

RelayForge is a durable webhook delivery service built to make failure visible and recoverable. It demonstrates how a service can deliver an event when a receiver is temporarily slow or unavailable. Producers publish once; RelayForge stores the work, signs each request, retries transient failures and keeps exhausted deliveries available for inspection and replay.

The API acknowledges an event without waiting for subscriber network calls. Database leases let multiple workers share the queue and recover abandoned work after a process crash.

Evidence

Engineering question	Repository evidence
Can two workers send the same due job concurrently?	PostgreSQL CI test starts competing workers and asserts that only one claim succeeds
What happens when a receiver returns 503?	Full-jitter retry policy honours Retry-After; the demo receiver fails twice before succeeding
How are forged or replayed requests handled?	HMAC-SHA256 covers timestamp and exact body; the example receiver rejects bad signatures and stale timestamps
What happens after a worker crashes?	Expiring leases make abandoned in-flight work eligible for another worker
Can an operator recover permanent failures?	Attempt history is persisted and dead-lettered deliveries have an explicit replay operation
Is the PostgreSQL path exercised?	GitHub Actions applies Alembic migrations and runs the concurrency test against PostgreSQL 17 on Python 3.11 and 3.12

For a focused review, follow the five-minute evaluation guide. Recovery recording · Verification and CV notes For diagnosing failed deliveries and worker recovery, use the local incident runbook.

What it demonstrates

• Python 3.11+, FastAPI, Pydantic, SQLAlchemy 2 and Alembic
• Durable background work with atomic claims and expiring worker leases
• At-least-once delivery, producer idempotency and endpoint-level deduplication
• Async HTTP I/O with bounded concurrency
• HMAC-SHA256 request signing over canonical JSON
• Full-jitter exponential backoff, Retry-After support and dead-letter replay
• SSRF-oriented target validation and redirect blocking
• Structured logs, request IDs, readiness checks and Prometheus-format metrics
• Unit, API and worker integration tests with mocked transports
• A runnable signed receiver that demonstrates transient failure and idempotent consumption
• Docker Compose, PostgreSQL-backed CI, non-root containers and GitHub Actions quality gates

Architecture

Producer
   |
   | POST /v1/events + idempotency key
   v
+-------------+       atomic claim + lease       +------------------+
| FastAPI API | --------------------------------> | Delivery workers |
+------+------+                                   +--------+---------+
       |                                                   |
       | event, delivery and audit rows                    | signed POST
       v                                                   v
+-------------+                                    Subscriber endpoints
| SQL database|
+-------------+
       ^
       | delivery status, attempts, replay
       |
   Operator API / metrics

The API and workers do not share in-memory queue state. A delivery is claimed with a conditional database update and a time-limited lease. If a worker dies after claiming work, another worker can reclaim it after the lease expires. Finalisation checks lease ownership so a stale worker cannot overwrite a newer result.

Run locally

Create the environment and install the application:

python -m venv .venv
.venv/Scripts/activate
python -m pip install -e ".[dev]"

On macOS or Linux, activate with:

source .venv/bin/activate

Set a non-default API key, apply the migrations, then start one API process and one worker:

set RELAYFORGE_API_KEY=local-development-key
relayforge migrate
relayforge api
relayforge worker

PowerShell users can set the key with:

$env:RELAYFORGE_API_KEY = "local-development-key"

Interactive OpenAPI documentation is available at http://127.0.0.1:8000/docs.

Run the failure-recovery demo

The demo overlay starts the platform plus a receiver that verifies signatures, rejects stale requests, deduplicates delivery IDs and deliberately returns 503 for the first two attempts.

docker compose -f docker-compose.yml -f docker-compose.demo.yml up --build

Use the requests in docs/evaluation-guide.md to register the receiver and publish an event. The delivery moves from pending to retrying and then succeeded, while the receiver records the event once.

Register a subscriber

Public HTTPS destinations are required by default. For a local receiver, explicitly set RELAYFORGE_ALLOW_PRIVATE_TARGETS=true and RELAYFORGE_ALLOW_INSECURE_HTTP=true.

curl -X POST http://127.0.0.1:8000/v1/endpoints ^
  -H "Content-Type: application/json" ^
  -H "X-API-Key: local-development-key" ^
  -d "{\"name\":\"orders\",\"url\":\"https://example.com/webhooks\",\"event_types\":[\"order.created\"]}"

The signing secret is returned only by the create response. Store it securely.

Publish an event

curl -X POST http://127.0.0.1:8000/v1/events ^
  -H "Content-Type: application/json" ^
  -H "X-API-Key: local-development-key" ^
  -d "{\"source\":\"checkout\",\"event_type\":\"order.created\",\"idempotency_key\":\"checkout-ord-42-v1\",\"payload\":{\"order_id\":\"ord-42\",\"total\":1299}}"

Reusing the same source and idempotency key with the same event returns the original event. Reusing it with a different type or payload returns HTTP 409.

Delivery contract

Each request body is a stable JSON envelope with id, type, created_at and data fields. RelayForge adds these headers:

Header	Purpose
X-RelayForge-Event-Id	Stable producer event identifier
X-RelayForge-Delivery-Id	Stable endpoint-specific delivery identifier
X-RelayForge-Attempt	One-based attempt number
X-RelayForge-Timestamp	Unix timestamp included in the signature
X-RelayForge-Signature	v1 HMAC-SHA256 signature

Consumers should verify the signature, reject stale timestamps and make processing idempotent using the delivery or event ID. At-least-once delivery means duplicates remain possible after ambiguous network failures.

Failure policy

Outcome	Action
2xx	Mark succeeded
408, 425, 429 or 5xx	Retry with full-jitter exponential backoff
Valid Retry-After	Wait at least that long, bounded by the configured maximum
Other 3xx or 4xx	Dead-letter immediately
Timeout or transport error	Retry
Unsafe target at delivery time	Dead-letter immediately
Attempts exhausted	Dead-letter

Dead-lettered deliveries can be inspected and replayed through POST /v1/deliveries/{delivery_id}/replay.

Operations

• GET /healthz checks process liveness.
• GET /readyz verifies database connectivity.
• GET /metrics exposes queue depth by state and the persisted attempt count.
• X-Request-Id is accepted or generated and returned on every response.
• Audit rows record endpoint changes, event publication, claims, retry scheduling, success, dead-lettering and manual replay.

For local evaluation, SQLite is the zero-setup default. Docker Compose uses PostgreSQL and runs one migration job plus separate API and worker containers.

docker compose up --build

Quality checks

ruff check .
ruff format --check .
mypy src
pytest

The test suite covers signing and tamper detection, SSRF rules, retry timing, subscription fan-out, idempotency conflicts, lease expiry, stale-worker protection, API error contracts, signed delivery, retry-to-success, dead-letter behaviour, receiver deduplication and competing PostgreSQL claims.

Engineering decisions and limits

• Database-backed work keeps the failure model inspectable and avoids requiring a broker for a local evaluation. PostgreSQL is the intended multi-worker store; SQLite is for development.
• Conditional updates provide exclusive claims without holding a database transaction open during network I/O.
• Endpoint secrets are stored in the application database for this version. A production deployment should envelope-encrypt them with a cloud KMS and support rotation.
• Each delivery resolves its target off the event loop and connects to a checked IP. HTTPS still verifies the original hostname. Redirects and environment proxies are disabled. Connections are not reused: different hostnames may share an IP. Only the first validated address is tried per attempt; normal retries resolve again. DNS timeout stops waiting, not the underlying OS lookup.
• Metrics are derived from persisted state, so they survive process restarts. High-volume deployments should export counters and histograms to a dedicated telemetry backend.

See docs/architecture.md, docs/threat-model.md and docs/adr/0001-database-backed-leases.md for the detailed reasoning.

Suggested CV wording

RelayForge - Python, FastAPI, SQLAlchemy, PostgreSQL, Docker

• Built a durable webhook platform with idempotent event ingestion, HMAC-signed delivery, leased multi-worker claims, bounded async concurrency and replayable dead-letter handling.
• Added full-jitter retries with Retry-After support, SSRF controls, audit history, Prometheus-format metrics, database migrations and automated API/worker integration tests.

Licence

MIT