PaperTrade production runs on Project Babbage private Kubernetes infrastructure and uses a private local registry for image pushes and cluster pulls.
These notes document the live Project Babbage production shape. Public self-hosters can use the manifests and scripts as references, but the GitHub Actions workflow and registry defaults assume private infrastructure and production secrets.
The runtime image is split into two layers of ownership:
p2ppsr/papertrade-runtime-base:<tag>contains Node.js plus the heavy document conversion stack: Poppler, LibreOffice Writer, and Calibre.p2ppsr/papertrade:<short-sha>-production-<date>contains the application build, production node modules, migrations, and static frontend assets.
The runtime base image should change only when the OS, Node major version, or document conversion tooling changes. Normal PaperTrade source deployments should build only the app image.
For a local Docker build outside Project Babbage infrastructure:
docker build -f Dockerfile.runtime-base -t papertrade-runtime-base:local .
docker build -t papertrade:local .Production builds pass RUNTIME_BASE_IMAGE explicitly, so the public Dockerfile
default remains usable for local builders without changing the cluster workflow.
Production runs two replicas on separate nodes. Both use the internal S3 API
for publication files, rendered pages, avatars, and appearance assets. The
papertrade-s3-credentials Secret supplies a bucket-specific access key;
database, wallet, and S3 credentials are never stored in this repository.
/data/papertrade is an emptyDir scratch filesystem used only for upload,
conversion, rendering, and OCR work.
The deployment has a PDB with minAvailable: 1 and hard hostname anti-affinity.
Before node maintenance, require two Ready endpoints on separate nodes and a
healthy four-member object store. Existing filesystem content must be copied
from a read-only source mount without delete or sync semantics before enabling
the S3-backed deployment. Retain the source PVC and PV through cutover
validation and the first verified off-site backup.
scripts/k8s/build-local-image.sh uses the managed Linux/amd64 rootless
Docker daemon through the self-hosted runner. Its existing registry credentials
and TLS trust are required; anonymous insecure Kaniko builds are retired.
The persistent daemon reuses application dependency layers. Deliberate runtime
rebuilds use --pull --no-cache so distribution security updates are refreshed.
Every runtime base and application image is resolved to an immutable digest.
The workflow installs Python 3.13 for the guarded promotion and verifies every required runner command before building or writing Kubernetes resources.
The production scanner reads a Docker archive of the pulled deployment digest. The remote daemon and runner Pod have separate filesystems. The workflow extracts Trivy from its digest-pinned Linux/amd64 image and executes the binary on the runner, without starting a scanner container or mounting the workspace remotely. Production and Docker credential environment variables are removed for scanning. The existing critical/fixable-high policy remains mandatory before promotion.
Build only the normal app image:
scripts/k8s/build-local-image.shRebuild the heavy runtime base image deliberately:
BUILD_TARGET=runtime-base scripts/k8s/build-local-image.shSeed both the runtime base and app image in one run:
BUILD_TARGET=all scripts/k8s/build-local-image.shDeploy a previously built app image tag:
IMAGE_TAG=<tag> SQL_DATABASE_USER=<user> SQL_DATABASE_PASSWORD=<password> scripts/k8s/deploy-local.sh.github/workflows/deploy-production-local.yml runs on private self-hosted
runner labels:
linux-amd64dockerkubectllocal-registry
The workflow accepts an optional source_sha and a build_runtime_base switch.
Keep build_runtime_base=false for normal deploys. Set it to true only after
reviewing changes to Dockerfile.runtime-base, Node major versions, or document
conversion dependencies.
.github/workflows/image-security.yml builds the complete runtime image on a
GitHub-hosted Linux/amd64 runner for every runtime-input change, once a week,
and on manual dispatch. It scans the exact built image with digest-pinned
Trivy 0.73.0, retains the JSON report, and rejects any critical occurrence
or any high occurrence for which the distribution publishes a fixed version.
An unfixable critical can pass only when its exact CVE, binary package, and
installed version appear in
.github/security/trivy-critical-allowlist.json with a current review,
Debian-tracker source, risk rationale, and unexpired deadline. New, expired,
stale, or newly fixable criticals fail the build. Exception records are short:
the initial Debian Trixie set expires on 2026-09-07, so a weekly scan cannot
turn a temporary upstream wait into permanent acceptance.
The 2026-08-24 review accepted eight such occurrences for at most fourteen days. Debian classifies the GLib, Mbed TLS, libxml2, Perl Archive::Tar, 32-bit Perl regex, and Pillow findings as minor/no-DSA or postponed in Trixie. The runtime does not expose the affected D-Bus introspection, Mbed TLS termination, Perl archive extraction, or enormous/32-bit Perl regex paths. PaperTrade does process untrusted documents, so the libxml2 and Pillow exceptions remain deliberately short even though their specific XML and McIDAS AREA paths are not supported application inputs. Remove an exception as soon as its finding disappears; the gate rejects a stale record rather than silently accumulating waivers.
The scanner gate runs outside the production cluster. This keeps large LibreOffice and Calibre rebuild downloads off the Evans Creek Starlink links and prevents a security candidate build from competing with production pods. The production deploy remains a separate, explicitly dispatched workflow.
Production builds must pass the runtime policy on their exact immutable image
digest. deploy-local.sh requires IMAGE_TAG and IMAGE_DIGEST; the deployment
workflow supplies both from its build output and retains the scan and rollout
evidence, including on failure. The shared protocol-state upgrade has the
additive schema migration and compatibility requirements described below.
promote-guarded.py creates two candidate replicas on distinct nodes behind a
private Service, with a PDB and a copy of the existing production egress boundary.
Both replicas must serve health, status, catalog and a real stored free-page PNG;
an anonymous paid-page request must still be denied, and a synthetic authenticated
client must receive the configured signed 402 challenge with spending disabled.
The candidate's ten-second
preStop hook is exercised by withdrawing one candidate while another node serves
100 consecutive requests. Two exact-image Ready replicas must return before
promotion. Public root, health, catalog and stored-page probes run throughout.
The public Service then selects the verified candidate pool. Its EndpointSlices must name those exact Pods. Only after ten seconds of withdrawal may the legacy Deployment change, which also protects old Pods that did not have a drain hook. After the canonical pool observes its new generation and has two Ready, available replicas at the exact digest, both pass content checks and its PDB permits one disruption, traffic returns to it. The temporary pool drains before removal.
A failed or ambiguous cutover keeps the candidate pool intact. A failed canonical rollout leaves the two verified candidates serving; inspect the latched failure, repair or roll back the canonical Deployment, verify it, then explicitly switch back before removing candidates. Never rerun over a surviving candidate pool or delete that pool while the public Service selects it. A failure before cutover removes only the isolated candidate resources and preserves the old public pool. Network-ops fleet gates and independent public probes remain required around the workflow.
Migration 202609240001_shared_protocol_state.cjs creates auth_sessions,
auth_message_nonces and payment_replays before serving requests. Existing
wallet, content, purchase and payout rows are unchanged. MySQL protocol tables
use ASCII binary collation so distinct case-sensitive base64 nonces cannot
collapse onto the same primary key. Every replica must use the same application
database and server wallet identity. Session/nonce handling uses the published
KnexSessionManager; initial-request claims are capped at 256 per identity.
Expired sessions and orphaned nonces are pruned hourly with no overlapping prune
in one process. Cleanup failure is logged without granting authentication.
Both page and admin-funding payment middleware use the same atomic transaction-ID
claim table. Claims have no expiration: pruning accepted transaction IDs could
reopen replay of old payments. A duplicate returns false; database failures
propagate so the middleware fails closed. Keep these rows through restarts,
upgrades, restores and application rollback. The migration deliberately refuses
down; rolling back code must not delete protocol state.
Before promotion, take and verify an encrypted application-database backup. Rehearse the additive migration and check the MySQL table collations/primary keys. Old code ignores the extra tables, but old process-local sessions do not become shared: finish the guarded cutover before accepting replicated authentication. Do not remove the tables or roll back replay state while serving payments.
Candidate acceptance must include a non-spending authenticated paid-page request
that obtains a signed 402 challenge with the configured amount, with handshake
and requests deliberately sent to different replicas. The synthetic client's
createAction must throw before spending. Anonymous paid-page denial and free
rendered routes remain required. A real bounded paid test is separate operator
authorization and must verify the purchase ledger and entitlement, not merely an
HTTP status. On any failure, preserve the evidence and halt the release wave;
green free routes do not establish payment acceptance.