diff --git a/docs/PROJECT_STATE.md b/docs/PROJECT_STATE.md index 0a7bc672..47df1028 100644 --- a/docs/PROJECT_STATE.md +++ b/docs/PROJECT_STATE.md @@ -6,7 +6,7 @@ > to read **only this file** and continue immediately. Updated after every > milestone and every significant architectural step. -_Last updated: 2026-09-05 — M15 Increment 52: deterministic analysis-cache cold-race test._ +_Last updated: 2026-09-07 — PR #51: backup/restore drill safety regressions._ Prior: _Last updated: 2026-09-05 — M15 Increment 51: Signature B mechanism isolation and diagnostic hardening._ @@ -4123,6 +4123,13 @@ Per package: `cd packages/ && npm install && npm run build && npm test`. - **Recorded gaps**: GraphQL `Player` has no `teams` field despite ADR-0073's Context claiming it; five pre-existing design-system findings in `style.css` are reported, not repaired (fixing drift inside a feature PR is how a design-system change ships unreviewed). - Detailed in `docs/adr/0074-social-ui-profile.md`. +## PR #51 — Backup/restore drill safety regressions — 2026-09-07 + +- Fixed the restore tooling scope error and made every nonzero restore exit fatal. Both dump formats require the baseline's exported snapshot; unreadable tables fail the baseline instead of losing count coverage. +- Bounded generated target names to 63 ASCII bytes, reject oversized explicit names and connection query overrides, escape catalog-derived table identifiers, and preserve row counts for special property names. +- Reserve backup files exclusively and clean up only resources created by the drill. Preserve restore and cleanup errors together while continuing other cleanup. Redact connection secrets from diagnostics, including CLI argument errors. +- Verify append-only protection and valid, ready HNSW indexes on their specific public-schema relations. Added database-boundary and disposable-file regressions for native/Docker custom/plain orchestration and failure paths; live integration remains opt-in and was not run against an existing database. + diff --git a/docs/runbooks/backup-restore-drill.md b/docs/runbooks/backup-restore-drill.md new file mode 100644 index 00000000..c7e7341c --- /dev/null +++ b/docs/runbooks/backup-restore-drill.md @@ -0,0 +1,272 @@ +# Gambit — Database Backup & Restore Drill Runbook + +> **Audience:** Operators, SREs, Database Administrators.
+> **Frequency:** Monthly scheduled drill, pre-release rehearsal for major schema migrations, disaster recovery validation.
+> **Target RTO:** < 15 minutes for 100k games.
+> **Target RPO:** 0 data loss for committed transactions (append-only event store). + +--- + +## 1. Objectives & Overview + +Gambit's system of record relies on PostgreSQL 16 with the `citext` and `vector` (pgvector) extensions. The primary source of truth is the append-only `game_events` event store, supported by relational projections (`users`, `credentials`, `games`, `ratings`, `tournaments`, `search_embeddings`). + +This runbook defines the operational procedure to: +1. Generate a consistent backup of the application database without downtime. +2. Restore the backup into a brand-new, isolated target database. +3. Validate that all durable application data, triggers, indexes, and extensions survived. +4. Ensure safety: destructive actions are strictly guarded against touching the primary or production databases. + +--- + +## 2. Automated Drill Execution + +The platform includes an automated, operator-usable drill tool: [`scripts/db-backup-restore-drill.mjs`](../../scripts/db-backup-restore-drill.mjs). + +### 2.1 Basic Usage + +Run against the default `DATABASE_URL` or an explicit source URL: + +```bash +# Run with default environment DATABASE_URL +node scripts/db-backup-restore-drill.mjs + +# Run with explicit source URL +node scripts/db-backup-restore-drill.mjs --source-url "postgres://gambit:secret@localhost:5432/gambit" +``` + +The script will automatically: +1. Connect to the source and record a baseline of tables, row counts, migrations, and sample data. +2. Resolve pg tooling (native `pg_dump`/`pg_restore` if installed, or automatic Docker `pgvector/pgvector:pg16` fallback). +3. Create a custom-format archive (`-Fc`). +4. Validate the backup file size and `PGDMP` header magic. +5. Create a disposable isolated target database (e.g. `gambit_backup_drill_restore__`). +6. Restore the backup into the isolated database. +7. Perform deep structural and functional verification. +8. Drop the isolated target database and clean up the temporary dump file. +9. Print a structured diagnostic summary. + +### 2.2 CLI Options & Flags + +| Option | Default | Description | +|---|---|---| +| `--source-url ` | `$DATABASE_URL` | Source database URL (defaults to DATABASE_URL) | +| `--target-url ` | Auto-generated | Explicit target database URL (must be isolated) | +| `--target-db-name ` | Auto-generated | Target database name (default: auto-generated isolated name) | +| `--backup-file ` | Temp file | Path for backup dump file (default: temporary file) | +| `--keep-backup` | `false` | Preserve the backup dump file after the drill | +| `--keep-target` | `false` | Preserve the restored target database after the drill | +| `--format ` | `custom` | pg_dump format (default: custom) | +| `--use-docker` | `auto` | Force execution of pg tools via Docker container | +| `--docker-image ` | `pgvector/pgvector:pg16` | Docker image for pg tools (default: pgvector/pgvector:pg16) | +| `--allow-custom-target-name`| `false` | Permit target name without default isolation markers | +| `--json` | `false` | Output drill report in JSON format | +| `--help` | `false` | Show help message | + +The backup path must not already exist. The drill reserves a new file exclusively +and removes only the file it created. Target names are limited to 63 ASCII bytes; +generated names retain their isolation marker and unique suffix within that limit. +Connection URL query parameters are restricted to `sslmode`, `sslcert`, `sslkey`, +and `sslrootcert`, so the JavaScript client and PostgreSQL tools use the same +host, port, and credentials. + +Both dump formats use the source transaction's exported snapshot. Snapshot export +or row-count failures abort the drill. Every nonzero restore exit is fatal, +including warning-only or localized diagnostics. Cleanup is attempted after a +failure; a failed database drop or backup removal also fails the drill, and combined +restore and cleanup failures are reported together. + +### 2.3 Retaining the Target Database for Forensic Inspection + +When diagnosing schema discrepancies or inspecting restore behavior, instruct the drill to keep the restored database: + +```bash +node scripts/db-backup-restore-drill.mjs \ + --source-url "postgres://gambit:secret@localhost:5432/gambit" \ + --keep-target \ + --keep-backup +``` + +Output will report the exact target database name created: +```text +Target: postgres://gambit:***@localhost:5432/gambit_backup_drill_restore_1788672281592_96c80ce5 +Backup: /tmp/gambit_backup_1788672281593_2706.dump +``` + +When finished with forensic analysis, drop the database manually: +```sql +DROP DATABASE "gambit_backup_drill_restore_1788672281592_96c80ce5" WITH (FORCE); +``` + +--- + +## 3. Manual Operator Walkthrough + +If the automated script is unavailable, perform the drill manually using standard PostgreSQL client utilities (`pg_dump`, `pg_restore`, `psql`). + +### Step 1: Create Backup + +Generate a PostgreSQL custom archive format backup. The custom format includes a table of contents (TOC) for selective restore and parallel processing: + +```bash +pg_dump \ + -h "${PGHOST:-localhost}" \ + -p "${PGPORT:-5432}" \ + -U "${PGUSER:-gambit}" \ + -d "${PGDATABASE:-gambit}" \ + -F c \ + -f "/tmp/gambit_manual_drill_$(date +%s).dump" +``` + +### Step 2: Validate Backup File Integrity + +Ensure the dump file is non-empty and starts with the PostgreSQL dump magic bytes (`PGDMP`): + +```bash +# Check size +ls -lh /tmp/gambit_manual_drill_*.dump + +# Check header magic (should print PGDMP) +head -c 5 /tmp/gambit_manual_drill_*.dump +``` + +### Step 3: Provision Clean Isolated Target Database + +Connect to the PostgreSQL administrative database (`postgres` or `template1`) and create a dedicated drill database: + +```bash +TARGET_DB="gambit_backup_drill_restore_$(date +%s)" + +psql -h "${PGHOST:-localhost}" -p "${PGPORT:-5432}" -U "${PGUSER:-gambit}" -d postgres -c " + CREATE DATABASE \"${TARGET_DB}\"; +" +``` + +### Step 4: Restore into Isolated Target + +Restore the custom archive into the target database: + +```bash +BACKUP_FILE=$(ls -t /tmp/gambit_manual_drill_*.dump | head -n 1) + +pg_restore \ + -h "${PGHOST:-localhost}" \ + -p "${PGPORT:-5432}" \ + -U "${PGUSER:-gambit}" \ + -d "${TARGET_DB}" \ + --clean \ + --if-exists \ + "${BACKUP_FILE}" +``` + +### Step 5: Verification Checklist + +Connect to the restored database (`${TARGET_DB}`) and execute the following checks: + +#### 1. Extension Verification +Verify that both `citext` and `vector` extensions exist: +```sql +SELECT extname, extversion FROM pg_extension WHERE extname IN ('citext', 'vector'); +-- Expected: 2 rows (citext and vector) +``` + +#### 2. Schema Migrations Ledger +Confirm that the migrations ledger exists in both the source and restored +databases before comparing it: +```sql +SELECT to_regclass('public.schema_migrations') IS NOT NULL AS present; +``` +Both queries must return `present = true`. If either database is missing the +table, stop the drill and investigate before continuing. + +Then run the same query against the source and restored databases: +```sql +SELECT version, name, checksum, state +FROM schema_migrations +ORDER BY version; +``` +The two ordered result sets must match exactly, and every row must have +`state = 'applied'`. + +#### 3. Append-Only Trigger Verification +Verify that the `game_events` immutability trigger is active by testing an `UPDATE`: +```sql +BEGIN; +UPDATE game_events SET seq = seq WHERE game_id IN (SELECT game_id FROM game_events LIMIT 1); +-- Expected error: "game_events is append-only (UPDATE.game_events attempted)" +ROLLBACK; +``` +If the statement succeeds without raising an exception, the trigger is missing or inactive! + +#### 4. Durable State Row Counts +Compare row counts between source and restored databases: +```sql +SELECT 'users' AS tbl, count(*) FROM users +UNION ALL +SELECT 'game_events', count(*) FROM game_events +UNION ALL +SELECT 'games', count(*) FROM games +UNION ALL +SELECT 'tournaments', count(*) FROM tournaments +UNION ALL +SELECT 'ratings', count(*) FROM ratings +UNION ALL +SELECT 'search_embeddings', count(*) FROM search_embeddings; +``` + +#### 5. Vector Query Functionality +Verify pgvector cosine distance operations and HNSW indexing: +```sql +-- Test cosine distance operator (<=>) +SELECT id, embedding <=> embedding AS distance FROM search_embeddings LIMIT 1; +-- Verify HNSW index +SELECT i.relname, am.amname, ix.indisvalid, ix.indisready +FROM pg_index ix +JOIN pg_class i ON i.oid = ix.indexrelid +JOIN pg_class t ON t.oid = ix.indrelid +JOIN pg_am am ON i.relam = am.oid +WHERE t.oid = 'public.search_embeddings'::regclass + AND am.amname = 'hnsw' + AND ix.indisvalid = true + AND ix.indisready = true; +``` + +### Step 6: Cleanup Isolated Target + +After verification succeeds, reconnect to the administrative `postgres` +database before dropping the temporary drill database. PostgreSQL cannot drop +the database used by the current session, even with `WITH (FORCE)`: + +```bash +psql -h "${PGHOST:-localhost}" -p "${PGPORT:-5432}" -U "${PGUSER:-gambit}" -d postgres -c " + DROP DATABASE \"${TARGET_DB}\" WITH (FORCE); +" +``` + +--- + +## 4. Safety & Isolation Guardrails + +1. **Never Overwrite Source:** The drill script explicitly compares the normalized source and target URLs. If the host, port, and database name match, execution aborts immediately. +2. **Protected Databases:** Destructive commands will refuse to drop databases named `gambit`, `postgres`, `template1`, `production`, `master`, or `main`. +3. **Naming Convention:** Target databases must contain an isolation indicator (`drill`, `restore`, `disposable`, `test`, or `isolated`) unless the `--allow-custom-target-name` flag is explicitly provided. +4. **Credential Security:** Database URLs and known connection passwords are masked in diagnostics. PostgreSQL tool subprocesses receive passwords through `PGPASSWORD`. Prefer setting `DATABASE_URL` when invoking the drill; a URL supplied through `--source-url` or `--target-url` is still visible in the drill process's arguments. + +The structural checks require the append-only trigger on `public.game_events` +and a valid, ready HNSW index on `public.search_embeddings`; identically named +objects on another relation do not satisfy verification. The tests in +`scripts/test/backup-restore-safety.test.mjs` exercise orchestration and failure +paths without contacting a database. The separate live integration test remains +opt-in through `DATABASE_URL`; use a disposable, migrated database for that test. + +--- + +## 5. Troubleshooting Failed Restores + +| Symptom | Probable Cause | Corrective Action | +|---|---|---| +| `extension "vector" is not available` | Target PostgreSQL instance lacks `pgvector` library | Install `postgresql-16-pgvector` package or use `pgvector/pgvector:pg16` image. | +| `Trigger on game_events failed: mutation was not blocked` | Restore command stripped or disabled triggers | Ensure `pg_restore` did not run with `--disable-triggers` without re-enabling them. | +| `Row count mismatch for table "X"` | Partial dump or table-level exclusion | Ensure `pg_dump` was run for the whole database without `--schema-only` or `--exclude-table`. | +| `Migration checksum mismatch` | Working copy newline translation or modified migration | Run `npm run check:ci-parity` and ensure canonical LF newlines in migration files. | +| `Backup file does not exist or empty` | Permissions or disk space exhaustion | Check disk space in `/tmp` and file write permissions for PostgreSQL process. | diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs new file mode 100644 index 00000000..bd03e0a5 --- /dev/null +++ b/scripts/db-backup-restore-drill.mjs @@ -0,0 +1,1133 @@ +#!/usr/bin/env node +/** + * @file scripts/db-backup-restore-drill.mjs + * + * Reproducible, operator-usable PostgreSQL backup and restore verification drill. + * + * What this script does: + * 1. Validates connection parameters and enforces target database isolation guardrails. + * 2. Connects to the source application database and captures a baseline of durable state: + * - Schema migrations ledger and SHA-256 checksums + * - Required extensions (citext, vector) + * - Critical application tables and exact row counts + * - Representative durable sample records (users, game events, tournaments, etc.) + * 3. Creates a standard PostgreSQL backup using pg_dump (-Fc custom archive format with TOC). + * 4. Validates the backup artifact (file size, header magic "PGDMP", integrity). + * 5. Provisions a clean, isolated target database (never overwriting source). + * 6. Restores the backup into the isolated target using pg_restore. + * 7. Performs deep structural and functional verification: + * - All required extensions are active (citext, vector) + * - Migration count and checksums match source byte-for-byte + * - All critical application tables are present + * - Row counts across all durable tables match 100% + * - Sample durable records match source data + * - Append-only trigger (game_events_block_mutate) is active and blocks UPDATE/DELETE + * - pgvector operations and HNSW indexes are valid and queryable + * - Target database is functional and writable + * 8. Cleans up isolated target database and temporary dump file (unless flags request retention). + * 9. Emits a structured diagnostic audit report. + * + * Security: + * - Passwords and credentials are NEVER logged or printed. + * - PGPASSWORD is passed via process environment, not command-line arguments. + * - Destructive actions are restricted to isolated targets; refusing any target matching source. + * + * Usage: + * node scripts/db-backup-restore-drill.mjs [options] + * + * Options: + * --source-url Source database URL (defaults to DATABASE_URL) + * --target-url Explicit target database URL (must be isolated) + * --target-db-name Target database name (default: auto-generated isolated name) + * --backup-file Path for backup dump file (default: temporary file) + * --keep-backup Preserve the backup dump file after the drill + * --keep-target Preserve the restored target database after the drill + * --format pg_dump format (default: custom) + * --use-docker Force execution of pg tools via Docker container + * --docker-image Docker image for pg tools (default: pgvector/pgvector:pg16) + * --allow-custom-target-name Permit target name without default isolation markers + * --json Output drill report in JSON format + * --help Show this help message + */ + +import { execFileSync, spawnSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { existsSync, statSync, unlinkSync, readFileSync, openSync, readSync, closeSync, createReadStream } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve, dirname, basename } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { lookup } from 'node:dns/promises'; +import { pipeline } from 'node:stream/promises'; +import pg from 'pg'; + +const { Pool, Client } = pg; + +/** Critical application tables that hold durable state or lookup vocabularies. */ +export const CRITICAL_APPLICATION_TABLES = [ + 'schema_migrations', + 'variants', + 'terminations', + 'users', + 'credentials', + 'webauthn_credentials', + 'sessions', + 'roles', + 'ratings', + 'seeks', + 'game_events', + 'games', + 'tournaments', + 'audit_log', + 'search_documents', + 'search_embeddings', +]; + +/** Required PostgreSQL extensions for the Gambit platform. */ +export const REQUIRED_EXTENSIONS = ['citext', 'vector']; + +/** Names of databases that must never be targeted for destructive drop/overwrite. */ +const PROTECTED_DATABASE_NAMES = new Set([ + 'gambit', + 'postgres', + 'template0', + 'template1', + 'production', + 'prod', + 'master', + 'main', +]); + +/** Isolation keywords required in target database names unless explicitly overridden. */ +const ISOLATION_MARKERS = /(?:drill|restore|disposable|test|isolated)/i; + +/** + * Mask passwords in database connection strings for safe logging and error reporting. + */ +export function sanitizeDatabaseUrl(urlString) { + if (typeof urlString !== 'string' || !urlString) { + return urlString || ''; + } + try { + const parsed = new URL(urlString); + if (parsed.password) { + parsed.password = '***'; + } + for (const key of parsed.searchParams.keys()) { + if (/password|secret|token/i.test(key)) parsed.searchParams.set(key, '***'); + } + parsed.hash = ''; + return parsed.toString(); + } catch { + // Regex fallback for partially malformed or non-standard connection strings + return urlString.replace(/(:\/\/)([^:@]+)(?::([^@]+))?(@)/, '$1$2:***$4'); + } +} + +/** Redact known connection secrets and embedded URLs from process diagnostics. */ +function sanitizeDiagnostic(message, urls) { + let result = String(message).replace(/postgres(?:ql)?:\/\/[^\s]+/gi, sanitizeDatabaseUrl); + const secrets = new Set(); + for (const url of urls) { + try { + const parsed = new URL(url); + if (parsed.password) { + secrets.add(parsed.password); + secrets.add(decodeURIComponent(parsed.password)); + } + for (const [key, value] of parsed.searchParams) { + if (value && /password|secret|token/i.test(key)) secrets.add(value); + } + } catch { /* Invalid URLs are handled by argument validation. */ } + } + for (const secret of [...secrets].sort((a, b) => b.length - a.length)) { + result = result.replaceAll(secret, '***'); + } + return result; +} + +/** + * Generate a unique, isolated database name for the restore drill. + */ +export function generateIsolatedDbName(base = 'gambit') { + const timestamp = Date.now(); + const random = Math.floor(Math.random() * 0xffffffff).toString(16).padStart(8, '0'); + const suffix = `_backup_drill_restore_${timestamp}_${random}`; + const prefix = base.replace(/[^a-zA-Z0-9_-]/g, '_') || 'gambit'; + return `${prefix.slice(0, 63 - suffix.length)}${suffix}`; +} + +export function parseDatabaseUrl(urlString) { + const parsed = new URL(urlString); + const dbName = parsed.pathname.replace(/^\//, '') || 'postgres'; + const sslParams = {}; + const supportedQueryParams = new Set(['sslmode', 'sslcert', 'sslkey', 'sslrootcert']); + for (const [key, value] of parsed.searchParams.entries()) { + if (!supportedQueryParams.has(key)) throw new Error('Unsupported database URL query parameter; only sslmode, sslcert, sslkey, and sslrootcert are allowed'); + const envKey = 'PG' + key.toUpperCase(); + sslParams[envKey] = value; + } + return { + host: parsed.hostname || 'localhost', + port: parsed.port ? parseInt(parsed.port, 10) : 5432, + user: decodeURIComponent(parsed.username || 'postgres'), + password: decodeURIComponent(parsed.password || ''), + database: decodeURIComponent(dbName), + searchParams: parsed.searchParams, + sslParams, + protocol: parsed.protocol, + }; +} + +/** + * Construct a connection URL replacing only the database name. + */ +export function urlWithDatabase(urlString, databaseName) { + const parsed = new URL(urlString); + parsed.pathname = `/${encodeURIComponent(databaseName)}`; + return parsed.toString(); +} + +/** + * Enforce target isolation guardrails to guarantee the drill cannot overwrite or drop source/production DBs. + */ +export async function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { + if (!sourceUrl) { + throw new Error('Source database URL is required'); + } + if (!targetUrl) { + throw new Error('Target database URL is required'); + } + + const source = parseDatabaseUrl(sourceUrl); + const target = parseDatabaseUrl(targetUrl); + + if (!/^[a-zA-Z0-9_-]+$/.test(target.database)) { + throw new Error('Invalid target database name format'); + } + if (Buffer.byteLength(target.database, 'utf8') > 63) { + throw new Error('Target database name must not exceed 63 bytes'); + } + + // 1. URLs must not be identical + if (sourceUrl.trim() === targetUrl.trim()) { + throw new Error( + `Target database URL must not be identical to source database URL: ${sanitizeDatabaseUrl(targetUrl)}`, + ); + } + + const resolveHost = async (host) => { + try { + return (await lookup(host)).address; + } catch { + return host; + } + }; + + const sourceIp = await resolveHost(source.host); + const targetIp = await resolveHost(target.host); + + // 2. Target must not target the same database on the same host/port + if ( + sourceIp === targetIp && + source.port === target.port && + source.database.toLowerCase() === target.database.toLowerCase() + ) { + throw new Error( + `Target database name "${target.database}" matches source database name on the same host (${target.host}:${target.port}). Refusing unsafe operation.`, + ); + } + + // 3. Target must not be a protected system or production database name + if (PROTECTED_DATABASE_NAMES.has(target.database.toLowerCase())) { + throw new Error( + `Target database name "${target.database}" is a protected or non-isolated database. Target must be a dedicated disposable drill database.`, + ); + } + + // 4. Target name must contain an isolation marker unless explicitly overridden + if (!ISOLATION_MARKERS.test(target.database) && !options.allowCustomTargetName) { + throw new Error( + `Target database name "${target.database}" does not contain an isolated drill marker (such as "drill", "restore", "test", "disposable", "isolated"). Refusing to use potentially unsafe target. Supply --allow-custom-target-name if this is intentional.`, + ); + } + + return { + isolated: true, + sourceDbName: source.database, + targetDbName: target.database, + }; +} + +/** + * Validate a backup file's existence, size, header magic, and digest. + */ +export async function validateBackupFile(filePath, format = 'custom') { + if (!existsSync(filePath)) { + throw new Error(`Backup file does not exist: ${filePath}`); + } + + const stats = statSync(filePath); + if (stats.size === 0) { + throw new Error(`Backup file is empty (0 bytes): ${filePath}`); + } + + // Check header magic for custom format + if (format === 'custom') { + if (stats.size < 5) { + throw new Error(`Invalid custom-format backup: file is too small (${stats.size} bytes)`); + } + const fd = openSync(filePath, 'r'); + const headerBuf = Buffer.alloc(5); + try { + readSync(fd, headerBuf, 0, 5, 0); + } finally { + closeSync(fd); + } + if (headerBuf.toString('ascii') !== 'PGDMP') { + throw new Error( + 'Invalid custom-format backup: missing PostgreSQL dump magic header "PGDMP". The backup may be corrupted or created in an incompatible format.', + ); + } + } + + // Compute SHA-256 digest + const hash = createHash('sha256'); + await pipeline(createReadStream(filePath), hash); + const sha256 = hash.digest('hex'); + + return { + valid: true, + filePath, + sizeBytes: stats.size, + sha256, + }; +} + +export function resolvePgTooling(options = {}) { + const execSyncFn = options.execSyncFn || execFileSync; + const forceDocker = options.useDocker === true || options.useDocker === 'true'; + const dockerImage = options.dockerImage || 'pgvector/pgvector:pg16'; + + let hasNativePgDump = false; + let hasNativePgRestore = false; + let hasNativePsql = false; + + if (!forceDocker) { + try { + execSyncFn('pg_dump', ['--version'], { stdio: 'ignore' }); + hasNativePgDump = true; + } catch {} + try { + execSyncFn('pg_restore', ['--version'], { stdio: 'ignore' }); + hasNativePgRestore = true; + } catch {} + try { + execSyncFn('psql', ['--version'], { stdio: 'ignore' }); + hasNativePsql = true; + } catch {} + } + + const isPlain = (options.format || 'custom') === 'plain'; + const hasRequiredNativeTools = isPlain + ? (hasNativePgDump && hasNativePsql) + : (hasNativePgDump && hasNativePgRestore); + + if (hasRequiredNativeTools) { + return { + type: 'native', + runDump: (args, env) => execSyncFn('pg_dump', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + runRestore: (args, env) => execSyncFn('pg_restore', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + runPsql: (args, env) => execSyncFn('psql', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + }; + } + + // Fallback: Check if Docker is available + let hasDocker = false; + try { + execSyncFn('docker', ['--version'], { stdio: 'ignore' }); + hasDocker = true; + } catch {} + + if (!hasDocker) { + const requiredTools = isPlain ? 'pg_dump, psql' : 'pg_dump, pg_restore'; + throw new Error( + `Neither native PostgreSQL tools (${requiredTools}) nor Docker are available in PATH. Please install postgresql-client or ensure Docker is running.`, + ); + } + + return { + type: 'docker', + dockerImage, + runDump: (args, env, mountDir) => { + const dockerArgs = ['run', '--rm']; + if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD'); + for (const key of Object.keys(env)) { + if (key.startsWith('PGSSL')) dockerArgs.push('-e', key); + } + if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); + // Network host so it can reach localhost postgres + if (process.platform === 'linux') { + dockerArgs.push('--net=host'); + } + dockerArgs.push(dockerImage, 'pg_dump', ...args); + return execSyncFn('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); + }, + runRestore: (args, env, mountDir) => { + const dockerArgs = ['run', '--rm']; + if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD'); + for (const key of Object.keys(env)) { + if (key.startsWith('PGSSL')) dockerArgs.push('-e', key); + } + if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); + if (process.platform === 'linux') { + dockerArgs.push('--net=host'); + } + dockerArgs.push(dockerImage, 'pg_restore', ...args); + return execSyncFn('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); + }, + runPsql: (args, env, mountDir) => { + const dockerArgs = ['run', '--rm']; + if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD'); + for (const key of Object.keys(env)) { + if (key.startsWith('PGSSL')) dockerArgs.push('-e', key); + } + if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); + if (process.platform === 'linux') { + dockerArgs.push('--net=host'); + } + dockerArgs.push(dockerImage, 'psql', ...args); + return execSyncFn('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); + }, + }; +} + +/** + * Reject every failed pg_restore invocation, regardless of diagnostic language or content. + */ +export function parsePgRestoreError(err) { + throw err; +} + +/** + * Capture source database state baseline before running backup. + */ +export async function collectSourceBaseline(pool, existingClient = null) { + const client = existingClient || (await pool.connect()); + const shouldManageTx = !existingClient; + if (shouldManageTx) { + await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); + } + try { + // 1. Extensions with versions + const extRes = await client.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); + const extensions = extRes.rows.map((r) => ({ extname: r.extname, extversion: r.extversion })); + + // 2. Schema migrations + const hasMigrationsTable = await client.query( + "SELECT to_regclass('schema_migrations') IS NOT NULL AS present", + ); + let migrations = []; + if (hasMigrationsTable.rows[0]?.present) { + const migRes = await client.query( + 'SELECT version, name, checksum, state FROM schema_migrations ORDER BY version', + ); + migrations = migRes.rows; + } + + // 3. Existing tables in public schema + const tableRes = await client.query( + "SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename", + ); + const tables = tableRes.rows.map((r) => r.tablename); + + // 4. Row counts across critical tables + const rowCounts = Object.create(null); + for (const table of tables) { + const countRes = await client.query(`SELECT COUNT(*) AS count FROM "${table.replace(/"/g, '""')}"`); + rowCounts[table] = parseInt(countRes.rows[0].count, 10); + } + + // 5. Sample records for integrity comparison + const sampleData = {}; + if (tables.includes('users') && (rowCounts['users'] || 0) > 0) { + const sampleUsers = await client.query('SELECT id, handle FROM users ORDER BY created_at LIMIT 5'); + sampleData.users = sampleUsers.rows; + } + if (tables.includes('game_events') && (rowCounts['game_events'] || 0) > 0) { + const sampleEvents = await client.query( + 'SELECT game_id, seq, type FROM game_events ORDER BY server_ts DESC LIMIT 5', + ); + sampleData.game_events = sampleEvents.rows; + } + if (tables.includes('tournaments') && (rowCounts['tournaments'] || 0) > 0) { + const sampleTournaments = await client.query( + 'SELECT id, name, format FROM tournaments ORDER BY created_at DESC LIMIT 5', + ); + sampleData.tournaments = sampleTournaments.rows; + } + + return { + extensions, + migrations, + tables, + rowCounts, + sampleData, + }; + } finally { + if (shouldManageTx) { + await client.query('ROLLBACK'); + client.release(); + } + } +} + +/** + * Deep structural and functional verification of restored target database against source baseline. + */ +export async function verifyRestoredDatabase(sourceBaseline, targetPool, options = {}) { + const checks = []; + const failures = []; + + const recordCheck = (name, passed, detail) => { + checks.push({ name, passed, detail }); + if (!passed) { + failures.push({ name, detail }); + } + }; + + // Check 1: Extensions + const extRes = await targetPool.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); + const targetExtMap = new Map(); + for (const r of extRes.rows || []) { + targetExtMap.set(r.extname, r.extversion || null); + } + const targetExts = new Set(targetExtMap.keys()); + + for (const requiredExt of REQUIRED_EXTENSIONS) { + const present = targetExtMap.has(requiredExt); + recordCheck( + `Required Extension: ${requiredExt}`, + present, + present ? 'Installed and active' : `Missing required extension in restored database: ${requiredExt}`, + ); + if (!present) { + throw new Error(`Missing required extension in restored database: ${requiredExt}`); + } + } + + // Compare every source extension and version with restored database + for (const srcExt of sourceBaseline.extensions || []) { + const name = typeof srcExt === 'string' ? srcExt : srcExt.extname; + const version = typeof srcExt === 'string' ? null : (srcExt.extversion || null); + const present = targetExtMap.has(name); + const targetVersion = targetExtMap.get(name); + const versionMatch = !version || !targetVersion || targetVersion === version; + recordCheck( + `Source Extension: ${name}`, + present && versionMatch, + present + ? (versionMatch ? `Installed version ${targetVersion || 'active'}` : `Version mismatch: source ${version} vs restored ${targetVersion}`) + : `Missing source extension in restored database: ${name}`, + ); + if (!present) { + throw new Error(`Missing source extension in restored database: ${name}`); + } + if (!versionMatch) { + throw new Error(`Extension version mismatch in restored database for ${name}: source ${version} vs restored ${targetVersion}`); + } + } + + // Check 2: Schema migrations ledger + if (sourceBaseline.migrations && sourceBaseline.migrations.length > 0) { + const migTableRes = await targetPool.query( + "SELECT to_regclass('schema_migrations') IS NOT NULL AS present", + ); + if (!migTableRes.rows[0]?.present) { + recordCheck('Migrations Ledger', false, 'Table schema_migrations does not exist in restored database'); + throw new Error('Table schema_migrations does not exist in restored database'); + } + + const targetMigRes = await targetPool.query( + 'SELECT version, name, checksum, state FROM schema_migrations ORDER BY version', + ); + const targetMigrations = targetMigRes.rows; + + if (targetMigrations.length !== sourceBaseline.migrations.length) { + const msg = `Migration count mismatch: source had ${sourceBaseline.migrations.length}, restored has ${targetMigrations.length}`; + recordCheck('Migration Count', false, msg); + throw new Error(msg); + } + recordCheck( + 'Migration Count', + true, + `All ${sourceBaseline.migrations.length} migration records restored`, + ); + + const targetMigMap = new Map(targetMigrations.map((m) => [m.version, m])); + for (const srcMig of sourceBaseline.migrations) { + const tgtMig = targetMigMap.get(srcMig.version); + if (!tgtMig) { + throw new Error(`Restored database is missing recorded migration version ${srcMig.version}`); + } + if (tgtMig.checksum !== srcMig.checksum || tgtMig.name !== srcMig.name || tgtMig.state !== srcMig.state || tgtMig.state !== 'applied') { + const msg = `Migration ${srcMig.version} mismatch: source="${srcMig.checksum}/${srcMig.name}/${srcMig.state}", restored="${tgtMig.checksum}/${tgtMig.name}/${tgtMig.state}" (must be 'applied')`; + recordCheck(`Migration Checksum ${srcMig.version}`, false, msg); + throw new Error(msg); + } + } + recordCheck('Migration Checksums', true, 'All migration ledger checksums match source byte-for-byte'); + } + + // Check 3: Tables + const tgtTableRes = await targetPool.query( + "SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename", + ); + const targetTables = new Set(tgtTableRes.rows.map((r) => r.tablename)); + + for (const expectedTable of new Set([...sourceBaseline.tables, ...CRITICAL_APPLICATION_TABLES])) { + const present = targetTables.has(expectedTable); + if (!present) { + const msg = `Missing required table in restored database: ${expectedTable}`; + recordCheck(`Table: ${expectedTable}`, false, msg); + throw new Error(msg); + } + } + recordCheck( + 'Table Structure', + true, + `All ${sourceBaseline.tables.length} tables present in restored database`, + ); + + // Check 4: Row Counts + for (const [table, expectedCount] of Object.entries(sourceBaseline.rowCounts)) { + if (!targetTables.has(table)) continue; + const countRes = await targetPool.query(`SELECT COUNT(*) AS count FROM "${table.replace(/"/g, '""')}"`); + const actualCount = parseInt(countRes.rows[0].count, 10); + if (actualCount !== expectedCount) { + const msg = `Row count mismatch for table "${table}": source had ${expectedCount}, restored has ${actualCount}`; + recordCheck(`Row Count: ${table}`, false, msg); + throw new Error(msg); + } + } + recordCheck('Row Counts', true, 'All table row counts match source database 100%'); + + // Check 5: Sample Data Identity + if (sourceBaseline.sampleData?.users?.length) { + const userSample = sourceBaseline.sampleData.users[0]; + const userCheck = await targetPool.query('SELECT id, handle FROM users WHERE id = $1', [userSample.id]); + if (userCheck.rows.length === 0 || userCheck.rows[0].handle !== userSample.handle) { + throw new Error(`Sample user record ${userSample.id} (${userSample.handle}) not found or mismatched in restored database`); + } + recordCheck('Sample Data: Users', true, `Sample user ${userSample.handle} verified`); + } + + if (sourceBaseline.sampleData?.game_events?.length) { + const eventSample = sourceBaseline.sampleData.game_events[0]; + const eventCheck = await targetPool.query( + 'SELECT game_id, seq, type FROM game_events WHERE game_id = $1 AND seq = $2', + [eventSample.game_id, eventSample.seq], + ); + if (eventCheck.rows.length === 0 || eventCheck.rows[0].type !== eventSample.type) { + throw new Error(`Sample game_event ${eventSample.game_id}#${eventSample.seq} not found or mismatched in restored database`); + } + recordCheck('Sample Data: Game Events', true, `Sample game event ${eventSample.type} verified`); + } + + // Check 6: Append-only trigger enforcement on game_events + if (targetTables.has('game_events')) { + let triggerActive = false; + const client = await targetPool.connect(); + try { + await client.query('BEGIN'); + const res = await client.query( + 'UPDATE game_events SET seq = seq WHERE game_id IN (SELECT game_id FROM game_events LIMIT 1)', + ); + if (res.rowCount && res.rowCount > 0) { + triggerActive = false; + } else { + // Table was empty; verify trigger registration in pg_trigger catalog + const trigRes = await client.query( + "SELECT tgname FROM pg_trigger WHERE tgrelid = 'public.game_events'::regclass AND tgname = 'game_events_block_mutate' AND tgenabled = 'O'", + ); + triggerActive = trigRes.rows.length > 0; + } + } catch (err) { + if ( + err.message && + err.message.includes('game_events is append-only') && err.code === 'P0001' + ) { + triggerActive = true; + } else { + throw err; + } + } finally { + await client.query('ROLLBACK'); + client.release(); + } + + if (!triggerActive) { + const msg = 'Append-only trigger on game_events failed: mutation was not blocked'; + recordCheck('Trigger: game_events append-only', false, msg); + throw new Error(msg); + } + recordCheck('Trigger: game_events append-only', true, 'Mutation blocked by trigger game_events_block_mutate'); + } + + // Check 7: Semantic search / pgvector functional test + if (targetTables.has('search_embeddings') && targetExts.has('vector')) { + try { + // Test vector operator syntax and index validity + const testVec = `[${new Array(256).fill(0.1).join(',')}]`; + await targetPool.query('SELECT $1::vector(256) <=> $1::vector(256) AS dist', [testVec]); + + const idxRes = await targetPool.query(` + SELECT i.relname AS index_name, am.amname AS access_method, ix.indisvalid, ix.indisready + FROM pg_index ix + JOIN pg_class i ON i.oid = ix.indexrelid + JOIN pg_class t ON t.oid = ix.indrelid + JOIN pg_am am ON i.relam = am.oid + WHERE t.oid = 'public.search_embeddings'::regclass AND am.amname = 'hnsw' + `); + if (!idxRes.rows.some(index => index.indisvalid === true && index.indisready === true)) { + throw new Error('Valid and ready HNSW index missing on search_embeddings table'); + } + + recordCheck('pgvector Functionality', true, 'Vector cosine operator (<=>) and HNSW index functional'); + } catch (err) { + const msg = `Vector functionality verification failed: ${err.message}`; + recordCheck('pgvector Functionality', false, msg); + throw new Error(msg); + } + } + + return { + passed: failures.length === 0, + checks, + failures, + }; +} + +/** + * CLI Argument parser. + */ +export function parseArgs(args = process.argv.slice(2)) { + const result = { + sourceUrl: process.env.BACKUP_DRILL_SOURCE_URL || process.env.DATABASE_URL || '', + targetUrl: process.env.BACKUP_DRILL_TARGET_URL || '', + targetDbName: '', + backupFile: '', + keepBackup: false, + keepTarget: false, + format: 'custom', + useDocker: 'auto', + dockerImage: 'pgvector/pgvector:pg16', + allowCustomTargetName: false, + json: false, + help: false, + }; + + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + if (arg === '--source-url' && i + 1 < args.length) { + result.sourceUrl = args[++i]; + } else if (arg === '--target-url' && i + 1 < args.length) { + result.targetUrl = args[++i]; + } else if (arg === '--target-db-name' && i + 1 < args.length) { + result.targetDbName = args[++i]; + } else if (arg === '--backup-file' && i + 1 < args.length) { + result.backupFile = args[++i]; + } else if (arg === '--keep-backup') { + result.keepBackup = true; + } else if (arg === '--keep-target') { + result.keepTarget = true; + } else if (arg === '--format' && i + 1 < args.length) { + result.format = args[++i]; + if (result.format !== 'custom' && result.format !== 'plain') { + throw new Error(`Invalid format "${result.format}". Only "custom" and "plain" are allowed.`); + } + } else if (arg === '--use-docker') { + result.useDocker = true; + } else if (arg === '--no-docker') { + result.useDocker = false; + } else if (arg === '--docker-image' && i + 1 < args.length) { + result.dockerImage = args[++i]; + } else if (arg === '--allow-custom-target-name') { + result.allowCustomTargetName = true; + } else if (arg === '--json') { + result.json = true; + } else if (arg === '--help' || arg === '-h') { + result.help = true; + } + } + + if (result.sourceUrl && !result.targetUrl) { + const isolatedName = result.targetDbName || generateIsolatedDbName('gambit'); + result.targetUrl = urlWithDatabase(result.sourceUrl, isolatedName); + } + + return result; +} + +/** + * Execute full backup, isolated restore, and verification drill. + */ +export async function runBackupRestoreDrill(options = {}) { + const startTime = Date.now(); + const report = { + startedAt: new Date(startTime).toISOString(), + source: '', + target: '', + backupFile: '', + backupSizeBytes: 0, + backupSha256: '', + timings: {}, + checks: [], + success: false, + }; + + const parsedSource = parseDatabaseUrl(options.sourceUrl); + const targetUrl = options.targetUrl || urlWithDatabase( + options.sourceUrl, + options.targetDbName || generateIsolatedDbName(parsedSource.database), + ); + const parsedTarget = parseDatabaseUrl(targetUrl); + + report.source = sanitizeDatabaseUrl(options.sourceUrl); + report.target = sanitizeDatabaseUrl(targetUrl); + + // Validate isolation + await validateTargetIsolation(options.sourceUrl, targetUrl, { + allowCustomTargetName: options.allowCustomTargetName, + }); + + const sourcePool = new Pool({ connectionString: options.sourceUrl, max: 2 }); + let targetPool = null; + let adminClient = null; + let targetCreatedByThisRun = false; + let backupCreatedByThisRun = false; + let drillError = null; + const cleanupErrors = []; + + const rawBackupPath = + options.backupFile || + join(tmpdir(), `gambit_backup_${Date.now()}_${Math.floor(Math.random() * 0xffff).toString(16)}.dump`); + const backupPath = resolve(rawBackupPath); + const backupDir = dirname(backupPath); + const backupFileName = basename(backupPath); + report.backupFile = backupPath; + + const log = (msg) => { + if (!options.json) { + console.log(`[drill] ${sanitizeDiagnostic(msg, [options.sourceUrl, targetUrl])}`); + } + }; + + try { + // Reserve the destination exclusively before any tool can overwrite it. + const backupFd = openSync(backupPath, 'wx', 0o600); + backupCreatedByThisRun = true; + closeSync(backupFd); + const tooling = resolvePgTooling(options); + const isCustom = (options.format || 'custom') === 'custom'; + // 1. Capture source baseline and export snapshot + log(`Capturing source baseline from ${report.source}...`); + const baselineClient = await sourcePool.connect(); + let snapshotId = null; + let sourceBaseline = null; + let baselineError = null; + const baselineCleanupErrors = []; + try { + await baselineClient.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); + const snapRes = await baselineClient.query('SELECT pg_export_snapshot() AS snap'); + snapshotId = snapRes.rows[0]?.snap; + if (!snapshotId) throw new Error('Source did not provide an exported snapshot; refusing an inconsistent drill'); + sourceBaseline = await collectSourceBaseline(sourcePool, baselineClient); + log(`Source baseline captured: ${sourceBaseline.tables.length} tables, ${sourceBaseline.migrations.length} migrations.`); + + // 2. Resolve tooling + log(`Resolved PostgreSQL tooling (${tooling.type})...`); + + // 3. Perform backup using pg_dump + log(`Creating ${options.format || 'custom'} backup to ${backupPath}...`); + const dumpStart = Date.now(); + + if (tooling.type === 'native') { + const dumpArgs = [ + '-h', parsedSource.host, + '-p', String(parsedSource.port), + '-U', parsedSource.user, + '-d', parsedSource.database, + '-F', isCustom ? 'c' : 'p', + '--no-owner', + '--no-acl', + '-f', backupPath, + ]; + if (snapshotId) { + dumpArgs.push(`--snapshot=${snapshotId}`); + } + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }); + } else { + // Docker mode + const hostForDocker = + parsedSource.host === 'localhost' || parsedSource.host === '127.0.0.1' + ? (process.platform === 'linux' ? '127.0.0.1' : 'host.docker.internal') + : parsedSource.host; + + const dumpArgs = [ + '-h', hostForDocker, + '-p', String(parsedSource.port), + '-U', parsedSource.user, + '-d', parsedSource.database, + '-F', isCustom ? 'c' : 'p', + '--no-owner', + '--no-acl', + '-f', `/work/${backupFileName}`, + ]; + if (snapshotId) { + dumpArgs.push(`--snapshot=${snapshotId}`); + } + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }, backupDir); + } + report.timings.backupMs = Date.now() - dumpStart; + log(`Backup completed in ${report.timings.backupMs}ms.`); + } catch (err) { + baselineError = err; + } finally { + await baselineClient.query('ROLLBACK').catch(err => baselineCleanupErrors.push(err)); + try { + baselineClient.release(); + } catch (err) { + baselineCleanupErrors.push(err); + } + } + const baselineErrors = [...(baselineError ? [baselineError] : []), ...baselineCleanupErrors]; + if (baselineErrors.length === 1) throw baselineErrors[0]; + if (baselineErrors.length > 1) { + throw new AggregateError( + baselineErrors, + 'Source baseline, dump, or transaction cleanup failed: ' + baselineErrors.map(error => error?.message || String(error)).join('; '), + ); + } + + // 4. Validate backup file + const backupMeta = await validateBackupFile(backupPath, options.format || 'custom'); + report.backupSizeBytes = backupMeta.sizeBytes; + report.backupSha256 = backupMeta.sha256; + log(`Backup file validated: ${backupMeta.sizeBytes} bytes (SHA-256: ${backupMeta.sha256.substring(0, 12)}...).`); + + // 5. Create isolated target database + log(`Provisioning isolated target database "${parsedTarget.database}"...`); + const adminUrl = urlWithDatabase(targetUrl, 'postgres'); + adminClient = new Client({ connectionString: adminUrl, statement_timeout: 10000 }); + try { + await adminClient.connect(); + } catch { + // Try template1 if postgres db is not accessible + adminClient = new Client({ connectionString: urlWithDatabase(targetUrl, 'template1'), statement_timeout: 10000 }); + await adminClient.connect(); + } + + await adminClient.query(`SET statement_timeout = 10000`); + await adminClient.query(`CREATE DATABASE "` + parsedTarget.database.replace(/"/g, '""') + `"`); + targetCreatedByThisRun = true; + log(`Target database "${parsedTarget.database}" created.`); + + // 6. Restore backup into target + log(`Restoring backup into ${report.target}...`); + const restoreStart = Date.now(); + if (tooling.type === 'native') { + if (isCustom) { + const restoreArgs = [ + '-h', parsedTarget.host, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '--clean', + '--if-exists', + '--no-owner', + '--no-acl', + '--exit-on-error', + backupPath, + ]; + try { + tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }); + } catch (err) { + parsePgRestoreError(err); + } + } else { + const psqlArgs = [ + '-h', parsedTarget.host, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '-v', 'ON_ERROR_STOP=1', + '-f', backupPath, + ]; + tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }); + } + } else { + // Docker mode + const hostForDocker = + parsedTarget.host === 'localhost' || parsedTarget.host === '127.0.0.1' + ? (process.platform === 'linux' ? '127.0.0.1' : 'host.docker.internal') + : parsedTarget.host; + + if (isCustom) { + const restoreArgs = [ + '-h', hostForDocker, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '--clean', + '--if-exists', + '--no-owner', + '--no-acl', + '--exit-on-error', + `/work/${backupFileName}`, + ]; + try { + tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }, backupDir); + } catch (err) { + parsePgRestoreError(err); + } + } else { + const psqlArgs = [ + '-h', hostForDocker, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '-v', 'ON_ERROR_STOP=1', + '-f', `/work/${backupFileName}`, + ]; + tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }, backupDir); + } + } + report.timings.restoreMs = Date.now() - restoreStart; + log(`Restore completed in ${report.timings.restoreMs}ms.`); + + // 7. Verify restored database + log('Running comprehensive structural and functional verification...'); + const verifyStart = Date.now(); + targetPool = new Pool({ connectionString: targetUrl, max: 2 }); + const verifyResult = await verifyRestoredDatabase(sourceBaseline, targetPool, options); + report.timings.verifyMs = Date.now() - verifyStart; + report.checks = verifyResult.checks; + + log(`Verification passed: ${verifyResult.checks.length} checks succeeded.`); + report.success = true; + } catch (err) { + drillError = err; + } finally { + // Teardown connections + await sourcePool.end().catch(err => cleanupErrors.push(err)); + if (targetPool) { + await targetPool.end().catch(err => cleanupErrors.push(err)); + } + + // Teardown target database if not keeping + if (adminClient) { + if (!options.keepTarget && parsedTarget.database && targetCreatedByThisRun) { + try { + log(`Cleaning up isolated target database "${parsedTarget.database}"...`); + await adminClient.query(`DROP DATABASE IF EXISTS "` + parsedTarget.database.replace(/"/g, '""') + `" WITH (FORCE)`); + log('Target database dropped.'); + } catch (err) { + cleanupErrors.push(err); + } + } + await adminClient.end().catch(err => cleanupErrors.push(err)); + } + + // Remove backup file if not keeping + if (!options.keepBackup && backupCreatedByThisRun && existsSync(backupPath)) { + try { + unlinkSync(backupPath); + } catch (err) { + cleanupErrors.push(err); + } + } + } + + const errors = [...(drillError ? [drillError] : []), ...cleanupErrors]; + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError( + errors, + 'Backup/restore drill and cleanup failures: ' + errors.map(error => error?.message || String(error)).join('; '), + ); + } + + report.timings.totalMs = Date.now() - startTime; + report.completedAt = new Date().toISOString(); + + return report; +} + +// CLI entry point +const isDirectExecution = + process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url); + +if (isDirectExecution) { + let args; + try { + args = parseArgs(process.argv.slice(2)); + } catch { + console.error('Invalid drill arguments. Check database URLs and use --format custom or plain.'); + process.exit(1); + } + + if (args.help || !args.sourceUrl) { + console.log(` +Gambit PostgreSQL Backup & Restore Drill Runner + +Usage: + node scripts/db-backup-restore-drill.mjs [options] + +Options: + --source-url Source database URL (defaults to DATABASE_URL) + --target-url Explicit target database URL (must be isolated) + --target-db-name Target database name (default: auto-generated isolated name) + --backup-file Path for backup dump file (default: temporary file) + --keep-backup Preserve the backup dump file after the drill + --keep-target Preserve the restored target database after the drill + --format pg_dump format (default: custom) + --use-docker Force execution of pg tools via Docker container + --docker-image Docker image for pg tools (default: pgvector/pgvector:pg16) + --allow-custom-target-name Permit target name without default isolation markers + --json Output drill report in JSON format + --help Show this help message + `); + process.exit(args.help ? 0 : 1); + } + + runBackupRestoreDrill(args) + .then((report) => { + if (args.json) { + process.stdout.write(JSON.stringify(report, null, 2) + '\n', () => process.exit(0)); + return; + } else { + console.log('\n========================================'); + console.log('✅ BACKUP & RESTORE DRILL SUCCESSFUL'); + console.log('========================================'); + console.log(`Source: ${report.source}`); + console.log(`Target: ${report.target}`); + console.log(`Backup Size: ${(report.backupSizeBytes / 1024).toFixed(2)} KB`); + console.log(`Backup SHA-256: ${report.backupSha256}`); + console.log(`Backup Duration: ${report.timings.backupMs}ms`); + console.log(`Restore Duration:${report.timings.restoreMs}ms`); + console.log(`Verify Duration: ${report.timings.verifyMs}ms`); + console.log(`Total Duration: ${report.timings.totalMs}ms`); + console.log(`Checks Passed: ${report.checks.length}/${report.checks.length}`); + console.log('========================================\n'); + } + process.exit(0); + }) + .catch((err) => { + console.error('\n========================================'); + console.error('❌ BACKUP & RESTORE DRILL FAILED'); + console.error('========================================'); + console.error(`Error: ${sanitizeDiagnostic(err.message, [args.sourceUrl, args.targetUrl])}`); + console.error('========================================\n'); + process.exit(1); + }); +} diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs new file mode 100644 index 00000000..11ffa89d --- /dev/null +++ b/scripts/test/backup-restore-drill.test.mjs @@ -0,0 +1,655 @@ +/** + * Automated tests for the Postgres backup/restore drill and verification engine. + * + * Verifies: + * 1. Security: credential masking in logs/diagnostics, no leaking passwords. + * 2. Isolation: fails safe if target matches source or is not an isolated drill target. + * 3. Backup validation: detects missing, empty, or corrupt backup files. + * 4. Verification checks: detects missing extensions, missing migration ledgers, + * checksum drift, missing tables, row count discrepancies, and missing/inactive + * append-only triggers on game_events. + * 5. CLI argument parsing: flags, defaults, and environment fallbacks. + */ + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { writeFileSync, unlinkSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { + sanitizeDatabaseUrl, + validateTargetIsolation, + validateBackupFile, + verifyRestoredDatabase, + parseArgs, + generateIsolatedDbName, + runBackupRestoreDrill, + resolvePgTooling, + CRITICAL_APPLICATION_TABLES, + REQUIRED_EXTENSIONS, + parsePgRestoreError, +} from '../db-backup-restore-drill.mjs'; + +test('security: sanitizeDatabaseUrl masks plaintext passwords in postgres URLs', () => { + const plainUrl = 'postgres://gambit:my_super_secret_pw@db.production.internal:5432/gambit'; + const sanitized = sanitizeDatabaseUrl(plainUrl); + assert.equal(sanitized, 'postgres://gambit:***@db.production.internal:5432/gambit'); + assert.equal(sanitized.includes('my_super_secret_pw'), false); +}); + +test('security: sanitizeDatabaseUrl handles URLs with URL-encoded special characters', () => { + const complexUrl = 'postgresql://admin%40corp:p%40ss%3Aword%21@127.0.0.1:5432/gambit_prod?sslmode=require'; + const sanitized = sanitizeDatabaseUrl(complexUrl); + assert.equal(sanitized.includes('p%40ss%3Aword%21'), false); + assert.equal(sanitized, 'postgresql://admin%40corp:***@127.0.0.1:5432/gambit_prod?sslmode=require'); +}); + +test('security: sanitizeDatabaseUrl safely handles URLs without passwords', () => { + const noPwUrl = 'postgres://gambit@localhost:5432/gambit_dev'; + assert.equal(sanitizeDatabaseUrl(noPwUrl), 'postgres://gambit@localhost:5432/gambit_dev'); +}); + +test('security: sanitizeDatabaseUrl handles non-URL strings and errors safely', () => { + assert.equal(sanitizeDatabaseUrl('not a url'), 'not a url'); + assert.equal(sanitizeDatabaseUrl(''), ''); +}); + +test('isolation: validateTargetIsolation rejects when target URL matches source URL', async () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const target = 'postgres://gambit:pass@localhost:5432/gambit'; + + await assert.rejects( + async () => await validateTargetIsolation(source, target), + /Target database URL must not be identical to source database URL/, + ); +}); + +test('isolation: validateTargetIsolation rejects when source and target have identical DB name on same host/port', async () => { + const source = 'postgres://gambit:pass1@localhost:5432/gambit'; + const target = 'postgres://gambit_admin:pass2@localhost:5432/gambit'; + + await assert.rejects( + async () => await validateTargetIsolation(source, target), + /Target database name "gambit" matches source database name on the same host/, + ); +}); + +test('isolation: validateTargetIsolation rejects dangerous production or system databases as target', async () => { + const source = 'postgres://gambit:pass@localhost:5432/source_test'; + for (const dangerous of ['gambit', 'postgres', 'template1', 'template0', 'production']) { + const target = `postgres://gambit:pass@localhost:5432/${dangerous}`; + await assert.rejects( + async () => await validateTargetIsolation(source, target), + /Target database name .* is a protected or non-isolated database/, + ); + } +}); + +test('isolation: validateTargetIsolation rejects target names without isolated naming markers unless explicitly allowed', async () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const target = 'postgres://gambit:pass@localhost:5432/arbitrary_name'; + + await assert.rejects( + async () => await validateTargetIsolation(source, target, { allowCustomTargetName: false }), + /Target database name "arbitrary_name" does not contain an isolated drill marker/, + ); + + // When explicitly permitted via allowCustomTargetName flag + const result = await validateTargetIsolation(source, target, { allowCustomTargetName: true }); + assert.equal(result.isolated, true); + assert.equal(result.targetDbName, 'arbitrary_name'); +}); + +test('isolation: validateTargetIsolation accepts valid isolated database names', async () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const validTargets = [ + 'gambit_backup_drill_restore', + 'gambit_backup_drill_restore_1720000000_abcd', + 'test_db_restore', + 'gambit_disposable_drill', + 'gambit_isolated_restore', + ]; + + for (const name of validTargets) { + const target = `postgres://gambit:pass@localhost:5432/${name}`; + const validated = await validateTargetIsolation(source, target); + assert.equal(validated.isolated, true); + assert.equal(validated.targetDbName, name); + } +}); + +test('isolation: generateIsolatedDbName generates prefixed unique name', async () => { + const name1 = generateIsolatedDbName('gambit'); + const name2 = generateIsolatedDbName('gambit'); + assert.match(name1, /^gambit_backup_drill_restore_\d+_[a-f0-9]+$/); + assert.notEqual(name1, name2); +}); + +test('isolation: validateTargetIsolation rejects malicious injection payloads in target URL', async () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const maliciousTarget = 'postgres://gambit:pass@localhost:5432/test" OR 1=1; DROP DATABASE production; --'; + + await assert.rejects( + async () => await validateTargetIsolation(source, maliciousTarget, { allowCustomTargetName: true }), + /Invalid target database name format/ + ); +}); + +test('backup validation: validateBackupFile rejects non-existent file', async () => { + const nonExistent = join(tmpdir(), 'non-existent-backup-file-12345.dump'); + await assert.rejects( + async () => await validateBackupFile(nonExistent, 'custom'), + /Backup file does not exist/, + ); +}); + +test('backup validation: validateBackupFile rejects 0-byte empty file', async () => { + const emptyFile = join(tmpdir(), `test-empty-backup-${Date.now()}.dump`); + writeFileSync(emptyFile, Buffer.alloc(0)); + try { + await assert.rejects( + async () => await validateBackupFile(emptyFile, 'custom'), + /Backup file is empty \(0 bytes\)/, + ); + } finally { + unlinkSync(emptyFile); + } +}); + +test('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', async () => { + const corruptFile = join(tmpdir(), `test-corrupt-backup-${Date.now()}.dump`); + writeFileSync(corruptFile, Buffer.from('NOT_A_VALID_PG_DUMP_HEADER')); + try { + await assert.rejects( + async () => await validateBackupFile(corruptFile, 'custom'), + /Invalid custom-format backup: missing PostgreSQL dump magic header "PGDMP"/, + ); + } finally { + unlinkSync(corruptFile); + } +}); + +test('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', async () => { + const validFile = join(tmpdir(), `test-valid-backup-${Date.now()}.dump`); + const header = Buffer.from('PGDMP\x01\x10\x00\x01'); + writeFileSync(validFile, header); + try { + const meta = await validateBackupFile(validFile, 'custom'); + assert.equal(meta.valid, true); + assert.equal(meta.sizeBytes, header.length); + assert.equal(typeof meta.sha256, 'string'); + assert.equal(meta.sha256.length, 64); + } finally { + unlinkSync(validFile); + } +}); + +test('verification engine: detects missing extensions in restored database', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['users', 'games'], + rowCounts: { users: 2, games: 1 }, + sampleData: {}, + }; + + // Mock target database pool missing the 'vector' extension + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext', extversion: '1.6' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Missing required extension in restored database: vector/); + return true; + }, + ); +}); + +test('verification engine: detects extension version mismatch between source and restored database', async () => { + const sourceBaseline = { + extensions: [{ extname: 'citext', extversion: '1.6' }, { extname: 'vector', extversion: '0.5.1' }], + migrations: [], + tables: [], + rowCounts: {}, + sampleData: {}, + }; + + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext', extversion: '1.6' }, { extname: 'vector', extversion: '0.4.0' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Extension version mismatch in restored database for vector: source 0.5.1 vs restored 0.4.0/); + return true; + }, + ); +}); + +test('verification engine: detects missing schema_migrations table', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users'], + rowCounts: { schema_migrations: 1, users: 1 }, + sampleData: {}, + }; + + // Mock target where schema_migrations table does not exist + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: false }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Table schema_migrations does not exist in restored database/); + return true; + }, + ); +}); + +test('verification engine: detects missing schema_migrations table or count mismatch', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [ + { version: 1, name: '0001_init.sql', checksum: 'abc1', state: 'applied' }, + { version: 2, name: '0002_seek_color.sql', checksum: 'abc2', state: 'applied' }, + ], + tables: ['schema_migrations', 'users'], + rowCounts: { schema_migrations: 2, users: 1 }, + sampleData: {}, + }; + + // Mock target where only migration 1 is present + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc1', state: 'applied' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Migration count mismatch: source had 2, restored has 1/); + return true; + }, + ); +}); + +test('verification engine: detects corrupted or modified migration checksum in restored database', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [ + { version: 1, name: '0001_init.sql', checksum: 'original_checksum_123', state: 'applied' }, + ], + tables: ['schema_migrations'], + rowCounts: { schema_migrations: 1 }, + sampleData: {}, + }; + + // Mock target where migration 1 has altered checksum + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'corrupted_checksum_999', state: 'applied' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Migration 1 mismatch/); + return true; + }, + ); +}); + +test('verification engine: detects missing tables in restored database', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'games', 'tournaments'], + rowCounts: { schema_migrations: 1, users: 2, games: 1, tournaments: 1 }, + sampleData: {}, + }; + + // Mock target where 'tournaments' table is missing + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { rows: [{ tablename: 'schema_migrations' }, { tablename: 'users' }, { tablename: 'games' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Missing required table in restored database: tournaments/); + return true; + }, + ); +}); + +test('verification engine: detects row count mismatch in critical tables', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'game_events'], + rowCounts: { schema_migrations: 1, users: 5, game_events: 100 }, + sampleData: {}, + }; + + // Mock target where game_events only has 90 rows restored (incomplete restore!) + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { rows: [...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t })), ...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))] }; + } + if (text.includes('COUNT(*) FROM "users"') || text.includes('COUNT(*) AS count FROM "users"')) { + return { rows: [{ count: '5' }] }; + } + if (text.includes('COUNT(*) FROM "game_events"') || text.includes('COUNT(*) AS count FROM "game_events"')) { + return { rows: [{ count: '90' }] }; // 90 != 100 + } + if (text.includes('COUNT(*) FROM "schema_migrations"') || text.includes('COUNT(*) AS count FROM "schema_migrations"')) { + return { rows: [{ count: '1' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Row count mismatch for table "game_events": source had 100, restored has 90/); + return true; + }, + ); +}); + +test('verification engine: detects missing or inactive append-only trigger on game_events', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'game_events'], + rowCounts: { schema_migrations: 1, users: 1, game_events: 1 }, + sampleData: {}, + }; + + // Mock target where table exists and row count matches, but trigger probe does NOT throw + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { rows: [...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t })), ...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))] }; + } + if (text.includes('COUNT(*)')) { + return { rows: [{ count: '1' }] }; + } + if (text.includes('UPDATE game_events')) { + // Trigger did NOT fire! Update silently returned 1 row updated! + return { rowCount: 1 }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Append-only trigger on game_events failed: mutation was not blocked/); + return true; + }, + ); +}); + +test('verification engine: passes when all structural and functional checks succeed', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'game_events', 'variants'], + rowCounts: { schema_migrations: 1, users: 1, game_events: 1, variants: 8 }, + sampleData: { + users: [{ id: 'u1', handle: 'alice' }], + game_events: [{ game_id: 'g1', seq: 0, type: 'MovePlayed' }], + }, + }; + + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { + rows: CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t })), + }; + } + if (text.includes('COUNT(*)')) { + if (text.includes('variants')) return { rows: [{ count: '8' }] }; + return { rows: [{ count: '1' }] }; + } + if (text.includes('SELECT id, handle FROM users')) { + return { rows: [{ id: 'u1', handle: 'alice' }] }; + } + if (text.includes('SELECT game_id, seq, type FROM game_events')) { + return { rows: [{ game_id: 'g1', seq: 0, type: 'MovePlayed' }] }; + } + if (text.includes('UPDATE game_events')) { + // Trigger correctly blocks update + const error = new Error('game_events is append-only (UPDATE.game_events attempted)'); + error.code = 'P0001'; + throw error; + } + if (text.includes('search_embeddings')) { + return { rows: [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw', indisvalid: true, indisready: true }] }; + } + return { rows: [] }; + }, + }; + + const report = await verifyRestoredDatabase(sourceBaseline, mockTargetPool); + assert.equal(report.passed, true); + assert.ok(report.checks.length >= 5); + assert.equal(report.failures.length, 0); +}); + +test('cli: parseArgs parses options, flags, and environment fallbacks', () => { + const customArgs = [ + '--source-url', 'postgres://u:p@localhost:5432/src', + '--target-url', 'postgres://u:p@localhost:5432/gambit_backup_drill_restore', + '--backup-file', '/tmp/my-backup.dump', + '--keep-backup', + '--keep-target', + '--format', 'custom', + '--json', + ]; + + const parsed = parseArgs(customArgs); + assert.equal(parsed.sourceUrl, 'postgres://u:p@localhost:5432/src'); + assert.equal(parsed.targetUrl, 'postgres://u:p@localhost:5432/gambit_backup_drill_restore'); + assert.equal(parsed.backupFile, '/tmp/my-backup.dump'); + assert.equal(parsed.keepBackup, true); + assert.equal(parsed.keepTarget, true); + assert.equal(parsed.format, 'custom'); + assert.equal(parsed.json, true); +}); + +test('cli: parseArgs defaults to safe isolated target when not specified', () => { + const originalEnv = process.env.BACKUP_DRILL_TARGET_URL; + delete process.env.BACKUP_DRILL_TARGET_URL; + try { + const parsed = parseArgs(['--source-url', 'postgres://u:p@localhost:5432/gambit']); + assert.equal(parsed.sourceUrl, 'postgres://u:p@localhost:5432/gambit'); + assert.match(parsed.targetUrl, /^postgres:\/\/u:p@localhost:5432\/gambit_backup_drill_restore_\d+_[a-f0-9]+$/); + assert.equal(parsed.keepBackup, false); + assert.equal(parsed.keepTarget, false); + assert.equal(parsed.format, 'custom'); + } finally { + if (originalEnv !== undefined) { + process.env.BACKUP_DRILL_TARGET_URL = originalEnv; + } + } +}); + +test('tooling: resolvePgTooling accepts options and detects native or docker runner', () => { + const mockExec = (cmd, args) => { return Buffer.from('mock version'); }; + + const customTooling = resolvePgTooling({ format: 'custom', execSyncFn: mockExec }); + assert.ok(customTooling.type === 'native' || customTooling.type === 'docker'); + assert.equal(typeof customTooling.runDump, 'function'); + assert.equal(typeof customTooling.runRestore, 'function'); + assert.equal(typeof customTooling.runPsql, 'function'); + + const plainTooling = resolvePgTooling({ format: 'plain', execSyncFn: mockExec }); + assert.ok(plainTooling.type === 'native' || plainTooling.type === 'docker'); +}); + +test('pg_restore: parsePgRestoreError rejects nonzero status even with warning-only diagnostics', () => { + const err = new Error('Command failed: pg_restore exit code 1'); + err.status = 1; + err.stderr = Buffer.from( + 'pg_restore: warning: errors ignored on restore: 1\n' + + 'pg_restore: warning: could not execute query: ERROR: schema "public" does not exist' + ); + + assert.throws(() => parsePgRestoreError(err), error => error === err); +}); + +test('pg_restore: parsePgRestoreError throws on real errors including missing objects', () => { + const err = new Error('Command failed: pg_restore exit code 1'); + err.status = 1; + err.stderr = Buffer.from( + 'pg_restore: warning: errors ignored on restore: 1\n' + + 'pg_restore: error: could not execute query: ERROR: role "postgres" does not exist' + ); + + assert.throws(() => parsePgRestoreError(err), /Command failed/); +}); + +test('verification engine: checks REQUIRED_EXTENSIONS even when not present in source baseline', async () => { + const sourceBaseline = { + extensions: [], // Missing from source! + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users'], + rowCounts: { schema_migrations: 1, users: 1 }, + sampleData: {}, + }; + + // Mock target database pool where required extension "vector" is missing + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext', extversion: '1.6' }] }; // "vector" missing + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Missing required extension in restored database: vector/); + return true; + }, + ); +}); + +test( + 'integration: full backup, isolated restore, and verification drill against live Postgres', + { skip: process.env.DATABASE_URL ? false : 'DATABASE_URL not set' }, + async () => { + const sourceUrl = process.env.DATABASE_URL; + const report = await runBackupRestoreDrill({ + sourceUrl, + keepTarget: false, + keepBackup: false, + }); + assert.equal(report.success, true); + assert.ok(report.checks.length > 0); + }, +); diff --git a/scripts/test/backup-restore-safety.test.mjs b/scripts/test/backup-restore-safety.test.mjs new file mode 100644 index 00000000..e37185a6 --- /dev/null +++ b/scripts/test/backup-restore-safety.test.mjs @@ -0,0 +1,286 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, writeFileSync, readFileSync, existsSync, rmSync, mkdirSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { basename, join } from 'node:path'; +import pg from 'pg'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; +import { + runBackupRestoreDrill, + CRITICAL_APPLICATION_TABLES, + generateIsolatedDbName, + validateTargetIsolation, + collectSourceBaseline, + sanitizeDatabaseUrl, +} from '../db-backup-restore-drill.mjs'; + +/** + * Build a disposable drill harness that replaces only database and subprocess + * boundaries while exercising the real validation, orchestration, and cleanup. + */ +function drillFixture(t, behavior = {}) { + const directory = mkdtempSync(join(tmpdir(), 'backup-drill-test-')); + t.after(() => rmSync(directory, { recursive: true, force: true })); + const backupFile = join(directory, 'backup.dump'); + const events = []; + /** Return complete catalog/data responses and record every SQL side effect. */ + const databaseQuery = async (sql) => { + events.push(sql); + if (sql === 'ROLLBACK' && behavior.rollbackError) throw behavior.rollbackError; + if (sql.includes('pg_export_snapshot')) return { rows: behavior.missingSnapshot ? [] : [{ snap: '00000001-00000001-1' }] }; + if (sql.includes('pg_extension')) return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + if (sql.includes('to_regclass')) return { rows: [{ present: false }] }; + if (sql.includes('pg_tables')) return { rows: CRITICAL_APPLICATION_TABLES.map(tablename => ({ tablename })) }; + if (sql.includes('COUNT(*)')) return { rows: [{ count: '0' }] }; + if (sql.includes('UPDATE game_events') && behavior.unrelatedTriggerError) throw Object.assign(new Error('unrelated trigger failure'), { code: 'P0001' }); + if (sql.includes('pg_trigger')) return { rows: behavior.otherTriggerRelation && sql.includes("tgrelid = 'public.game_events'::regclass") ? [] : [{ tgname: 'game_events_block_mutate' }] }; + if (sql.includes('pg_index')) return { rows: behavior.otherIndexRelation && sql.includes("t.oid = 'public.search_embeddings'::regclass") ? [] : [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw', indisvalid: !behavior.invalidIndex, indisready: true }] }; + return { rows: [], rowCount: 0 }; + }; + t.mock.method(pg.Pool.prototype, 'connect', async () => { + if (behavior.connectError) throw behavior.connectError; + return { + query: databaseQuery, + release() { + events.push('release'); + if (behavior.releaseError) throw behavior.releaseError; + }, + }; + }); + t.mock.method(pg.Pool.prototype, 'query', databaseQuery); + t.mock.method(pg.Pool.prototype, 'end', async () => { events.push('pool.end'); }); + t.mock.method(pg.Client.prototype, 'connect', async () => { events.push('admin.connect'); }); + t.mock.method(pg.Client.prototype, 'end', async () => { events.push('admin.end'); }); + t.mock.method(pg.Client.prototype, 'query', async (sql) => { + events.push(sql); + if (sql.startsWith('CREATE DATABASE') && behavior.createError) throw behavior.createError; + if (sql.startsWith('DROP DATABASE') && behavior.dropError) throw behavior.dropError; + return { rows: [] }; + }); + /** Simulate PostgreSQL tooling while materializing a real disposable dump file. */ + const execSyncFn = (command, args) => { + if (args.includes('--version')) return Buffer.from('test tooling'); + const executable = command === 'docker' ? args.find(arg => ['pg_dump', 'pg_restore', 'psql'].includes(arg)) : command; + events.push({ executable, args }); + if (executable === 'pg_dump') { + const dumpPath = args[args.indexOf('-f') + 1]; + writeFileSync(command === 'docker' ? join(directory, basename(dumpPath)) : dumpPath, 'PGDMP test archive'); + } else if (behavior.restoreError) { + throw behavior.restoreError; + } else if (behavior.backupCleanupError) { + rmSync(backupFile); + mkdirSync(backupFile); + } + return Buffer.alloc(0); + }; + return { + events, backupFile, + options: { + sourceUrl: 'postgres://u:test-password@127.0.0.1/source_test', + targetUrl: 'postgres://u:test-password@127.0.0.1/isolated_restore_test', + backupFile, json: true, execSyncFn, + }, + }; +} + +for (const stderr of ['', 'pg_restore: warning: errors ignored on restore: 1', 'pg_restore: erreur: restauration incomplète']) { + test(`restore errors: nonzero exit fails closed with diagnostics ${JSON.stringify(stderr)}`, async t => { + const restoreError = Object.assign(new Error('restore command failed'), { status: 1, stderr: Buffer.from(stderr) }); + const fixture = drillFixture(t, { restoreError }); + await assert.rejects(runBackupRestoreDrill(fixture.options), error => error === restoreError); + assert.equal(existsSync(fixture.backupFile), false); + assert.ok(fixture.events.some(event => typeof event === 'string' && event.startsWith('DROP DATABASE'))); + }); +} + +test('identifiers: generated names retain isolation and uniqueness suffix within 63 bytes', () => { + for (const base of ['x'.repeat(63), '測試'.repeat(32), 'strange " source']) { + const name = generateIsolatedDbName(base); + assert.ok(Buffer.byteLength(name) <= 63, `generated identifier has ${Buffer.byteLength(name)} bytes`); + assert.match(name, /^[a-zA-Z0-9_-]+_backup_drill_restore_\d+_[a-f0-9]{8}$/); + } +}); + +test('ownership: an existing backup file survives an early failed drill', async t => { + const fixture = drillFixture(t, { connectError: new Error('source unavailable') }); + writeFileSync(fixture.backupFile, 'operator-owned backup'); + await assert.rejects(runBackupRestoreDrill(fixture.options)); + assert.equal(existsSync(fixture.backupFile), true); + assert.equal(readFileSync(fixture.backupFile, 'utf8'), 'operator-owned backup'); +}); + +test('ownership: an existing backup file is refused before it can be overwritten', async t => { + const fixture = drillFixture(t); + writeFileSync(fixture.backupFile, 'operator-owned backup'); + await assert.rejects(runBackupRestoreDrill(fixture.options), /exist/i); + assert.equal(readFileSync(fixture.backupFile, 'utf8'), 'operator-owned backup'); + assert.equal(fixture.events.some(event => event.executable === 'pg_dump'), false); +}); + +test('ownership: CREATE failure never drops a target this drill does not own', async t => { + const createError = new Error('database already exists'); + const fixture = drillFixture(t, { createError }); + await assert.rejects(runBackupRestoreDrill(fixture.options), error => error === createError); + assert.equal(fixture.events.some(event => typeof event === 'string' && event.startsWith('DROP DATABASE')), false); + assert.equal(existsSync(fixture.backupFile), false); +}); + +test('cleanup: drop failure preserves the restore failure and still closes admin and removes backup', async t => { + const restoreError = new Error('restore failed'); + const dropError = new Error('drop failed'); + const fixture = drillFixture(t, { restoreError, dropError }); + await assert.rejects(runBackupRestoreDrill(fixture.options), error => { + assert.ok(error instanceof AggregateError); + assert.deepEqual(error.errors, [restoreError, dropError]); + return true; + }); + assert.ok(fixture.events.includes('admin.end')); + assert.equal(existsSync(fixture.backupFile), false); +}); + +test('cleanup: baseline rollback and release failures do not mask the dump failure or skip outer cleanup', async t => { + const dumpError = new Error('dump failed'); + const rollbackError = new Error('rollback failed'); + const releaseError = new Error('release failed'); + const fixture = drillFixture(t, { restoreError: dumpError, rollbackError, releaseError }); + const options = { + ...fixture.options, + execSyncFn(command, args) { + if (args.includes('--version')) return Buffer.from('test tooling'); + if (command === 'pg_dump') throw dumpError; + return Buffer.alloc(0); + }, + }; + + await assert.rejects(runBackupRestoreDrill(options), error => { + assert.ok(error instanceof AggregateError); + assert.deepEqual(error.errors, [dumpError, rollbackError, releaseError]); + return true; + }); + assert.ok(fixture.events.includes('pool.end')); + assert.equal(existsSync(fixture.backupFile), false); +}); + +test('cleanup: backup removal failure prevents a success report', async t => { + const fixture = drillFixture(t, { backupCleanupError: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /directory|EPERM|EISDIR/i); + assert.ok(fixture.events.includes('admin.end')); +}); + +test('snapshot: plain dumps use the same exported snapshot as the baseline', async t => { + const fixture = drillFixture(t); + await runBackupRestoreDrill({ ...fixture.options, format: 'plain' }); + const dump = fixture.events.find(event => event.executable === 'pg_dump'); + assert.ok(dump.args.includes('--snapshot=00000001-00000001-1')); +}); + +test('snapshot: no exported snapshot aborts before creating a dump or target', async t => { + const fixture = drillFixture(t, { missingSnapshot: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /snapshot/i); + assert.equal(fixture.events.some(event => event.executable === 'pg_dump'), false); + assert.ok(fixture.events.includes('release')); +}); + +test('connection: query overrides cannot send pg clients and tools to different servers', async () => { + const source = 'postgres://u:p@127.0.0.1/source_test'; + for (const query of ['host=elsewhere', 'port=5433', 'user=other', 'password=query-secret', 'options=-csearch_path=other']) { + await assert.rejects(validateTargetIsolation(source, `postgres://u:p@127.0.0.1/restore_test?${query}`), /parameter/i); + } +}); + +test('security: URL diagnostics redact query passwords and fragments', () => { + const result = sanitizeDatabaseUrl('postgres://u:authority-secret@127.0.0.1/source_test?password=query-secret#fragment-secret'); + assert.doesNotMatch(result, /authority-secret|query-secret|fragment-secret/); +}); + +test('security: CLI never echoes credentials from malformed arguments', () => { + const script = fileURLToPath(new URL('../db-backup-restore-drill.mjs', import.meta.url)); + const result = spawnSync(process.execPath, [script, '--format', 'postgres://u:cli-regression-secret@invalid/db'], { + encoding: 'utf8', env: { ...process.env, DATABASE_URL: '', BACKUP_DRILL_SOURCE_URL: '', BACKUP_DRILL_TARGET_URL: '' }, + }); + assert.equal(result.status, 1); + assert.doesNotMatch(result.stdout + result.stderr, /cli-regression-secret/); +}); + +test('security: CLI failure diagnostics redact known connection passwords outside URLs', () => { + const script = fileURLToPath(new URL('../db-backup-restore-drill.mjs', import.meta.url)); + const result = spawnSync(process.execPath, [script, + '--source-url', 'postgres://u:production@127.0.0.1/source_test', + '--target-url', 'postgres://u:production@127.0.0.1/production', + ], { encoding: 'utf8' }); + assert.equal(result.status, 1); + assert.doesNotMatch(result.stdout + result.stderr, /production/); +}); + +test('baseline: unreadable table fails instead of silently omitting its row count', async () => { + const failure = new Error('permission denied for table'); + const client = { + async query(sql) { + if (sql.includes('pg_extension')) return { rows: [] }; + if (sql.includes('to_regclass')) return { rows: [{ present: false }] }; + if (sql.includes('pg_tables')) return { rows: [{ tablename: 'durable_data' }] }; + if (sql.includes('COUNT(*)')) throw failure; + throw new Error(`unexpected query ${sql}`); + }, + }; + await assert.rejects(collectSourceBaseline(null, client), error => error === failure); +}); + +for (const table of ['with"quote', '__proto__']) test(`baseline: ${table} preserves its exact row count`, async () => { + const client = { + async query(sql) { + if (sql.includes('pg_extension')) return { rows: [] }; + if (sql.includes('to_regclass')) return { rows: [{ present: false }] }; + if (sql.includes('pg_tables')) return { rows: [{ tablename: table }] }; + if (sql === 'SELECT COUNT(*) AS count FROM "with""quote"') return { rows: [{ count: '3' }] }; + if (sql === 'SELECT COUNT(*) AS count FROM "__proto__"') return { rows: [{ count: '7' }] }; + throw new Error('invalid quoted identifier'); + }, + }; + const baseline = await collectSourceBaseline(null, client); + assert.equal(Object.hasOwn(baseline.rowCounts, table), true); + assert.equal(baseline.rowCounts[table], table === '__proto__' ? 7 : 3); +}); + +test('verification: an unrelated P0001 error does not prove append-only protection', async t => { + const fixture = drillFixture(t, { unrelatedTriggerError: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /unrelated trigger failure/); +}); + +test('verification: an invalid HNSW index cannot pass verification', async t => { + const fixture = drillFixture(t, { invalidIndex: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /HNSW.*index/i); +}); + +test('verification: a trigger on another relation cannot protect empty public.game_events', async t => { + const fixture = drillFixture(t, { otherTriggerRelation: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /Append-only trigger/); +}); + +test('verification: an index on another schema cannot verify public.search_embeddings', async t => { + const fixture = drillFixture(t, { otherIndexRelation: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /HNSW.*index/i); +}); + +test('identifiers: explicit target cannot rely on PostgreSQL identifier truncation', async () => { + const source = 'postgres://u:p@127.0.0.1/source_test'; + await assert.rejects(validateTargetIsolation(source, `postgres://u:p@127.0.0.1/${'x'.repeat(63)}_restore`, { allowCustomTargetName: true }), /63/); + const boundary = 'restore_' + 'x'.repeat(55); + assert.equal((await validateTargetIsolation(source, `postgres://u:p@127.0.0.1/${boundary}`)).targetDbName, boundary); +}); + +for (const format of ['custom', 'plain']) { + for (const useDocker of [false, true]) { + test(`orchestration: ${format} restore completes with ${useDocker ? 'docker' : 'native'} tooling`, async t => { + const fixture = drillFixture(t); + const report = await runBackupRestoreDrill({ ...fixture.options, format, useDocker }); + assert.equal(report.success, true); + const restore = fixture.events.find(event => event.executable === (format === 'custom' ? 'pg_restore' : 'psql')); + assert.ok(restore); + if (format === 'custom') assert.ok(restore.args.includes('--exit-on-error')); + assert.ok(fixture.events.some(event => typeof event === 'string' && event.startsWith('DROP DATABASE'))); + assert.equal(existsSync(fixture.backupFile), false); + }); + } +}