From 75082fc36aad85acb664d8b653fce41095a1010f Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Sun, 6 Sep 2026 08:30:33 +0300 Subject: [PATCH 01/14] ops(db): add reproducible backup and restore verification --- docs/runbooks/backup-restore-drill.md | 222 +++++ scripts/db-backup-restore-drill.mjs | 928 +++++++++++++++++++++ scripts/test/backup-restore-drill.test.mjs | 540 ++++++++++++ 3 files changed, 1690 insertions(+) create mode 100644 docs/runbooks/backup-restore-drill.md create mode 100644 scripts/db-backup-restore-drill.mjs create mode 100644 scripts/test/backup-restore-drill.test.mjs diff --git a/docs/runbooks/backup-restore-drill.md b/docs/runbooks/backup-restore-drill.md new file mode 100644 index 00000000..eac7c787 --- /dev/null +++ b/docs/runbooks/backup-restore-drill.md @@ -0,0 +1,222 @@ +# Gambit — Database Backup & Restore Drill Runbook + +> **Audience:** Operators, SREs, Database Administrators.
+> **Frequency:** Monthly scheduled drill, pre-release rehearsal for major schema migrations, disaster recovery validation.
+> **Target RTO:** < 15 minutes for 100k games.
+> **Target RPO:** 0 data loss for committed transactions (append-only event store). + +--- + +## 1. Objectives & Overview + +Gambit's system of record relies on PostgreSQL 16 with the `citext` and `vector` (pgvector) extensions. The primary source of truth is the append-only `game_events` event store, supported by relational projections (`users`, `credentials`, `games`, `ratings`, `tournaments`, `search_embeddings`). + +This runbook defines the operational procedure to: +1. Generate a consistent backup of the application database without downtime. +2. Restore the backup into a brand-new, isolated target database. +3. Validate that all durable application data, triggers, indexes, and extensions survived. +4. Ensure safety: destructive actions are strictly guarded against touching the primary or production databases. + +--- + +## 2. Automated Drill Execution + +The platform includes an automated, operator-usable drill tool: [`scripts/db-backup-restore-drill.mjs`](../../scripts/db-backup-restore-drill.mjs). + +### 2.1 Basic Usage + +Run against the default `DATABASE_URL` or an explicit source URL: + +```bash +# Run with default environment DATABASE_URL +node scripts/db-backup-restore-drill.mjs + +# Run with explicit source URL +node scripts/db-backup-restore-drill.mjs --source-url "postgres://gambit:secret@localhost:5432/gambit" +``` + +The script will automatically: +1. Connect to the source and record a baseline of tables, row counts, migrations, and sample data. +2. Resolve pg tooling (native `pg_dump`/`pg_restore` if installed, or automatic Docker `pgvector/pgvector:pg16` fallback). +3. Create a custom-format archive (`-Fc`). +4. Validate the backup file size and `PGDMP` header magic. +5. Create a disposable isolated target database (e.g. `gambit_backup_drill_restore__`). +6. Restore the backup into the isolated database. +7. Perform deep structural and functional verification. +8. Drop the isolated target database and clean up the temporary dump file. +9. Print a structured diagnostic summary. + +### 2.2 CLI Options & Flags + +| Option | Default | Description | +|---|---|---| +| `--source-url ` | `$DATABASE_URL` | Source database connection URL | +| `--target-url ` | Auto-generated | Target database connection URL (must be isolated) | +| `--backup-file ` | Temp file | Path to write backup dump file | +| `--keep-backup` | `false` | Preserve the backup dump file after the drill completes | +| `--keep-target` | `false` | Preserve the restored target DB for forensic inspection | +| `--format ` | `custom` | Dump format (`custom` = `-Fc`, `plain` = SQL) | +| `--use-docker` | `auto` | Force execution inside `pgvector/pgvector:pg16` container | +| `--allow-custom-target-name`| `false` | Allow target DB name without default isolation markers | +| `--json` | `false` | Output machine-readable JSON report | + +### 2.3 Retaining the Target Database for Forensic Inspection + +When diagnosing schema discrepancies or inspecting restore behavior, instruct the drill to keep the restored database: + +```bash +node scripts/db-backup-restore-drill.mjs \ + --source-url "postgres://gambit:secret@localhost:5432/gambit" \ + --keep-target \ + --keep-backup +``` + +Output will report the exact target database name created: +``` +Target: postgres://gambit:***@localhost:5432/gambit_backup_drill_restore_1788672281592_96c80ce5 +Backup: /tmp/gambit_backup_1788672281593_2706.dump +``` + +When finished with forensic analysis, drop the database manually: +```sql +DROP DATABASE "gambit_backup_drill_restore_1788672281592_96c80ce5" WITH (FORCE); +``` + +--- + +## 3. Manual Operator Walkthrough + +If the automated script is unavailable, perform the drill manually using standard PostgreSQL client utilities (`pg_dump`, `pg_restore`, `psql`). + +### Step 1: Create Backup + +Generate a PostgreSQL custom archive format backup. The custom format includes a table of contents (TOC) for selective restore and parallel processing: + +```bash +pg_dump \ + -h "${PGHOST:-localhost}" \ + -p "${PGPORT:-5432}" \ + -U "${PGUSER:-gambit}" \ + -d "${PGDATABASE:-gambit}" \ + -F c \ + -f "/tmp/gambit_manual_drill_$(date +%s).dump" +``` + +### Step 2: Validate Backup File Integrity + +Ensure the dump file is non-empty and starts with the PostgreSQL dump magic bytes (`PGDMP`): + +```bash +# Check size +ls -lh /tmp/gambit_manual_drill_*.dump + +# Check header magic (should print PGDMP) +head -c 5 /tmp/gambit_manual_drill_*.dump +``` + +### Step 3: Provision Clean Isolated Target Database + +Connect to the PostgreSQL administrative database (`postgres` or `template1`) and create a dedicated drill database: + +```bash +TARGET_DB="gambit_backup_drill_restore_$(date +%s)" + +psql -h "${PGHOST:-localhost}" -p "${PGPORT:-5432}" -U "${PGUSER:-gambit}" -d postgres -c " + CREATE DATABASE \"${TARGET_DB}\"; +" +``` + +### Step 4: Restore into Isolated Target + +Restore the custom archive into the target database: + +```bash +pg_restore \ + -h "${PGHOST:-localhost}" \ + -p "${PGPORT:-5432}" \ + -U "${PGUSER:-gambit}" \ + -d "${TARGET_DB}" \ + --clean \ + --if-exists \ + "/tmp/gambit_manual_drill_*.dump" +``` + +### Step 5: Verification Checklist + +Connect to the restored database (`${TARGET_DB}`) and execute the following checks: + +#### 1. Extension Verification +Verify that both `citext` and `vector` extensions exist: +```sql +SELECT extname, extversion FROM pg_extension WHERE extname IN ('citext', 'vector'); +-- Expected: 2 rows (citext and vector) +``` + +#### 2. Schema Migrations Ledger +Verify that all migrations are present and checksums match the source: +```sql +SELECT count(*) FROM schema_migrations WHERE state = 'applied'; +-- Must match count of migration files in packages/persistence/migrations/ (e.g. 31) +``` + +#### 3. Append-Only Trigger Verification +Verify that the `game_events` immutability trigger is active by testing an `UPDATE`: +```sql +BEGIN; +UPDATE game_events SET seq = seq WHERE game_id IN (SELECT game_id FROM game_events LIMIT 1); +-- Expected error: "game_events is append-only (UPDATE.game_events attempted)" +ROLLBACK; +``` +If the statement succeeds without raising an exception, the trigger is missing or inactive! + +#### 4. Durable State Row Counts +Compare row counts between source and restored databases: +```sql +SELECT 'users' AS tbl, count(*) FROM users +UNION ALL +SELECT 'game_events', count(*) FROM game_events +UNION ALL +SELECT 'games', count(*) FROM games +UNION ALL +SELECT 'tournaments', count(*) FROM tournaments +UNION ALL +SELECT 'ratings', count(*) FROM ratings +UNION ALL +SELECT 'search_embeddings', count(*) FROM search_embeddings; +``` + +#### 5. Vector Query Functionality +Verify pgvector cosine distance operations and HNSW indexing: +```sql +-- Test distance operator (<->) +SELECT id, embedding <-> embedding AS distance FROM search_embeddings LIMIT 1; +``` + +### Step 6: Cleanup Isolated Target + +After verification succeeds, drop the temporary drill database: + +```sql +DROP DATABASE "gambit_backup_drill_restore_" WITH (FORCE); +``` + +--- + +## 4. Safety & Isolation Guardrails + +1. **Never Overwrite Source:** The drill script explicitly compares the normalized source and target URLs. If the host, port, and database name match, execution aborts immediately. +2. **Protected Databases:** Destructive commands will refuse to drop databases named `gambit`, `postgres`, `template1`, `production`, `master`, or `main`. +3. **Naming Convention:** Target databases must contain an isolation indicator (`drill`, `restore`, `disposable`, `test`, or `isolated`) unless the `--allow-custom-target-name` flag is explicitly provided. +4. **Credential Security:** Database URLs are masked in all logs and console output (`postgres://user:***@host:port/db`). Passwords are passed via environment variables (`PGPASSWORD`), never exposed in `ps` process arguments. + +--- + +## 5. Troubleshooting Failed Restores + +| Symptom | Probable Cause | Corrective Action | +|---|---|---| +| `extension "vector" is not available` | Target PostgreSQL instance lacks `pgvector` library | Install `postgresql-16-pgvector` package or use `pgvector/pgvector:pg16` image. | +| `Trigger on game_events failed: mutation was not blocked` | Restore command stripped or disabled triggers | Ensure `pg_restore` did not run with `--disable-triggers` without re-enabling them. | +| `Row count mismatch for table "X"` | Partial dump or table-level exclusion | Ensure `pg_dump` was run for the whole database without `--schema-only` or `--exclude-table`. | +| `Migration checksum mismatch` | Working copy newline translation or modified migration | Run `npm run check:ci-parity` and ensure canonical LF newlines in migration files. | +| `Backup file does not exist or empty` | Permissions or disk space exhaustion | Check disk space in `/tmp` and file write permissions for PostgreSQL process. | diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs new file mode 100644 index 00000000..0224105f --- /dev/null +++ b/scripts/db-backup-restore-drill.mjs @@ -0,0 +1,928 @@ +#!/usr/bin/env node +/** + * @file scripts/db-backup-restore-drill.mjs + * + * Reproducible, operator-usable PostgreSQL backup and restore verification drill. + * + * What this script does: + * 1. Validates connection parameters and enforces target database isolation guardrails. + * 2. Connects to the source application database and captures a baseline of durable state: + * - Schema migrations ledger and SHA-256 checksums + * - Required extensions (citext, vector) + * - Critical application tables and exact row counts + * - Representative durable sample records (users, game events, tournaments, etc.) + * 3. Creates a standard PostgreSQL backup using pg_dump (-Fc custom archive format with TOC). + * 4. Validates the backup artifact (file size, header magic "PGDMP", integrity). + * 5. Provisions a clean, isolated target database (never overwriting source). + * 6. Restores the backup into the isolated target using pg_restore. + * 7. Performs deep structural and functional verification: + * - All required extensions are active (citext, vector) + * - Migration count and checksums match source byte-for-byte + * - All critical application tables are present + * - Row counts across all durable tables match 100% + * - Sample durable records match source data + * - Append-only trigger (game_events_block_mutate) is active and blocks UPDATE/DELETE + * - pgvector operations and HNSW indexes are valid and queryable + * - Target database is functional and writable + * 8. Cleans up isolated target database and temporary dump file (unless flags request retention). + * 9. Emits a structured diagnostic audit report. + * + * Security: + * - Passwords and credentials are NEVER logged or printed. + * - PGPASSWORD is passed via process environment, not command-line arguments. + * - Destructive actions are restricted to isolated targets; refusing any target matching source. + * + * Usage: + * node scripts/db-backup-restore-drill.mjs [options] + * + * Options: + * --source-url Source database URL (defaults to DATABASE_URL) + * --target-url Explicit target database URL (must be isolated) + * --target-db-name Target database name (default: auto-generated isolated name) + * --backup-file Path for backup dump file (default: temporary file) + * --keep-backup Preserve the backup dump file after the drill + * --keep-target Preserve the restored target database after the drill + * --format pg_dump format (default: custom) + * --use-docker Force execution of pg tools via Docker container + * --docker-image Docker image for pg tools (default: pgvector/pgvector:pg16) + * --allow-custom-target-name Permit target name without default isolation markers + * --json Output drill report in JSON format + * --help Show this help message + */ + +import { execFileSync, spawnSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { existsSync, statSync, unlinkSync, readFileSync, openSync, readSync, closeSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join, resolve, dirname, basename } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import pg from 'pg'; + +const { Pool, Client } = pg; + +/** Critical application tables that hold durable state or lookup vocabularies. */ +export const CRITICAL_APPLICATION_TABLES = [ + 'schema_migrations', + 'variants', + 'terminations', + 'users', + 'credentials', + 'webauthn_credentials', + 'sessions', + 'roles', + 'ratings', + 'seeks', + 'game_events', + 'games', + 'tournaments', + 'audit_log', + 'search_documents', + 'search_embeddings', +]; + +/** Required PostgreSQL extensions for the Gambit platform. */ +export const REQUIRED_EXTENSIONS = ['citext', 'vector']; + +/** Names of databases that must never be targeted for destructive drop/overwrite. */ +const PROTECTED_DATABASE_NAMES = new Set([ + 'gambit', + 'postgres', + 'template0', + 'template1', + 'production', + 'prod', + 'master', + 'main', +]); + +/** Isolation keywords required in target database names unless explicitly overridden. */ +const ISOLATION_MARKERS = /(?:drill|restore|disposable|test|isolated)/i; + +/** + * Mask passwords in database connection strings for safe logging and error reporting. + */ +export function sanitizeDatabaseUrl(urlString) { + if (typeof urlString !== 'string' || !urlString) { + return urlString || ''; + } + try { + const parsed = new URL(urlString); + if (parsed.password) { + parsed.password = '***'; + } + return parsed.toString(); + } catch { + // Regex fallback for partially malformed or non-standard connection strings + return urlString.replace(/(:\/\/)([^:@]+)(?::([^@]+))?(@)/, '$1$2:***$4'); + } +} + +/** + * Generate a unique, isolated database name for the restore drill. + */ +export function generateIsolatedDbName(base = 'gambit') { + const timestamp = Date.now(); + const random = Math.floor(Math.random() * 0xffffffff).toString(16).padStart(8, '0'); + return `${base}_backup_drill_restore_${timestamp}_${random}`; +} + +/** + * Parse a PostgreSQL connection string into host, port, user, password, and database. + */ +export function parseDatabaseUrl(urlString) { + const parsed = new URL(urlString); + const dbName = parsed.pathname.replace(/^\//, '') || 'postgres'; + return { + host: parsed.hostname || 'localhost', + port: parsed.port ? parseInt(parsed.port, 10) : 5432, + user: decodeURIComponent(parsed.username || 'postgres'), + password: decodeURIComponent(parsed.password || ''), + database: decodeURIComponent(dbName), + searchParams: parsed.searchParams, + protocol: parsed.protocol, + }; +} + +/** + * Construct a connection URL replacing only the database name. + */ +export function urlWithDatabase(urlString, databaseName) { + const parsed = new URL(urlString); + parsed.pathname = `/${encodeURIComponent(databaseName)}`; + return parsed.toString(); +} + +/** + * Enforce target isolation guardrails to guarantee the drill cannot overwrite or drop source/production DBs. + */ +export function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { + if (!sourceUrl) { + throw new Error('Source database URL is required'); + } + if (!targetUrl) { + throw new Error('Target database URL is required'); + } + + const source = parseDatabaseUrl(sourceUrl); + const target = parseDatabaseUrl(targetUrl); + + // 1. URLs must not be identical + if (sourceUrl.trim() === targetUrl.trim()) { + throw new Error( + `Target database URL must not be identical to source database URL: ${sanitizeDatabaseUrl(targetUrl)}`, + ); + } + + // 2. Target must not target the same database on the same host/port + if ( + source.host.toLowerCase() === target.host.toLowerCase() && + source.port === target.port && + source.database.toLowerCase() === target.database.toLowerCase() + ) { + throw new Error( + `Target database name "${target.database}" matches source database name on the same host (${target.host}:${target.port}). Refusing unsafe operation.`, + ); + } + + // 3. Target must not be a protected system or production database name + if (PROTECTED_DATABASE_NAMES.has(target.database.toLowerCase())) { + throw new Error( + `Target database name "${target.database}" is a protected or non-isolated database. Target must be a dedicated disposable drill database.`, + ); + } + + // 4. Target name must contain an isolation marker unless explicitly overridden + if (!ISOLATION_MARKERS.test(target.database) && !options.allowCustomTargetName) { + throw new Error( + `Target database name "${target.database}" does not contain an isolated drill marker (such as "drill", "restore", "test", "disposable", "isolated"). Refusing to use potentially unsafe target. Supply --allow-custom-target-name if this is intentional.`, + ); + } + + return { + isolated: true, + sourceDbName: source.database, + targetDbName: target.database, + }; +} + +/** + * Validate a backup file's existence, size, header magic, and digest. + */ +export function validateBackupFile(filePath, format = 'custom') { + if (!existsSync(filePath)) { + throw new Error(`Backup file does not exist: ${filePath}`); + } + + const stats = statSync(filePath); + if (stats.size === 0) { + throw new Error(`Backup file is empty (0 bytes): ${filePath}`); + } + + // Check header magic for custom format + if (format === 'custom') { + if (stats.size < 5) { + throw new Error(`Invalid custom-format backup: file is too small (${stats.size} bytes)`); + } + const fd = openSync(filePath, 'r'); + const headerBuf = Buffer.alloc(5); + try { + readSync(fd, headerBuf, 0, 5, 0); + } finally { + closeSync(fd); + } + if (headerBuf.toString('ascii') !== 'PGDMP') { + throw new Error( + 'Invalid custom-format backup: missing PostgreSQL dump magic header "PGDMP". The backup may be corrupted or created in an incompatible format.', + ); + } + } + + // Compute SHA-256 digest + const fileBytes = readFileSync(filePath); + const sha256 = createHash('sha256').update(fileBytes).digest('hex'); + + return { + valid: true, + filePath, + sizeBytes: stats.size, + sha256, + }; +} + +/** + * Inspect installed PostgreSQL tools or fallback to Docker. + */ +export function resolvePgTooling(options = {}) { + const forceDocker = options.useDocker === true || options.useDocker === 'true'; + const dockerImage = options.dockerImage || 'pgvector/pgvector:pg16'; + + let hasNativePgDump = false; + let hasNativePgRestore = false; + let hasNativePsql = false; + + if (!forceDocker) { + try { + execFileSync('pg_dump', ['--version'], { stdio: 'ignore' }); + hasNativePgDump = true; + } catch {} + try { + execFileSync('pg_restore', ['--version'], { stdio: 'ignore' }); + hasNativePgRestore = true; + } catch {} + try { + execFileSync('psql', ['--version'], { stdio: 'ignore' }); + hasNativePsql = true; + } catch {} + } + + if (hasNativePgDump && hasNativePgRestore) { + return { + type: 'native', + runDump: (args, env) => execFileSync('pg_dump', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + runRestore: (args, env) => execFileSync('pg_restore', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + runPsql: (args, env) => execFileSync('psql', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + }; + } + + // Fallback: Check if Docker is available + let hasDocker = false; + try { + execFileSync('docker', ['--version'], { stdio: 'ignore' }); + hasDocker = true; + } catch {} + + if (!hasDocker) { + throw new Error( + 'Neither native PostgreSQL tools (pg_dump, pg_restore) nor Docker are available in PATH. Please install postgresql-client or ensure Docker is running.', + ); + } + + return { + type: 'docker', + dockerImage, + runDump: (args, env, mountDir) => { + const dockerArgs = ['run', '--rm']; + if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`); + if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); + // Network host so it can reach localhost postgres + if (process.platform === 'linux') { + dockerArgs.push('--net=host'); + } + dockerArgs.push(dockerImage, 'pg_dump', ...args); + return execFileSync('docker', dockerArgs, { stdio: 'pipe' }); + }, + runRestore: (args, env, mountDir) => { + const dockerArgs = ['run', '--rm']; + if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`); + if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); + if (process.platform === 'linux') { + dockerArgs.push('--net=host'); + } + dockerArgs.push(dockerImage, 'pg_restore', ...args); + return execFileSync('docker', dockerArgs, { stdio: 'pipe' }); + }, + runPsql: (args, env, mountDir) => { + const dockerArgs = ['run', '--rm']; + if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`); + if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); + if (process.platform === 'linux') { + dockerArgs.push('--net=host'); + } + dockerArgs.push(dockerImage, 'psql', ...args); + return execFileSync('docker', dockerArgs, { stdio: 'pipe' }); + }, + }; +} + +/** + * Capture source database state baseline before running backup. + */ +export async function collectSourceBaseline(pool) { + // 1. Extensions + const extRes = await pool.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); + const extensions = extRes.rows.map((r) => r.extname); + + // 2. Schema migrations + const hasMigrationsTable = await pool.query( + "SELECT to_regclass('schema_migrations') IS NOT NULL AS present", + ); + let migrations = []; + if (hasMigrationsTable.rows[0]?.present) { + const migRes = await pool.query( + 'SELECT version, name, checksum, state FROM schema_migrations ORDER BY version', + ); + migrations = migRes.rows; + } + + // 3. Existing tables in public schema + const tableRes = await pool.query( + "SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename", + ); + const tables = tableRes.rows.map((r) => r.tablename); + + // 4. Row counts across critical tables + const rowCounts = {}; + for (const table of tables) { + try { + const countRes = await pool.query(`SELECT COUNT(*) AS count FROM "${table}"`); + rowCounts[table] = parseInt(countRes.rows[0].count, 10); + } catch { + // Non-readable table or view + } + } + + // 5. Sample records for integrity comparison + const sampleData = {}; + if (tables.includes('users') && (rowCounts['users'] || 0) > 0) { + const sampleUsers = await pool.query('SELECT id, handle FROM users ORDER BY created_at LIMIT 5'); + sampleData.users = sampleUsers.rows; + } + if (tables.includes('game_events') && (rowCounts['game_events'] || 0) > 0) { + const sampleEvents = await pool.query( + 'SELECT game_id, seq, type FROM game_events ORDER BY server_ts DESC LIMIT 5', + ); + sampleData.game_events = sampleEvents.rows; + } + if (tables.includes('tournaments') && (rowCounts['tournaments'] || 0) > 0) { + const sampleTournaments = await pool.query( + 'SELECT id, name, format FROM tournaments ORDER BY created_at DESC LIMIT 5', + ); + sampleData.tournaments = sampleTournaments.rows; + } + + return { + extensions, + migrations, + tables, + rowCounts, + sampleData, + }; +} + +/** + * Deep structural and functional verification of restored target database against source baseline. + */ +export async function verifyRestoredDatabase(sourceBaseline, targetPool, options = {}) { + const checks = []; + const failures = []; + + const recordCheck = (name, passed, detail) => { + checks.push({ name, passed, detail }); + if (!passed) { + failures.push({ name, detail }); + } + }; + + // Check 1: Extensions + const extRes = await targetPool.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); + const targetExts = new Set(extRes.rows.map((r) => r.extname)); + + for (const requiredExt of REQUIRED_EXTENSIONS) { + if (sourceBaseline.extensions.includes(requiredExt)) { + const present = targetExts.has(requiredExt); + recordCheck( + `Extension: ${requiredExt}`, + present, + present ? 'Installed and active' : `Missing required extension in restored database: ${requiredExt}`, + ); + if (!present) { + throw new Error(`Missing required extension in restored database: ${requiredExt}`); + } + } + } + + // Check 2: Schema migrations ledger + if (sourceBaseline.migrations && sourceBaseline.migrations.length > 0) { + const migTableRes = await targetPool.query( + "SELECT to_regclass('schema_migrations') IS NOT NULL AS present", + ); + if (!migTableRes.rows[0]?.present) { + recordCheck('Migrations Ledger', false, 'Table schema_migrations does not exist in restored database'); + throw new Error('Table schema_migrations does not exist in restored database'); + } + + const targetMigRes = await targetPool.query( + 'SELECT version, name, checksum, state FROM schema_migrations ORDER BY version', + ); + const targetMigrations = targetMigRes.rows; + + if (targetMigrations.length !== sourceBaseline.migrations.length) { + const msg = `Migration count mismatch: source had ${sourceBaseline.migrations.length}, restored has ${targetMigrations.length}`; + recordCheck('Migration Count', false, msg); + throw new Error(msg); + } + recordCheck( + 'Migration Count', + true, + `All ${sourceBaseline.migrations.length} migration records restored`, + ); + + const targetMigMap = new Map(targetMigrations.map((m) => [m.version, m])); + for (const srcMig of sourceBaseline.migrations) { + const tgtMig = targetMigMap.get(srcMig.version); + if (!tgtMig) { + throw new Error(`Restored database is missing recorded migration version ${srcMig.version}`); + } + if (tgtMig.checksum !== srcMig.checksum) { + const msg = `Migration ${srcMig.version} checksum mismatch: source="${srcMig.checksum}", restored="${tgtMig.checksum}"`; + recordCheck(`Migration Checksum ${srcMig.version}`, false, msg); + throw new Error(msg); + } + } + recordCheck('Migration Checksums', true, 'All migration ledger checksums match source byte-for-byte'); + } + + // Check 3: Tables + const tgtTableRes = await targetPool.query( + "SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename", + ); + const targetTables = new Set(tgtTableRes.rows.map((r) => r.tablename)); + + for (const expectedTable of sourceBaseline.tables) { + const present = targetTables.has(expectedTable); + if (!present) { + const msg = `Missing required table in restored database: ${expectedTable}`; + recordCheck(`Table: ${expectedTable}`, false, msg); + throw new Error(msg); + } + } + recordCheck( + 'Table Structure', + true, + `All ${sourceBaseline.tables.length} tables present in restored database`, + ); + + // Check 4: Row Counts + for (const [table, expectedCount] of Object.entries(sourceBaseline.rowCounts)) { + if (!targetTables.has(table)) continue; + const countRes = await targetPool.query(`SELECT COUNT(*) AS count FROM "${table}"`); + const actualCount = parseInt(countRes.rows[0].count, 10); + if (actualCount !== expectedCount) { + const msg = `Row count mismatch for table "${table}": source had ${expectedCount}, restored has ${actualCount}`; + recordCheck(`Row Count: ${table}`, false, msg); + throw new Error(msg); + } + } + recordCheck('Row Counts', true, 'All table row counts match source database 100%'); + + // Check 5: Sample Data Identity + if (sourceBaseline.sampleData?.users?.length) { + const userSample = sourceBaseline.sampleData.users[0]; + const userCheck = await targetPool.query('SELECT id, handle FROM users WHERE id = $1', [userSample.id]); + if (userCheck.rows.length === 0 || userCheck.rows[0].handle !== userSample.handle) { + throw new Error(`Sample user record ${userSample.id} (${userSample.handle}) not found or mismatched in restored database`); + } + recordCheck('Sample Data: Users', true, `Sample user ${userSample.handle} verified`); + } + + if (sourceBaseline.sampleData?.game_events?.length) { + const eventSample = sourceBaseline.sampleData.game_events[0]; + const eventCheck = await targetPool.query( + 'SELECT game_id, seq, type FROM game_events WHERE game_id = $1 AND seq = $2', + [eventSample.game_id, eventSample.seq], + ); + if (eventCheck.rows.length === 0 || eventCheck.rows[0].type !== eventSample.type) { + throw new Error(`Sample game_event ${eventSample.game_id}#${eventSample.seq} not found or mismatched in restored database`); + } + recordCheck('Sample Data: Game Events', true, `Sample game event ${eventSample.type} verified`); + } + + // Check 6: Append-only trigger enforcement on game_events + if (targetTables.has('game_events')) { + let triggerActive = false; + try { + const res = await targetPool.query( + 'UPDATE game_events SET seq = seq WHERE game_id IN (SELECT game_id FROM game_events LIMIT 1)', + ); + if (res.rowCount && res.rowCount > 0) { + triggerActive = false; + } else { + // Table was empty; verify trigger registration in pg_trigger catalog + const trigRes = await targetPool.query( + "SELECT tgname FROM pg_trigger WHERE tgname = 'game_events_block_mutate' AND tgenabled = 'O'", + ); + triggerActive = trigRes.rows.length > 0; + } + } catch (err) { + if ( + err.message && + (err.message.includes('game_events is append-only') || err.code === 'P0001') + ) { + triggerActive = true; + } else { + throw err; + } + } + + if (!triggerActive) { + const msg = 'Append-only trigger on game_events failed: mutation was not blocked'; + recordCheck('Trigger: game_events append-only', false, msg); + throw new Error(msg); + } + recordCheck('Trigger: game_events append-only', true, 'Mutation blocked by trigger game_events_block_mutate'); + } + + // Check 7: Semantic search / pgvector functional test + if (targetTables.has('search_embeddings') && targetExts.has('vector')) { + try { + // Test vector operator syntax and index validity + const testVec = `[${new Array(256).fill(0.1).join(',')}]`; + await targetPool.query('SELECT $1::vector(256) <-> $1::vector(256) AS dist', [testVec]); + recordCheck('pgvector Functionality', true, 'Vector distance operator (<->) functional'); + } catch (err) { + const msg = `Vector functionality verification failed: ${err.message}`; + recordCheck('pgvector Functionality', false, msg); + throw new Error(msg); + } + } + + return { + passed: failures.length === 0, + checks, + failures, + }; +} + +/** + * CLI Argument parser. + */ +export function parseArgs(args = process.argv.slice(2)) { + const result = { + sourceUrl: process.env.BACKUP_DRILL_SOURCE_URL || process.env.DATABASE_URL || '', + targetUrl: process.env.BACKUP_DRILL_TARGET_URL || '', + targetDbName: '', + backupFile: '', + keepBackup: false, + keepTarget: false, + format: 'custom', + useDocker: 'auto', + dockerImage: 'pgvector/pgvector:pg16', + allowCustomTargetName: false, + json: false, + help: false, + }; + + for (let i = 0; i < args.length; i++) { + const arg = args[i]; + if (arg === '--source-url' && i + 1 < args.length) { + result.sourceUrl = args[++i]; + } else if (arg === '--target-url' && i + 1 < args.length) { + result.targetUrl = args[++i]; + } else if (arg === '--target-db-name' && i + 1 < args.length) { + result.targetDbName = args[++i]; + } else if (arg === '--backup-file' && i + 1 < args.length) { + result.backupFile = args[++i]; + } else if (arg === '--keep-backup') { + result.keepBackup = true; + } else if (arg === '--keep-target') { + result.keepTarget = true; + } else if (arg === '--format' && i + 1 < args.length) { + result.format = args[++i]; + } else if (arg === '--use-docker') { + result.useDocker = true; + } else if (arg === '--no-docker') { + result.useDocker = false; + } else if (arg === '--docker-image' && i + 1 < args.length) { + result.dockerImage = args[++i]; + } else if (arg === '--allow-custom-target-name') { + result.allowCustomTargetName = true; + } else if (arg === '--json') { + result.json = true; + } else if (arg === '--help' || arg === '-h') { + result.help = true; + } + } + + if (result.sourceUrl && !result.targetUrl) { + const isolatedName = result.targetDbName || generateIsolatedDbName('gambit'); + result.targetUrl = urlWithDatabase(result.sourceUrl, isolatedName); + } + + return result; +} + +/** + * Execute full backup, isolated restore, and verification drill. + */ +export async function runBackupRestoreDrill(options = {}) { + const startTime = Date.now(); + const report = { + startedAt: new Date(startTime).toISOString(), + source: '', + target: '', + backupFile: '', + backupSizeBytes: 0, + backupSha256: '', + timings: {}, + checks: [], + success: false, + }; + + const parsedSource = parseDatabaseUrl(options.sourceUrl); + const parsedTarget = parseDatabaseUrl(options.targetUrl); + + report.source = sanitizeDatabaseUrl(options.sourceUrl); + report.target = sanitizeDatabaseUrl(options.targetUrl); + + // Validate isolation + validateTargetIsolation(options.sourceUrl, options.targetUrl, { + allowCustomTargetName: options.allowCustomTargetName, + }); + + const sourcePool = new Pool({ connectionString: options.sourceUrl, max: 2 }); + let targetPool = null; + let adminClient = null; + + const rawBackupPath = + options.backupFile || + join(tmpdir(), `gambit_backup_${Date.now()}_${Math.floor(Math.random() * 0xffff).toString(16)}.dump`); + const backupPath = resolve(rawBackupPath); + const backupDir = dirname(backupPath); + const backupFileName = basename(backupPath); + report.backupFile = backupPath; + + const log = (msg) => { + if (!options.json) { + console.log(`[drill] ${msg}`); + } + }; + + try { + // 1. Capture source baseline + log(`Capturing source baseline from ${report.source}...`); + const sourceBaseline = await collectSourceBaseline(sourcePool); + log(`Source baseline captured: ${sourceBaseline.tables.length} tables, ${sourceBaseline.migrations.length} migrations.`); + + // 2. Resolve tooling + const tooling = resolvePgTooling(options); + log(`Resolved PostgreSQL tooling (${tooling.type})...`); + + // 3. Perform backup using pg_dump + log(`Creating ${options.format || 'custom'} backup to ${backupPath}...`); + const dumpStart = Date.now(); + const isCustom = (options.format || 'custom') === 'custom'; + + if (tooling.type === 'native') { + const dumpArgs = [ + '-h', parsedSource.host, + '-p', String(parsedSource.port), + '-U', parsedSource.user, + '-d', parsedSource.database, + '-F', isCustom ? 'c' : 'p', + '-f', backupPath, + ]; + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password }); + } else { + // Docker mode + const hostForDocker = + parsedSource.host === 'localhost' || parsedSource.host === '127.0.0.1' + ? (process.platform === 'linux' ? '127.0.0.1' : 'host.docker.internal') + : parsedSource.host; + + const dumpArgs = [ + '-h', hostForDocker, + '-p', String(parsedSource.port), + '-U', parsedSource.user, + '-d', parsedSource.database, + '-F', isCustom ? 'c' : 'p', + '-f', `/work/${backupFileName}`, + ]; + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password }, backupDir); + } + report.timings.backupMs = Date.now() - dumpStart; + log(`Backup completed in ${report.timings.backupMs}ms.`); + + // 4. Validate backup file + const backupMeta = validateBackupFile(backupPath, options.format || 'custom'); + report.backupSizeBytes = backupMeta.sizeBytes; + report.backupSha256 = backupMeta.sha256; + log(`Backup file validated: ${backupMeta.sizeBytes} bytes (SHA-256: ${backupMeta.sha256.substring(0, 12)}...).`); + + // 5. Create isolated target database + log(`Provisioning isolated target database "${parsedTarget.database}"...`); + const adminUrl = urlWithDatabase(options.sourceUrl, 'postgres'); + adminClient = new Client({ connectionString: adminUrl }); + try { + await adminClient.connect(); + } catch { + // Try template1 if postgres db is not accessible + adminClient = new Client({ connectionString: urlWithDatabase(options.sourceUrl, 'template1') }); + await adminClient.connect(); + } + + await adminClient.query(`CREATE DATABASE "${parsedTarget.database}"`); + log(`Target database "${parsedTarget.database}" created.`); + + // 6. Restore backup into target + log(`Restoring backup into ${report.target}...`); + const restoreStart = Date.now(); + if (tooling.type === 'native') { + if (isCustom) { + const restoreArgs = [ + '-h', parsedTarget.host, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '--clean', + '--if-exists', + backupPath, + ]; + try { + tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password }); + } catch (err) { + // pg_restore may exit with non-critical warnings on drops of non-existent objects + if (!err.message?.includes('exit code 1')) throw err; + } + } else { + const psqlArgs = [ + '-h', parsedTarget.host, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '-f', backupPath, + ]; + tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password }); + } + } else { + // Docker mode + const hostForDocker = + parsedTarget.host === 'localhost' || parsedTarget.host === '127.0.0.1' + ? (process.platform === 'linux' ? '127.0.0.1' : 'host.docker.internal') + : parsedTarget.host; + + if (isCustom) { + const restoreArgs = [ + '-h', hostForDocker, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '--clean', + '--if-exists', + `/work/${backupFileName}`, + ]; + try { + tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password }, backupDir); + } catch (err) { + if (!err.message?.includes('exit code 1')) throw err; + } + } else { + const psqlArgs = [ + '-h', hostForDocker, + '-p', String(parsedTarget.port), + '-U', parsedTarget.user, + '-d', parsedTarget.database, + '-f', `/work/${backupFileName}`, + ]; + tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password }, backupDir); + } + } + report.timings.restoreMs = Date.now() - restoreStart; + log(`Restore completed in ${report.timings.restoreMs}ms.`); + + // 7. Verify restored database + log('Running comprehensive structural and functional verification...'); + const verifyStart = Date.now(); + targetPool = new Pool({ connectionString: options.targetUrl, max: 2 }); + const verifyResult = await verifyRestoredDatabase(sourceBaseline, targetPool, options); + report.timings.verifyMs = Date.now() - verifyStart; + report.checks = verifyResult.checks; + + log(`Verification passed: ${verifyResult.checks.length} checks succeeded.`); + report.success = true; + } finally { + // Teardown connections + await sourcePool.end().catch(() => {}); + if (targetPool) { + await targetPool.end().catch(() => {}); + } + + // Teardown target database if not keeping + if (adminClient) { + if (!options.keepTarget && parsedTarget.database) { + try { + log(`Cleaning up isolated target database "${parsedTarget.database}"...`); + await adminClient.query(`DROP DATABASE IF EXISTS "${parsedTarget.database}" WITH (FORCE)`); + log('Target database dropped.'); + } catch (err) { + log(`Warning: Failed to drop isolated target database: ${err.message}`); + } + } + await adminClient.end().catch(() => {}); + } + + // Remove backup file if not keeping + if (!options.keepBackup && existsSync(backupPath)) { + try { + unlinkSync(backupPath); + } catch {} + } + } + + report.timings.totalMs = Date.now() - startTime; + report.completedAt = new Date().toISOString(); + + return report; +} + +// CLI entry point +const isDirectExecution = + process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url); + +if (isDirectExecution) { + const args = parseArgs(process.argv.slice(2)); + + if (args.help || !args.sourceUrl) { + console.log(` +Gambit PostgreSQL Backup & Restore Drill Runner + +Usage: + node scripts/db-backup-restore-drill.mjs [options] + +Options: + --source-url Source database URL (defaults to DATABASE_URL) + --target-url Explicit target database URL (must be isolated) + --target-db-name Target database name (default: auto-generated isolated name) + --backup-file Path for backup dump file (default: temporary file) + --keep-backup Preserve the backup dump file after the drill + --keep-target Preserve the restored target database after the drill + --format pg_dump format (default: custom) + --use-docker Force execution of pg tools via Docker container + --docker-image Docker image for pg tools (default: pgvector/pgvector:pg16) + --allow-custom-target-name Permit target name without default isolation markers + --json Output drill report in JSON format + --help Show this help message + `); + process.exit(args.help ? 0 : 1); + } + + runBackupRestoreDrill(args) + .then((report) => { + if (args.json) { + console.log(JSON.stringify(report, null, 2)); + } else { + console.log('\n========================================'); + console.log('✅ BACKUP & RESTORE DRILL SUCCESSFUL'); + console.log('========================================'); + console.log(`Source: ${report.source}`); + console.log(`Target: ${report.target}`); + console.log(`Backup Size: ${(report.backupSizeBytes / 1024).toFixed(2)} KB`); + console.log(`Backup SHA-256: ${report.backupSha256}`); + console.log(`Backup Duration: ${report.timings.backupMs}ms`); + console.log(`Restore Duration:${report.timings.restoreMs}ms`); + console.log(`Verify Duration: ${report.timings.verifyMs}ms`); + console.log(`Total Duration: ${report.timings.totalMs}ms`); + console.log(`Checks Passed: ${report.checks.length}/${report.checks.length}`); + console.log('========================================\n'); + } + process.exit(0); + }) + .catch((err) => { + console.error('\n========================================'); + console.error('❌ BACKUP & RESTORE DRILL FAILED'); + console.error('========================================'); + console.error(`Error: ${err.message}`); + if (err.stack) console.error(err.stack); + console.error('========================================\n'); + process.exit(1); + }); +} diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs new file mode 100644 index 00000000..0f4cd918 --- /dev/null +++ b/scripts/test/backup-restore-drill.test.mjs @@ -0,0 +1,540 @@ +/** + * Automated tests for the Postgres backup/restore drill and verification engine. + * + * Verifies: + * 1. Security: credential masking in logs/diagnostics, no leaking passwords. + * 2. Isolation: fails safe if target matches source or is not an isolated drill target. + * 3. Backup validation: detects missing, empty, or corrupt backup files. + * 4. Verification checks: detects missing extensions, missing migration ledgers, + * checksum drift, missing tables, row count discrepancies, and missing/inactive + * append-only triggers on game_events. + * 5. CLI argument parsing: flags, defaults, and environment fallbacks. + */ + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { writeFileSync, unlinkSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + +import { + sanitizeDatabaseUrl, + validateTargetIsolation, + validateBackupFile, + verifyRestoredDatabase, + parseArgs, + generateIsolatedDbName, + runBackupRestoreDrill, + CRITICAL_APPLICATION_TABLES, + REQUIRED_EXTENSIONS, +} from '../db-backup-restore-drill.mjs'; + +test('security: sanitizeDatabaseUrl masks plaintext passwords in postgres URLs', () => { + const plainUrl = 'postgres://gambit:my_super_secret_pw@db.production.internal:5432/gambit'; + const sanitized = sanitizeDatabaseUrl(plainUrl); + assert.equal(sanitized, 'postgres://gambit:***@db.production.internal:5432/gambit'); + assert.equal(sanitized.includes('my_super_secret_pw'), false); +}); + +test('security: sanitizeDatabaseUrl handles URLs with URL-encoded special characters', () => { + const complexUrl = 'postgresql://admin%40corp:p%40ss%3Aword%21@127.0.0.1:5432/gambit_prod?sslmode=require'; + const sanitized = sanitizeDatabaseUrl(complexUrl); + assert.equal(sanitized.includes('p%40ss%3Aword%21'), false); + assert.equal(sanitized, 'postgresql://admin%40corp:***@127.0.0.1:5432/gambit_prod?sslmode=require'); +}); + +test('security: sanitizeDatabaseUrl safely handles URLs without passwords', () => { + const noPwUrl = 'postgres://gambit@localhost:5432/gambit_dev'; + assert.equal(sanitizeDatabaseUrl(noPwUrl), 'postgres://gambit@localhost:5432/gambit_dev'); +}); + +test('security: sanitizeDatabaseUrl handles non-URL strings and errors safely', () => { + assert.equal(sanitizeDatabaseUrl('not a url'), 'not a url'); + assert.equal(sanitizeDatabaseUrl(''), ''); +}); + +test('isolation: validateTargetIsolation rejects when target URL matches source URL', () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const target = 'postgres://gambit:pass@localhost:5432/gambit'; + + assert.throws( + () => validateTargetIsolation(source, target), + /Target database URL must not be identical to source database URL/, + ); +}); + +test('isolation: validateTargetIsolation rejects when source and target have identical DB name on same host/port', () => { + const source = 'postgres://gambit:pass1@localhost:5432/gambit'; + const target = 'postgres://gambit_admin:pass2@localhost:5432/gambit'; + + assert.throws( + () => validateTargetIsolation(source, target), + /Target database name "gambit" matches source database name on the same host/, + ); +}); + +test('isolation: validateTargetIsolation rejects dangerous production or system databases as target', () => { + const source = 'postgres://gambit:pass@localhost:5432/source_test'; + for (const dangerous of ['gambit', 'postgres', 'template1', 'template0', 'production']) { + const target = `postgres://gambit:pass@localhost:5432/${dangerous}`; + assert.throws( + () => validateTargetIsolation(source, target), + /Target database name .* is a protected or non-isolated database/, + ); + } +}); + +test('isolation: validateTargetIsolation rejects target names without isolated naming markers unless explicitly allowed', () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const target = 'postgres://gambit:pass@localhost:5432/arbitrary_name'; + + assert.throws( + () => validateTargetIsolation(source, target, { allowCustomTargetName: false }), + /Target database name "arbitrary_name" does not contain an isolated drill marker/, + ); + + // When explicitly permitted via allowCustomTargetName flag + const result = validateTargetIsolation(source, target, { allowCustomTargetName: true }); + assert.equal(result.isolated, true); + assert.equal(result.targetDbName, 'arbitrary_name'); +}); + +test('isolation: validateTargetIsolation accepts valid isolated database names', () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const validTargets = [ + 'gambit_backup_drill_restore', + 'gambit_backup_drill_restore_1720000000_abcd', + 'test_db_restore', + 'gambit_disposable_drill', + 'gambit_isolated_restore', + ]; + + for (const name of validTargets) { + const target = `postgres://gambit:pass@localhost:5432/${name}`; + const validated = validateTargetIsolation(source, target); + assert.equal(validated.isolated, true); + assert.equal(validated.targetDbName, name); + } +}); + +test('isolation: generateIsolatedDbName generates prefixed unique name', () => { + const name1 = generateIsolatedDbName('gambit'); + const name2 = generateIsolatedDbName('gambit'); + assert.match(name1, /^gambit_backup_drill_restore_\d+_[a-f0-9]+$/); + assert.notEqual(name1, name2); +}); + +test('backup validation: validateBackupFile rejects non-existent file', () => { + const nonExistent = join(tmpdir(), 'non-existent-backup-file-12345.dump'); + assert.throws( + () => validateBackupFile(nonExistent, 'custom'), + /Backup file does not exist/, + ); +}); + +test('backup validation: validateBackupFile rejects 0-byte empty file', () => { + const emptyFile = join(tmpdir(), `test-empty-backup-${Date.now()}.dump`); + writeFileSync(emptyFile, Buffer.alloc(0)); + try { + assert.throws( + () => validateBackupFile(emptyFile, 'custom'), + /Backup file is empty \(0 bytes\)/, + ); + } finally { + unlinkSync(emptyFile); + } +}); + +test('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', () => { + const corruptFile = join(tmpdir(), `test-corrupt-backup-${Date.now()}.dump`); + writeFileSync(corruptFile, Buffer.from('NOT_A_VALID_PG_DUMP_HEADER')); + try { + assert.throws( + () => validateBackupFile(corruptFile, 'custom'), + /Invalid custom-format backup: missing PostgreSQL dump magic header "PGDMP"/, + ); + } finally { + unlinkSync(corruptFile); + } +}); + +test('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', () => { + const validFile = join(tmpdir(), `test-valid-backup-${Date.now()}.dump`); + const header = Buffer.from('PGDMP\x01\x10\x00\x01'); + writeFileSync(validFile, header); + try { + const meta = validateBackupFile(validFile, 'custom'); + assert.equal(meta.valid, true); + assert.equal(meta.sizeBytes, header.length); + assert.equal(typeof meta.sha256, 'string'); + assert.equal(meta.sha256.length, 64); + } finally { + unlinkSync(validFile); + } +}); + +test('verification engine: detects missing extensions in restored database', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['users', 'games'], + rowCounts: { users: 2, games: 1 }, + sampleData: {}, + }; + + // Mock target database pool missing the 'vector' extension + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext', extversion: '1.6' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Missing required extension in restored database: vector/); + return true; + }, + ); +}); + +test('verification engine: detects missing schema_migrations table', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users'], + rowCounts: { schema_migrations: 1, users: 1 }, + sampleData: {}, + }; + + // Mock target where schema_migrations table does not exist + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: false }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Table schema_migrations does not exist in restored database/); + return true; + }, + ); +}); + +test('verification engine: detects missing schema_migrations table or count mismatch', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [ + { version: 1, name: '0001_init.sql', checksum: 'abc1', state: 'applied' }, + { version: 2, name: '0002_seek_color.sql', checksum: 'abc2', state: 'applied' }, + ], + tables: ['schema_migrations', 'users'], + rowCounts: { schema_migrations: 2, users: 1 }, + sampleData: {}, + }; + + // Mock target where only migration 1 is present + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc1', state: 'applied' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Migration count mismatch: source had 2, restored has 1/); + return true; + }, + ); +}); + +test('verification engine: detects corrupted or modified migration checksum in restored database', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [ + { version: 1, name: '0001_init.sql', checksum: 'original_checksum_123', state: 'applied' }, + ], + tables: ['schema_migrations'], + rowCounts: { schema_migrations: 1 }, + sampleData: {}, + }; + + // Mock target where migration 1 has altered checksum + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'corrupted_checksum_999', state: 'applied' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Migration 1 checksum mismatch/); + return true; + }, + ); +}); + +test('verification engine: detects missing tables in restored database', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'games', 'tournaments'], + rowCounts: { schema_migrations: 1, users: 2, games: 1, tournaments: 1 }, + sampleData: {}, + }; + + // Mock target where 'tournaments' table is missing + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { rows: [{ tablename: 'schema_migrations' }, { tablename: 'users' }, { tablename: 'games' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Missing required table in restored database: tournaments/); + return true; + }, + ); +}); + +test('verification engine: detects row count mismatch in critical tables', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'game_events'], + rowCounts: { schema_migrations: 1, users: 5, game_events: 100 }, + sampleData: {}, + }; + + // Mock target where game_events only has 90 rows restored (incomplete restore!) + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { rows: [{ tablename: 'schema_migrations' }, { tablename: 'users' }, { tablename: 'game_events' }] }; + } + if (text.includes('COUNT(*) FROM "users"') || text.includes('COUNT(*) AS count FROM "users"')) { + return { rows: [{ count: '5' }] }; + } + if (text.includes('COUNT(*) FROM "game_events"') || text.includes('COUNT(*) AS count FROM "game_events"')) { + return { rows: [{ count: '90' }] }; // 90 != 100 + } + if (text.includes('COUNT(*) FROM "schema_migrations"') || text.includes('COUNT(*) AS count FROM "schema_migrations"')) { + return { rows: [{ count: '1' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Row count mismatch for table "game_events": source had 100, restored has 90/); + return true; + }, + ); +}); + +test('verification engine: detects missing or inactive append-only trigger on game_events', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'game_events'], + rowCounts: { schema_migrations: 1, users: 1, game_events: 1 }, + sampleData: {}, + }; + + // Mock target where table exists and row count matches, but trigger probe does NOT throw + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { rows: [{ tablename: 'schema_migrations' }, { tablename: 'users' }, { tablename: 'game_events' }] }; + } + if (text.includes('COUNT(*)')) { + return { rows: [{ count: '1' }] }; + } + if (text.includes('UPDATE game_events')) { + // Trigger did NOT fire! Update silently returned 1 row updated! + return { rowCount: 1 }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Append-only trigger on game_events failed: mutation was not blocked/); + return true; + }, + ); +}); + +test('verification engine: passes when all structural and functional checks succeed', async () => { + const sourceBaseline = { + extensions: ['citext', 'vector'], + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users', 'game_events', 'variants'], + rowCounts: { schema_migrations: 1, users: 1, game_events: 1, variants: 8 }, + sampleData: { + users: [{ id: 'u1', handle: 'alice' }], + game_events: [{ game_id: 'g1', seq: 0, type: 'MovePlayed' }], + }, + }; + + const mockTargetPool = { + async query(text, params) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + } + if (text.includes("to_regclass('schema_migrations')")) { + return { rows: [{ present: true }] }; + } + if (text.includes('SELECT version, name, checksum, state FROM schema_migrations')) { + return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; + } + if (text.includes('information_schema.tables') || text.includes('pg_tables')) { + return { + rows: [ + { tablename: 'schema_migrations' }, + { tablename: 'users' }, + { tablename: 'game_events' }, + { tablename: 'variants' }, + ], + }; + } + if (text.includes('COUNT(*)')) { + if (text.includes('variants')) return { rows: [{ count: '8' }] }; + return { rows: [{ count: '1' }] }; + } + if (text.includes('SELECT id, handle FROM users')) { + return { rows: [{ id: 'u1', handle: 'alice' }] }; + } + if (text.includes('SELECT game_id, seq, type FROM game_events')) { + return { rows: [{ game_id: 'g1', seq: 0, type: 'MovePlayed' }] }; + } + if (text.includes('UPDATE game_events')) { + // Trigger correctly blocks update + const error = new Error('game_events is append-only (UPDATE.game_events attempted)'); + error.code = 'P0001'; + throw error; + } + if (text.includes('search_embeddings')) { + return { rows: [] }; + } + return { rows: [] }; + }, + }; + + const report = await verifyRestoredDatabase(sourceBaseline, mockTargetPool); + assert.equal(report.passed, true); + assert.ok(report.checks.length >= 5); + assert.equal(report.failures.length, 0); +}); + +test('cli: parseArgs parses options, flags, and environment fallbacks', () => { + const customArgs = [ + '--source-url', 'postgres://u:p@localhost:5432/src', + '--target-url', 'postgres://u:p@localhost:5432/gambit_backup_drill_restore', + '--backup-file', '/tmp/my-backup.dump', + '--keep-backup', + '--keep-target', + '--format', 'custom', + '--json', + ]; + + const parsed = parseArgs(customArgs); + assert.equal(parsed.sourceUrl, 'postgres://u:p@localhost:5432/src'); + assert.equal(parsed.targetUrl, 'postgres://u:p@localhost:5432/gambit_backup_drill_restore'); + assert.equal(parsed.backupFile, '/tmp/my-backup.dump'); + assert.equal(parsed.keepBackup, true); + assert.equal(parsed.keepTarget, true); + assert.equal(parsed.format, 'custom'); + assert.equal(parsed.json, true); +}); + +test('cli: parseArgs defaults to safe isolated target when not specified', () => { + const parsed = parseArgs(['--source-url', 'postgres://u:p@localhost:5432/gambit']); + assert.equal(parsed.sourceUrl, 'postgres://u:p@localhost:5432/gambit'); + assert.match(parsed.targetUrl, /^postgres:\/\/u:p@localhost:5432\/gambit_backup_drill_restore_\d+_[a-f0-9]+$/); + assert.equal(parsed.keepBackup, false); + assert.equal(parsed.keepTarget, false); + assert.equal(parsed.format, 'custom'); +}); + +test( + 'integration: full backup, isolated restore, and verification drill against live Postgres', + { skip: process.env.DATABASE_URL ? false : 'DATABASE_URL not set' }, + async () => { + const sourceUrl = process.env.DATABASE_URL; + const report = await runBackupRestoreDrill({ + sourceUrl, + keepTarget: false, + keepBackup: false, + }); + assert.equal(report.success, true); + assert.ok(report.checks.length > 0); + }, +); From eaeada18ec43630936c629d8944232f882599a3a Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Sun, 6 Sep 2026 13:53:42 +0300 Subject: [PATCH 02/14] fix(db): derive isolated targetUrl when omitted and handle pg_restore status 1 --- scripts/db-backup-restore-drill.mjs | 32 ++++++++++++++++------ scripts/test/backup-restore-drill.test.mjs | 20 ++++++++++---- 2 files changed, 37 insertions(+), 15 deletions(-) diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index 0224105f..b42f41b0 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -659,13 +659,17 @@ export async function runBackupRestoreDrill(options = {}) { }; const parsedSource = parseDatabaseUrl(options.sourceUrl); - const parsedTarget = parseDatabaseUrl(options.targetUrl); + const targetUrl = options.targetUrl || urlWithDatabase( + options.sourceUrl, + `${parsedSource.database}_backup_drill_restore_${Date.now()}_${Math.floor(Math.random() * 0xffff).toString(16)}`, + ); + const parsedTarget = parseDatabaseUrl(targetUrl); report.source = sanitizeDatabaseUrl(options.sourceUrl); - report.target = sanitizeDatabaseUrl(options.targetUrl); + report.target = sanitizeDatabaseUrl(targetUrl); // Validate isolation - validateTargetIsolation(options.sourceUrl, options.targetUrl, { + validateTargetIsolation(options.sourceUrl, targetUrl, { allowCustomTargetName: options.allowCustomTargetName, }); @@ -740,13 +744,13 @@ export async function runBackupRestoreDrill(options = {}) { // 5. Create isolated target database log(`Provisioning isolated target database "${parsedTarget.database}"...`); - const adminUrl = urlWithDatabase(options.sourceUrl, 'postgres'); + const adminUrl = urlWithDatabase(targetUrl, 'postgres'); adminClient = new Client({ connectionString: adminUrl }); try { await adminClient.connect(); } catch { // Try template1 if postgres db is not accessible - adminClient = new Client({ connectionString: urlWithDatabase(options.sourceUrl, 'template1') }); + adminClient = new Client({ connectionString: urlWithDatabase(targetUrl, 'template1') }); await adminClient.connect(); } @@ -770,8 +774,13 @@ export async function runBackupRestoreDrill(options = {}) { try { tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password }); } catch (err) { - // pg_restore may exit with non-critical warnings on drops of non-existent objects - if (!err.message?.includes('exit code 1')) throw err; + // pg_restore may exit with status 1 for non-critical warnings on drops of non-existent objects + const status = err?.status ?? err?.code; + if (status === 1 || err?.message?.includes('exit code 1')) { + if (err?.stderr) log(`pg_restore notice: ${err.stderr.toString().trim()}`); + } else { + throw err; + } } } else { const psqlArgs = [ @@ -803,7 +812,12 @@ export async function runBackupRestoreDrill(options = {}) { try { tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password }, backupDir); } catch (err) { - if (!err.message?.includes('exit code 1')) throw err; + const status = err?.status ?? err?.code; + if (status === 1 || err?.message?.includes('exit code 1')) { + if (err?.stderr) log(`pg_restore notice: ${err.stderr.toString().trim()}`); + } else { + throw err; + } } } else { const psqlArgs = [ @@ -822,7 +836,7 @@ export async function runBackupRestoreDrill(options = {}) { // 7. Verify restored database log('Running comprehensive structural and functional verification...'); const verifyStart = Date.now(); - targetPool = new Pool({ connectionString: options.targetUrl, max: 2 }); + targetPool = new Pool({ connectionString: targetUrl, max: 2 }); const verifyResult = await verifyRestoredDatabase(sourceBaseline, targetPool, options); report.timings.verifyMs = Date.now() - verifyStart; report.checks = verifyResult.checks; diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index 0f4cd918..27f2a466 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -516,12 +516,20 @@ test('cli: parseArgs parses options, flags, and environment fallbacks', () => { }); test('cli: parseArgs defaults to safe isolated target when not specified', () => { - const parsed = parseArgs(['--source-url', 'postgres://u:p@localhost:5432/gambit']); - assert.equal(parsed.sourceUrl, 'postgres://u:p@localhost:5432/gambit'); - assert.match(parsed.targetUrl, /^postgres:\/\/u:p@localhost:5432\/gambit_backup_drill_restore_\d+_[a-f0-9]+$/); - assert.equal(parsed.keepBackup, false); - assert.equal(parsed.keepTarget, false); - assert.equal(parsed.format, 'custom'); + const originalEnv = process.env.BACKUP_DRILL_TARGET_URL; + delete process.env.BACKUP_DRILL_TARGET_URL; + try { + const parsed = parseArgs(['--source-url', 'postgres://u:p@localhost:5432/gambit']); + assert.equal(parsed.sourceUrl, 'postgres://u:p@localhost:5432/gambit'); + assert.match(parsed.targetUrl, /^postgres:\/\/u:p@localhost:5432\/gambit_backup_drill_restore_\d+_[a-f0-9]+$/); + assert.equal(parsed.keepBackup, false); + assert.equal(parsed.keepTarget, false); + assert.equal(parsed.format, 'custom'); + } finally { + if (originalEnv !== undefined) { + process.env.BACKUP_DRILL_TARGET_URL = originalEnv; + } + } }); test( From 5de23dfc46a26f1352bcddf75b87473e328d3bb5 Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Sun, 6 Sep 2026 14:00:24 +0300 Subject: [PATCH 03/14] fix(db): ensure psql availability when plain format is requested --- scripts/db-backup-restore-drill.mjs | 10 ++++++++-- scripts/test/backup-restore-drill.test.mjs | 12 ++++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index b42f41b0..52068729 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -275,7 +275,12 @@ export function resolvePgTooling(options = {}) { } catch {} } - if (hasNativePgDump && hasNativePgRestore) { + const isPlain = (options.format || 'custom') === 'plain'; + const hasRequiredNativeTools = isPlain + ? (hasNativePgDump && hasNativePsql) + : (hasNativePgDump && hasNativePgRestore); + + if (hasRequiredNativeTools) { return { type: 'native', runDump: (args, env) => execFileSync('pg_dump', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), @@ -292,8 +297,9 @@ export function resolvePgTooling(options = {}) { } catch {} if (!hasDocker) { + const requiredTools = isPlain ? 'pg_dump, psql' : 'pg_dump, pg_restore'; throw new Error( - 'Neither native PostgreSQL tools (pg_dump, pg_restore) nor Docker are available in PATH. Please install postgresql-client or ensure Docker is running.', + `Neither native PostgreSQL tools (${requiredTools}) nor Docker are available in PATH. Please install postgresql-client or ensure Docker is running.`, ); } diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index 27f2a466..b04ba6a4 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -25,6 +25,7 @@ import { parseArgs, generateIsolatedDbName, runBackupRestoreDrill, + resolvePgTooling, CRITICAL_APPLICATION_TABLES, REQUIRED_EXTENSIONS, } from '../db-backup-restore-drill.mjs'; @@ -532,6 +533,17 @@ test('cli: parseArgs defaults to safe isolated target when not specified', () => } }); +test('tooling: resolvePgTooling accepts options and detects native or docker runner', () => { + const customTooling = resolvePgTooling({ format: 'custom' }); + assert.ok(customTooling.type === 'native' || customTooling.type === 'docker'); + assert.equal(typeof customTooling.runDump, 'function'); + assert.equal(typeof customTooling.runRestore, 'function'); + assert.equal(typeof customTooling.runPsql, 'function'); + + const plainTooling = resolvePgTooling({ format: 'plain' }); + assert.ok(plainTooling.type === 'native' || plainTooling.type === 'docker'); +}); + test( 'integration: full backup, isolated restore, and verification drill against live Postgres', { skip: process.env.DATABASE_URL ? false : 'DATABASE_URL not set' }, From c1d9ec57169d756142f629a5f2c3e180721bfa9d Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Sun, 6 Sep 2026 14:52:09 +0300 Subject: [PATCH 04/14] Remediate PR #51 defects (backup-restore drill) --- apply.py | 80 ++++++ docs/runbooks/backup-restore-drill.md | 33 ++- fix-tests.js | 63 +++++ fix-tests2.js | 45 ++++ fix-tests3.js | 6 + fix-tests4.js | 9 + patch.js | 28 +++ scripts/db-backup-restore-drill.mjs | 274 +++++++++++++-------- scripts/test/backup-restore-drill.test.mjs | 93 ++++--- 9 files changed, 474 insertions(+), 157 deletions(-) create mode 100644 apply.py create mode 100644 fix-tests.js create mode 100644 fix-tests2.js create mode 100644 fix-tests3.js create mode 100644 fix-tests4.js create mode 100644 patch.js diff --git a/apply.py b/apply.py new file mode 100644 index 00000000..ef9275d2 --- /dev/null +++ b/apply.py @@ -0,0 +1,80 @@ +import re +import sys +import os + +print("Applying patches...") + +mjs_path = 'scripts/db-backup-restore-drill.mjs' +with open(mjs_path, 'r', encoding='utf-8') as f: + content = f.read() + +# 1. DATABASE OWNERSHIP & DESTRUCTIVE CLEANUP +# Only run destructive cleanup if targetCreatedByThisRun = true +content = content.replace( + 'await adminClient.query(`CREATE DATABASE "${parsedTarget.database}"`);', + 'await adminClient.query(`CREATE DATABASE """${parsedTarget.database.replace(/"/g, \'""\')}"""`);\n targetCreatedByThisRun = true;' +) +content = content.replace( + 'let adminClient = null;', + 'let adminClient = null;\n let targetCreatedByThisRun = false;' +) + +# 2. TARGET HOST +# Admin queries must connect to the same target host/server as restore +# Right now it uses parsedTarget but connects to targetUrl, 'postgres'. +# Already implemented as: const adminUrl = urlWithDatabase(targetUrl, 'postgres'); which preserves host. +# But wait, step 5 says admin client connects. Yes, adminUrl preserves host. + +# 3. IDENTIFIER INJECTION +# Validate DB name format +content = content.replace( + 'export async function validateTargetIsolation(sourceUrl, targetUrl, options = {}) {', + '''export async function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { + if (!/^[a-zA-Z0-9_-]+$/.test(parseDatabaseUrl(targetUrl).database)) { + throw new Error('Invalid target database name format'); + }''' +) + +# Quote identifiers +# (Already partly done in step 1, what about drop?) +content = re.sub( + r'DROP DATABASE "([^"]+)"', + r'DROP DATABASE """\1"""', + content +) + +# 4. HOST ALIAS / SOURCE ISOLATION +# done by node command earlier + +# 5. CREDENTIAL EXPOSURE +# Pass '-e', 'PGPASSWORD' in docker, not '-e PGPASSWORD=...' +content = content.replace( + "if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`);", + "if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD');" +) + +# 6. CONNECTION SECURITY OPTIONS +# urlWithDatabase should preserve searchParams (SSL) +content = content.replace( + 'parsed.pathname = `/${encodeURIComponent(databaseName)}`;\n return parsed.toString();', + 'parsed.pathname = `/${encodeURIComponent(databaseName)}`;\n return parsed.toString();' +) +# wait, parseDatabaseUrl doesn't preserve search params if we rebuild, but urlWithDatabase uses URL object so it preserves them automatically. + +# 7. BASELINE SNAPSHOT CONSISTENCY +# capture baseline in transaction or just REPEATABLE READ? +content = content.replace( + 'export async function collectSourceBaseline(pool) {', + "export async function collectSourceBaseline(pool) {\n const client = await pool.connect();\n await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY');\n try {" +) +content = content.replace( + ' return {', + " } finally {\n await client.query('ROLLBACK');\n client.release();\n }\n return {" +) +# Replace pool.query with client.query in collectSourceBaseline +# Actually that would need a full regex. I will just do a simpler search/replace. + +with open(mjs_path, 'w', encoding='utf-8') as f: + f.write(content) + +print("Done") diff --git a/docs/runbooks/backup-restore-drill.md b/docs/runbooks/backup-restore-drill.md index eac7c787..f5ea86c3 100644 --- a/docs/runbooks/backup-restore-drill.md +++ b/docs/runbooks/backup-restore-drill.md @@ -50,15 +50,18 @@ The script will automatically: | Option | Default | Description | |---|---|---| -| `--source-url ` | `$DATABASE_URL` | Source database connection URL | -| `--target-url ` | Auto-generated | Target database connection URL (must be isolated) | -| `--backup-file ` | Temp file | Path to write backup dump file | -| `--keep-backup` | `false` | Preserve the backup dump file after the drill completes | -| `--keep-target` | `false` | Preserve the restored target DB for forensic inspection | -| `--format ` | `custom` | Dump format (`custom` = `-Fc`, `plain` = SQL) | -| `--use-docker` | `auto` | Force execution inside `pgvector/pgvector:pg16` container | -| `--allow-custom-target-name`| `false` | Allow target DB name without default isolation markers | -| `--json` | `false` | Output machine-readable JSON report | +| `--source-url ` | `$DATABASE_URL` | Source database URL (defaults to DATABASE_URL) | +| `--target-url ` | Auto-generated | Explicit target database URL (must be isolated) | +| `--target-db-name ` | Auto-generated | Target database name (default: auto-generated isolated name) | +| `--backup-file ` | Temp file | Path for backup dump file (default: temporary file) | +| `--keep-backup` | `false` | Preserve the backup dump file after the drill | +| `--keep-target` | `false` | Preserve the restored target database after the drill | +| `--format ` | `custom` | pg_dump format (default: custom) | +| `--use-docker` | `auto` | Force execution of pg tools via Docker container | +| `--docker-image ` | `pgvector/pgvector:pg16` | Docker image for pg tools (default: pgvector/pgvector:pg16) | +| `--allow-custom-target-name`| `false` | Permit target name without default isolation markers | +| `--json` | `false` | Output drill report in JSON format | +| `--help` | `false` | Show help message | ### 2.3 Retaining the Target Database for Forensic Inspection @@ -72,7 +75,7 @@ node scripts/db-backup-restore-drill.mjs \ ``` Output will report the exact target database name created: -``` +```text Target: postgres://gambit:***@localhost:5432/gambit_backup_drill_restore_1788672281592_96c80ce5 Backup: /tmp/gambit_backup_1788672281593_2706.dump ``` @@ -131,6 +134,8 @@ psql -h "${PGHOST:-localhost}" -p "${PGPORT:-5432}" -U "${PGUSER:-gambit}" -d po Restore the custom archive into the target database: ```bash +BACKUP_FILE=$(ls -t /tmp/gambit_manual_drill_*.dump | head -n 1) + pg_restore \ -h "${PGHOST:-localhost}" \ -p "${PGPORT:-5432}" \ @@ -138,7 +143,7 @@ pg_restore \ -d "${TARGET_DB}" \ --clean \ --if-exists \ - "/tmp/gambit_manual_drill_*.dump" + "${BACKUP_FILE}" ``` ### Step 5: Verification Checklist @@ -188,8 +193,10 @@ SELECT 'search_embeddings', count(*) FROM search_embeddings; #### 5. Vector Query Functionality Verify pgvector cosine distance operations and HNSW indexing: ```sql --- Test distance operator (<->) -SELECT id, embedding <-> embedding AS distance FROM search_embeddings LIMIT 1; +-- Test cosine distance operator (<=>) +SELECT id, embedding <=> embedding AS distance FROM search_embeddings LIMIT 1; +-- Verify HNSW index +SELECT i.relname, am.amname FROM pg_index ix JOIN pg_class i ON i.oid = ix.indexrelid JOIN pg_class t ON t.oid = ix.indrelid JOIN pg_am am ON i.relam = am.oid WHERE t.relname = 'search_embeddings' AND am.amname = 'hnsw'; ``` ### Step 6: Cleanup Isolated Target diff --git a/fix-tests.js b/fix-tests.js new file mode 100644 index 00000000..16dc9a93 --- /dev/null +++ b/fix-tests.js @@ -0,0 +1,63 @@ +const fs = require('fs'); +let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); + +// validateBackupFile is async now +content = content.replace( + /test\('backup validation: validateBackupFile rejects non-existent file', \(\) => {/g, + "test('backup validation: validateBackupFile rejects non-existent file', async () => {" +); +content = content.replace( + /assert\.throws\(\n\s*\(\) => validateBackupFile\(nonExistent, 'custom'\),/g, + "await assert.rejects(\n async () => await validateBackupFile(nonExistent, 'custom')," +); + +content = content.replace( + /test\('backup validation: validateBackupFile rejects 0-byte empty file', \(\) => {/g, + "test('backup validation: validateBackupFile rejects 0-byte empty file', async () => {" +); +content = content.replace( + /assert\.throws\(\n\s*\(\) => validateBackupFile\(emptyFile, 'custom'\),/g, + "await assert.rejects(\n async () => await validateBackupFile(emptyFile, 'custom')," +); + +content = content.replace( + /test\('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', \(\) => {/g, + "test('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', async () => {" +); +content = content.replace( + /assert\.throws\(\n\s*\(\) => validateBackupFile\(corruptFile, 'custom'\),/g, + "await assert.rejects(\n async () => await validateBackupFile(corruptFile, 'custom')," +); + +content = content.replace( + /test\('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', \(\) => {/g, + "test('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', async () => {" +); +content = content.replace( + /const meta = validateBackupFile\(validFile, 'custom'\);/g, + "const meta = await validateBackupFile(validFile, 'custom');" +); + +// Checksum regex fix +content = content.replace( + /\/Migration 1 checksum mismatch\//g, + "/Migration 1 mismatch/" +); + +// Missing tables due to CRITICAL_APPLICATION_TABLES checking all of them +content = content.replace( + /\{ tablename: 'users' \}, \{ tablename: 'game_events' \}/g, + "{ tablename: 'users' }, { tablename: 'game_events' }, ...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))" +); + +content = content.replace( + /\{ tablename: 'schema_migrations' \}, \{ tablename: 'users' \}, \{ tablename: 'game_events' \}/g, + "...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))" +); + +content = content.replace( + /rows: \[\n\s*\{ tablename: 'schema_migrations' \},\n\s*\{ tablename: 'users' \},\n\s*\{ tablename: 'game_events' \},\n\s*\{ tablename: 'variants' \},\n\s*\]/g, + "rows: CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))" +); + +fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/fix-tests2.js b/fix-tests2.js new file mode 100644 index 00000000..cc923737 --- /dev/null +++ b/fix-tests2.js @@ -0,0 +1,45 @@ +const fs = require('fs'); +let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); + +content = content.replace( + /validateTargetIsolation\(source, target\)/g, + 'await validateTargetIsolation(source, target)' +); +content = content.replace( + /validateTargetIsolation\(source, target, /g, + 'await validateTargetIsolation(source, target, ' +); + +content = content.replace( + /const validated = /g, + 'const validated = await ' +); + +content = content.replace( + /test\('isolation:.*\(\) => \{/g, + (match) => match.replace('() => {', 'async () => {') +); + +// fix assert.throws for validateTargetIsolation +content = content.replace( + /assert\.throws\(\n\s*\(\) => await validateTargetIsolation/g, + "await assert.rejects(\n async () => await validateTargetIsolation" +); +content = content.replace( + /assert\.throws\(\n\s*\(\) => validateTargetIsolation/g, + "await assert.rejects(\n async () => await validateTargetIsolation" +); + +// fix missing awaits for validateTargetIsolation inside blocks +content = content.replace( + /await await validateTargetIsolation/g, + "await validateTargetIsolation" +); + +// Mock pool connect +content = content.replace( + /async query\(text, params\) \{/g, + "async connect() { return { query: this.query, release: () => {} }; },\n async query(text, params) {" +); + +fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/fix-tests3.js b/fix-tests3.js new file mode 100644 index 00000000..8f1cf206 --- /dev/null +++ b/fix-tests3.js @@ -0,0 +1,6 @@ +const fs = require('fs'); +let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); + +content = content.replace(/async async \(\) =>/g, 'async () =>'); + +fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/fix-tests4.js b/fix-tests4.js new file mode 100644 index 00000000..30720fed --- /dev/null +++ b/fix-tests4.js @@ -0,0 +1,9 @@ +const fs = require('fs'); +let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); + +content = content.replace( + /if \(text\.includes\('search_embeddings'\)\) \{\n\s*return \{ rows: \[\] \};\n\s*\}/g, + "if (text.includes('search_embeddings')) {\n return { rows: [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw' }] };\n }" +); + +fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/patch.js b/patch.js new file mode 100644 index 00000000..1bcb0009 --- /dev/null +++ b/patch.js @@ -0,0 +1,28 @@ +const fs = require('fs'); + +function patchFile(file, edits) { + let content = fs.readFileSync(file, 'utf8'); + for (const [search, replace] of edits) { + if (!content.includes(search)) { + console.error(`Could not find search string in ${file}:\n${search.substring(0, 50)}...`); + } else { + content = content.replace(search, replace); + } + } + fs.writeFileSync(file, content); + console.log(`Patched ${file}`); +} + +const editsMjs = [ + // 1. STREAM HASHING for validateBackupFile + [ + `export function validateBackupFile`, + `import { pipeline } from 'node:stream/promises';\nexport async function validateBackupFile` + ], + [ + ` // Compute SHA-256 digest\n const fileBytes = readFileSync(filePath);\n const sha256 = createHash('sha256').update(fileBytes).digest('hex');\n\n return {`, + ` // Compute SHA-256 digest\n const hash = createHash('sha256');\n await pipeline(fs.createReadStream(filePath), hash);\n const sha256 = hash.digest('hex');\n\n return {` + ] +]; + +patchFile('scripts/db-backup-restore-drill.mjs', editsMjs); diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index 52068729..f69d47f3 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -52,10 +52,12 @@ import { execFileSync, spawnSync } from 'node:child_process'; import { createHash } from 'node:crypto'; -import { existsSync, statSync, unlinkSync, readFileSync, openSync, readSync, closeSync } from 'node:fs'; +import { existsSync, statSync, unlinkSync, readFileSync, openSync, readSync, closeSync, createReadStream } from 'node:fs'; import { tmpdir } from 'node:os'; import { join, resolve, dirname, basename } from 'node:path'; import { fileURLToPath } from 'node:url'; +import { lookup } from 'node:dns/promises'; +import { pipeline } from 'node:stream/promises'; import pg from 'pg'; const { Pool, Client } = pg; @@ -126,12 +128,16 @@ export function generateIsolatedDbName(base = 'gambit') { return `${base}_backup_drill_restore_${timestamp}_${random}`; } -/** - * Parse a PostgreSQL connection string into host, port, user, password, and database. - */ export function parseDatabaseUrl(urlString) { const parsed = new URL(urlString); const dbName = parsed.pathname.replace(/^\//, '') || 'postgres'; + const sslParams = {}; + for (const [key, value] of parsed.searchParams.entries()) { + if (key.startsWith('ssl')) { + const envKey = 'PG' + key.toUpperCase(); + sslParams[envKey] = value; + } + } return { host: parsed.hostname || 'localhost', port: parsed.port ? parseInt(parsed.port, 10) : 5432, @@ -139,6 +145,7 @@ export function parseDatabaseUrl(urlString) { password: decodeURIComponent(parsed.password || ''), database: decodeURIComponent(dbName), searchParams: parsed.searchParams, + sslParams, protocol: parsed.protocol, }; } @@ -155,7 +162,7 @@ export function urlWithDatabase(urlString, databaseName) { /** * Enforce target isolation guardrails to guarantee the drill cannot overwrite or drop source/production DBs. */ -export function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { +export async function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { if (!sourceUrl) { throw new Error('Source database URL is required'); } @@ -165,6 +172,10 @@ export function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { const source = parseDatabaseUrl(sourceUrl); const target = parseDatabaseUrl(targetUrl); + + if (!/^[a-zA-Z0-9_-]+$/.test(target.database)) { + throw new Error('Invalid target database name format'); + } // 1. URLs must not be identical if (sourceUrl.trim() === targetUrl.trim()) { @@ -173,9 +184,20 @@ export function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { ); } + const resolveHost = async (host) => { + try { + return (await lookup(host)).address; + } catch { + return host; + } + }; + + const sourceIp = await resolveHost(source.host); + const targetIp = await resolveHost(target.host); + // 2. Target must not target the same database on the same host/port if ( - source.host.toLowerCase() === target.host.toLowerCase() && + sourceIp === targetIp && source.port === target.port && source.database.toLowerCase() === target.database.toLowerCase() ) { @@ -208,7 +230,7 @@ export function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { /** * Validate a backup file's existence, size, header magic, and digest. */ -export function validateBackupFile(filePath, format = 'custom') { +export async function validateBackupFile(filePath, format = 'custom') { if (!existsSync(filePath)) { throw new Error(`Backup file does not exist: ${filePath}`); } @@ -238,8 +260,9 @@ export function validateBackupFile(filePath, format = 'custom') { } // Compute SHA-256 digest - const fileBytes = readFileSync(filePath); - const sha256 = createHash('sha256').update(fileBytes).digest('hex'); + const hash = createHash('sha256'); + await pipeline(createReadStream(filePath), hash); + const sha256 = hash.digest('hex'); return { valid: true, @@ -249,10 +272,8 @@ export function validateBackupFile(filePath, format = 'custom') { }; } -/** - * Inspect installed PostgreSQL tools or fallback to Docker. - */ export function resolvePgTooling(options = {}) { + const execSyncFn = options.execSyncFn || execFileSync; const forceDocker = options.useDocker === true || options.useDocker === 'true'; const dockerImage = options.dockerImage || 'pgvector/pgvector:pg16'; @@ -262,15 +283,15 @@ export function resolvePgTooling(options = {}) { if (!forceDocker) { try { - execFileSync('pg_dump', ['--version'], { stdio: 'ignore' }); + execSyncFn('pg_dump', ['--version'], { stdio: 'ignore' }); hasNativePgDump = true; } catch {} try { - execFileSync('pg_restore', ['--version'], { stdio: 'ignore' }); + execSyncFn('pg_restore', ['--version'], { stdio: 'ignore' }); hasNativePgRestore = true; } catch {} try { - execFileSync('psql', ['--version'], { stdio: 'ignore' }); + execSyncFn('psql', ['--version'], { stdio: 'ignore' }); hasNativePsql = true; } catch {} } @@ -292,7 +313,7 @@ export function resolvePgTooling(options = {}) { // Fallback: Check if Docker is available let hasDocker = false; try { - execFileSync('docker', ['--version'], { stdio: 'ignore' }); + execSyncFn('docker', ['--version'], { stdio: 'ignore' }); hasDocker = true; } catch {} @@ -308,34 +329,43 @@ export function resolvePgTooling(options = {}) { dockerImage, runDump: (args, env, mountDir) => { const dockerArgs = ['run', '--rm']; - if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`); + if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD'); + for (const key of Object.keys(env)) { + if (key.startsWith('PGSSL')) dockerArgs.push('-e', key); + } if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); // Network host so it can reach localhost postgres if (process.platform === 'linux') { dockerArgs.push('--net=host'); } dockerArgs.push(dockerImage, 'pg_dump', ...args); - return execFileSync('docker', dockerArgs, { stdio: 'pipe' }); + return execFileSync('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); }, runRestore: (args, env, mountDir) => { const dockerArgs = ['run', '--rm']; - if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`); + if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD'); + for (const key of Object.keys(env)) { + if (key.startsWith('PGSSL')) dockerArgs.push('-e', key); + } if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); if (process.platform === 'linux') { dockerArgs.push('--net=host'); } dockerArgs.push(dockerImage, 'pg_restore', ...args); - return execFileSync('docker', dockerArgs, { stdio: 'pipe' }); + return execFileSync('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); }, runPsql: (args, env, mountDir) => { const dockerArgs = ['run', '--rm']; - if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`); + if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD'); + for (const key of Object.keys(env)) { + if (key.startsWith('PGSSL')) dockerArgs.push('-e', key); + } if (mountDir) dockerArgs.push('-v', `${mountDir}:/work`); if (process.platform === 'linux') { dockerArgs.push('--net=host'); } dockerArgs.push(dockerImage, 'psql', ...args); - return execFileSync('docker', dockerArgs, { stdio: 'pipe' }); + return execFileSync('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); }, }; } @@ -344,65 +374,72 @@ export function resolvePgTooling(options = {}) { * Capture source database state baseline before running backup. */ export async function collectSourceBaseline(pool) { - // 1. Extensions - const extRes = await pool.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); - const extensions = extRes.rows.map((r) => r.extname); + const client = await pool.connect(); + await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); + try { + // 1. Extensions + const extRes = await client.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); + const extensions = extRes.rows.map((r) => r.extname); - // 2. Schema migrations - const hasMigrationsTable = await pool.query( - "SELECT to_regclass('schema_migrations') IS NOT NULL AS present", - ); - let migrations = []; - if (hasMigrationsTable.rows[0]?.present) { - const migRes = await pool.query( - 'SELECT version, name, checksum, state FROM schema_migrations ORDER BY version', + // 2. Schema migrations + const hasMigrationsTable = await client.query( + "SELECT to_regclass('schema_migrations') IS NOT NULL AS present", ); - migrations = migRes.rows; - } + let migrations = []; + if (hasMigrationsTable.rows[0]?.present) { + const migRes = await client.query( + 'SELECT version, name, checksum, state FROM schema_migrations ORDER BY version', + ); + migrations = migRes.rows; + } - // 3. Existing tables in public schema - const tableRes = await pool.query( - "SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename", - ); - const tables = tableRes.rows.map((r) => r.tablename); + // 3. Existing tables in public schema + const tableRes = await client.query( + "SELECT tablename FROM pg_tables WHERE schemaname = 'public' ORDER BY tablename", + ); + const tables = tableRes.rows.map((r) => r.tablename); - // 4. Row counts across critical tables - const rowCounts = {}; - for (const table of tables) { - try { - const countRes = await pool.query(`SELECT COUNT(*) AS count FROM "${table}"`); - rowCounts[table] = parseInt(countRes.rows[0].count, 10); - } catch { - // Non-readable table or view + // 4. Row counts across critical tables + const rowCounts = {}; + for (const table of tables) { + try { + const countRes = await client.query(`SELECT COUNT(*) AS count FROM "${table}"`); + rowCounts[table] = parseInt(countRes.rows[0].count, 10); + } catch { + // Non-readable table or view + } } - } - // 5. Sample records for integrity comparison - const sampleData = {}; - if (tables.includes('users') && (rowCounts['users'] || 0) > 0) { - const sampleUsers = await pool.query('SELECT id, handle FROM users ORDER BY created_at LIMIT 5'); - sampleData.users = sampleUsers.rows; - } - if (tables.includes('game_events') && (rowCounts['game_events'] || 0) > 0) { - const sampleEvents = await pool.query( - 'SELECT game_id, seq, type FROM game_events ORDER BY server_ts DESC LIMIT 5', - ); - sampleData.game_events = sampleEvents.rows; - } - if (tables.includes('tournaments') && (rowCounts['tournaments'] || 0) > 0) { - const sampleTournaments = await pool.query( - 'SELECT id, name, format FROM tournaments ORDER BY created_at DESC LIMIT 5', - ); - sampleData.tournaments = sampleTournaments.rows; - } + // 5. Sample records for integrity comparison + const sampleData = {}; + if (tables.includes('users') && (rowCounts['users'] || 0) > 0) { + const sampleUsers = await client.query('SELECT id, handle FROM users ORDER BY created_at LIMIT 5'); + sampleData.users = sampleUsers.rows; + } + if (tables.includes('game_events') && (rowCounts['game_events'] || 0) > 0) { + const sampleEvents = await client.query( + 'SELECT game_id, seq, type FROM game_events ORDER BY server_ts DESC LIMIT 5', + ); + sampleData.game_events = sampleEvents.rows; + } + if (tables.includes('tournaments') && (rowCounts['tournaments'] || 0) > 0) { + const sampleTournaments = await client.query( + 'SELECT id, name, format FROM tournaments ORDER BY created_at DESC LIMIT 5', + ); + sampleData.tournaments = sampleTournaments.rows; + } - return { - extensions, - migrations, - tables, - rowCounts, - sampleData, - }; + return { + extensions, + migrations, + tables, + rowCounts, + sampleData, + }; + } finally { + await client.query('ROLLBACK'); + client.release(); + } } /** @@ -424,16 +461,14 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options const targetExts = new Set(extRes.rows.map((r) => r.extname)); for (const requiredExt of REQUIRED_EXTENSIONS) { - if (sourceBaseline.extensions.includes(requiredExt)) { - const present = targetExts.has(requiredExt); - recordCheck( - `Extension: ${requiredExt}`, - present, - present ? 'Installed and active' : `Missing required extension in restored database: ${requiredExt}`, - ); - if (!present) { - throw new Error(`Missing required extension in restored database: ${requiredExt}`); - } + const present = targetExts.has(requiredExt); + recordCheck( + `Extension: ${requiredExt}`, + present, + present ? 'Installed and active' : `Missing required extension in restored database: ${requiredExt}`, + ); + if (!present) { + throw new Error(`Missing required extension in restored database: ${requiredExt}`); } } @@ -469,8 +504,8 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options if (!tgtMig) { throw new Error(`Restored database is missing recorded migration version ${srcMig.version}`); } - if (tgtMig.checksum !== srcMig.checksum) { - const msg = `Migration ${srcMig.version} checksum mismatch: source="${srcMig.checksum}", restored="${tgtMig.checksum}"`; + if (tgtMig.checksum !== srcMig.checksum || tgtMig.name !== srcMig.name || tgtMig.state !== srcMig.state || tgtMig.state !== 'applied') { + const msg = `Migration ${srcMig.version} mismatch: source="${srcMig.checksum}/${srcMig.name}/${srcMig.state}", restored="${tgtMig.checksum}/${tgtMig.name}/${tgtMig.state}" (must be 'applied')`; recordCheck(`Migration Checksum ${srcMig.version}`, false, msg); throw new Error(msg); } @@ -484,7 +519,7 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options ); const targetTables = new Set(tgtTableRes.rows.map((r) => r.tablename)); - for (const expectedTable of sourceBaseline.tables) { + for (const expectedTable of new Set([...sourceBaseline.tables, ...CRITICAL_APPLICATION_TABLES])) { const present = targetTables.has(expectedTable); if (!present) { const msg = `Missing required table in restored database: ${expectedTable}`; @@ -536,15 +571,17 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options // Check 6: Append-only trigger enforcement on game_events if (targetTables.has('game_events')) { let triggerActive = false; + const client = await targetPool.connect(); try { - const res = await targetPool.query( + await client.query('BEGIN'); + const res = await client.query( 'UPDATE game_events SET seq = seq WHERE game_id IN (SELECT game_id FROM game_events LIMIT 1)', ); if (res.rowCount && res.rowCount > 0) { triggerActive = false; } else { // Table was empty; verify trigger registration in pg_trigger catalog - const trigRes = await targetPool.query( + const trigRes = await client.query( "SELECT tgname FROM pg_trigger WHERE tgname = 'game_events_block_mutate' AND tgenabled = 'O'", ); triggerActive = trigRes.rows.length > 0; @@ -558,6 +595,9 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options } else { throw err; } + } finally { + await client.query('ROLLBACK'); + client.release(); } if (!triggerActive) { @@ -573,8 +613,21 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options try { // Test vector operator syntax and index validity const testVec = `[${new Array(256).fill(0.1).join(',')}]`; - await targetPool.query('SELECT $1::vector(256) <-> $1::vector(256) AS dist', [testVec]); - recordCheck('pgvector Functionality', true, 'Vector distance operator (<->) functional'); + await targetPool.query('SELECT $1::vector(256) <=> $1::vector(256) AS dist', [testVec]); + + const idxRes = await targetPool.query(` + SELECT i.relname AS index_name, am.amname AS access_method + FROM pg_index ix + JOIN pg_class i ON i.oid = ix.indexrelid + JOIN pg_class t ON t.oid = ix.indrelid + JOIN pg_am am ON i.relam = am.oid + WHERE t.relname = 'search_embeddings' AND am.amname = 'hnsw' + `); + if (idxRes.rows.length === 0) { + throw new Error('HNSW index missing on search_embeddings table'); + } + + recordCheck('pgvector Functionality', true, 'Vector cosine operator (<=>) and HNSW index functional'); } catch (err) { const msg = `Vector functionality verification failed: ${err.message}`; recordCheck('pgvector Functionality', false, msg); @@ -624,6 +677,9 @@ export function parseArgs(args = process.argv.slice(2)) { result.keepTarget = true; } else if (arg === '--format' && i + 1 < args.length) { result.format = args[++i]; + if (result.format !== 'custom' && result.format !== 'plain') { + throw new Error(`Invalid format "${result.format}". Only "custom" and "plain" are allowed.`); + } } else if (arg === '--use-docker') { result.useDocker = true; } else if (arg === '--no-docker') { @@ -675,13 +731,14 @@ export async function runBackupRestoreDrill(options = {}) { report.target = sanitizeDatabaseUrl(targetUrl); // Validate isolation - validateTargetIsolation(options.sourceUrl, targetUrl, { + await validateTargetIsolation(options.sourceUrl, targetUrl, { allowCustomTargetName: options.allowCustomTargetName, }); const sourcePool = new Pool({ connectionString: options.sourceUrl, max: 2 }); let targetPool = null; let adminClient = null; + let targetCreatedByThisRun = false; const rawBackupPath = options.backupFile || @@ -721,7 +778,7 @@ export async function runBackupRestoreDrill(options = {}) { '-F', isCustom ? 'c' : 'p', '-f', backupPath, ]; - tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password }); + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }); } else { // Docker mode const hostForDocker = @@ -737,13 +794,13 @@ export async function runBackupRestoreDrill(options = {}) { '-F', isCustom ? 'c' : 'p', '-f', `/work/${backupFileName}`, ]; - tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password }, backupDir); + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }, backupDir); } report.timings.backupMs = Date.now() - dumpStart; log(`Backup completed in ${report.timings.backupMs}ms.`); // 4. Validate backup file - const backupMeta = validateBackupFile(backupPath, options.format || 'custom'); + const backupMeta = await validateBackupFile(backupPath, options.format || 'custom'); report.backupSizeBytes = backupMeta.sizeBytes; report.backupSha256 = backupMeta.sha256; log(`Backup file validated: ${backupMeta.sizeBytes} bytes (SHA-256: ${backupMeta.sha256.substring(0, 12)}...).`); @@ -751,16 +808,18 @@ export async function runBackupRestoreDrill(options = {}) { // 5. Create isolated target database log(`Provisioning isolated target database "${parsedTarget.database}"...`); const adminUrl = urlWithDatabase(targetUrl, 'postgres'); - adminClient = new Client({ connectionString: adminUrl }); + adminClient = new Client({ connectionString: adminUrl, statement_timeout: 10000 }); try { await adminClient.connect(); } catch { // Try template1 if postgres db is not accessible - adminClient = new Client({ connectionString: urlWithDatabase(targetUrl, 'template1') }); + adminClient = new Client({ connectionString: urlWithDatabase(targetUrl, 'template1'), statement_timeout: 10000 }); await adminClient.connect(); } - await adminClient.query(`CREATE DATABASE "${parsedTarget.database}"`); + await adminClient.query(`SET statement_timeout = 10000`); + await adminClient.query(`CREATE DATABASE "` + parsedTarget.database.replace(/"/g, '""') + `"`); + targetCreatedByThisRun = true; log(`Target database "${parsedTarget.database}" created.`); // 6. Restore backup into target @@ -778,7 +837,7 @@ export async function runBackupRestoreDrill(options = {}) { backupPath, ]; try { - tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password }); + tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }); } catch (err) { // pg_restore may exit with status 1 for non-critical warnings on drops of non-existent objects const status = err?.status ?? err?.code; @@ -794,9 +853,10 @@ export async function runBackupRestoreDrill(options = {}) { '-p', String(parsedTarget.port), '-U', parsedTarget.user, '-d', parsedTarget.database, + '-v', 'ON_ERROR_STOP=1', '-f', backupPath, ]; - tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password }); + tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }); } } else { // Docker mode @@ -816,7 +876,7 @@ export async function runBackupRestoreDrill(options = {}) { `/work/${backupFileName}`, ]; try { - tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password }, backupDir); + tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }, backupDir); } catch (err) { const status = err?.status ?? err?.code; if (status === 1 || err?.message?.includes('exit code 1')) { @@ -831,9 +891,10 @@ export async function runBackupRestoreDrill(options = {}) { '-p', String(parsedTarget.port), '-U', parsedTarget.user, '-d', parsedTarget.database, + '-v', 'ON_ERROR_STOP=1', '-f', `/work/${backupFileName}`, ]; - tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password }, backupDir); + tooling.runPsql(psqlArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }, backupDir); } } report.timings.restoreMs = Date.now() - restoreStart; @@ -858,13 +919,15 @@ export async function runBackupRestoreDrill(options = {}) { // Teardown target database if not keeping if (adminClient) { - if (!options.keepTarget && parsedTarget.database) { + if (!options.keepTarget && parsedTarget.database && targetCreatedByThisRun) { try { log(`Cleaning up isolated target database "${parsedTarget.database}"...`); - await adminClient.query(`DROP DATABASE IF EXISTS "${parsedTarget.database}" WITH (FORCE)`); + await adminClient.query(`DROP DATABASE IF EXISTS "` + parsedTarget.database.replace(/"/g, '""') + `" WITH (FORCE)`); log('Target database dropped.'); } catch (err) { log(`Warning: Failed to drop isolated target database: ${err.message}`); + report.success = false; + throw err; } } await adminClient.end().catch(() => {}); @@ -918,7 +981,8 @@ Options: runBackupRestoreDrill(args) .then((report) => { if (args.json) { - console.log(JSON.stringify(report, null, 2)); + process.stdout.write(JSON.stringify(report, null, 2) + '\n', () => process.exit(0)); + return; } else { console.log('\n========================================'); console.log('✅ BACKUP & RESTORE DRILL SUCCESSFUL'); diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index b04ba6a4..40c1777a 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -54,53 +54,53 @@ test('security: sanitizeDatabaseUrl handles non-URL strings and errors safely', assert.equal(sanitizeDatabaseUrl(''), ''); }); -test('isolation: validateTargetIsolation rejects when target URL matches source URL', () => { +test('isolation: validateTargetIsolation rejects when target URL matches source URL', async () => { const source = 'postgres://gambit:pass@localhost:5432/gambit'; const target = 'postgres://gambit:pass@localhost:5432/gambit'; - assert.throws( - () => validateTargetIsolation(source, target), + await assert.rejects( + async () => await validateTargetIsolation(source, target), /Target database URL must not be identical to source database URL/, ); }); -test('isolation: validateTargetIsolation rejects when source and target have identical DB name on same host/port', () => { +test('isolation: validateTargetIsolation rejects when source and target have identical DB name on same host/port', async () => { const source = 'postgres://gambit:pass1@localhost:5432/gambit'; const target = 'postgres://gambit_admin:pass2@localhost:5432/gambit'; - assert.throws( - () => validateTargetIsolation(source, target), + await assert.rejects( + async () => await validateTargetIsolation(source, target), /Target database name "gambit" matches source database name on the same host/, ); }); -test('isolation: validateTargetIsolation rejects dangerous production or system databases as target', () => { +test('isolation: validateTargetIsolation rejects dangerous production or system databases as target', async () => { const source = 'postgres://gambit:pass@localhost:5432/source_test'; for (const dangerous of ['gambit', 'postgres', 'template1', 'template0', 'production']) { const target = `postgres://gambit:pass@localhost:5432/${dangerous}`; - assert.throws( - () => validateTargetIsolation(source, target), + await assert.rejects( + async () => await validateTargetIsolation(source, target), /Target database name .* is a protected or non-isolated database/, ); } }); -test('isolation: validateTargetIsolation rejects target names without isolated naming markers unless explicitly allowed', () => { +test('isolation: validateTargetIsolation rejects target names without isolated naming markers unless explicitly allowed', async () => { const source = 'postgres://gambit:pass@localhost:5432/gambit'; const target = 'postgres://gambit:pass@localhost:5432/arbitrary_name'; - assert.throws( - () => validateTargetIsolation(source, target, { allowCustomTargetName: false }), + await assert.rejects( + async () => await validateTargetIsolation(source, target, { allowCustomTargetName: false }), /Target database name "arbitrary_name" does not contain an isolated drill marker/, ); // When explicitly permitted via allowCustomTargetName flag - const result = validateTargetIsolation(source, target, { allowCustomTargetName: true }); + const result = await validateTargetIsolation(source, target, { allowCustomTargetName: true }); assert.equal(result.isolated, true); assert.equal(result.targetDbName, 'arbitrary_name'); }); -test('isolation: validateTargetIsolation accepts valid isolated database names', () => { +test('isolation: validateTargetIsolation accepts valid isolated database names', async () => { const source = 'postgres://gambit:pass@localhost:5432/gambit'; const validTargets = [ 'gambit_backup_drill_restore', @@ -112,33 +112,43 @@ test('isolation: validateTargetIsolation accepts valid isolated database names', for (const name of validTargets) { const target = `postgres://gambit:pass@localhost:5432/${name}`; - const validated = validateTargetIsolation(source, target); + const validated = await validateTargetIsolation(source, target); assert.equal(validated.isolated, true); assert.equal(validated.targetDbName, name); } }); -test('isolation: generateIsolatedDbName generates prefixed unique name', () => { +test('isolation: generateIsolatedDbName generates prefixed unique name', async () => { const name1 = generateIsolatedDbName('gambit'); const name2 = generateIsolatedDbName('gambit'); assert.match(name1, /^gambit_backup_drill_restore_\d+_[a-f0-9]+$/); assert.notEqual(name1, name2); }); -test('backup validation: validateBackupFile rejects non-existent file', () => { +test('isolation: validateTargetIsolation rejects malicious injection payloads in target URL', async () => { + const source = 'postgres://gambit:pass@localhost:5432/gambit'; + const maliciousTarget = 'postgres://gambit:pass@localhost:5432/test" OR 1=1; DROP DATABASE production; --'; + + await assert.rejects( + async () => await validateTargetIsolation(source, maliciousTarget, { allowCustomTargetName: true }), + /Invalid target database name format/ + ); +}); + +test('backup validation: validateBackupFile rejects non-existent file', async () => { const nonExistent = join(tmpdir(), 'non-existent-backup-file-12345.dump'); - assert.throws( - () => validateBackupFile(nonExistent, 'custom'), + await assert.rejects( + async () => await validateBackupFile(nonExistent, 'custom'), /Backup file does not exist/, ); }); -test('backup validation: validateBackupFile rejects 0-byte empty file', () => { +test('backup validation: validateBackupFile rejects 0-byte empty file', async () => { const emptyFile = join(tmpdir(), `test-empty-backup-${Date.now()}.dump`); writeFileSync(emptyFile, Buffer.alloc(0)); try { - assert.throws( - () => validateBackupFile(emptyFile, 'custom'), + await assert.rejects( + async () => await validateBackupFile(emptyFile, 'custom'), /Backup file is empty \(0 bytes\)/, ); } finally { @@ -146,12 +156,12 @@ test('backup validation: validateBackupFile rejects 0-byte empty file', () => { } }); -test('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', () => { +test('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', async () => { const corruptFile = join(tmpdir(), `test-corrupt-backup-${Date.now()}.dump`); writeFileSync(corruptFile, Buffer.from('NOT_A_VALID_PG_DUMP_HEADER')); try { - assert.throws( - () => validateBackupFile(corruptFile, 'custom'), + await assert.rejects( + async () => await validateBackupFile(corruptFile, 'custom'), /Invalid custom-format backup: missing PostgreSQL dump magic header "PGDMP"/, ); } finally { @@ -159,12 +169,12 @@ test('backup validation: validateBackupFile rejects corrupted custom format arch } }); -test('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', () => { +test('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', async () => { const validFile = join(tmpdir(), `test-valid-backup-${Date.now()}.dump`); const header = Buffer.from('PGDMP\x01\x10\x00\x01'); writeFileSync(validFile, header); try { - const meta = validateBackupFile(validFile, 'custom'); + const meta = await validateBackupFile(validFile, 'custom'); assert.equal(meta.valid, true); assert.equal(meta.sizeBytes, header.length); assert.equal(typeof meta.sha256, 'string'); @@ -185,6 +195,7 @@ test('verification engine: detects missing extensions in restored database', asy // Mock target database pool missing the 'vector' extension const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext', extversion: '1.6' }] }; @@ -213,6 +224,7 @@ test('verification engine: detects missing schema_migrations table', async () => // Mock target where schema_migrations table does not exist const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; @@ -247,6 +259,7 @@ test('verification engine: detects missing schema_migrations table or count mism // Mock target where only migration 1 is present const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; @@ -283,6 +296,7 @@ test('verification engine: detects corrupted or modified migration checksum in r // Mock target where migration 1 has altered checksum const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; @@ -300,7 +314,7 @@ test('verification engine: detects corrupted or modified migration checksum in r await assert.rejects( () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), (err) => { - assert.match(err.message, /Migration 1 checksum mismatch/); + assert.match(err.message, /Migration 1 mismatch/); return true; }, ); @@ -317,6 +331,7 @@ test('verification engine: detects missing tables in restored database', async ( // Mock target where 'tournaments' table is missing const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; @@ -354,6 +369,7 @@ test('verification engine: detects row count mismatch in critical tables', async // Mock target where game_events only has 90 rows restored (incomplete restore!) const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; @@ -365,7 +381,7 @@ test('verification engine: detects row count mismatch in critical tables', async return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; } if (text.includes('information_schema.tables') || text.includes('pg_tables')) { - return { rows: [{ tablename: 'schema_migrations' }, { tablename: 'users' }, { tablename: 'game_events' }] }; + return { rows: [...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t })), ...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))] }; } if (text.includes('COUNT(*) FROM "users"') || text.includes('COUNT(*) AS count FROM "users"')) { return { rows: [{ count: '5' }] }; @@ -400,6 +416,7 @@ test('verification engine: detects missing or inactive append-only trigger on ga // Mock target where table exists and row count matches, but trigger probe does NOT throw const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; @@ -411,7 +428,7 @@ test('verification engine: detects missing or inactive append-only trigger on ga return { rows: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }] }; } if (text.includes('information_schema.tables') || text.includes('pg_tables')) { - return { rows: [{ tablename: 'schema_migrations' }, { tablename: 'users' }, { tablename: 'game_events' }] }; + return { rows: [...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t })), ...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))] }; } if (text.includes('COUNT(*)')) { return { rows: [{ count: '1' }] }; @@ -446,6 +463,7 @@ test('verification engine: passes when all structural and functional checks succ }; const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, async query(text, params) { if (text.includes('pg_extension')) { return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; @@ -458,12 +476,7 @@ test('verification engine: passes when all structural and functional checks succ } if (text.includes('information_schema.tables') || text.includes('pg_tables')) { return { - rows: [ - { tablename: 'schema_migrations' }, - { tablename: 'users' }, - { tablename: 'game_events' }, - { tablename: 'variants' }, - ], + rows: CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t })), }; } if (text.includes('COUNT(*)')) { @@ -483,7 +496,7 @@ test('verification engine: passes when all structural and functional checks succ throw error; } if (text.includes('search_embeddings')) { - return { rows: [] }; + return { rows: [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw' }] }; } return { rows: [] }; }, @@ -534,13 +547,15 @@ test('cli: parseArgs defaults to safe isolated target when not specified', () => }); test('tooling: resolvePgTooling accepts options and detects native or docker runner', () => { - const customTooling = resolvePgTooling({ format: 'custom' }); + const mockExec = (cmd, args) => { return Buffer.from('mock version'); }; + + const customTooling = resolvePgTooling({ format: 'custom', execSyncFn: mockExec }); assert.ok(customTooling.type === 'native' || customTooling.type === 'docker'); assert.equal(typeof customTooling.runDump, 'function'); assert.equal(typeof customTooling.runRestore, 'function'); assert.equal(typeof customTooling.runPsql, 'function'); - const plainTooling = resolvePgTooling({ format: 'plain' }); + const plainTooling = resolvePgTooling({ format: 'plain', execSyncFn: mockExec }); assert.ok(plainTooling.type === 'native' || plainTooling.type === 'docker'); }); From 045e067703498ef8006b9c7ecb3ef6b6fc410eb8 Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Sun, 6 Sep 2026 14:53:01 +0300 Subject: [PATCH 05/14] chore: clean up temporary test helper scripts --- apply.py | 80 --------------------------------------------------- fix-tests.js | 63 ---------------------------------------- fix-tests2.js | 45 ----------------------------- fix-tests3.js | 6 ---- fix-tests4.js | 9 ------ patch.js | 28 ------------------ 6 files changed, 231 deletions(-) delete mode 100644 apply.py delete mode 100644 fix-tests.js delete mode 100644 fix-tests2.js delete mode 100644 fix-tests3.js delete mode 100644 fix-tests4.js delete mode 100644 patch.js diff --git a/apply.py b/apply.py deleted file mode 100644 index ef9275d2..00000000 --- a/apply.py +++ /dev/null @@ -1,80 +0,0 @@ -import re -import sys -import os - -print("Applying patches...") - -mjs_path = 'scripts/db-backup-restore-drill.mjs' -with open(mjs_path, 'r', encoding='utf-8') as f: - content = f.read() - -# 1. DATABASE OWNERSHIP & DESTRUCTIVE CLEANUP -# Only run destructive cleanup if targetCreatedByThisRun = true -content = content.replace( - 'await adminClient.query(`CREATE DATABASE "${parsedTarget.database}"`);', - 'await adminClient.query(`CREATE DATABASE """${parsedTarget.database.replace(/"/g, \'""\')}"""`);\n targetCreatedByThisRun = true;' -) -content = content.replace( - 'let adminClient = null;', - 'let adminClient = null;\n let targetCreatedByThisRun = false;' -) - -# 2. TARGET HOST -# Admin queries must connect to the same target host/server as restore -# Right now it uses parsedTarget but connects to targetUrl, 'postgres'. -# Already implemented as: const adminUrl = urlWithDatabase(targetUrl, 'postgres'); which preserves host. -# But wait, step 5 says admin client connects. Yes, adminUrl preserves host. - -# 3. IDENTIFIER INJECTION -# Validate DB name format -content = content.replace( - 'export async function validateTargetIsolation(sourceUrl, targetUrl, options = {}) {', - '''export async function validateTargetIsolation(sourceUrl, targetUrl, options = {}) { - if (!/^[a-zA-Z0-9_-]+$/.test(parseDatabaseUrl(targetUrl).database)) { - throw new Error('Invalid target database name format'); - }''' -) - -# Quote identifiers -# (Already partly done in step 1, what about drop?) -content = re.sub( - r'DROP DATABASE "([^"]+)"', - r'DROP DATABASE """\1"""', - content -) - -# 4. HOST ALIAS / SOURCE ISOLATION -# done by node command earlier - -# 5. CREDENTIAL EXPOSURE -# Pass '-e', 'PGPASSWORD' in docker, not '-e PGPASSWORD=...' -content = content.replace( - "if (env.PGPASSWORD) dockerArgs.push('-e', `PGPASSWORD=${env.PGPASSWORD}`);", - "if (env.PGPASSWORD) dockerArgs.push('-e', 'PGPASSWORD');" -) - -# 6. CONNECTION SECURITY OPTIONS -# urlWithDatabase should preserve searchParams (SSL) -content = content.replace( - 'parsed.pathname = `/${encodeURIComponent(databaseName)}`;\n return parsed.toString();', - 'parsed.pathname = `/${encodeURIComponent(databaseName)}`;\n return parsed.toString();' -) -# wait, parseDatabaseUrl doesn't preserve search params if we rebuild, but urlWithDatabase uses URL object so it preserves them automatically. - -# 7. BASELINE SNAPSHOT CONSISTENCY -# capture baseline in transaction or just REPEATABLE READ? -content = content.replace( - 'export async function collectSourceBaseline(pool) {', - "export async function collectSourceBaseline(pool) {\n const client = await pool.connect();\n await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY');\n try {" -) -content = content.replace( - ' return {', - " } finally {\n await client.query('ROLLBACK');\n client.release();\n }\n return {" -) -# Replace pool.query with client.query in collectSourceBaseline -# Actually that would need a full regex. I will just do a simpler search/replace. - -with open(mjs_path, 'w', encoding='utf-8') as f: - f.write(content) - -print("Done") diff --git a/fix-tests.js b/fix-tests.js deleted file mode 100644 index 16dc9a93..00000000 --- a/fix-tests.js +++ /dev/null @@ -1,63 +0,0 @@ -const fs = require('fs'); -let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); - -// validateBackupFile is async now -content = content.replace( - /test\('backup validation: validateBackupFile rejects non-existent file', \(\) => {/g, - "test('backup validation: validateBackupFile rejects non-existent file', async () => {" -); -content = content.replace( - /assert\.throws\(\n\s*\(\) => validateBackupFile\(nonExistent, 'custom'\),/g, - "await assert.rejects(\n async () => await validateBackupFile(nonExistent, 'custom')," -); - -content = content.replace( - /test\('backup validation: validateBackupFile rejects 0-byte empty file', \(\) => {/g, - "test('backup validation: validateBackupFile rejects 0-byte empty file', async () => {" -); -content = content.replace( - /assert\.throws\(\n\s*\(\) => validateBackupFile\(emptyFile, 'custom'\),/g, - "await assert.rejects(\n async () => await validateBackupFile(emptyFile, 'custom')," -); - -content = content.replace( - /test\('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', \(\) => {/g, - "test('backup validation: validateBackupFile rejects corrupted custom format archive without PGDMP header', async () => {" -); -content = content.replace( - /assert\.throws\(\n\s*\(\) => validateBackupFile\(corruptFile, 'custom'\),/g, - "await assert.rejects(\n async () => await validateBackupFile(corruptFile, 'custom')," -); - -content = content.replace( - /test\('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', \(\) => {/g, - "test('backup validation: validateBackupFile accepts valid custom format archive with PGDMP header', async () => {" -); -content = content.replace( - /const meta = validateBackupFile\(validFile, 'custom'\);/g, - "const meta = await validateBackupFile(validFile, 'custom');" -); - -// Checksum regex fix -content = content.replace( - /\/Migration 1 checksum mismatch\//g, - "/Migration 1 mismatch/" -); - -// Missing tables due to CRITICAL_APPLICATION_TABLES checking all of them -content = content.replace( - /\{ tablename: 'users' \}, \{ tablename: 'game_events' \}/g, - "{ tablename: 'users' }, { tablename: 'game_events' }, ...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))" -); - -content = content.replace( - /\{ tablename: 'schema_migrations' \}, \{ tablename: 'users' \}, \{ tablename: 'game_events' \}/g, - "...CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))" -); - -content = content.replace( - /rows: \[\n\s*\{ tablename: 'schema_migrations' \},\n\s*\{ tablename: 'users' \},\n\s*\{ tablename: 'game_events' \},\n\s*\{ tablename: 'variants' \},\n\s*\]/g, - "rows: CRITICAL_APPLICATION_TABLES.map(t => ({ tablename: t }))" -); - -fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/fix-tests2.js b/fix-tests2.js deleted file mode 100644 index cc923737..00000000 --- a/fix-tests2.js +++ /dev/null @@ -1,45 +0,0 @@ -const fs = require('fs'); -let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); - -content = content.replace( - /validateTargetIsolation\(source, target\)/g, - 'await validateTargetIsolation(source, target)' -); -content = content.replace( - /validateTargetIsolation\(source, target, /g, - 'await validateTargetIsolation(source, target, ' -); - -content = content.replace( - /const validated = /g, - 'const validated = await ' -); - -content = content.replace( - /test\('isolation:.*\(\) => \{/g, - (match) => match.replace('() => {', 'async () => {') -); - -// fix assert.throws for validateTargetIsolation -content = content.replace( - /assert\.throws\(\n\s*\(\) => await validateTargetIsolation/g, - "await assert.rejects(\n async () => await validateTargetIsolation" -); -content = content.replace( - /assert\.throws\(\n\s*\(\) => validateTargetIsolation/g, - "await assert.rejects(\n async () => await validateTargetIsolation" -); - -// fix missing awaits for validateTargetIsolation inside blocks -content = content.replace( - /await await validateTargetIsolation/g, - "await validateTargetIsolation" -); - -// Mock pool connect -content = content.replace( - /async query\(text, params\) \{/g, - "async connect() { return { query: this.query, release: () => {} }; },\n async query(text, params) {" -); - -fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/fix-tests3.js b/fix-tests3.js deleted file mode 100644 index 8f1cf206..00000000 --- a/fix-tests3.js +++ /dev/null @@ -1,6 +0,0 @@ -const fs = require('fs'); -let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); - -content = content.replace(/async async \(\) =>/g, 'async () =>'); - -fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/fix-tests4.js b/fix-tests4.js deleted file mode 100644 index 30720fed..00000000 --- a/fix-tests4.js +++ /dev/null @@ -1,9 +0,0 @@ -const fs = require('fs'); -let content = fs.readFileSync('scripts/test/backup-restore-drill.test.mjs', 'utf8'); - -content = content.replace( - /if \(text\.includes\('search_embeddings'\)\) \{\n\s*return \{ rows: \[\] \};\n\s*\}/g, - "if (text.includes('search_embeddings')) {\n return { rows: [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw' }] };\n }" -); - -fs.writeFileSync('scripts/test/backup-restore-drill.test.mjs', content); diff --git a/patch.js b/patch.js deleted file mode 100644 index 1bcb0009..00000000 --- a/patch.js +++ /dev/null @@ -1,28 +0,0 @@ -const fs = require('fs'); - -function patchFile(file, edits) { - let content = fs.readFileSync(file, 'utf8'); - for (const [search, replace] of edits) { - if (!content.includes(search)) { - console.error(`Could not find search string in ${file}:\n${search.substring(0, 50)}...`); - } else { - content = content.replace(search, replace); - } - } - fs.writeFileSync(file, content); - console.log(`Patched ${file}`); -} - -const editsMjs = [ - // 1. STREAM HASHING for validateBackupFile - [ - `export function validateBackupFile`, - `import { pipeline } from 'node:stream/promises';\nexport async function validateBackupFile` - ], - [ - ` // Compute SHA-256 digest\n const fileBytes = readFileSync(filePath);\n const sha256 = createHash('sha256').update(fileBytes).digest('hex');\n\n return {`, - ` // Compute SHA-256 digest\n const hash = createHash('sha256');\n await pipeline(fs.createReadStream(filePath), hash);\n const sha256 = hash.digest('hex');\n\n return {` - ] -]; - -patchFile('scripts/db-backup-restore-drill.mjs', editsMjs); From c1ee1aff6ef38c74cb4edec4e0e0c239d43ba6cb Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Sun, 6 Sep 2026 15:01:19 +0300 Subject: [PATCH 06/14] fix(db): verify all source extensions and enforce snapshot consistency across dump --- scripts/db-backup-restore-drill.mjs | 154 ++++++++++++++------- scripts/test/backup-restore-drill.test.mjs | 28 ++++ 2 files changed, 131 insertions(+), 51 deletions(-) diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index f69d47f3..37d0bb27 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -373,13 +373,16 @@ export function resolvePgTooling(options = {}) { /** * Capture source database state baseline before running backup. */ -export async function collectSourceBaseline(pool) { - const client = await pool.connect(); - await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); +export async function collectSourceBaseline(pool, existingClient = null) { + const client = existingClient || (await pool.connect()); + const shouldManageTx = !existingClient; + if (shouldManageTx) { + await client.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); + } try { - // 1. Extensions + // 1. Extensions with versions const extRes = await client.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); - const extensions = extRes.rows.map((r) => r.extname); + const extensions = extRes.rows.map((r) => ({ extname: r.extname, extversion: r.extversion })); // 2. Schema migrations const hasMigrationsTable = await client.query( @@ -437,8 +440,10 @@ export async function collectSourceBaseline(pool) { sampleData, }; } finally { - await client.query('ROLLBACK'); - client.release(); + if (shouldManageTx) { + await client.query('ROLLBACK'); + client.release(); + } } } @@ -458,12 +463,16 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options // Check 1: Extensions const extRes = await targetPool.query('SELECT extname, extversion FROM pg_extension ORDER BY extname'); - const targetExts = new Set(extRes.rows.map((r) => r.extname)); + const targetExtMap = new Map(); + for (const r of extRes.rows || []) { + targetExtMap.set(r.extname, r.extversion || null); + } + const targetExts = new Set(targetExtMap.keys()); for (const requiredExt of REQUIRED_EXTENSIONS) { - const present = targetExts.has(requiredExt); + const present = targetExtMap.has(requiredExt); recordCheck( - `Extension: ${requiredExt}`, + `Required Extension: ${requiredExt}`, present, present ? 'Installed and active' : `Missing required extension in restored database: ${requiredExt}`, ); @@ -472,6 +481,28 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options } } + // Compare every source extension and version with restored database + for (const srcExt of sourceBaseline.extensions || []) { + const name = typeof srcExt === 'string' ? srcExt : srcExt.extname; + const version = typeof srcExt === 'string' ? null : (srcExt.extversion || null); + const present = targetExtMap.has(name); + const targetVersion = targetExtMap.get(name); + const versionMatch = !version || !targetVersion || targetVersion === version; + recordCheck( + `Source Extension: ${name}`, + present && versionMatch, + present + ? (versionMatch ? `Installed version ${targetVersion || 'active'}` : `Version mismatch: source ${version} vs restored ${targetVersion}`) + : `Missing source extension in restored database: ${name}`, + ); + if (!present) { + throw new Error(`Missing source extension in restored database: ${name}`); + } + if (!versionMatch) { + throw new Error(`Extension version mismatch in restored database for ${name}: source ${version} vs restored ${targetVersion}`); + } + } + // Check 2: Schema migrations ledger if (sourceBaseline.migrations && sourceBaseline.migrations.length > 0) { const migTableRes = await targetPool.query( @@ -755,49 +786,70 @@ export async function runBackupRestoreDrill(options = {}) { }; try { - // 1. Capture source baseline + // 1. Capture source baseline and export snapshot log(`Capturing source baseline from ${report.source}...`); - const sourceBaseline = await collectSourceBaseline(sourcePool); - log(`Source baseline captured: ${sourceBaseline.tables.length} tables, ${sourceBaseline.migrations.length} migrations.`); - - // 2. Resolve tooling - const tooling = resolvePgTooling(options); - log(`Resolved PostgreSQL tooling (${tooling.type})...`); - - // 3. Perform backup using pg_dump - log(`Creating ${options.format || 'custom'} backup to ${backupPath}...`); - const dumpStart = Date.now(); - const isCustom = (options.format || 'custom') === 'custom'; + const baselineClient = await sourcePool.connect(); + let snapshotId = null; + let sourceBaseline = null; + try { + await baselineClient.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); + try { + const snapRes = await baselineClient.query('SELECT pg_export_snapshot() AS snap'); + snapshotId = snapRes.rows[0]?.snap || null; + } catch { + // Snapshot export might not be supported in some test or replica configurations + } + sourceBaseline = await collectSourceBaseline(sourcePool, baselineClient); + log(`Source baseline captured: ${sourceBaseline.tables.length} tables, ${sourceBaseline.migrations.length} migrations.`); + + // 2. Resolve tooling + const tooling = resolvePgTooling(options); + log(`Resolved PostgreSQL tooling (${tooling.type})...`); + + // 3. Perform backup using pg_dump + log(`Creating ${options.format || 'custom'} backup to ${backupPath}...`); + const dumpStart = Date.now(); + const isCustom = (options.format || 'custom') === 'custom'; + + if (tooling.type === 'native') { + const dumpArgs = [ + '-h', parsedSource.host, + '-p', String(parsedSource.port), + '-U', parsedSource.user, + '-d', parsedSource.database, + '-F', isCustom ? 'c' : 'p', + '-f', backupPath, + ]; + if (snapshotId && isCustom) { + dumpArgs.push(`--snapshot=${snapshotId}`); + } + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }); + } else { + // Docker mode + const hostForDocker = + parsedSource.host === 'localhost' || parsedSource.host === '127.0.0.1' + ? (process.platform === 'linux' ? '127.0.0.1' : 'host.docker.internal') + : parsedSource.host; - if (tooling.type === 'native') { - const dumpArgs = [ - '-h', parsedSource.host, - '-p', String(parsedSource.port), - '-U', parsedSource.user, - '-d', parsedSource.database, - '-F', isCustom ? 'c' : 'p', - '-f', backupPath, - ]; - tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }); - } else { - // Docker mode - const hostForDocker = - parsedSource.host === 'localhost' || parsedSource.host === '127.0.0.1' - ? (process.platform === 'linux' ? '127.0.0.1' : 'host.docker.internal') - : parsedSource.host; - - const dumpArgs = [ - '-h', hostForDocker, - '-p', String(parsedSource.port), - '-U', parsedSource.user, - '-d', parsedSource.database, - '-F', isCustom ? 'c' : 'p', - '-f', `/work/${backupFileName}`, - ]; - tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }, backupDir); - } - report.timings.backupMs = Date.now() - dumpStart; - log(`Backup completed in ${report.timings.backupMs}ms.`); + const dumpArgs = [ + '-h', hostForDocker, + '-p', String(parsedSource.port), + '-U', parsedSource.user, + '-d', parsedSource.database, + '-F', isCustom ? 'c' : 'p', + '-f', `/work/${backupFileName}`, + ]; + if (snapshotId && isCustom) { + dumpArgs.push(`--snapshot=${snapshotId}`); + } + tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }, backupDir); + } + report.timings.backupMs = Date.now() - dumpStart; + log(`Backup completed in ${report.timings.backupMs}ms.`); + } finally { + await baselineClient.query('ROLLBACK').catch(() => {}); + baselineClient.release(); + } // 4. Validate backup file const backupMeta = await validateBackupFile(backupPath, options.format || 'custom'); diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index 40c1777a..5f350983 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -213,6 +213,34 @@ test('verification engine: detects missing extensions in restored database', asy ); }); +test('verification engine: detects extension version mismatch between source and restored database', async () => { + const sourceBaseline = { + extensions: [{ extname: 'citext', extversion: '1.6' }, { extname: 'vector', extversion: '0.5.1' }], + migrations: [], + tables: [], + rowCounts: {}, + sampleData: {}, + }; + + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext', extversion: '1.6' }, { extname: 'vector', extversion: '0.4.0' }] }; + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Extension version mismatch in restored database for vector: source 0.5.1 vs restored 0.4.0/); + return true; + }, + ); +}); + test('verification engine: detects missing schema_migrations table', async () => { const sourceBaseline = { extensions: ['citext', 'vector'], From 97e2e9e0a3e7d4746289000079195b6984e2f707 Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Sun, 6 Sep 2026 15:39:58 +0300 Subject: [PATCH 07/14] fix(drill): distinguish pg_restore warnings from real errors; enforce required extensions unconditionally --- scripts/db-backup-restore-drill.mjs | 51 ++++++++++++-------- scripts/test/backup-restore-drill.test.mjs | 55 ++++++++++++++++++++++ 2 files changed, 87 insertions(+), 19 deletions(-) diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index 37d0bb27..5077b61b 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -304,9 +304,9 @@ export function resolvePgTooling(options = {}) { if (hasRequiredNativeTools) { return { type: 'native', - runDump: (args, env) => execFileSync('pg_dump', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), - runRestore: (args, env) => execFileSync('pg_restore', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), - runPsql: (args, env) => execFileSync('psql', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + runDump: (args, env) => execSyncFn('pg_dump', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + runRestore: (args, env) => execSyncFn('pg_restore', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), + runPsql: (args, env) => execSyncFn('psql', args, { env: { ...process.env, ...env }, stdio: 'pipe' }), }; } @@ -339,7 +339,7 @@ export function resolvePgTooling(options = {}) { dockerArgs.push('--net=host'); } dockerArgs.push(dockerImage, 'pg_dump', ...args); - return execFileSync('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); + return execSyncFn('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); }, runRestore: (args, env, mountDir) => { const dockerArgs = ['run', '--rm']; @@ -352,7 +352,7 @@ export function resolvePgTooling(options = {}) { dockerArgs.push('--net=host'); } dockerArgs.push(dockerImage, 'pg_restore', ...args); - return execFileSync('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); + return execSyncFn('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); }, runPsql: (args, env, mountDir) => { const dockerArgs = ['run', '--rm']; @@ -365,11 +365,33 @@ export function resolvePgTooling(options = {}) { dockerArgs.push('--net=host'); } dockerArgs.push(dockerImage, 'psql', ...args); - return execFileSync('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); + return execSyncFn('docker', dockerArgs, { env: { ...process.env, ...env }, stdio: 'pipe' }); }, }; } +/** + * Parse pg_restore errors to distinguish harmless drop-if-not-exists warnings from real failures. + */ +export function parsePgRestoreError(err) { + const status = err?.status ?? err?.code; + if (status === 1 || err?.message?.includes('exit code 1')) { + const stderrStr = err?.stderr ? err.stderr.toString().trim() : ''; + if (stderrStr) { + const lines = stderrStr.split('\n'); + const hasRealError = lines.some(line => { + return line.includes('pg_restore: error:') && !/does not exist|no existe/i.test(line); + }); + if (hasRealError) { + throw err; + } + return stderrStr; + } + return ''; + } + throw err; +} + /** * Capture source database state baseline before running backup. */ @@ -891,13 +913,8 @@ export async function runBackupRestoreDrill(options = {}) { try { tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }); } catch (err) { - // pg_restore may exit with status 1 for non-critical warnings on drops of non-existent objects - const status = err?.status ?? err?.code; - if (status === 1 || err?.message?.includes('exit code 1')) { - if (err?.stderr) log(`pg_restore notice: ${err.stderr.toString().trim()}`); - } else { - throw err; - } + const notice = parsePgRestoreError(err); + if (notice) log(`pg_restore notice: ${notice}`); } } else { const psqlArgs = [ @@ -930,12 +947,8 @@ export async function runBackupRestoreDrill(options = {}) { try { tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }, backupDir); } catch (err) { - const status = err?.status ?? err?.code; - if (status === 1 || err?.message?.includes('exit code 1')) { - if (err?.stderr) log(`pg_restore notice: ${err.stderr.toString().trim()}`); - } else { - throw err; - } + const notice = parsePgRestoreError(err); + if (notice) log(`pg_restore notice: ${notice}`); } } else { const psqlArgs = [ diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index 5f350983..7b7aa294 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -28,6 +28,7 @@ import { resolvePgTooling, CRITICAL_APPLICATION_TABLES, REQUIRED_EXTENSIONS, + parsePgRestoreError, } from '../db-backup-restore-drill.mjs'; test('security: sanitizeDatabaseUrl masks plaintext passwords in postgres URLs', () => { @@ -587,6 +588,60 @@ test('tooling: resolvePgTooling accepts options and detects native or docker run assert.ok(plainTooling.type === 'native' || plainTooling.type === 'docker'); }); +test('pg_restore: parsePgRestoreError treats "does not exist" warnings as benign', () => { + const err = new Error('Command failed: pg_restore exit code 1'); + err.status = 1; + err.stderr = Buffer.from( + 'pg_restore: warning: errors ignored on restore: 1\n' + + 'pg_restore: error: could not execute query: ERROR: schema "public" does not exist\n' + + 'pg_restore: error: could not execute query: ERROR: role "postgres" does not exist' + ); + + const notice = parsePgRestoreError(err); + assert.ok(notice.includes('does not exist')); +}); + +test('pg_restore: parsePgRestoreError throws on real errors', () => { + const err = new Error('Command failed: pg_restore exit code 1'); + err.status = 1; + err.stderr = Buffer.from( + 'pg_restore: warning: errors ignored on restore: 1\n' + + 'pg_restore: error: could not execute query: ERROR: syntax error at or near "SELECT"\n' + + 'pg_restore: error: could not execute query: ERROR: role "postgres" does not exist' + ); + + assert.throws(() => parsePgRestoreError(err), /Command failed/); +}); + +test('verification engine: checks REQUIRED_EXTENSIONS even when not present in source baseline', async () => { + const sourceBaseline = { + extensions: [], // Missing from source! + migrations: [{ version: 1, name: '0001_init.sql', checksum: 'abc', state: 'applied' }], + tables: ['schema_migrations', 'users'], + rowCounts: { schema_migrations: 1, users: 1 }, + sampleData: {}, + }; + + // Mock target database pool where required extension "vector" is missing + const mockTargetPool = { + async connect() { return { query: this.query, release: () => {} }; }, + async query(text) { + if (text.includes('pg_extension')) { + return { rows: [{ extname: 'citext', extversion: '1.6' }] }; // "vector" missing + } + return { rows: [] }; + }, + }; + + await assert.rejects( + () => verifyRestoredDatabase(sourceBaseline, mockTargetPool), + (err) => { + assert.match(err.message, /Missing required extension in restored database: vector/); + return true; + }, + ); +}); + test( 'integration: full backup, isolated restore, and verification drill against live Postgres', { skip: process.env.DATABASE_URL ? false : 'DATABASE_URL not set' }, From 5ca188c6ae75bc173ee871b3adef38289284a820 Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Mon, 7 Sep 2026 10:04:00 +0300 Subject: [PATCH 08/14] fix(drill): treat all pg_restore: error: lines as fatal; add --no-owner --no-acl --- scripts/db-backup-restore-drill.mjs | 6 +++++- scripts/test/backup-restore-drill.test.mjs | 8 +++----- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index 5077b61b..96b1cac1 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -380,7 +380,7 @@ export function parsePgRestoreError(err) { if (stderrStr) { const lines = stderrStr.split('\n'); const hasRealError = lines.some(line => { - return line.includes('pg_restore: error:') && !/does not exist|no existe/i.test(line); + return line.includes('pg_restore: error:'); }); if (hasRealError) { throw err; @@ -908,6 +908,8 @@ export async function runBackupRestoreDrill(options = {}) { '-d', parsedTarget.database, '--clean', '--if-exists', + '--no-owner', + '--no-acl', backupPath, ]; try { @@ -942,6 +944,8 @@ export async function runBackupRestoreDrill(options = {}) { '-d', parsedTarget.database, '--clean', '--if-exists', + '--no-owner', + '--no-acl', `/work/${backupFileName}`, ]; try { diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index 7b7aa294..0ef7a731 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -588,25 +588,23 @@ test('tooling: resolvePgTooling accepts options and detects native or docker run assert.ok(plainTooling.type === 'native' || plainTooling.type === 'docker'); }); -test('pg_restore: parsePgRestoreError treats "does not exist" warnings as benign', () => { +test('pg_restore: parsePgRestoreError treats non-error warnings as benign', () => { const err = new Error('Command failed: pg_restore exit code 1'); err.status = 1; err.stderr = Buffer.from( 'pg_restore: warning: errors ignored on restore: 1\n' + - 'pg_restore: error: could not execute query: ERROR: schema "public" does not exist\n' + - 'pg_restore: error: could not execute query: ERROR: role "postgres" does not exist' + 'pg_restore: warning: could not execute query: ERROR: schema "public" does not exist' ); const notice = parsePgRestoreError(err); assert.ok(notice.includes('does not exist')); }); -test('pg_restore: parsePgRestoreError throws on real errors', () => { +test('pg_restore: parsePgRestoreError throws on real errors including missing objects', () => { const err = new Error('Command failed: pg_restore exit code 1'); err.status = 1; err.stderr = Buffer.from( 'pg_restore: warning: errors ignored on restore: 1\n' + - 'pg_restore: error: could not execute query: ERROR: syntax error at or near "SELECT"\n' + 'pg_restore: error: could not execute query: ERROR: role "postgres" does not exist' ); From 6b243f687246bb083824a9cdc0e54fed4c87ab55 Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Mon, 7 Sep 2026 15:51:38 +0300 Subject: [PATCH 09/14] fix(drill): add --no-owner --no-acl to pg_dump invocations so dump file excludes ownership commands --- scripts/db-backup-restore-drill.mjs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index 96b1cac1..2d0f4896 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -840,6 +840,8 @@ export async function runBackupRestoreDrill(options = {}) { '-U', parsedSource.user, '-d', parsedSource.database, '-F', isCustom ? 'c' : 'p', + '--no-owner', + '--no-acl', '-f', backupPath, ]; if (snapshotId && isCustom) { @@ -859,6 +861,8 @@ export async function runBackupRestoreDrill(options = {}) { '-U', parsedSource.user, '-d', parsedSource.database, '-F', isCustom ? 'c' : 'p', + '--no-owner', + '--no-acl', '-f', `/work/${backupFileName}`, ]; if (snapshotId && isCustom) { From 9c6e2eb863f93289d2801035c1ad2168fa0601cb Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Tue, 8 Sep 2026 09:31:27 +0300 Subject: [PATCH 10/14] fix(backup): fail closed across restore drill --- docs/PROJECT_STATE.md | 9 +- docs/runbooks/backup-restore-drill.md | 22 +- scripts/db-backup-restore-drill.mjs | 176 +++++++----- scripts/test/backup-restore-drill.test.mjs | 7 +- scripts/test/backup-restore-safety.test.mjs | 286 ++++++++++++++++++++ 5 files changed, 430 insertions(+), 70 deletions(-) create mode 100644 scripts/test/backup-restore-safety.test.mjs diff --git a/docs/PROJECT_STATE.md b/docs/PROJECT_STATE.md index 0a7bc672..47df1028 100644 --- a/docs/PROJECT_STATE.md +++ b/docs/PROJECT_STATE.md @@ -6,7 +6,7 @@ > to read **only this file** and continue immediately. Updated after every > milestone and every significant architectural step. -_Last updated: 2026-09-05 — M15 Increment 52: deterministic analysis-cache cold-race test._ +_Last updated: 2026-09-07 — PR #51: backup/restore drill safety regressions._ Prior: _Last updated: 2026-09-05 — M15 Increment 51: Signature B mechanism isolation and diagnostic hardening._ @@ -4123,6 +4123,13 @@ Per package: `cd packages/ && npm install && npm run build && npm test`. - **Recorded gaps**: GraphQL `Player` has no `teams` field despite ADR-0073's Context claiming it; five pre-existing design-system findings in `style.css` are reported, not repaired (fixing drift inside a feature PR is how a design-system change ships unreviewed). - Detailed in `docs/adr/0074-social-ui-profile.md`. +## PR #51 — Backup/restore drill safety regressions — 2026-09-07 + +- Fixed the restore tooling scope error and made every nonzero restore exit fatal. Both dump formats require the baseline's exported snapshot; unreadable tables fail the baseline instead of losing count coverage. +- Bounded generated target names to 63 ASCII bytes, reject oversized explicit names and connection query overrides, escape catalog-derived table identifiers, and preserve row counts for special property names. +- Reserve backup files exclusively and clean up only resources created by the drill. Preserve restore and cleanup errors together while continuing other cleanup. Redact connection secrets from diagnostics, including CLI argument errors. +- Verify append-only protection and valid, ready HNSW indexes on their specific public-schema relations. Added database-boundary and disposable-file regressions for native/Docker custom/plain orchestration and failure paths; live integration remains opt-in and was not run against an existing database. + diff --git a/docs/runbooks/backup-restore-drill.md b/docs/runbooks/backup-restore-drill.md index f5ea86c3..1bae8b0a 100644 --- a/docs/runbooks/backup-restore-drill.md +++ b/docs/runbooks/backup-restore-drill.md @@ -63,6 +63,19 @@ The script will automatically: | `--json` | `false` | Output drill report in JSON format | | `--help` | `false` | Show help message | +The backup path must not already exist. The drill reserves a new file exclusively +and removes only the file it created. Target names are limited to 63 ASCII bytes; +generated names retain their isolation marker and unique suffix within that limit. +Connection URL query parameters are restricted to `sslmode`, `sslcert`, `sslkey`, +and `sslrootcert`, so the JavaScript client and PostgreSQL tools use the same +host, port, and credentials. + +Both dump formats use the source transaction's exported snapshot. Snapshot export +or row-count failures abort the drill. Every nonzero restore exit is fatal, +including warning-only or localized diagnostics. Cleanup is attempted after a +failure; a failed database drop or backup removal also fails the drill, and combined +restore and cleanup failures are reported together. + ### 2.3 Retaining the Target Database for Forensic Inspection When diagnosing schema discrepancies or inspecting restore behavior, instruct the drill to keep the restored database: @@ -214,7 +227,14 @@ DROP DATABASE "gambit_backup_drill_restore_" WITH (FORCE); 1. **Never Overwrite Source:** The drill script explicitly compares the normalized source and target URLs. If the host, port, and database name match, execution aborts immediately. 2. **Protected Databases:** Destructive commands will refuse to drop databases named `gambit`, `postgres`, `template1`, `production`, `master`, or `main`. 3. **Naming Convention:** Target databases must contain an isolation indicator (`drill`, `restore`, `disposable`, `test`, or `isolated`) unless the `--allow-custom-target-name` flag is explicitly provided. -4. **Credential Security:** Database URLs are masked in all logs and console output (`postgres://user:***@host:port/db`). Passwords are passed via environment variables (`PGPASSWORD`), never exposed in `ps` process arguments. +4. **Credential Security:** Database URLs and known connection passwords are masked in diagnostics. PostgreSQL tool subprocesses receive passwords through `PGPASSWORD`. Prefer setting `DATABASE_URL` when invoking the drill; a URL supplied through `--source-url` or `--target-url` is still visible in the drill process's arguments. + +The structural checks require the append-only trigger on `public.game_events` +and a valid, ready HNSW index on `public.search_embeddings`; identically named +objects on another relation do not satisfy verification. The tests in +`scripts/test/backup-restore-safety.test.mjs` exercise orchestration and failure +paths without contacting a database. The separate live integration test remains +opt-in through `DATABASE_URL`; use a disposable, migrated database for that test. --- diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index 2d0f4896..b16a783a 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -112,6 +112,10 @@ export function sanitizeDatabaseUrl(urlString) { if (parsed.password) { parsed.password = '***'; } + for (const key of parsed.searchParams.keys()) { + if (/password|secret|token/i.test(key)) parsed.searchParams.set(key, '***'); + } + parsed.hash = ''; return parsed.toString(); } catch { // Regex fallback for partially malformed or non-standard connection strings @@ -119,24 +123,48 @@ export function sanitizeDatabaseUrl(urlString) { } } +/** Redact known connection secrets and embedded URLs from process diagnostics. */ +function sanitizeDiagnostic(message, urls) { + let result = String(message).replace(/postgres(?:ql)?:\/\/[^\s]+/gi, sanitizeDatabaseUrl); + const secrets = new Set(); + for (const url of urls) { + try { + const parsed = new URL(url); + if (parsed.password) { + secrets.add(parsed.password); + secrets.add(decodeURIComponent(parsed.password)); + } + for (const [key, value] of parsed.searchParams) { + if (value && /password|secret|token/i.test(key)) secrets.add(value); + } + } catch { /* Invalid URLs are handled by argument validation. */ } + } + for (const secret of [...secrets].sort((a, b) => b.length - a.length)) { + result = result.replaceAll(secret, '***'); + } + return result; +} + /** * Generate a unique, isolated database name for the restore drill. */ export function generateIsolatedDbName(base = 'gambit') { const timestamp = Date.now(); const random = Math.floor(Math.random() * 0xffffffff).toString(16).padStart(8, '0'); - return `${base}_backup_drill_restore_${timestamp}_${random}`; + const suffix = `_backup_drill_restore_${timestamp}_${random}`; + const prefix = base.replace(/[^a-zA-Z0-9_-]/g, '_') || 'gambit'; + return `${prefix.slice(0, 63 - suffix.length)}${suffix}`; } export function parseDatabaseUrl(urlString) { const parsed = new URL(urlString); const dbName = parsed.pathname.replace(/^\//, '') || 'postgres'; const sslParams = {}; + const supportedQueryParams = new Set(['sslmode', 'sslcert', 'sslkey', 'sslrootcert']); for (const [key, value] of parsed.searchParams.entries()) { - if (key.startsWith('ssl')) { - const envKey = 'PG' + key.toUpperCase(); - sslParams[envKey] = value; - } + if (!supportedQueryParams.has(key)) throw new Error('Unsupported database URL query parameter; only sslmode, sslcert, sslkey, and sslrootcert are allowed'); + const envKey = 'PG' + key.toUpperCase(); + sslParams[envKey] = value; } return { host: parsed.hostname || 'localhost', @@ -176,6 +204,9 @@ export async function validateTargetIsolation(sourceUrl, targetUrl, options = {} if (!/^[a-zA-Z0-9_-]+$/.test(target.database)) { throw new Error('Invalid target database name format'); } + if (Buffer.byteLength(target.database, 'utf8') > 63) { + throw new Error('Target database name must not exceed 63 bytes'); + } // 1. URLs must not be identical if (sourceUrl.trim() === targetUrl.trim()) { @@ -371,24 +402,9 @@ export function resolvePgTooling(options = {}) { } /** - * Parse pg_restore errors to distinguish harmless drop-if-not-exists warnings from real failures. + * Reject every failed pg_restore invocation, regardless of diagnostic language or content. */ export function parsePgRestoreError(err) { - const status = err?.status ?? err?.code; - if (status === 1 || err?.message?.includes('exit code 1')) { - const stderrStr = err?.stderr ? err.stderr.toString().trim() : ''; - if (stderrStr) { - const lines = stderrStr.split('\n'); - const hasRealError = lines.some(line => { - return line.includes('pg_restore: error:'); - }); - if (hasRealError) { - throw err; - } - return stderrStr; - } - return ''; - } throw err; } @@ -425,14 +441,10 @@ export async function collectSourceBaseline(pool, existingClient = null) { const tables = tableRes.rows.map((r) => r.tablename); // 4. Row counts across critical tables - const rowCounts = {}; + const rowCounts = Object.create(null); for (const table of tables) { - try { - const countRes = await client.query(`SELECT COUNT(*) AS count FROM "${table}"`); - rowCounts[table] = parseInt(countRes.rows[0].count, 10); - } catch { - // Non-readable table or view - } + const countRes = await client.query(`SELECT COUNT(*) AS count FROM "${table.replace(/"/g, '""')}"`); + rowCounts[table] = parseInt(countRes.rows[0].count, 10); } // 5. Sample records for integrity comparison @@ -589,7 +601,7 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options // Check 4: Row Counts for (const [table, expectedCount] of Object.entries(sourceBaseline.rowCounts)) { if (!targetTables.has(table)) continue; - const countRes = await targetPool.query(`SELECT COUNT(*) AS count FROM "${table}"`); + const countRes = await targetPool.query(`SELECT COUNT(*) AS count FROM "${table.replace(/"/g, '""')}"`); const actualCount = parseInt(countRes.rows[0].count, 10); if (actualCount !== expectedCount) { const msg = `Row count mismatch for table "${table}": source had ${expectedCount}, restored has ${actualCount}`; @@ -635,14 +647,14 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options } else { // Table was empty; verify trigger registration in pg_trigger catalog const trigRes = await client.query( - "SELECT tgname FROM pg_trigger WHERE tgname = 'game_events_block_mutate' AND tgenabled = 'O'", + "SELECT tgname FROM pg_trigger WHERE tgrelid = 'public.game_events'::regclass AND tgname = 'game_events_block_mutate' AND tgenabled = 'O'", ); triggerActive = trigRes.rows.length > 0; } } catch (err) { if ( err.message && - (err.message.includes('game_events is append-only') || err.code === 'P0001') + err.message.includes('game_events is append-only') && err.code === 'P0001' ) { triggerActive = true; } else { @@ -669,15 +681,15 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options await targetPool.query('SELECT $1::vector(256) <=> $1::vector(256) AS dist', [testVec]); const idxRes = await targetPool.query(` - SELECT i.relname AS index_name, am.amname AS access_method + SELECT i.relname AS index_name, am.amname AS access_method, ix.indisvalid, ix.indisready FROM pg_index ix JOIN pg_class i ON i.oid = ix.indexrelid JOIN pg_class t ON t.oid = ix.indrelid JOIN pg_am am ON i.relam = am.oid - WHERE t.relname = 'search_embeddings' AND am.amname = 'hnsw' + WHERE t.oid = 'public.search_embeddings'::regclass AND am.amname = 'hnsw' `); - if (idxRes.rows.length === 0) { - throw new Error('HNSW index missing on search_embeddings table'); + if (!idxRes.rows.some(index => index.indisvalid === true && index.indisready === true)) { + throw new Error('Valid and ready HNSW index missing on search_embeddings table'); } recordCheck('pgvector Functionality', true, 'Vector cosine operator (<=>) and HNSW index functional'); @@ -776,7 +788,7 @@ export async function runBackupRestoreDrill(options = {}) { const parsedSource = parseDatabaseUrl(options.sourceUrl); const targetUrl = options.targetUrl || urlWithDatabase( options.sourceUrl, - `${parsedSource.database}_backup_drill_restore_${Date.now()}_${Math.floor(Math.random() * 0xffff).toString(16)}`, + options.targetDbName || generateIsolatedDbName(parsedSource.database), ); const parsedTarget = parseDatabaseUrl(targetUrl); @@ -792,6 +804,9 @@ export async function runBackupRestoreDrill(options = {}) { let targetPool = null; let adminClient = null; let targetCreatedByThisRun = false; + let backupCreatedByThisRun = false; + let drillError = null; + const cleanupErrors = []; const rawBackupPath = options.backupFile || @@ -803,35 +818,38 @@ export async function runBackupRestoreDrill(options = {}) { const log = (msg) => { if (!options.json) { - console.log(`[drill] ${msg}`); + console.log(`[drill] ${sanitizeDiagnostic(msg, [options.sourceUrl, targetUrl])}`); } }; try { + // Reserve the destination exclusively before any tool can overwrite it. + const backupFd = openSync(backupPath, 'wx', 0o600); + backupCreatedByThisRun = true; + closeSync(backupFd); + const tooling = resolvePgTooling(options); + const isCustom = (options.format || 'custom') === 'custom'; // 1. Capture source baseline and export snapshot log(`Capturing source baseline from ${report.source}...`); const baselineClient = await sourcePool.connect(); let snapshotId = null; let sourceBaseline = null; + let baselineError = null; + const baselineCleanupErrors = []; try { await baselineClient.query('BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY'); - try { - const snapRes = await baselineClient.query('SELECT pg_export_snapshot() AS snap'); - snapshotId = snapRes.rows[0]?.snap || null; - } catch { - // Snapshot export might not be supported in some test or replica configurations - } + const snapRes = await baselineClient.query('SELECT pg_export_snapshot() AS snap'); + snapshotId = snapRes.rows[0]?.snap; + if (!snapshotId) throw new Error('Source did not provide an exported snapshot; refusing an inconsistent drill'); sourceBaseline = await collectSourceBaseline(sourcePool, baselineClient); log(`Source baseline captured: ${sourceBaseline.tables.length} tables, ${sourceBaseline.migrations.length} migrations.`); // 2. Resolve tooling - const tooling = resolvePgTooling(options); log(`Resolved PostgreSQL tooling (${tooling.type})...`); // 3. Perform backup using pg_dump log(`Creating ${options.format || 'custom'} backup to ${backupPath}...`); const dumpStart = Date.now(); - const isCustom = (options.format || 'custom') === 'custom'; if (tooling.type === 'native') { const dumpArgs = [ @@ -844,7 +862,7 @@ export async function runBackupRestoreDrill(options = {}) { '--no-acl', '-f', backupPath, ]; - if (snapshotId && isCustom) { + if (snapshotId) { dumpArgs.push(`--snapshot=${snapshotId}`); } tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }); @@ -865,16 +883,30 @@ export async function runBackupRestoreDrill(options = {}) { '--no-acl', '-f', `/work/${backupFileName}`, ]; - if (snapshotId && isCustom) { + if (snapshotId) { dumpArgs.push(`--snapshot=${snapshotId}`); } tooling.runDump(dumpArgs, { PGPASSWORD: parsedSource.password, ...parsedSource.sslParams }, backupDir); } report.timings.backupMs = Date.now() - dumpStart; log(`Backup completed in ${report.timings.backupMs}ms.`); + } catch (err) { + baselineError = err; } finally { - await baselineClient.query('ROLLBACK').catch(() => {}); - baselineClient.release(); + await baselineClient.query('ROLLBACK').catch(err => baselineCleanupErrors.push(err)); + try { + baselineClient.release(); + } catch (err) { + baselineCleanupErrors.push(err); + } + } + const baselineErrors = [...(baselineError ? [baselineError] : []), ...baselineCleanupErrors]; + if (baselineErrors.length === 1) throw baselineErrors[0]; + if (baselineErrors.length > 1) { + throw new AggregateError( + baselineErrors, + 'Source baseline, dump, or transaction cleanup failed: ' + baselineErrors.map(error => error?.message || String(error)).join('; '), + ); } // 4. Validate backup file @@ -914,13 +946,13 @@ export async function runBackupRestoreDrill(options = {}) { '--if-exists', '--no-owner', '--no-acl', + '--exit-on-error', backupPath, ]; try { tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }); } catch (err) { - const notice = parsePgRestoreError(err); - if (notice) log(`pg_restore notice: ${notice}`); + parsePgRestoreError(err); } } else { const psqlArgs = [ @@ -950,13 +982,13 @@ export async function runBackupRestoreDrill(options = {}) { '--if-exists', '--no-owner', '--no-acl', + '--exit-on-error', `/work/${backupFileName}`, ]; try { tooling.runRestore(restoreArgs, { PGPASSWORD: parsedTarget.password, ...parsedTarget.sslParams }, backupDir); } catch (err) { - const notice = parsePgRestoreError(err); - if (notice) log(`pg_restore notice: ${notice}`); + parsePgRestoreError(err); } } else { const psqlArgs = [ @@ -983,11 +1015,13 @@ export async function runBackupRestoreDrill(options = {}) { log(`Verification passed: ${verifyResult.checks.length} checks succeeded.`); report.success = true; + } catch (err) { + drillError = err; } finally { // Teardown connections - await sourcePool.end().catch(() => {}); + await sourcePool.end().catch(err => cleanupErrors.push(err)); if (targetPool) { - await targetPool.end().catch(() => {}); + await targetPool.end().catch(err => cleanupErrors.push(err)); } // Teardown target database if not keeping @@ -998,22 +1032,31 @@ export async function runBackupRestoreDrill(options = {}) { await adminClient.query(`DROP DATABASE IF EXISTS "` + parsedTarget.database.replace(/"/g, '""') + `" WITH (FORCE)`); log('Target database dropped.'); } catch (err) { - log(`Warning: Failed to drop isolated target database: ${err.message}`); - report.success = false; - throw err; + cleanupErrors.push(err); } } - await adminClient.end().catch(() => {}); + await adminClient.end().catch(err => cleanupErrors.push(err)); } // Remove backup file if not keeping - if (!options.keepBackup && existsSync(backupPath)) { + if (!options.keepBackup && backupCreatedByThisRun && existsSync(backupPath)) { try { unlinkSync(backupPath); - } catch {} + } catch (err) { + cleanupErrors.push(err); + } } } + const errors = [...(drillError ? [drillError] : []), ...cleanupErrors]; + if (errors.length === 1) throw errors[0]; + if (errors.length > 1) { + throw new AggregateError( + errors, + 'Backup/restore drill and cleanup failures: ' + errors.map(error => error?.message || String(error)).join('; '), + ); + } + report.timings.totalMs = Date.now() - startTime; report.completedAt = new Date().toISOString(); @@ -1025,7 +1068,13 @@ const isDirectExecution = process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url); if (isDirectExecution) { - const args = parseArgs(process.argv.slice(2)); + let args; + try { + args = parseArgs(process.argv.slice(2)); + } catch { + console.error('Invalid drill arguments. Check database URLs and use --format custom or plain.'); + process.exit(1); + } if (args.help || !args.sourceUrl) { console.log(` @@ -1077,8 +1126,7 @@ Options: console.error('\n========================================'); console.error('❌ BACKUP & RESTORE DRILL FAILED'); console.error('========================================'); - console.error(`Error: ${err.message}`); - if (err.stack) console.error(err.stack); + console.error(`Error: ${sanitizeDiagnostic(err.message, [args.sourceUrl, args.targetUrl])}`); console.error('========================================\n'); process.exit(1); }); diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index 0ef7a731..51a045cd 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -525,7 +525,7 @@ test('verification engine: passes when all structural and functional checks succ throw error; } if (text.includes('search_embeddings')) { - return { rows: [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw' }] }; + return { rows: [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw', indisvalid: true, indisready: true }] }; } return { rows: [] }; }, @@ -588,7 +588,7 @@ test('tooling: resolvePgTooling accepts options and detects native or docker run assert.ok(plainTooling.type === 'native' || plainTooling.type === 'docker'); }); -test('pg_restore: parsePgRestoreError treats non-error warnings as benign', () => { +test('pg_restore: parsePgRestoreError rejects nonzero status even with warning-only diagnostics', () => { const err = new Error('Command failed: pg_restore exit code 1'); err.status = 1; err.stderr = Buffer.from( @@ -596,8 +596,7 @@ test('pg_restore: parsePgRestoreError treats non-error warnings as benign', () = 'pg_restore: warning: could not execute query: ERROR: schema "public" does not exist' ); - const notice = parsePgRestoreError(err); - assert.ok(notice.includes('does not exist')); + assert.throws(() => parsePgRestoreError(err), error => error === err); }); test('pg_restore: parsePgRestoreError throws on real errors including missing objects', () => { diff --git a/scripts/test/backup-restore-safety.test.mjs b/scripts/test/backup-restore-safety.test.mjs new file mode 100644 index 00000000..e37185a6 --- /dev/null +++ b/scripts/test/backup-restore-safety.test.mjs @@ -0,0 +1,286 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, writeFileSync, readFileSync, existsSync, rmSync, mkdirSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { basename, join } from 'node:path'; +import pg from 'pg'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; +import { + runBackupRestoreDrill, + CRITICAL_APPLICATION_TABLES, + generateIsolatedDbName, + validateTargetIsolation, + collectSourceBaseline, + sanitizeDatabaseUrl, +} from '../db-backup-restore-drill.mjs'; + +/** + * Build a disposable drill harness that replaces only database and subprocess + * boundaries while exercising the real validation, orchestration, and cleanup. + */ +function drillFixture(t, behavior = {}) { + const directory = mkdtempSync(join(tmpdir(), 'backup-drill-test-')); + t.after(() => rmSync(directory, { recursive: true, force: true })); + const backupFile = join(directory, 'backup.dump'); + const events = []; + /** Return complete catalog/data responses and record every SQL side effect. */ + const databaseQuery = async (sql) => { + events.push(sql); + if (sql === 'ROLLBACK' && behavior.rollbackError) throw behavior.rollbackError; + if (sql.includes('pg_export_snapshot')) return { rows: behavior.missingSnapshot ? [] : [{ snap: '00000001-00000001-1' }] }; + if (sql.includes('pg_extension')) return { rows: [{ extname: 'citext' }, { extname: 'vector' }] }; + if (sql.includes('to_regclass')) return { rows: [{ present: false }] }; + if (sql.includes('pg_tables')) return { rows: CRITICAL_APPLICATION_TABLES.map(tablename => ({ tablename })) }; + if (sql.includes('COUNT(*)')) return { rows: [{ count: '0' }] }; + if (sql.includes('UPDATE game_events') && behavior.unrelatedTriggerError) throw Object.assign(new Error('unrelated trigger failure'), { code: 'P0001' }); + if (sql.includes('pg_trigger')) return { rows: behavior.otherTriggerRelation && sql.includes("tgrelid = 'public.game_events'::regclass") ? [] : [{ tgname: 'game_events_block_mutate' }] }; + if (sql.includes('pg_index')) return { rows: behavior.otherIndexRelation && sql.includes("t.oid = 'public.search_embeddings'::regclass") ? [] : [{ index_name: 'search_embeddings_hnsw_idx', access_method: 'hnsw', indisvalid: !behavior.invalidIndex, indisready: true }] }; + return { rows: [], rowCount: 0 }; + }; + t.mock.method(pg.Pool.prototype, 'connect', async () => { + if (behavior.connectError) throw behavior.connectError; + return { + query: databaseQuery, + release() { + events.push('release'); + if (behavior.releaseError) throw behavior.releaseError; + }, + }; + }); + t.mock.method(pg.Pool.prototype, 'query', databaseQuery); + t.mock.method(pg.Pool.prototype, 'end', async () => { events.push('pool.end'); }); + t.mock.method(pg.Client.prototype, 'connect', async () => { events.push('admin.connect'); }); + t.mock.method(pg.Client.prototype, 'end', async () => { events.push('admin.end'); }); + t.mock.method(pg.Client.prototype, 'query', async (sql) => { + events.push(sql); + if (sql.startsWith('CREATE DATABASE') && behavior.createError) throw behavior.createError; + if (sql.startsWith('DROP DATABASE') && behavior.dropError) throw behavior.dropError; + return { rows: [] }; + }); + /** Simulate PostgreSQL tooling while materializing a real disposable dump file. */ + const execSyncFn = (command, args) => { + if (args.includes('--version')) return Buffer.from('test tooling'); + const executable = command === 'docker' ? args.find(arg => ['pg_dump', 'pg_restore', 'psql'].includes(arg)) : command; + events.push({ executable, args }); + if (executable === 'pg_dump') { + const dumpPath = args[args.indexOf('-f') + 1]; + writeFileSync(command === 'docker' ? join(directory, basename(dumpPath)) : dumpPath, 'PGDMP test archive'); + } else if (behavior.restoreError) { + throw behavior.restoreError; + } else if (behavior.backupCleanupError) { + rmSync(backupFile); + mkdirSync(backupFile); + } + return Buffer.alloc(0); + }; + return { + events, backupFile, + options: { + sourceUrl: 'postgres://u:test-password@127.0.0.1/source_test', + targetUrl: 'postgres://u:test-password@127.0.0.1/isolated_restore_test', + backupFile, json: true, execSyncFn, + }, + }; +} + +for (const stderr of ['', 'pg_restore: warning: errors ignored on restore: 1', 'pg_restore: erreur: restauration incomplète']) { + test(`restore errors: nonzero exit fails closed with diagnostics ${JSON.stringify(stderr)}`, async t => { + const restoreError = Object.assign(new Error('restore command failed'), { status: 1, stderr: Buffer.from(stderr) }); + const fixture = drillFixture(t, { restoreError }); + await assert.rejects(runBackupRestoreDrill(fixture.options), error => error === restoreError); + assert.equal(existsSync(fixture.backupFile), false); + assert.ok(fixture.events.some(event => typeof event === 'string' && event.startsWith('DROP DATABASE'))); + }); +} + +test('identifiers: generated names retain isolation and uniqueness suffix within 63 bytes', () => { + for (const base of ['x'.repeat(63), '測試'.repeat(32), 'strange " source']) { + const name = generateIsolatedDbName(base); + assert.ok(Buffer.byteLength(name) <= 63, `generated identifier has ${Buffer.byteLength(name)} bytes`); + assert.match(name, /^[a-zA-Z0-9_-]+_backup_drill_restore_\d+_[a-f0-9]{8}$/); + } +}); + +test('ownership: an existing backup file survives an early failed drill', async t => { + const fixture = drillFixture(t, { connectError: new Error('source unavailable') }); + writeFileSync(fixture.backupFile, 'operator-owned backup'); + await assert.rejects(runBackupRestoreDrill(fixture.options)); + assert.equal(existsSync(fixture.backupFile), true); + assert.equal(readFileSync(fixture.backupFile, 'utf8'), 'operator-owned backup'); +}); + +test('ownership: an existing backup file is refused before it can be overwritten', async t => { + const fixture = drillFixture(t); + writeFileSync(fixture.backupFile, 'operator-owned backup'); + await assert.rejects(runBackupRestoreDrill(fixture.options), /exist/i); + assert.equal(readFileSync(fixture.backupFile, 'utf8'), 'operator-owned backup'); + assert.equal(fixture.events.some(event => event.executable === 'pg_dump'), false); +}); + +test('ownership: CREATE failure never drops a target this drill does not own', async t => { + const createError = new Error('database already exists'); + const fixture = drillFixture(t, { createError }); + await assert.rejects(runBackupRestoreDrill(fixture.options), error => error === createError); + assert.equal(fixture.events.some(event => typeof event === 'string' && event.startsWith('DROP DATABASE')), false); + assert.equal(existsSync(fixture.backupFile), false); +}); + +test('cleanup: drop failure preserves the restore failure and still closes admin and removes backup', async t => { + const restoreError = new Error('restore failed'); + const dropError = new Error('drop failed'); + const fixture = drillFixture(t, { restoreError, dropError }); + await assert.rejects(runBackupRestoreDrill(fixture.options), error => { + assert.ok(error instanceof AggregateError); + assert.deepEqual(error.errors, [restoreError, dropError]); + return true; + }); + assert.ok(fixture.events.includes('admin.end')); + assert.equal(existsSync(fixture.backupFile), false); +}); + +test('cleanup: baseline rollback and release failures do not mask the dump failure or skip outer cleanup', async t => { + const dumpError = new Error('dump failed'); + const rollbackError = new Error('rollback failed'); + const releaseError = new Error('release failed'); + const fixture = drillFixture(t, { restoreError: dumpError, rollbackError, releaseError }); + const options = { + ...fixture.options, + execSyncFn(command, args) { + if (args.includes('--version')) return Buffer.from('test tooling'); + if (command === 'pg_dump') throw dumpError; + return Buffer.alloc(0); + }, + }; + + await assert.rejects(runBackupRestoreDrill(options), error => { + assert.ok(error instanceof AggregateError); + assert.deepEqual(error.errors, [dumpError, rollbackError, releaseError]); + return true; + }); + assert.ok(fixture.events.includes('pool.end')); + assert.equal(existsSync(fixture.backupFile), false); +}); + +test('cleanup: backup removal failure prevents a success report', async t => { + const fixture = drillFixture(t, { backupCleanupError: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /directory|EPERM|EISDIR/i); + assert.ok(fixture.events.includes('admin.end')); +}); + +test('snapshot: plain dumps use the same exported snapshot as the baseline', async t => { + const fixture = drillFixture(t); + await runBackupRestoreDrill({ ...fixture.options, format: 'plain' }); + const dump = fixture.events.find(event => event.executable === 'pg_dump'); + assert.ok(dump.args.includes('--snapshot=00000001-00000001-1')); +}); + +test('snapshot: no exported snapshot aborts before creating a dump or target', async t => { + const fixture = drillFixture(t, { missingSnapshot: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /snapshot/i); + assert.equal(fixture.events.some(event => event.executable === 'pg_dump'), false); + assert.ok(fixture.events.includes('release')); +}); + +test('connection: query overrides cannot send pg clients and tools to different servers', async () => { + const source = 'postgres://u:p@127.0.0.1/source_test'; + for (const query of ['host=elsewhere', 'port=5433', 'user=other', 'password=query-secret', 'options=-csearch_path=other']) { + await assert.rejects(validateTargetIsolation(source, `postgres://u:p@127.0.0.1/restore_test?${query}`), /parameter/i); + } +}); + +test('security: URL diagnostics redact query passwords and fragments', () => { + const result = sanitizeDatabaseUrl('postgres://u:authority-secret@127.0.0.1/source_test?password=query-secret#fragment-secret'); + assert.doesNotMatch(result, /authority-secret|query-secret|fragment-secret/); +}); + +test('security: CLI never echoes credentials from malformed arguments', () => { + const script = fileURLToPath(new URL('../db-backup-restore-drill.mjs', import.meta.url)); + const result = spawnSync(process.execPath, [script, '--format', 'postgres://u:cli-regression-secret@invalid/db'], { + encoding: 'utf8', env: { ...process.env, DATABASE_URL: '', BACKUP_DRILL_SOURCE_URL: '', BACKUP_DRILL_TARGET_URL: '' }, + }); + assert.equal(result.status, 1); + assert.doesNotMatch(result.stdout + result.stderr, /cli-regression-secret/); +}); + +test('security: CLI failure diagnostics redact known connection passwords outside URLs', () => { + const script = fileURLToPath(new URL('../db-backup-restore-drill.mjs', import.meta.url)); + const result = spawnSync(process.execPath, [script, + '--source-url', 'postgres://u:production@127.0.0.1/source_test', + '--target-url', 'postgres://u:production@127.0.0.1/production', + ], { encoding: 'utf8' }); + assert.equal(result.status, 1); + assert.doesNotMatch(result.stdout + result.stderr, /production/); +}); + +test('baseline: unreadable table fails instead of silently omitting its row count', async () => { + const failure = new Error('permission denied for table'); + const client = { + async query(sql) { + if (sql.includes('pg_extension')) return { rows: [] }; + if (sql.includes('to_regclass')) return { rows: [{ present: false }] }; + if (sql.includes('pg_tables')) return { rows: [{ tablename: 'durable_data' }] }; + if (sql.includes('COUNT(*)')) throw failure; + throw new Error(`unexpected query ${sql}`); + }, + }; + await assert.rejects(collectSourceBaseline(null, client), error => error === failure); +}); + +for (const table of ['with"quote', '__proto__']) test(`baseline: ${table} preserves its exact row count`, async () => { + const client = { + async query(sql) { + if (sql.includes('pg_extension')) return { rows: [] }; + if (sql.includes('to_regclass')) return { rows: [{ present: false }] }; + if (sql.includes('pg_tables')) return { rows: [{ tablename: table }] }; + if (sql === 'SELECT COUNT(*) AS count FROM "with""quote"') return { rows: [{ count: '3' }] }; + if (sql === 'SELECT COUNT(*) AS count FROM "__proto__"') return { rows: [{ count: '7' }] }; + throw new Error('invalid quoted identifier'); + }, + }; + const baseline = await collectSourceBaseline(null, client); + assert.equal(Object.hasOwn(baseline.rowCounts, table), true); + assert.equal(baseline.rowCounts[table], table === '__proto__' ? 7 : 3); +}); + +test('verification: an unrelated P0001 error does not prove append-only protection', async t => { + const fixture = drillFixture(t, { unrelatedTriggerError: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /unrelated trigger failure/); +}); + +test('verification: an invalid HNSW index cannot pass verification', async t => { + const fixture = drillFixture(t, { invalidIndex: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /HNSW.*index/i); +}); + +test('verification: a trigger on another relation cannot protect empty public.game_events', async t => { + const fixture = drillFixture(t, { otherTriggerRelation: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /Append-only trigger/); +}); + +test('verification: an index on another schema cannot verify public.search_embeddings', async t => { + const fixture = drillFixture(t, { otherIndexRelation: true }); + await assert.rejects(runBackupRestoreDrill(fixture.options), /HNSW.*index/i); +}); + +test('identifiers: explicit target cannot rely on PostgreSQL identifier truncation', async () => { + const source = 'postgres://u:p@127.0.0.1/source_test'; + await assert.rejects(validateTargetIsolation(source, `postgres://u:p@127.0.0.1/${'x'.repeat(63)}_restore`, { allowCustomTargetName: true }), /63/); + const boundary = 'restore_' + 'x'.repeat(55); + assert.equal((await validateTargetIsolation(source, `postgres://u:p@127.0.0.1/${boundary}`)).targetDbName, boundary); +}); + +for (const format of ['custom', 'plain']) { + for (const useDocker of [false, true]) { + test(`orchestration: ${format} restore completes with ${useDocker ? 'docker' : 'native'} tooling`, async t => { + const fixture = drillFixture(t); + const report = await runBackupRestoreDrill({ ...fixture.options, format, useDocker }); + assert.equal(report.success, true); + const restore = fixture.events.find(event => event.executable === (format === 'custom' ? 'pg_restore' : 'psql')); + assert.ok(restore); + if (format === 'custom') assert.ok(restore.args.includes('--exit-on-error')); + assert.ok(fixture.events.some(event => typeof event === 'string' && event.startsWith('DROP DATABASE'))); + assert.equal(existsSync(fixture.backupFile), false); + }); + } +} From cad265912d8eaceddb8f3aff8f55c8626b8d693b Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Tue, 8 Sep 2026 14:46:24 +0300 Subject: [PATCH 11/14] style(drill): remove trailing whitespace --- scripts/db-backup-restore-drill.mjs | 6 +++--- scripts/test/backup-restore-drill.test.mjs | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/scripts/db-backup-restore-drill.mjs b/scripts/db-backup-restore-drill.mjs index b16a783a..bd03e0a5 100644 --- a/scripts/db-backup-restore-drill.mjs +++ b/scripts/db-backup-restore-drill.mjs @@ -200,7 +200,7 @@ export async function validateTargetIsolation(sourceUrl, targetUrl, options = {} const source = parseDatabaseUrl(sourceUrl); const target = parseDatabaseUrl(targetUrl); - + if (!/^[a-zA-Z0-9_-]+$/.test(target.database)) { throw new Error('Invalid target database name format'); } @@ -679,7 +679,7 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options // Test vector operator syntax and index validity const testVec = `[${new Array(256).fill(0.1).join(',')}]`; await targetPool.query('SELECT $1::vector(256) <=> $1::vector(256) AS dist', [testVec]); - + const idxRes = await targetPool.query(` SELECT i.relname AS index_name, am.amname AS access_method, ix.indisvalid, ix.indisready FROM pg_index ix @@ -691,7 +691,7 @@ export async function verifyRestoredDatabase(sourceBaseline, targetPool, options if (!idxRes.rows.some(index => index.indisvalid === true && index.indisready === true)) { throw new Error('Valid and ready HNSW index missing on search_embeddings table'); } - + recordCheck('pgvector Functionality', true, 'Vector cosine operator (<=>) and HNSW index functional'); } catch (err) { const msg = `Vector functionality verification failed: ${err.message}`; diff --git a/scripts/test/backup-restore-drill.test.mjs b/scripts/test/backup-restore-drill.test.mjs index 51a045cd..11ffa89d 100644 --- a/scripts/test/backup-restore-drill.test.mjs +++ b/scripts/test/backup-restore-drill.test.mjs @@ -129,7 +129,7 @@ test('isolation: generateIsolatedDbName generates prefixed unique name', async ( test('isolation: validateTargetIsolation rejects malicious injection payloads in target URL', async () => { const source = 'postgres://gambit:pass@localhost:5432/gambit'; const maliciousTarget = 'postgres://gambit:pass@localhost:5432/test" OR 1=1; DROP DATABASE production; --'; - + await assert.rejects( async () => await validateTargetIsolation(source, maliciousTarget, { allowCustomTargetName: true }), /Invalid target database name format/ @@ -577,7 +577,7 @@ test('cli: parseArgs defaults to safe isolated target when not specified', () => test('tooling: resolvePgTooling accepts options and detects native or docker runner', () => { const mockExec = (cmd, args) => { return Buffer.from('mock version'); }; - + const customTooling = resolvePgTooling({ format: 'custom', execSyncFn: mockExec }); assert.ok(customTooling.type === 'native' || customTooling.type === 'docker'); assert.equal(typeof customTooling.runDump, 'function'); @@ -595,7 +595,7 @@ test('pg_restore: parsePgRestoreError rejects nonzero status even with warning-o 'pg_restore: warning: errors ignored on restore: 1\n' + 'pg_restore: warning: could not execute query: ERROR: schema "public" does not exist' ); - + assert.throws(() => parsePgRestoreError(err), error => error === err); }); @@ -606,7 +606,7 @@ test('pg_restore: parsePgRestoreError throws on real errors including missing ob 'pg_restore: warning: errors ignored on restore: 1\n' + 'pg_restore: error: could not execute query: ERROR: role "postgres" does not exist' ); - + assert.throws(() => parsePgRestoreError(err), /Command failed/); }); From f1270c4fd885c6cdcf8583d2385c3997550d3b17 Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Tue, 8 Sep 2026 20:45:44 +0300 Subject: [PATCH 12/14] docs(db): harden manual HNSW verification --- docs/runbooks/backup-restore-drill.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/backup-restore-drill.md b/docs/runbooks/backup-restore-drill.md index 1bae8b0a..6633228d 100644 --- a/docs/runbooks/backup-restore-drill.md +++ b/docs/runbooks/backup-restore-drill.md @@ -209,7 +209,15 @@ Verify pgvector cosine distance operations and HNSW indexing: -- Test cosine distance operator (<=>) SELECT id, embedding <=> embedding AS distance FROM search_embeddings LIMIT 1; -- Verify HNSW index -SELECT i.relname, am.amname FROM pg_index ix JOIN pg_class i ON i.oid = ix.indexrelid JOIN pg_class t ON t.oid = ix.indrelid JOIN pg_am am ON i.relam = am.oid WHERE t.relname = 'search_embeddings' AND am.amname = 'hnsw'; +SELECT i.relname, am.amname, ix.indisvalid, ix.indisready +FROM pg_index ix +JOIN pg_class i ON i.oid = ix.indexrelid +JOIN pg_class t ON t.oid = ix.indrelid +JOIN pg_am am ON i.relam = am.oid +WHERE t.oid = 'public.search_embeddings'::regclass + AND am.amname = 'hnsw' + AND ix.indisvalid = true + AND ix.indisready = true; ``` ### Step 6: Cleanup Isolated Target From e4b72b6145922c8ebf3c695e1dc978c2dd28ef9a Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Wed, 9 Sep 2026 00:58:38 +0300 Subject: [PATCH 13/14] docs(db): align manual restore verification --- docs/runbooks/backup-restore-drill.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/runbooks/backup-restore-drill.md b/docs/runbooks/backup-restore-drill.md index 6633228d..895e4cbc 100644 --- a/docs/runbooks/backup-restore-drill.md +++ b/docs/runbooks/backup-restore-drill.md @@ -171,11 +171,14 @@ SELECT extname, extversion FROM pg_extension WHERE extname IN ('citext', 'vector ``` #### 2. Schema Migrations Ledger -Verify that all migrations are present and checksums match the source: +Run the same query against the source and restored databases: ```sql -SELECT count(*) FROM schema_migrations WHERE state = 'applied'; --- Must match count of migration files in packages/persistence/migrations/ (e.g. 31) +SELECT version, name, checksum, state +FROM schema_migrations +ORDER BY version; ``` +The two ordered result sets must match exactly, and every row must have +`state = 'applied'`. #### 3. Append-Only Trigger Verification Verify that the `game_events` immutability trigger is active by testing an `UPDATE`: @@ -222,10 +225,14 @@ WHERE t.oid = 'public.search_embeddings'::regclass ### Step 6: Cleanup Isolated Target -After verification succeeds, drop the temporary drill database: +After verification succeeds, reconnect to the administrative `postgres` +database before dropping the temporary drill database. PostgreSQL cannot drop +the database used by the current session, even with `WITH (FORCE)`: -```sql -DROP DATABASE "gambit_backup_drill_restore_" WITH (FORCE); +```bash +psql -h "${PGHOST:-localhost}" -p "${PGPORT:-5432}" -U "${PGUSER:-gambit}" -d postgres -c " + DROP DATABASE \"${TARGET_DB}\" WITH (FORCE); +" ``` --- From 7b226180d9cb6b10f2d7a4ba0a15646b7626951d Mon Sep 17 00:00:00 2001 From: Hussein Mohamed Date: Wed, 9 Sep 2026 04:40:35 +0300 Subject: [PATCH 14/14] docs: require migrations ledger preflight --- docs/runbooks/backup-restore-drill.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/backup-restore-drill.md b/docs/runbooks/backup-restore-drill.md index 895e4cbc..c7e7341c 100644 --- a/docs/runbooks/backup-restore-drill.md +++ b/docs/runbooks/backup-restore-drill.md @@ -171,7 +171,15 @@ SELECT extname, extversion FROM pg_extension WHERE extname IN ('citext', 'vector ``` #### 2. Schema Migrations Ledger -Run the same query against the source and restored databases: +Confirm that the migrations ledger exists in both the source and restored +databases before comparing it: +```sql +SELECT to_regclass('public.schema_migrations') IS NOT NULL AS present; +``` +Both queries must return `present = true`. If either database is missing the +table, stop the drill and investigate before continuing. + +Then run the same query against the source and restored databases: ```sql SELECT version, name, checksum, state FROM schema_migrations