diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index b1e2c021..d6846cb5 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1349,7 +1349,7 @@ Debt observed during M14. Each states what is known, not what is planned; items - **The published capability document could omit a composed feature, and the client offered controls that could not work (RESOLVED in M15 Increment 24 / ADR-0132).** Left open by Increment 23 above. `capabilitiesView` (`packages/api/src/presenters.ts`) took a hand-written `Pick`, so a feature never added to it was invisible to `GET /v1/capabilities` and nothing complained. ADR-0131 judged that "a narrower failure than a 503 — the feature works for anyone who calls the route directly", and deferred it on the grounds that closing it meant first deciding which optional dependencies are user-facing capabilities. **Both halves of that were wrong.** The failure is narrower only when the client can still reach the feature, and a live instance already existed where it could not: `GET /v1/search` served three modes from two independently-gated dependency sets behind one published `search` flag, so a deployment running the Helm chart’s `search.semanticEnabled: false` advertised search while the client offered two mode buttons whose every request answered 503. And the judgement, while genuinely underivable, can be made **unskippable**, which was the property wanted. `semanticSearch` is now published from both of its dependencies, `Exclude[0] | NotAPublishedCapability>` must be `never` so a new optional dependency cannot compile until someone gives it a flag or records why it has none, and a behavioural guard requires every capability source to change what the document publishes — because a key can sit in that parameter unread, which the compile-time half cannot see. The mutation ledger lives in ADR-0132 and is not restated here. - **The search surface was not gated on `capabilities.search`, so an absolute kill switch still showed a search box (RESOLVED in M15 Increment 24 / ADR-0132 §5).** `SEARCH_ENABLED=0` — the chart's `search.enabled: false`, an absolute kill switch per ADR-0055 — leaves `searchRepository` unconstructed and `GET /v1/search` answering 503 on every mode, keyword included. The entry point was the persistent header form in `packages/web/index.html`, present on every page; being a `
` rather than an `a[data-route]`, `NAV_CAPABILITY_MAP` could not reach it, which is why the first pass at Increment 24 gated the semantic and hybrid *modes* and left keyword ungated — the same defect class one mode over. Raised by the Qodo review of PR #155. **Fixed in the same increment rather than deferred:** the form now ships `hidden` and is revealed by `applySearchCapability` only on an explicit `search: true`; the route renders an honest unavailable notice and issues no request; and keyword search waits for the capability answer, reversing this increment's own earlier latency decision, because knowing whether a request is pointless requires having asked. A markup-contract test pins the `hidden` attribute, since the gate depends on it and every other test passes without it. - **Clicking a search mode discarded text typed since the page loaded (RESOLVED in a follow-up to M15 Increment 24 / ADR-0132).** `createModeInput` in `packages/web/src/app/search-mount.ts` closed over the query captured when the route mounted, so `navigateToSearchMode` navigated with the old term and the remount reset the input to match it. Type a new term into the header field, click **Semantic** without pressing enter, and the typed text was gone with no indication it had been discarded. Pre-existing — the closure predated Increment 24 and was untouched by it — and found by the adversarial review of PR #155 while reviewing the capability gate wrapped around the same control. **Resolved:** the query is now a `() => string` read when a mode is chosen rather than a string captured when the selector renders, matching what `main.ts`'s submit handler already does, and falling back to the mounted query only where the document has no header input. Two regression tests, one of which fails against the exact pre-fix closure. -- **`startHarness` drew ports `fetch` refuses (RESOLVED in ADR-0140); the unexplained whole-file failure is still open.** Two signatures were filed here, deliberately not as one cause, and that judgement held. **Signature A is resolved.** WHATWG Fetch blocks eighty-two ports and undici enforces the list on the port number alone, before opening a socket — so `server.listen(0)` could bind, listen and answer raw TCP while `fetch` still refused, surfacing as `TypeError: fetch failed` / `Error: bad port` at the harness’s first request rather than at the listen that caused it. Whether it can happen at all is a property of the host’s dynamic port range: a typical Linux CI range (32768–60999) contains no blocked port, while the Windows range in use here (1024–15000) contains nineteen, which is the whole of "green on CI, flaky locally". The guard that already existed was incomplete in a way that still failed — its hand-observed set of eighteen ports was the spec list intersected with one machine’s range **minus `6679`** — and it retried unboundedly, had no behaviour on exhaustion, and had been copy-pasted into `auth-signin-schema.integration.test.ts`, so the missing port had to be found twice. `packages/api/test/listen.ts` now owns port acquisition for both sites: the spec-complete eighty-two ports (verified by sweeping all 65535 through the real `fetch` on Node v24.15.0), a bounded twenty attempts, each rejected listener closed before the next is asked for, a guarded `address()` read in place of the `as AddressInfo` cast, and an exhaustion error naming the attempts and rejected ports and nothing else. **Signature B is not resolved and was not folded in.** A file still fails with `'test failed'`, no assertion, no stack and none of its own tests reported. Twenty consecutive full runs gave five failures: on pre-fix code one signature A (`auth.test.js`) and three signature B; on post-fix code one signature B and no signature A. The port fix removes A and leaves B exactly where it was, which is the evidence that they are two defects. Four different files were hit (`move-explanation-route`, `tournament-commentary-route`, `bot-detection-analyze`, `anti-cheat-analysis`), sharing no import beyond `./helpers`; each died in 589–703 ms with no test of its own reporting and no stderr. Refuted with evidence: ephemeral-port exhaustion (113 sockets in TIME_WAIT against a 13977-port range), a `Promise.race` loser becoming an unhandled rejection (`race` subscribes to every promise, confirmed on v24.15.0), a throwing `after`/`afterEach` hook (the affected files use none), and a double `close()` rejecting (awaited in a `finally`, it would be attributed to that test with a stack). A second bounded pass — twelve more full runs under the TAP reporter (which prints the file-level YAML block rather than collapsing it to `'test failed'`) with a preload recording `uncaughtException`, `unhandledRejection` and any non-zero exit — produced twelve clean runs and captured nothing, so the mechanism remains unread. No fix was invented for it. See ADR-0140 §4. +- **`startHarness` drew ports `fetch` refuses (RESOLVED in ADR-0140); the unexplained whole-file failure is still open.** Two signatures were filed here, deliberately not as one cause, and that judgement held. **Signature A is resolved.** WHATWG Fetch blocks eighty-two ports and undici enforces the list on the port number alone, before opening a socket — so `server.listen(0)` could bind, listen and answer raw TCP while `fetch` still refused, surfacing as `TypeError: fetch failed` / `Error: bad port` at the harness’s first request rather than at the listen that caused it. Whether it can happen at all is a property of the host’s dynamic port range: a typical Linux CI range (32768–60999) contains no blocked port, while the Windows range in use here (1024–15000) contains nineteen, which is the whole of "green on CI, flaky locally". The guard that already existed was incomplete in a way that still failed — its hand-observed set of eighteen ports was the spec list intersected with one machine’s range **minus `6679`** — and it retried unboundedly, had no behaviour on exhaustion, and had been copy-pasted into `auth-signin-schema.integration.test.ts`, so the missing port had to be found twice. `packages/api/test/listen.ts` now owns port acquisition for both sites: the spec-complete eighty-two ports (verified by sweeping all 65535 through the real `fetch` on Node v24.15.0), a bounded twenty attempts, each rejected listener closed before the next is asked for, a guarded `address()` read in place of the `as AddressInfo` cast, and an exhaustion error naming the attempts and rejected ports and nothing else. **Signature B is not resolved and was not folded in.** A file still fails with `'test failed'`, no assertion, no stack and none of its own tests reported. Twenty consecutive full runs gave five failures: on pre-fix code one signature A (`auth.test.js`) and three signature B; on post-fix code one signature B and no signature A. The port fix removes A and leaves B exactly where it was, which is the evidence that they are two defects. Four different files were hit (`move-explanation-route`, `tournament-commentary-route`, `bot-detection-analyze`, `anti-cheat-analysis`), sharing no import beyond `./helpers`; each died in 589–703 ms with no test of its own reporting and no stderr. Refuted with evidence: ephemeral-port exhaustion (113 sockets in TIME_WAIT against a 13977-port range), a `Promise.race` loser becoming an unhandled rejection (`race` subscribes to every promise, confirmed on v24.15.0), a throwing `after`/`afterEach` hook (the affected files use none), and a double `close()` rejecting (awaited in a `finally`, it would be attributed to that test with a stack). A second bounded pass — twelve more full runs under the TAP reporter with a preload recording `uncaughtException`, `unhandledRejection` and any non-zero exit — produced twelve clean runs and captured nothing at the time. **A follow-up increment (`claude/node-test-signature-b`) then captured the defect directly, three more times, on three files never previously implicated** (`rate-limit-atomicity`, `dependency-parity`, `studies-api`) — seven distinct files observed with this symptom to date. Occurrences across seven distinct files make a shared or cross-cutting path more plausible and make a defect confined to one test file less likely, but do not exclude file-specific inputs or lifecycle interactions. An instrumented preload (`packages/api/test/diagnostics/signature-b-preload.cjs`) hooking process-level events — `process.exit`, `process.abort`, `process.kill`, `uncaughtExceptionMonitor` (passively observing uncaught exceptions and fatal unhandled rejections), `warning`, `beforeExit`, and Node’s own unconditional `exit` — showed **none of the hooks active at the time fired** on any of the three historical captures (though `process.abort()` was not wrapped in those initial runs and is now covered for future occurrences). A synthetic `process.exit(1)`-before-registration fixture reproduces the identical silent shape; every other synthetic mechanism tried (a post-test async throw, an emitter `'error'` with its listener removed, a synchronous module-load throw, a delayed `SIGKILL`) prints a visibly different diagnostic line, stack, or partial test output that the real defect never shows. This narrows the investigated possibilities while leaving the root cause unresolved: the per-file child process (`node --test` spawns one per file, confirmed by distinct PIDs) was not terminated by `process.exit`, uncaught exceptions, or fatal unhandled rejections, and future runs with `process.abort` instrumentation will record whether abort was called through JS; an absent record narrows in-runtime JS termination but cannot alone prove external termination without corroborating child exit status/signal data or OS-level crash evidence (e.g. distinguishing an external kill or uncatchable signal from a native C++/V8 crash). The machine had roughly 2.5 GB of 15.7 GB RAM free at capture time with several other agents’ processes concurrently running, which is circumstantially consistent with resource contention, but no crash was recorded in the Windows Application or System event logs in that window, so the exact external trigger is still not established. No fix was invented — the forbidden responses (sleeps, whole-file retries, lowering concurrency) would only hide the unresolved root cause, whose origin is not yet established. See ADR-0140 §4. - **`ApiServer.listen` registered no `'error'` handler, so a failed bind hung and raised an uncaught event (RESOLVED in ADR-0140 §5).** `packages/api/src/server.ts` resolved its promise from the `listening` callback only and built it with no reject path. A bind failing asynchronously (`EADDRINUSE`, `EMFILE`) left the promise pending forever and, with no `'error'` listener on the `http.Server`, was re-raised as an uncaught exception. Found while investigating ADR-0140 and independently raised by the Qodo review of PR #21. **Resolved in the same increment** rather than deferred, because ADR-0140 §2's bounded, diagnosable acquisition is not true without it — the retry can only report a bind error if the listener it is handed rejects. A one-shot `'error'` listener now rejects and is removed once listening, so later server errors keep their previous semantics rather than being swallowed by a `reject` on a settled promise. The regression test fails against the exact pre-fix code, through an uncaught `ERR_UNHANDLED_ERROR`. - **`main.ts`'s controller-disposal list is manual, untested, and silently incomplete when a section is added (RESOLVED in Increment 25 / ADR-0092).** `run()` in `packages/web/src/main.ts` disposes the previous route's controllers by name, and its own comment says doing so "is what makes re-bootstrapping safe" — but adding a section to `bootstrap` and forgetting to add it there compiles, passes every gate, and leaks. Increment 23 shipped exactly that omission for `LearningController` and it was caught in PR review, not by a test. `main.ts` has no test coverage of any kind, so no section's disposal is verified. A structural fix (bootstrap returning its disposables as a collection, or a type-level exhaustiveness check keyed off the result type) would make the next omission a compile error; it is a refactor across ~15 return sites and belongs in its own increment. **Resolved in Increment 25 (ADR-0092):** extracted `createLifecycle` run loop in `lifecycle.ts`, defined `BootstrappedDisposables` and `DisposableKey` driving `DISPOSABLE_TEARDOWN_MAP: Record` for compile-time exhaustiveness, normalised `.dispose()` verb across all disposables, and cascaded `GameController.stop()` to `gameSync.stop()`. - **`stepView` sends the answers to the learner (RESOLVED in Increment 29 / ADR-0095).** `packages/api/src/presenters.ts` emits `expectedSan` on a move step and `correctIndex` on a quiz step, and `GET /v1/lessons/:id/steps` is the route the learner's own lesson page calls. Increment 23 omits both from the client-side types (`packages/web/src/api/models.ts`), so the app cannot render or grade against them and a future edit that tries becomes a compile error — but the fields are still on the wire and readable in devtools. The authoring routes legitimately need them returned to the author, so the fix is a learner-scoped step view (or a caller-dependent projection), not a deletion: an API contract decision with its own ADR. Nothing rated or rewarded depends on step progress today, so this is a wart rather than a breach. **Resolved in Increment 29 (ADR-0095):** added `LearnerStepView` / `learnerStepView` in `packages/api/src/presenters.ts` omitting `expectedSan` and `correctIndex`. `GET /v1/lessons/:id/steps` and `GET /v1/steps/:id` now check course authorship via `repo.getLesson` / `repo.getCourse`, returning full `stepView` to the author and `learnerStepView` to learners and anonymous callers. Updated OpenAPI schema and web model comments. The first attempt resolved authorship with a separate `getLesson` + `getCourse` after the step read, which doubled both routes from 3 SQL queries to 6 because `listSteps` / `getStep` had already made those reads internally and discarded the course; caught in the PR #92 review and fixed by adding `getStepWithCourse` / `listStepsWithCourse` to `LearningRepository`, which return what was already loaded. Both routes now make exactly one repository call, pinned by a counting-proxy test. diff --git a/docs/adr/0140-harness-ephemeral-port-acquisition.md b/docs/adr/0140-harness-ephemeral-port-acquisition.md index d62ed1d2..44a19d25 100644 --- a/docs/adr/0140-harness-ephemeral-port-acquisition.md +++ b/docs/adr/0140-harness-ephemeral-port-acquisition.md @@ -170,19 +170,103 @@ Hypotheses examined and **refuted** with evidence: throw would be attributed to that test with a stack, which is not the observed signature. -The leading unrefuted hypothesis is a process-level fault in the test child -after module load and before the first test reports — a keep-alive socket to a -closed harness erroring inside undici's pool, or an uncaught `'error'` event on -an `http.Server`. **One instance of that second shape has now been fixed** (§5), -and it produces exactly this signature when it fires: with the fix reverted, the -regression test for it fails through an uncaught `ERR_UNHANDLED_ERROR` rather -than an assertion. That makes it a plausible contributor, but not a -demonstrated cause — signature B was reproduced on files doing no failing bind, -and it was never observed to coincide with one. It stays open. - -Nothing above is strong enough to justify a fix, and a fix that cannot be shown -to remove a reproduction is indistinguishable from a coincidence. Signature B -stays open. +One instance of an uncaught `'error'` event on an `http.Server` **has since been +fixed** (§5): with that fix reverted, the regression test for it fails through +an uncaught `ERR_UNHANDLED_ERROR`. That made it a plausible contributor, but a +follow-up investigation (below) has since ruled it out as the mechanism behind +the observed signature specifically, even though it remained a real defect +worth fixing on its own merits. + +### Follow-up investigation: the failure mode is characterized, the trigger is not + +A later increment (`claude/node-test-signature-b`) instrumented selected +termination and error paths a child process can raise and captured three +further real occurrences directly, rather than reasoning from symptoms or +synthetic proxies alone. + +**Node's test runner spawns one child process per test file.** Verified +directly: two trivial files run together under +`node --test --test-concurrency=1` report distinct PIDs, neither matching the +parent's. The flag-free PID run is the evidence for default per-file process +isolation on the tested Node version, while the explicit `--test-isolation=process` +flag demonstrates that process isolation can also be explicitly selected. This process +boundary proves that a terminating child does not directly abort sibling test processes +in the runner, though shared host resources can still interact across processes. + +**The bare `'test failed'` with no stack and empty stderr is what Node's runner reports for a +child that calls `process.exit(1)` (or terminates silently) before registering any test.** This was reproduced directly: a synthetic file that calls +`process.exit(1)` before registering any test produces the identical reported +shape as the real defect — `✖ (Nms)` / `'test failed'`, zero individual +tests in the summary, nothing on stderr. Every OTHER synthetic mechanism tried +produces a visibly different shape: + +- An error thrown after a test's own promise settles (`setImmediate` inside a + passing test) — Node prints an explicit + `ℹ Error: ... generated asynchronous activity after the test ended ...` + diagnostic line, and the triggering test still shows `✔`. +- An `http.Server` emitting `'error'` with its listener already removed — + same diagnostic line, same visible `✔` on the test that created it. +- A promise rejected synchronously inside a test's own body — attributed to + that specific test, with a full stack. +- A process killed by `SIGKILL` after a test completes — that test's `✔` + still prints before the file dies. +- A synchronous throw at module load, before any `test()` call — prints a + full stack trace to stderr. + +None of these match. The real defect's log is completely silent before the +bare file-level failure: no diagnostic line, no stderr, no test names at all +for that file (not even the ones that would have run first). + +**A diagnostic preload +(`packages/api/test/diagnostics/signature-b-preload.cjs`) captures which of +those mechanisms fire on real occurrences, not synthetic ones.** It hooks +`process.exit`, `process.abort`, `process.kill`, `uncaughtExceptionMonitor` (a passive observer +that captures both uncaught exceptions and fatal unhandled rejections without registering active +listeners that alter Node's default crash handling), `warning`, `beforeExit`, and Node's +own unconditional `exit` event, writing a structured, redacted, timestamped +line to disk (before delegation for `process.exit`, `process.abort`, and `process.kill`, and during event delivery for `uncaughtExceptionMonitor`, `warning`, `beforeExit`, and `exit`) — bypassing stdout/stderr entirely so the test +reporter's own output is never touched. A bounded 20-run instrumented pass over +the full `packages/api` suite (chosen the same way as the original 20-run +sample: enough for >90% detection odds at the historically observed ~1-in-5 +rate) reproduced the defect 3 times, on **three files never previously +implicated** — `rate-limit-atomicity`, `dependency-parity`, `studies-api` — none +of the original four. Combined with the original four, that is **seven distinct files** observed with this symptom +to date. Occurrences across seven distinct files make a shared or cross-cutting path more +plausible and make a defect confined to one test file less likely, but do not exclude +file-specific inputs or lifecycle interactions. + +All three historical captures show the identical signature: **only the `start` and +`preload-installed` lines were logged. None of the hooks active at that time fired — +including `process.exit`, `process.kill`, `uncaughtExceptionMonitor`, `warning`, +`beforeExit`, and Node's `exit` event.** +Crucially, however, the diagnostic preload in use during those historical runs did +not yet wrap `process.abort()`. Because `process.abort()` terminates the process +immediately without emitting Node's `exit` event, those historical captures directly +ruled out `process.exit`, uncaught exceptions, and fatal unhandled rejections, but +could not categorically exclude an uninstrumented `process.abort()` or an external termination. + +**This narrows Signature B's investigated possibilities while leaving its root cause unresolved:** +`process.abort()` is now instrumented so any future occurrence will record whether an abort +was invoked through JS; an absent record narrows in-runtime JS termination but cannot alone +prove external termination without corroborating child exit status/signal data or OS-level +evidence (e.g. distinguishing an external kill or uncatchable signal from a native C++/V8 crash). +Two observations remain consistent with an +environmental (not code) origin: at the time of capture this development machine had roughly +2.5 GB of 15.7 GB RAM free, with several unrelated concurrent processes (other agents' worktrees +and dev servers) running; and `node --test` spawns dozens of child processes across a full +`packages/api` run, each loading the same large cross-package import graph, which is exactly the +pattern most exposed to transient resource contention. Neither observation is proof of a specific +external actor — no crash was recorded in the Windows Application or System event logs in the +capture window — so the exact trigger (OS scheduler, memory pressure, antivirus, or something +else entirely) is not established. + +**No fix is proposed.** The investigated JS mechanisms (`process.exit`, exceptions, rejections) +have been ruled out on the historical captures, `process.abort()` instrumentation is in place for +future occurrences, and the forbidden responses — sleeps, retries around the whole file, +swallowing errors, lowering concurrency to hide it — would suppress the symptom without touching +whatever the root cause turns out to be. Signature B stays open and unresolved. The diagnostic +preload is committed so the next occurrence — on this machine, in CI, or elsewhere — can be +captured with selected termination and error path instrumentation rather than re-deriving it. ## 5. `ApiServer.listen` rejects on a failed bind diff --git a/packages/api/package.json b/packages/api/package.json index a90dc536..d330a438 100644 --- a/packages/api/package.json +++ b/packages/api/package.json @@ -24,6 +24,7 @@ "build": "tsc -p tsconfig.json", "test": "tsc -p tsconfig.test.json && node --test --test-concurrency=1 \"dist-test/test/**/*.test.js\"", "test:analysis-smoke": "tsc -p tsconfig.test.json && node --test dist-test/test/analysis-stockfish-smoke.test.js dist-test/test/analysis-fairy-threecheck-smoke.test.js dist-test/test/analysis-real-stack.test.js", + "test:diagnostics:abort": "tsc -p tsconfig.test.json && node --test dist-test/test/diagnostics/signature-b-preload-abort.diag.js", "lint": "tsc -p tsconfig.json --noEmit", "openapi": "node dist/scripts/generate-openapi.js", "reindex-search": "node dist/scripts/reindex-search.js", diff --git a/packages/api/test/diagnostics/signature-b-preload-abort.diag.ts b/packages/api/test/diagnostics/signature-b-preload-abort.diag.ts new file mode 100644 index 00000000..80caa387 --- /dev/null +++ b/packages/api/test/diagnostics/signature-b-preload-abort.diag.ts @@ -0,0 +1,150 @@ +import { test } from 'node:test'; +import * as assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +const PRELOAD_PATH = path.resolve(__dirname, '../../../test/diagnostics/signature-b-preload.cjs'); + +interface DiagnosticRecord { + readonly kind: string; + readonly pid: number; + readonly ppid: number; + readonly code?: number; + readonly targetPid?: number; + readonly signal?: string | number; + readonly origin?: string; + readonly callerStack?: string; + readonly err?: { + readonly name?: string; + readonly message?: string; + readonly stack?: string; + readonly code?: string | number | boolean; + }; + readonly activeResources?: Record | null; +} + +/** + * Executes a synthetic test script under the diagnostic preload in an isolated temporary directory. + * Explicit diagnostic integration runner for real native abort validation. + * + * @param {string} code - JavaScript code to execute in the child process. + * @param {Record} [envOverride] - Optional environment variables to override. + * @returns {{ status: number | null, signal: NodeJS.Signals | null, logDir: string, records: readonly DiagnosticRecord[] }} + */ +function runDiagnosticChild( + code: string, + envOverride: Record = {}, +): { + readonly status: number | null; + readonly signal: NodeJS.Signals | null; + readonly logDir: string; + readonly targetLogDir: string; + readonly records: readonly DiagnosticRecord[]; +} { + const generatedLogDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sigb-abort-diag-')); + const rawLogDir = envOverride.SIGB_LOG_DIR; + const defaultPreloadLogDir = path.join(os.tmpdir(), 'sigb-diag'); + const targetLogDir = + rawLogDir === undefined + ? generatedLogDir + : rawLogDir === '' + ? defaultPreloadLogDir + : path.isAbsolute(rawLogDir) + ? rawLogDir + : path.resolve(generatedLogDir, rawLogDir); + const scriptPath = path.join(generatedLogDir, 'abort-target.cjs'); + fs.writeFileSync(scriptPath, code, 'utf8'); + + const args = ['--require', PRELOAD_PATH, scriptPath]; + const env = { + ...process.env, + ...envOverride, + SIGB_LOG_DIR: rawLogDir !== undefined ? rawLogDir : targetLogDir, + }; + + const result = + process.platform !== 'win32' + ? spawnSync('sh', ['-c', 'ulimit -c 0 2>/dev/null; exec "$@"', 'sh', process.execPath, ...args], { + cwd: generatedLogDir, + env, + encoding: 'utf8', + windowsHide: true, + }) + : spawnSync(process.execPath, args, { + cwd: generatedLogDir, + env, + encoding: 'utf8', + windowsHide: true, + }); + + const files = fs.existsSync(targetLogDir) + ? fs + .readdirSync(targetLogDir) + .filter( + (f) => + f.endsWith('.jsonl') && + (rawLogDir === '' && result.pid ? f.startsWith(`run-${result.pid}-`) : true), + ) + : []; + const records: DiagnosticRecord[] = []; + for (const file of files) { + const content = fs.readFileSync(path.join(targetLogDir, file), 'utf8'); + const lines = content.split('\n').map((l) => l.trim()).filter(Boolean); + for (const line of lines) { + records.push(JSON.parse(line) as DiagnosticRecord); + } + } + + return { + status: result.status, + signal: result.signal, + logDir: generatedLogDir, + targetLogDir, + records, + }; +} + +test('explicit diagnostic integration: real native process.abort() records synchronous diagnostic before termination', (t) => { + const { status, signal, records, logDir } = runDiagnosticChild(` + process.abort(); + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + // On Windows/POSIX, abort causes abnormal termination + assert.ok(status !== 0 || signal !== null, 'process terminates abnormally'); + assert.ok(records.some((r) => r.kind === 'preload-installed'), 'records preload-installed'); + const abortRecord = records.find((r) => r.kind === 'process.abort'); + assert.ok(abortRecord, 'synchronously records process.abort event'); + assert.ok(typeof abortRecord?.callerStack === 'string', 'includes callerStack in abort record'); + assert.ok(abortRecord?.callerStack?.includes('abort-call-site'), 'callerStack traces abort invocation site'); + + // Verify process.on('exit') does NOT fire on process.abort() + const exitRecord = records.find((r) => r.kind === 'exit'); + assert.equal(exitRecord, undefined, 'exit event must not fire on process.abort'); +}); + +test('explicit diagnostic integration: empty SIGB_LOG_DIR falls back to default directory without path split', (t) => { + const { status, signal, records, logDir, targetLogDir } = runDiagnosticChild( + ` + process.abort(); + `, + { SIGB_LOG_DIR: '' }, + ); + t.after(() => { + fs.rmSync(logDir, { recursive: true, force: true }); + if (fs.existsSync(targetLogDir) && targetLogDir !== logDir) { + const files = fs.readdirSync(targetLogDir).filter((f) => f.endsWith('.jsonl')); + for (const file of files) { + if (records.some((r) => file.startsWith(`run-${r.pid}-`))) { + fs.rmSync(path.join(targetLogDir, file), { force: true }); + } + } + } + }); + + assert.ok(status !== 0 || signal !== null, 'process terminates abnormally'); + assert.ok(records.some((r) => r.kind === 'preload-installed'), 'records preload-installed in default log dir'); + assert.ok(records.some((r) => r.kind === 'process.abort'), 'records process.abort in default log dir'); +}); diff --git a/packages/api/test/diagnostics/signature-b-preload.cjs b/packages/api/test/diagnostics/signature-b-preload.cjs new file mode 100644 index 00000000..65909dd2 --- /dev/null +++ b/packages/api/test/diagnostics/signature-b-preload.cjs @@ -0,0 +1,330 @@ +'use strict'; +/** + * Diagnostic preload for the "Signature B" node:test failure (docs/adr/0140 + * §4): a test file occasionally fails with a bare `'test failed'`, no + * assertion, no stack, and none of that file's own tests reported. Node's + * test runner spawns one child process per file; that exact message is its + * own hardcoded fallback (`ERR_TEST_FAILURE('test failed', kTestCodeFailure)` + * with `stack: undefined`) for when a child exits non-zero or signaled while + * no subtest recorded a failure — see `internal/test_runner/runner.js`. + * + * This module distinguishes the JS-catchable causes of that shape + * (`process.exit`, `process.abort`, an uncaught exception, a fatal unhandled + * rejection promoted by Node, an EventEmitter `'error'` event with no listener) + * from an external, uncatchable termination of the child (an OS-level kill or native crash), + * by hooking process-level lifecycle events and writing what fired to a structured log. + * + * Passive fatal rejection observation: Deliberately does NOT register active + * `unhandledRejection` or `rejectionHandled` listeners, because in Node.js + * attaching an `unhandledRejection` listener alters runtime semantics and suppresses + * default fatal crash behavior. Instead, `uncaughtExceptionMonitor` is registered + * to passively observe both `origin === 'uncaughtException'` and `origin === 'unhandledRejection'` + * without changing process exit codes or suppressing crash paths. + * + * Set SIGB_LOG_DIR to control where logs land (defaults under the OS temp + * dir). Logs are structured JSONL, one line per event, one file per child + * process. Nothing is written to stdout/stderr, so the test reporter's own + * output is never touched. + * + * Does not log request bodies, Authorization headers, cookie values, raw + * connection strings, or password hashes — see `redact`/`safeErr` below. + * + * Deliberately does NOT patch `EventEmitter.prototype.emit`. An earlier + * version did, to catch an unlistened `'error'` event before Node re-raises + * it; verified empirically to corrupt node:test's own internal `TestsStream` + * (itself an EventEmitter) and crash the PARENT `node --test` process on a + * trivial passing test with no Signature-B-related content at all — + * `--require` loads into every node process spawned with that flag, + * including the top-level runner, not just its per-file children. A global + * prototype patch is not a safe way to observe this system. An unhandled + * `'error'` event still surfaces below via `uncaughtExceptionMonitor`, + * Node's own re-raise of it, just without the emitter's constructor name. + */ + +// Usage (manual, not wired into `npm test`; run from packages/api): +// node --require ./test/diagnostics/signature-b-preload.cjs \ +// --test --test-concurrency=1 "dist-test/test/**/*.test.js" + +const fs = require('node:fs'); +const os = require('node:os'); +const path = require('node:path'); + +const LOG_DIR = process.env.SIGB_LOG_DIR || path.join(os.tmpdir(), 'sigb-diag'); +try { + fs.mkdirSync(LOG_DIR, { recursive: true, mode: 0o700 }); +} catch { + /* best-effort; missing dir just drops logging */ +} +const LOG_FILE = path.join(LOG_DIR, `run-${process.pid}-${Date.now()}.jsonl`); +let fd; +try { + fd = fs.openSync(LOG_FILE, 'a', 0o600); +} catch { + fd = null; +} + +const REDACT_PATTERNS = [ + [/sk-[a-zA-Z0-9_-]{10,}/g, '[REDACTED_API_KEY]'], + [/Bearer\s+[A-Za-z0-9._~+/-]+=*/g, 'Bearer [REDACTED_TOKEN]'], + [/(?:postgres|postgresql):\/\/[^:]+:[^@]+@/g, 'postgres://[REDACTED_CREDS]@'], +]; + +const SENSITIVE_KEY_RE = /(["']?)(password(?:[_\s-]?hash|Hash)?|secret|token|authorization|cookie)\1\s*[:=]\s*/gi; + +/** + * Scans and redacts sensitive key-value pairs, tracking quote escapes and arbitrary nested array/object depth. + * + * @param {string} str - Input string to redact. + * @returns {string} Sanitized string with sensitive values replaced by [REDACTED]. + */ +function redactSensitiveStructures(str) { + if (typeof str !== 'string') return str; + let out = ''; + let lastIndex = 0; + SENSITIVE_KEY_RE.lastIndex = 0; + + let match; + while ((match = SENSITIVE_KEY_RE.exec(str)) !== null) { + out += str.slice(lastIndex, match.index); + const keyPrefix = (match[1] || '') + match[2] + (match[1] || '') + '=[REDACTED]'; + out += keyPrefix; + + let i = SENSITIVE_KEY_RE.lastIndex; + if (i >= str.length) { + lastIndex = str.length; + break; + } + + const firstChar = str[i]; + if (firstChar === '"' || firstChar === "'") { + const quote = firstChar; + i++; + while (i < str.length) { + if (str[i] === '\\') { + i += 2; + } else if (str[i] === quote) { + i++; + break; + } else { + i++; + } + } + } else if (firstChar === '{' || firstChar === '[') { + const stack = [firstChar]; + i++; + let inQuote = null; + while (i < str.length && stack.length > 0) { + const char = str[i]; + if (inQuote) { + if (char === '\\') { + i += 2; + } else if (char === inQuote) { + inQuote = null; + i++; + } else { + i++; + } + } else { + if (char === '"' || char === "'") { + inQuote = char; + i++; + } else if (char === '{' || char === '[') { + stack.push(char); + i++; + } else if (char === '}' && stack[stack.length - 1] === '{') { + stack.pop(); + i++; + } else if (char === ']' && stack[stack.length - 1] === '[') { + stack.pop(); + i++; + } else { + i++; + } + } + } + } else { + while ( + i < str.length && + str[i] !== ',' && + str[i] !== '\r' && + str[i] !== '\n' && + str[i] !== '}' && + str[i] !== ']' + ) { + i++; + } + } + + lastIndex = i; + SENSITIVE_KEY_RE.lastIndex = i; + } + + out += str.slice(lastIndex); + return out; +} + +/** + * Redacts known sensitive patterns (API keys, bearer tokens, credentials, passwords, nested structures) + * from strings before writing to diagnostic logs. + * + * @param {unknown} value - The input value to redact. + * @returns {unknown} The sanitized value with sensitive patterns replaced by redaction placeholders. + */ +function redact(value) { + if (typeof value !== 'string') return value; + let out = value; + for (const [pattern, replacement] of REDACT_PATTERNS) out = out.replace(pattern, replacement); + out = redactSensitiveStructures(out); + return out; +} + +/** + * Safely normalizes error metadata values (code, syscall, errno) into a JSON-safe primitive or string. + * Prevents BigInt, circular references, or large object metadata from breaking diagnostic serialization. + * + * @param {unknown} val - The metadata value to normalize. + * @returns {string | number | boolean | null | undefined} A JSON-safe primitive representation. + */ +function normalizeMeta(val) { + if (val == null) return val; + if (typeof val === 'string' || typeof val === 'number' || typeof val === 'boolean') return val; + if (typeof val === 'bigint') return String(val); + try { + return redact(String(val)); + } catch { + return '[UNSERIALIZABLE]'; + } +} + +/** + * Safely serializes an Error or error-like object into a redacted, JSON-safe structure. + * Prevents circular reference crashes, BigInt serialization failures, and strips credentials. + * + * @param {unknown} err - The error or rejection value to serialize. + * @returns {unknown} A serialized error object or redacted primitive. + */ +function safeErr(err) { + if (err == null) return err; + if (typeof err !== 'object') { + if (typeof err === 'bigint') return String(err); + return redact(String(err)); + } + try { + return { + name: typeof err.name === 'string' ? redact(err.name) : undefined, + message: redact(String(err.message ?? '')), + stack: redact(String(err.stack ?? '')), + code: normalizeMeta(err.code), + syscall: normalizeMeta(err.syscall), + errno: normalizeMeta(err.errno), + }; + } catch { + return { name: 'SerializationError', message: '[UNSERIALIZABLE_ERROR]' }; + } +} + +/** + * Collects a non-identifying, aggregate count of currently active Node async resources + * using `process.getActiveResourcesInfo()` where available. + * + * @returns {Record | null} Resource counts by type, or null if unsupported. + */ +function activeResourceCounts() { + try { + if (typeof process.getActiveResourcesInfo !== 'function') return null; + const counts = {}; + for (const resource of process.getActiveResourcesInfo()) counts[resource] = (counts[resource] || 0) + 1; + return counts; + } catch { + return null; + } +} + +/** + * Synchronously appends a structured JSONL diagnostic record to the active log file. + * Synchronous writes ensure diagnostic records survive abrupt process termination. + * Serialization is guarded with BigInt replacer and try/catch to prevent dropped diagnostics. + * + * @param {string} kind - The event or lifecycle hook kind (e.g. 'start', 'process.abort'). + * @param {Record} fields - Event-specific payload fields. + * @returns {void} + */ +function write(kind, fields) { + if (!fd) return; + try { + const line = JSON.stringify( + { + kind, + pid: process.pid, + ppid: process.ppid, + testFile: Boolean(process.env.NODE_TEST_CONTEXT) ? (process.argv[1] || null) : null, + nodeTestContext: process.env.NODE_TEST_CONTEXT || null, + timeIso: new Date().toISOString(), + timeNs: process.hrtime.bigint().toString(), + ...fields, + }, + (_key, value) => (typeof value === 'bigint' ? value.toString() : value), + ); + fs.writeSync(fd, line + '\n'); + } catch { + /* best-effort; serialization or write failure dropped safely */ + } +} + +write('start', {}); + +const originalExit = process.exit.bind(process); +/** + * Patched wrapper around `process.exit` that synchronously records the exit code + * and redacted caller stack before invoking the original `process.exit`. + * + * @param {number} [code] - The process exit code. + * @returns {never} + */ +process.exit = function patchedExit(code) { + write('process.exit', { code, callerStack: redact(new Error('exit-call-site').stack) }); + return originalExit(code); +}; + +const originalAbort = typeof process.abort === 'function' ? process.abort.bind(process) : null; +if (originalAbort) { + /** + * Patched wrapper around `process.abort` that synchronously records the abort event + * and redacted caller stack before invoking the original `process.abort`. + * + * @param {...unknown} args - Any arguments forwarded to process.abort. + * @returns {never} + */ + process.abort = function patchedAbort(...args) { + write('process.abort', { callerStack: redact(new Error('abort-call-site').stack) }); + return originalAbort(...args); + }; +} + +const originalKill = process.kill.bind(process); +/** + * Patched wrapper around `process.kill` that synchronously records target PID and signal + * before delegating to the original `process.kill`. + * + * @param {number} pid - Target process ID. + * @param {string | number} [signal] - Signal to send. + * @returns {boolean} Result of original process.kill. + */ +process.kill = function patchedKill(pid, signal) { + write('process.kill', { targetPid: pid, signal, callerStack: redact(new Error('kill-call-site').stack) }); + return originalKill(pid, signal); +}; + +// Passive observer only: unlike 'uncaughtException' or 'unhandledRejection', registering +// 'uncaughtExceptionMonitor' does NOT suppress Node's default crash behavior, so it cannot +// itself change whether or how the process exits. Node emits it for both 'uncaughtException' +// and 'unhandledRejection' origins. +process.on('uncaughtExceptionMonitor', (err, origin) => { + write('uncaughtExceptionMonitor', { err: safeErr(err), origin, activeResources: activeResourceCounts() }); +}); + +process.on('warning', (warning) => write('warning', { err: safeErr(warning) })); +process.on('beforeExit', (code) => write('beforeExit', { code, activeResources: activeResourceCounts() })); +process.on('exit', (code) => write('exit', { code })); + +write('preload-installed', {}); diff --git a/packages/api/test/diagnostics/signature-b-preload.test.ts b/packages/api/test/diagnostics/signature-b-preload.test.ts new file mode 100644 index 00000000..9801e8d7 --- /dev/null +++ b/packages/api/test/diagnostics/signature-b-preload.test.ts @@ -0,0 +1,514 @@ +import { test } from 'node:test'; +import * as assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +const PRELOAD_PATH = path.resolve(__dirname, '../../../test/diagnostics/signature-b-preload.cjs'); + +interface DiagnosticRecord { + readonly kind: string; + readonly pid: number; + readonly ppid: number; + readonly testFile?: string | null; + readonly nodeTestContext?: string | null; + readonly code?: number; + readonly targetPid?: number; + readonly signal?: string | number; + readonly origin?: string; + readonly callerStack?: string; + readonly err?: { + readonly name?: string; + readonly message?: string; + readonly stack?: string; + readonly code?: string | number | boolean; + readonly syscall?: string | number | boolean; + readonly errno?: string | number | boolean; + }; + readonly activeResources?: Record | null; +} + +interface RunChildOptions { + readonly extraPreloads?: (dir: string) => readonly string[]; + readonly useNodeTestRunner?: boolean; +} + +/** + * Internal runner that executes a synthetic child script under the diagnostic preload + * in an isolated temporary directory with optional extra preload hooks. + */ +function runChild( + code: string, + envOverride: Record = {}, + options: RunChildOptions = {}, +): { + readonly status: number | null; + readonly signal: NodeJS.Signals | null; + readonly stdout: string; + readonly stderr: string; + readonly logDir: string; + readonly targetLogDir: string; + readonly records: readonly DiagnosticRecord[]; +} { + const generatedLogDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sigb-test-')); + const rawLogDir = envOverride.SIGB_LOG_DIR; + const defaultPreloadLogDir = path.join(os.tmpdir(), 'sigb-diag'); + const targetLogDir = + rawLogDir === undefined + ? generatedLogDir + : rawLogDir === '' + ? defaultPreloadLogDir + : path.isAbsolute(rawLogDir) + ? rawLogDir + : path.resolve(generatedLogDir, rawLogDir); + const scriptPath = path.join(generatedLogDir, 'test-target.cjs'); + fs.writeFileSync(scriptPath, code, 'utf8'); + + const extraPreloads = options.extraPreloads ? options.extraPreloads(generatedLogDir) : []; + const preloads = [...extraPreloads, PRELOAD_PATH]; + const args = preloads.flatMap((p) => ['--require', p]); + if (options.useNodeTestRunner) { + args.push('--test'); + } + args.push(scriptPath); + + const env: Record = { + ...process.env, + ...envOverride, + SIGB_LOG_DIR: rawLogDir !== undefined ? rawLogDir : targetLogDir, + }; + delete env.NODE_TEST_CONTEXT; + for (const [k, v] of Object.entries(envOverride)) { + env[k] = v; + } + + const result = spawnSync(process.execPath, args, { + cwd: generatedLogDir, + env, + encoding: 'utf8', + windowsHide: true, + }); + + const files = fs.existsSync(targetLogDir) + ? fs + .readdirSync(targetLogDir) + .filter( + (f) => + f.endsWith('.jsonl') && + (rawLogDir === '' && result.pid ? f.startsWith(`run-${result.pid}-`) : true), + ) + : []; + const records: DiagnosticRecord[] = []; + for (const file of files) { + const content = fs.readFileSync(path.join(targetLogDir, file), 'utf8'); + const lines = content.split('\n').map((l) => l.trim()).filter(Boolean); + for (const line of lines) { + records.push(JSON.parse(line) as DiagnosticRecord); + } + } + + return { + status: result.status, + signal: result.signal, + stdout: result.stdout, + stderr: result.stderr, + logDir: generatedLogDir, + targetLogDir, + records, + }; +} + +/** + * Executes a synthetic test script under the diagnostic preload in an isolated temporary directory. + * Sets `cwd: generatedLogDir` and isolates diagnostic log directory per invocation. + * + * @param {string} code - JavaScript code to execute in the child process. + * @param {Record} [envOverride] - Optional environment variables to override. + * @param {boolean} [useNodeTestRunner=false] - Whether to invoke via `node --test` runner instead of plain node. + * @returns {{ status: number | null, signal: NodeJS.Signals | null, stdout: string, stderr: string, logDir: string, records: readonly DiagnosticRecord[] }} + */ +function runWithPreload( + code: string, + envOverride: Record = {}, + useNodeTestRunner = false, +) { + return runChild(code, envOverride, { useNodeTestRunner }); +} + +/** + * Executes a synthetic child script with a test-only safe abort shim loaded BEFORE the real preload. + * + * Execution order: + * 1. `safe-abort-shim.cjs` replaces native `process.abort` with a harmless sentinel that logs to stdout and calls `process.exit(99)`. + * 2. `signature-b-preload.cjs` loads second, capturing the sentinel function as `originalAbort` and installing `patchedAbort`. + * 3. The child script runs and calls `process.abort()`. + * 4. `patchedAbort` synchronously writes the `process.abort` JSONL diagnostic record and delegates to `originalAbort`. + * 5. The harmless sentinel runs and calls `process.exit(99)`, preventing native `process.abort()` / OS crash dumps. + */ +function runWithSafeAbortShimAndPreload( + code: string, + envOverride: Record = {}, +) { + return runChild(code, envOverride, { + extraPreloads: (dir: string) => { + const shimPath = path.join(dir, 'safe-abort-shim.cjs'); + fs.writeFileSync( + shimPath, + ` + const fs = require('node:fs'); + process.abort = function harmlessSentinelAbort(...args) { + fs.writeSync(1, '__SAFE_ABORT_SENTINEL_INVOKED__\\n'); + process.exit(99); + }; + `, + 'utf8', + ); + return [shimPath]; + }, + }); +} + +test('diagnostic preload: normal shutdown records lifecycle events', (t) => { + const { status, records, logDir } = runWithPreload(` + // clean normal execution + const x = 1 + 1; + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + assert.equal(status, 0); + const kinds = records.map((r) => r.kind); + assert.ok(kinds.includes('start'), 'records start'); + assert.ok(kinds.includes('preload-installed'), 'records preload-installed'); + assert.ok(kinds.includes('beforeExit'), 'records beforeExit'); + assert.ok(kinds.includes('exit'), 'records exit'); +}); + +test('diagnostic preload: process.exit(42) records exit code and call-site stack before exit', (t) => { + const { status, records, logDir } = runWithPreload(` + process.exit(42); + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + assert.equal(status, 42); + const exitRecord = records.find((r) => r.kind === 'process.exit'); + assert.ok(exitRecord, 'records process.exit'); + assert.equal(exitRecord?.code, 42); + assert.ok(typeof exitRecord?.callerStack === 'string', 'includes callerStack'); + assert.ok(exitRecord?.callerStack?.includes('exit-call-site'), 'stack traces exit invocation site'); +}); + +test('diagnostic preload: process.abort wrapper synchronously writes record and delegates at runtime without native abort', (t) => { + // Safe runtime execution test: + // 1. safe-abort-shim.cjs preloads first and overrides native process.abort with harmless sentinel. + // 2. signature-b-preload.cjs preloads second, capturing the sentinel as originalAbort and installing patchedAbort. + // 3. Child script calls process.abort(). + // 4. patchedAbort synchronously writes the JSONL diagnostic record to SIGB_LOG_DIR before delegating to the sentinel. + // 5. Sentinel receives delegation and exits with status 99. + // 6. Child process does NOT execute native abort (no core dumps or OS crash handlers). + const { status, signal, stdout, records, logDir } = runWithSafeAbortShimAndPreload(` + // Verify runtime wrapper identity before calling + if (process.abort.name !== 'patchedAbort') { + process.exit(101); + } + // Call process.abort() + process.abort(); + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + // Process exited cleanly via harmless sentinel delegation + assert.equal(signal, null, 'must not be terminated by signal'); + assert.equal(status, 99, 'harmless sentinel was executed via originalAbort delegation'); + assert.ok(stdout.includes('__SAFE_ABORT_SENTINEL_INVOKED__'), 'sentinel output verified on stdout'); + + // Preload lifecycle and abort records were written + const kinds = records.map((r) => r.kind); + assert.ok(kinds.includes('start'), 'records start'); + assert.ok(kinds.includes('preload-installed'), 'records preload-installed'); + const abortRecord = records.find((r) => r.kind === 'process.abort'); + assert.ok(abortRecord, 'synchronously records process.abort event to JSONL'); + assert.ok(typeof abortRecord?.callerStack === 'string', 'includes callerStack in abort record'); + assert.ok(abortRecord?.callerStack?.includes('test-target'), 'callerStack traces abort invocation site'); +}); + +test('diagnostic preload: uncaughtExceptionMonitor passively captures error without suppressing crash', (t) => { + const { status, records, logDir } = runWithPreload(` + throw new Error('synthetic fatal crash with Bearer secret-tok-12345'); + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + assert.notEqual(status, 0); + const uncaught = records.find((r) => r.kind === 'uncaughtExceptionMonitor'); + assert.ok(uncaught, 'records uncaughtExceptionMonitor'); + assert.equal(uncaught?.origin, 'uncaughtException'); + assert.ok(uncaught?.err?.message?.includes('Bearer [REDACTED_TOKEN]'), 'redacts token in error message'); +}); + +test('diagnostic preload: fatal unhandled rejection passively captured under node --test runner', (t) => { + // Verifies that under node --test runner, an unhandled rejection is fatal (exits non-zero), + // is passively recorded by uncaughtExceptionMonitor with origin === 'unhandledRejection', + // and the preload does NOT register an active listener that swallows the crash. + const { status, records, logDir } = runWithPreload(` + Promise.reject(new Error('synthetic unhandled rejection: password="mysecretpassword"')); + `, {}, true); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + assert.notEqual(status, 0, 'node --test runner must fail on unhandled rejection'); + const rejectionRecord = records.find( + (r) => r.kind === 'uncaughtExceptionMonitor' && r.origin === 'unhandledRejection', + ); + assert.ok(rejectionRecord, 'records unhandledRejection origin via uncaughtExceptionMonitor'); + assert.ok(rejectionRecord?.err?.message?.includes('password=[REDACTED]'), 'redacts password in rejection reason'); +}); + +test('diagnostic preload: safeErr serialization does not throw on BigInt or circular metadata and preserves primitive codes', (t) => { + const { records, logDir } = runWithPreload(` + const circular = { name: 'circular' }; + circular.self = circular; + + const errWithBigInt = new Error('error with bigint code'); + errWithBigInt.code = 1n; + errWithBigInt.errno = 42n; + + const errWithCircular = new Error('error with circular metadata'); + errWithCircular.code = circular; + + const normalErr = new Error('normal error'); + normalErr.code = 'EPIPE'; + normalErr.syscall = 'write'; + normalErr.errno = -32; + + process.emitWarning(errWithBigInt); + process.emitWarning(errWithCircular); + process.emitWarning(normalErr); + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + const warnings = records.filter((r) => r.kind === 'warning'); + assert.ok(warnings.length >= 3, 'all 3 emitted warning records successfully serialized and written'); + + const bigIntWarning = warnings.find((w) => w.err?.message?.includes('bigint')); + assert.ok(bigIntWarning, 'bigint warning logged'); + assert.equal(bigIntWarning?.err?.code, '1', 'bigint code converted to string'); + assert.equal(bigIntWarning?.err?.errno, '42', 'bigint errno converted to string'); + + const circularWarning = warnings.find((w) => w.err?.message?.includes('circular')); + assert.ok(circularWarning, 'circular warning logged'); + + const normalWarning = warnings.find((w) => w.err?.message?.includes('normal error')); + assert.ok(normalWarning, 'normal warning logged'); + assert.equal(normalWarning?.err?.code, 'EPIPE', 'string code preserved verbatim'); + assert.equal(normalWarning?.err?.syscall, 'write', 'string syscall preserved verbatim'); + assert.equal(normalWarning?.err?.errno, -32, 'numeric errno preserved verbatim'); +}); + +test('diagnostic preload: process.kill records target PID and signal', (t) => { + const { records, logDir } = runWithPreload(` + try { + process.kill(process.pid, 0); + } catch {} + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + const killRecord = records.find((r) => r.kind === 'process.kill'); + assert.ok(killRecord, 'records process.kill'); + assert.equal(killRecord?.signal, 0); + assert.ok(killRecord?.callerStack?.includes('kill-call-site'), 'traces kill call site'); +}); + +test('diagnostic preload: redacts postgres/postgresql credentials, tokens, Authorization headers, cookies, quoted secrets, password hashes, and JSON-quoted fields', (t) => { + const { records, logDir } = runWithPreload(` + const parts = [ + 'Connect failed to postgresql://app_user:s3cr3tpass@db.internal:5432/chess', + 'postgres://admin:supersecret@10.0.0.1:5432/main', + 'sk-1234567890abcdef12345', + 'password="correct \\\\\\"horse\\\\\\" battery staple"', + "secret='top \\\\\\'secret\\\\\\' key phrase'", + 'passwordHash="$2b$12$e8uq4abcdefghijklmnopqrstuvwxyz1234567890"', + "password_hash='$2b$12$snakecasehash1234567890abcdefghijklmn'", + 'password hash: "spacedhash1234567890abcdefghijklmn"', + '{"authorization": "Bearer json_auth_token_xyz", "cookie": "session=json_cookie_val", "password": "json_password_secret", "password_hash": "$2b$12$json_hash_val"}', + '{"cookie":["theme=dark","session=raw-secret-cookie-val"]}', + '{"token":["tok1","secret-array-token-val"]}', + '{"authorization":{"type":"Bearer","token":"nested-secret-auth-token"}}', + '{"authorization":{"meta":{"type":"Bearer"},"credential":"deeply-nested-raw-secret"}}', + 'Authorization: Basic dXNlcjpwYXNzd29yZA==,', + 'Cookie: session=xyz123; token=abc456; other=789', + ]; + const err = new Error(parts.join(' and ')); + err.name = 'CustomError password="supersecretname"'; + throw err; + `); + t.after(() => fs.rmSync(logDir, { recursive: true, force: true })); + + const uncaught = records.find((r) => r.kind === 'uncaughtExceptionMonitor'); + assert.ok(uncaught, 'uncaught record exists'); + const msg = uncaught?.err?.message ?? ''; + const stack = uncaught?.err?.stack ?? ''; + const errName = uncaught?.err?.name ?? ''; + assert.ok(errName.includes('password=[REDACTED]'), 'redacts password in error name'); + assert.ok(!errName.includes('supersecretname'), 'raw password not present in error name'); + assert.ok(msg.includes('postgres://[REDACTED_CREDS]@db.internal:5432/chess'), 'redacts postgresql credentials'); + assert.ok(msg.includes('postgres://[REDACTED_CREDS]@10.0.0.1:5432/main'), 'redacts postgres credentials'); + assert.ok(msg.includes('[REDACTED_API_KEY]'), 'redacts OpenAI-style api key'); + assert.ok(msg.includes('password=[REDACTED]'), 'redacts double-quoted password with escaped quotes'); + assert.ok(msg.includes('secret=[REDACTED]'), 'redacts single-quoted secret with escaped quotes'); + assert.ok(msg.includes('passwordHash=[REDACTED]'), 'redacts passwordHash'); + assert.ok(msg.includes('password_hash=[REDACTED]'), 'redacts password_hash'); + assert.ok(msg.includes('password hash=[REDACTED]'), 'redacts password hash'); + assert.ok(msg.includes('"authorization"=[REDACTED]'), 'redacts JSON-quoted authorization'); + assert.ok(msg.includes('"cookie"=[REDACTED]'), 'redacts JSON-quoted cookie'); + assert.ok(msg.includes('"password"=[REDACTED]'), 'redacts JSON-quoted password'); + assert.ok(msg.includes('"password_hash"=[REDACTED]'), 'redacts JSON-quoted password_hash'); + assert.ok(msg.includes('"token"=[REDACTED]'), 'redacts JSON array-valued token'); + assert.ok(msg.includes('Authorization=[REDACTED]'), 'redacts Basic authorization header'); + assert.ok(msg.includes('Cookie=[REDACTED]'), 'redacts multi-cookie header'); + assert.ok(!msg.includes('s3cr3tpass'), 'raw password 1 not present'); + assert.ok(!msg.includes('supersecret'), 'raw password 2 not present'); + assert.ok(!msg.includes('horse'), 'raw quoted password content with escaped quotes not present'); + assert.ok(!msg.includes('key phrase'), 'raw single-quoted secret content with escaped quotes not present'); + assert.ok(!msg.includes('$2b$12$e8uq4abcdefghijklmnopqrstuvwxyz1234567890'), 'raw passwordHash absent from message'); + assert.ok(!msg.includes('$2b$12$snakecasehash1234567890abcdefghijklmn'), 'raw password_hash absent from message'); + assert.ok(!msg.includes('spacedhash1234567890abcdefghijklmn'), 'raw password hash absent from message'); + assert.ok(!msg.includes('json_auth_token_xyz'), 'raw JSON authorization absent from message'); + assert.ok(!msg.includes('json_cookie_val'), 'raw JSON cookie absent from message'); + assert.ok(!msg.includes('json_password_secret'), 'raw JSON password absent from message'); + assert.ok(!msg.includes('$2b$12$json_hash_val'), 'raw JSON password_hash absent from message'); + assert.ok(!msg.includes('raw-secret-cookie-val'), 'raw array cookie value absent from message'); + assert.ok(!msg.includes('secret-array-token-val'), 'raw array token value absent from message'); + assert.ok(!msg.includes('nested-secret-auth-token'), 'raw nested object auth token absent from message'); + assert.ok(!msg.includes('deeply-nested-raw-secret'), 'deeply nested credential absent from message'); + assert.ok(!stack.includes('$2b$12$e8uq4abcdefghijklmnopqrstuvwxyz1234567890'), 'raw passwordHash absent from stack'); + assert.ok(!stack.includes('$2b$12$snakecasehash1234567890abcdefghijklmn'), 'raw password_hash absent from stack'); + assert.ok(!stack.includes('spacedhash1234567890abcdefghijklmn'), 'raw password hash absent from stack'); + assert.ok(!stack.includes('json_auth_token_xyz'), 'raw JSON authorization absent from stack'); + assert.ok(!stack.includes('json_cookie_val'), 'raw JSON cookie absent from stack'); + assert.ok(!stack.includes('json_password_secret'), 'raw JSON password absent from stack'); + assert.ok(!stack.includes('$2b$12$json_hash_val'), 'raw JSON password_hash absent from stack'); + assert.ok(!stack.includes('raw-secret-cookie-val'), 'raw array cookie value absent from stack'); + assert.ok(!stack.includes('secret-array-token-val'), 'raw array token value absent from stack'); + assert.ok(!stack.includes('nested-secret-auth-token'), 'raw nested object auth token absent from stack'); + assert.ok(!stack.includes('deeply-nested-raw-secret'), 'deeply nested credential absent from stack'); + assert.ok(!msg.includes('dXNlcjpwYXNzd29yZA=='), 'raw basic auth credential not present'); + assert.ok(!msg.includes('session=xyz123'), 'raw cookie session not present'); + assert.ok(!msg.includes('token=abc456'), 'raw cookie token not present'); + assert.ok(!msg.includes('sk-1234567890abcdef12345'), 'raw key not present'); +}); + +test('diagnostic preload: directory mode 0700 and log file mode 0600', (t) => { + const targetDir = path.join(os.tmpdir(), `sigb-mode-test-${Date.now()}-${Math.random().toString(36).slice(2)}`); + const { logDir } = runWithPreload('const a = 1;', { SIGB_LOG_DIR: targetDir }); + t.after(() => { + fs.rmSync(logDir, { recursive: true, force: true }); + fs.rmSync(targetDir, { recursive: true, force: true }); + }); + + assert.ok(fs.existsSync(targetDir), 'target directory was created'); + const files = fs.readdirSync(targetDir).filter((f) => f.endsWith('.jsonl')); + assert.ok(files.length > 0, 'created log file'); + + const preloadSource = fs.readFileSync(PRELOAD_PATH, 'utf8'); + assert.ok(preloadSource.includes('0o700'), 'mkdirSync specifies 0o700 mode'); + assert.ok(preloadSource.includes('0o600'), 'openSync specifies 0o600 mode'); + + if (process.platform !== 'win32') { + const dirStat = fs.statSync(targetDir); + const fileStat = fs.statSync(path.join(targetDir, files[0])); + // On POSIX, check mode bits + const dirMode = dirStat.mode & 0o777; + const fileMode = fileStat.mode & 0o777; + assert.equal(dirMode, 0o700, `directory mode should be 0700, got ${dirMode.toString(8)}`); + assert.equal(fileMode, 0o600, `file mode should be 0600, got ${fileMode.toString(8)}`); + } +}); + +test('diagnostic preload: pre-existing directory does not fail or crash', (t) => { + const existingDir = fs.mkdtempSync(path.join(os.tmpdir(), 'sigb-existing-')); + const { status, records, logDir } = runWithPreload('const ok = true;', { SIGB_LOG_DIR: existingDir }); + t.after(() => { + fs.rmSync(logDir, { recursive: true, force: true }); + fs.rmSync(existingDir, { recursive: true, force: true }); + }); + + assert.equal(status, 0, 'runs cleanly with pre-existing directory'); + assert.ok(records.length > 0, 'wrote diagnostic records to pre-existing directory'); +}); + +test('diagnostic preload: relative SIGB_LOG_DIR resolves correctly', (t) => { + const relDir = `./tmp-sigb-rel-${Date.now()}-${Math.random().toString(36).slice(2)}`; + const { records, logDir } = runWithPreload('const relOk = true;', { SIGB_LOG_DIR: relDir }); + const absDir = path.resolve(logDir, relDir); + t.after(() => { + fs.rmSync(logDir, { recursive: true, force: true }); + fs.rmSync(absDir, { recursive: true, force: true }); + }); + + assert.ok(fs.existsSync(absDir), 'creates directory at resolved relative path in child cwd'); + assert.ok(records.length > 0, 'reads child diagnostic records from relative SIGB_LOG_DIR'); +}); + +test('diagnostic preload: empty SIGB_LOG_DIR falls back to default directory without path split', (t) => { + const { status, records, logDir, targetLogDir } = runWithPreload('const ok = true;', { SIGB_LOG_DIR: '' }); + t.after(() => { + fs.rmSync(logDir, { recursive: true, force: true }); + if (fs.existsSync(targetLogDir) && targetLogDir !== logDir) { + const files = fs.readdirSync(targetLogDir).filter((f) => f.endsWith('.jsonl')); + for (const file of files) { + if (records.some((r) => file.startsWith(`run-${r.pid}-`))) { + fs.rmSync(path.join(targetLogDir, file), { force: true }); + } + } + } + }); + + assert.equal(status, 0, 'runs cleanly with empty SIGB_LOG_DIR override'); + assert.ok(records.length > 0, 'wrote and read diagnostic records from fallback default directory'); + assert.ok(records.some((r) => r.kind === 'preload-installed'), 'records preload-installed in fallback directory'); +}); + +test('diagnostic preload: usage glob pattern matches test files without literal backslash and excludes diag files', () => { + const correctGlob = 'dist-test/test/**/*.test.js'; + const badEscapedGlob = 'dist-test/test/**\\/*.test.js'; + const diagFilePath = 'dist-test/test/diagnostics/signature-b-preload-abort.diag.js'; + + // Bad glob contains a literal backslash before slash + assert.ok(badEscapedGlob.includes('\\/'), 'bad glob contains literal escaped slash'); + assert.ok(!correctGlob.includes('\\/'), 'correct glob contains clean POSIX path separator'); + assert.ok(correctGlob.startsWith('dist-test/test/'), 'correct glob targets compiled test files'); + + // Verify that the default test glob pattern excludes .diag.js explicit diagnostic fixtures + assert.ok(!diagFilePath.endsWith('.test.js'), 'diag file is excluded from default *.test.js discovery'); + + const preloadSource = fs.readFileSync(PRELOAD_PATH, 'utf8'); + assert.ok(preloadSource.includes('"dist-test/test/**/*.test.js"'), 'preload usage specifies unescaped glob'); + assert.ok(!preloadSource.includes('**\\/*.test.js'), 'preload usage does not contain escaped backslash in glob'); +}); + +test('diagnostic preload: captures testFile when NODE_TEST_CONTEXT is "child" or "child-v8"', (t) => { + const childRun = runWithPreload('const ok = true;', { NODE_TEST_CONTEXT: 'child' }); + t.after(() => fs.rmSync(childRun.logDir, { recursive: true, force: true })); + + const childRecord = childRun.records.find((r) => r.kind === 'preload-installed'); + assert.ok(childRecord, 'preload-installed record exists for child context'); + assert.equal(childRecord?.nodeTestContext, 'child', 'records nodeTestContext'); + assert.ok(typeof childRecord?.testFile === 'string', 'testFile is a string for child context'); + assert.ok(childRecord?.testFile?.includes('test-target.cjs'), 'testFile captures script path for child context'); + + const v8Run = runWithPreload('const ok = true;', { NODE_TEST_CONTEXT: 'child-v8' }); + t.after(() => fs.rmSync(v8Run.logDir, { recursive: true, force: true })); + + const v8Record = v8Run.records.find((r) => r.kind === 'preload-installed'); + assert.ok(v8Record, 'preload-installed record exists for child-v8 context'); + assert.equal(v8Record?.nodeTestContext, 'child-v8', 'records nodeTestContext'); + assert.ok(typeof v8Record?.testFile === 'string', 'testFile is a string for child-v8 context'); + assert.ok(v8Record?.testFile?.includes('test-target.cjs'), 'testFile captures script path for child-v8 context'); + + const rootRun = runWithPreload('const ok = true;'); + t.after(() => fs.rmSync(rootRun.logDir, { recursive: true, force: true })); + + const rootRecord = rootRun.records.find((r) => r.kind === 'preload-installed'); + assert.ok(rootRecord, 'preload-installed record exists for root context'); + assert.equal(rootRecord?.nodeTestContext, null, 'nodeTestContext is null for non-runner process'); + assert.equal(rootRecord?.testFile, null, 'testFile is null for non-runner process'); +}); +