-
Notifications
You must be signed in to change notification settings - Fork 0
ci: enforce strict zero-test-skip architecture and environment suite partitioning #57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
622c8f8
ci: enforce strict zero-test-skip architecture and environment suite …
sayed710 f03a841
ci: refine zero-skip summary parser and migrate test database in back…
sayed710 903d400
ci: address review findings for signal handling, test compilation, an…
sayed710 27c3c66
ci: close pool before backup-restore drill to prevent connection term…
sayed710 c3e3572
ci: enforce positive test output, partition posix suites, self-contai…
sayed710 27712ec
fix(ci): harden backup-restore drill against teardown error and remov…
sayed710 77767f3
ci: anchor test-count and skip-count parsers to genuine reporter lines
sayed710 f43d350
ci: aggregate test counts and skips across multi-summary test outputs
sayed710 4b67b12
ci: enforce centralized repository-wide hermetic zero-skip orchestrat…
sayed710 5ebeda5
ci: reject TODO and cancelled test metrics across zero-skip enforcer …
sayed710 979d9b8
ci: reconcile TAP directives, raw failures, and per-summary zero-test…
sayed710 81e08de
ci: support unnumbered TAP directives and harden against escaped hash…
sayed710 40d3be2
ci: enforce universal discovery, runner reachability, and stream buff…
sayed710 b6bc8c7
ci: mechanically derive runner reachability from manifests and bound …
sayed710 e123c76
ci: separate stream state in runner, validate playwright config mecha…
sayed710 6374a1a
fix(scripts): handle **/ as zero or more path segments in fallback gl…
sayed710 57a2d83
fix(topology): fail closed on unparseable or non-literal Playwright t…
sayed710 0555ebe
fix(topology): restrict Playwright testDir and testMatch extraction t…
sayed710 bb41a55
fix(topology): derive Playwright reachability via authoritative test …
sayed710 6228891
fix(topology): guard scriptCmd before checking runner type (CodeRabbit)
sayed710 a085542
merge: integrate current main into PR 57
sayed710 9ca26a9
fix(ci): close strict zero-skip enforcement gaps
sayed710 d12180a
docs(test): correct adapter suite guidance
sayed710 39b46d9
fix: enforce per-suite zero-skip accounting
sayed710 5ac8200
fix(ci): close zero-skip parser and offline e2e gaps
sayed710 dc9f764
fix(ci): reconcile nested TAP against top-level plan
sayed710 0610ef2
fix(ci): recognize TAP descriptions and refresh Anthropic test model
sayed710 4a80927
fix(ci): distinguish spec application logs from TAP plans
sayed710 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| name: Live Provider Contract Tests | ||
|
|
||
| # Dedicated, manual-only workflow for live third-party AI provider contract tests. | ||
| # These tests make real network calls to OpenAI / Anthropic APIs and require live API keys. | ||
| # Per repository policy: | ||
| # - They MUST NOT run in automatic PR CI. | ||
| # - They MUST NOT be marked as passed when credentials are unavailable. | ||
| # - They execute strictly on demand when credentials are provided in repository secrets. | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| live-provider-contract: | ||
| name: live provider contract tests (provisioned providers) | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Node 22.x | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '22.x' | ||
| cache: npm | ||
|
|
||
| - name: Install dependencies (reproducible) | ||
| run: npm ci | ||
|
|
||
| - name: Build (dependency order) | ||
| run: npm run build | ||
|
|
||
| - name: Detect provisioned credentials | ||
| id: credentials | ||
| run: | | ||
| has_openai=false | ||
| has_anthropic=false | ||
| if [ -n "${OPENAI_API_KEY}" ]; then has_openai=true; fi | ||
| if [ -n "${ANTHROPIC_API_KEY}" ]; then has_anthropic=true; fi | ||
| echo "has_openai=${has_openai}" >> "$GITHUB_OUTPUT" | ||
| echo "has_anthropic=${has_anthropic}" >> "$GITHUB_OUTPUT" | ||
| if [ "${has_openai}" = false ] && [ "${has_anthropic}" = false ]; then | ||
| echo "::error::At least one live-provider credential must be configured." | ||
| exit 1 | ||
| fi | ||
| env: | ||
| OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | ||
|
|
||
| - name: Test AI orchestrator OpenAI contract | ||
| if: steps.credentials.outputs.has_openai == 'true' | ||
| run: npm run test:live-provider:openai --workspace @chess-platform/ai-orchestrator | ||
| env: | ||
| OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} | ||
|
|
||
| - name: Test AI orchestrator Anthropic contract | ||
| if: steps.credentials.outputs.has_anthropic == 'true' | ||
| run: npm run test:live-provider:anthropic --workspace @chess-platform/ai-orchestrator | ||
| env: | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | ||
|
|
||
| - name: Test AI features OpenAI contract | ||
| if: steps.credentials.outputs.has_openai == 'true' | ||
| run: npm run test:live-provider:openai --workspace @chess-platform/ai-features | ||
| env: | ||
| OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} | ||
|
|
||
| - name: Test AI features Anthropic contract | ||
| if: steps.credentials.outputs.has_anthropic == 'true' | ||
| run: npm run test:live-provider:anthropic --workspace @chess-platform/ai-features | ||
| env: | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| import { test } from 'node:test'; | ||
| import assert from 'node:assert/strict'; | ||
| import { spawn } from 'node:child_process'; | ||
| import { once } from 'node:events'; | ||
| import { mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; | ||
| import { tmpdir } from 'node:os'; | ||
| import { dirname, join } from 'node:path'; | ||
| import { fileURLToPath, pathToFileURL } from 'node:url'; | ||
|
|
||
| const REPO_ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..', '..'); | ||
|
|
||
| const armScript = (dir, body, options = 'undefined') => ` | ||
| import { existsSync, writeFileSync } from 'node:fs'; | ||
| import { armFailureEvidence, buildEvidence, clearEvidence, writeEvidence } from ${JSON.stringify( | ||
| pathToFileURL(join(REPO_ROOT, 'scripts/lib/run-evidence.mjs')).href, | ||
| )}; | ||
| const DIR = ${JSON.stringify(dir)}; | ||
| const FILE = 'evidence.json'; | ||
| const build = (exitCode) => buildEvidence({ | ||
| harness: 'test', outcome: 'aborted', exitCode, | ||
| startedAt: '2026-08-27T10:00:00.000Z', finishedAt: '2026-08-27T10:00:01.000Z', | ||
| }); | ||
| clearEvidence(DIR, FILE); | ||
| const fallback = armFailureEvidence(DIR, FILE, build, ${options}); | ||
| ${body} | ||
| `; | ||
|
|
||
| test('a real SIGTERM leaves the artifact and still terminates by the signal', async () => { | ||
| const dir = mkdtempSync(join(tmpdir(), 'gambit-evidence-')); | ||
| const file = join(dir, 'child.mjs'); | ||
| writeFileSync(file, armScript(dir, "console.log('armed');\nsetTimeout(() => {}, 60_000);"), 'utf8'); | ||
|
|
||
| const child = spawn(process.execPath, [file], { cwd: REPO_ROOT, encoding: 'utf8' }); | ||
| await once(child.stdout, 'data'); | ||
| child.kill('SIGTERM'); | ||
| const [code, signal] = await once(child, 'exit'); | ||
|
|
||
| assert.equal( | ||
| signal, | ||
| 'SIGTERM', | ||
| 'the handler re-raises after writing, so the process still dies BY the signal rather than ' + | ||
| 'turning an interrupt into an ordinary exit', | ||
| ); | ||
| assert.equal(code, null); | ||
| assert.equal(JSON.parse(readFileSync(join(dir, 'evidence.json'), 'utf8')).exitCode, 143); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.