test(cockpit): repair the rotted manual specs and guard them in CI - #853
Merged
Conversation
The *.manual.ts tier is unreachable: all 38 cockpit playwright configs use
testMatch '**/*.spec.ts', and nothing in any config, project target, or
workflow references it. 34 live-LLM specs that nobody runs and CI cannot
see.
Audited all 34. Ports were fine (0 drift vs cockpit/ports.mjs) and no
vacuous neg-only assertions remained. But 10 asserted UI copy that exists
NOWHERE in the repo:
- deep-agents/{filesystem,memory,planning,sandboxes,skills,subagents}:
welcome-prompt strings replaced by empty-state copy ('No delegations
yet', 'No plan yet', ...)
- langgraph/durable-execution: 'Execution Status' -> 'Pipeline'
- langgraph/memory: 'Agent Memory' -> 'Learned Facts'
- {langgraph,ag-ui}/interrupts: asserted an approvals sidebar that no
longer exists — those examples moved to a chat-approval-card modal with
welcome suggestions. Now assert the suggestion, and the test name no
longer describes a sidebar.
Repairing them isn't enough — nothing would stop the next drift. Since the
specs can't run in CI (real model, per-example dev server), this adds a
tripwire that can: apps/cockpit/scripts/manual-spec-freshness.spec.ts
checks every text and element selector a manual spec asserts still exists
in the source. It lives in apps/cockpit because
NX It's time to update Nx 🎉
Your repository uses a higher version of Nx (22.5.1) than your global CLI version (21.5.2)
For more information, see https://nx.dev/more-concepts/global-nx
> nx run cockpit:test
NX Successfully ran target test for project cockpit actually
runs in CI — libs/e2e-harness would have reproduced the original problem,
since it isn't in the Library job's LIBS list.
69 assertions (34 specs x 2 checks + a self-guard that fails if the glob
ever stops matching the tier). Mutation-tested: reintroducing the exact
stale string this commit removes turns it red and names the file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
blove
enabled auto-merge (squash)
August 29, 2026 21:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
*.manual.tstier is unreachable. All 38 cockpit playwright configs usetestMatch: '**/*.spec.ts', and nothing in any config, project target, or workflow references it. That's 34 live-LLM specs nobody runs and CI cannot see.Audit of all 34
Two things came back clean, and I'd rather report that than pad the diff:
localhost:NNNNstill matches itscockpit/ports.mjsassignment.10 of 34 asserted UI copy that exists nowhere in the repo:
No delegations yet,No plan yet, …)Execution StatusPipelineAgent MemoryLearned FactsNo pending approvalschat-approval-cardmodal + welcome suggestionsThe interrupts pair is the worst: the test name promised "approvals sidebar", and the UI hasn't had one for a long time. Those now assert the welcome suggestion, and the test names no longer describe a sidebar.
The part that matters
Repairing 10 files fixes today and changes nothing about tomorrow — the tier would drift again the same way, silently. Running the specs in CI isn't viable (real model, per-example dev server), but checking that what they assert still exists is cheap.
apps/cockpit/scripts/manual-spec-freshness.spec.tsverifies every text assertion and element selector in every manual spec still appears in the source. It lives inapps/cockpitdeliberately:nx test cockpitruns in CI. My first instinct waslibs/e2e-harness, which would have reproduced the original bug exactly — that project isn't in the Library job'sLIBSlist, so its tests don't run either.69 assertions — 34 specs × 2 checks, plus a self-guard asserting the glob still matches >20 specs, so the suite can't quietly pass over an empty list.
Mutation-tested: reintroducing the exact stale string this PR removes turns it red and names the offending file.
Note on local verification
nx build cockpitfails in my worktree, and I confirmed it fails identically on pristine main with zero changes (Next.js can't infer the workspace root; the worktree'snode_modulesis incomplete). Environmental, not from this diff — which touches only a vitest spec underscripts/and text inside*.manual.tsfiles, neither of which Next compiles.nx test cockpitpasses. CI does a cleannpm ciand will exercise the build properly.🤖 Generated with Claude Code