Skip to content

test: rework bundle-policy skips into expectation updates for the bundled tree - #182

Merged
ryan-dyer-sp merged 2 commits into
brightfire/999239d745d/bundle-all-pluginsfrom
skip-bundle-policy-tests
Sep 8, 2026
Merged

ryan-dyer-sp merged 2 commits into
brightfire/999239d745d/bundle-all-pluginsfrom
skip-bundle-policy-tests

Conversation

@vashbrightfire

@vashbrightfire vashbrightfire Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

What Problem This Solves

Doctrine (per maintainer ruling): bundle-all-plugins is an intentional contract change — all plugins ship in the core tarball. Tests are evaluated per-assertion, not per-test: old-contract assertions are removed or surgically dropped (Rule 1: whole case = externality/exclusion → skip or delete; Rule 2: exclusion is one portion of a broader valid contract → keep the rest, drop the condition; Rule 3: bundled-content presence/correctness → unskip and update expected values to patched-tree truth, so the test enforces the new contract). Real bundle-content regressions are never masked.

The original head of this branch blanket-skipped every bundle-all-affected test with a generic "asserts upstream's external-plugin bundling policy" comment. The codex review (7 comments) showed most of those tests assert contracts that survive bundle-all-plugins — they just needed their expectations updated to the patched tree — while two carried test files were foreign to this base entirely. This rework converts the skips into expectation updates, deletes the dead tests, and keeps only skips whose asserted policy is genuinely reversed by bundle-all (each verified by unskipping and running it).

Why This Change Was Made

Squashed to a single commit (29753e7) implementing all 7 review findings:

  1. Startup activation inventory (bundled-plugin-metadata): expected inventory updated to the verified 34-plugin onStartup set (adds imap, openai, visitor-access); empty-config Gateway startup inventory updated (cua-computer/openai in, diagnostics-otel out — its enabledByDefault is removed by the patch deliberately). Note: the review's named examples verified differently — brave/slack declare onStartup: false and diagnostics-otel was already listed; the real gaps were the three added ids.
  2. release-check sidecars: dist/extensions/slack/runtime-api.js flipped from not.toContain to toContain — listBundledPluginPackArtifacts() now includes it.
  3. Foreign tests removed: test/scripts/write-unified-entry-dts.test.ts and test/scripts/tsdown-declaration-fixture.ts are git rm'd — neither exists at base 999239d745d and scripts/write-unified-entry-dts.ts does not exist anywhere in the tree.
  4. Channel docs index: docs/channels/index.md regenerated via pnpm channels:catalog:gen (23 channels reclassified "official plugin" → "bundled plugin"); docs.json nav unchanged; guard test active.
  5. Static asset inventory (runtime-postbuild): diffs/diffs-language-pack viewer runtime assets added; source assertions flipped positive; no-fs-scan assertion kept active.
  6. Publisher inventory (release-plan-producer): counts updated to verified values — 4 npm (core packages only) and 0 ClawHub, since isPluginExternalPublicationDeferred() treats bundledDist: true as publication-deferred; test retitled to match.
  7. Slack configured-state contract (bundled-plugin-metadata): ./configured-state / hasConfiguredSlackChannelState expectations restored (still declared in extensions/slack/package.json); test active.

Bonus converts (same evidence standard, tests whose real contracts survive):

  • openclaw-npm-postpublish-verify: acpx moved from the "must not flag" list to the expected-missing list (now a bundled extension root); file fully green, zero skips.
  • bundled-plugin-metadata "reflects bundled manifest edits": restored to the upstream base shape (clearPluginMetadataLifecycleCaches between reads) instead of the older foreign variant the patch had swept in.

Follow-up commit 19ee23d8ff2 applies the per-assertion doctrine:

  • Rule 2 surgery (bundled-plugin-build-entries): the six mixed cases are active again — their bundled build-entry structure assertions (entry presence for acpx/googlechat/line, amazon-bedrock/-mantle/anthropic-vertex, copilot/openshell/slack/tokenjuice, synthetic, imessage; Docker-selection parsing, exact entry mapping, deterministic ordering, reorder stability, pack-artifact selection-independence) are kept, and only the superseded pack-artifact exclusion conditions were removed. test/scripts/bundled-plugin-build-entries.test.ts now: 26 passed / 7 Rule-1 skips.
  • Rule 3 (runtime sidecar path baseline): regenerated scripts/lib/bundled-runtime-sidecar-paths.json via pnpm runtime-sidecars:gen — it is generated output derived from bundled plugin metadata plus the root package.json files list (both changed by the patch), not product source — and the baseline guard is active. bundled-plugin-metadata.test.ts now: 37/37 green, zero skips; all consumer suites green (openclaw-npm-postpublish-verify, release-check, test-projects, and the src/infra/update-global*/package-update-steps* family: 283/283).

Remaining skips — all Rule 1 (entire assertion = external-plugin exclusion, the policy this patch reverses; each unskipped and run to confirm): the seven pure-exclusion build-entries cases (byteplus/cohere/meta/mistral/novita/opencode/xiaomi, vydra, comfy, teams-meetings/zoom-meetings, duckduckgo, voyage, volcengine), tsdown dist-graph exclusion (amazon-bedrock/-mantle out of root dist graph), release-check Docker-selected external plugin trees, official external provider endpoint mirroring (no dist-excluded plugins remain to mirror), and dual-published plugin selectability (the publication inventory is now covered by the active Rule-3 publisher test).

User Impact

None — test files and generated docs only. No product code changes. The regenerated channel docs index now truthfully states which channels ship bundled in the core package.

Evidence

Full six-patch stack rebuilt fresh from 999239d745d (upstream-test-fixes 5cb15f9a095 + slack-mrkdwn + cli-http-fallback 2966a901bec + webhook-sessiontarget-support + otel-improvements + bundle-all-plugins 43c8b3b545 + this branch 29753e70051), clean replay with zero conflicts. Final sweep of every test file touched by the patch branch:

  • bundled-plugin-metadata.test.ts 36 passed / 1 justified skip; release-check.test.ts 62 passed / 1 kept skip; runtime-postbuild.test.ts 40/40; official-channel-catalog.test.ts 20 passed / 1 canonical skip; release-plan-producer.test.ts green; openclaw-npm-postpublish-verify.test.ts 57/57 green; doc-baseline.integration.test.ts fully green; all other patch-touched files green (see PR comments for per-file numbers).
  • Local-only divergences, all verified pre-existing and unrelated to this branch: missing-configured-plugin-install.test.ts fails identically on the clean unpatched base (live gateway owns the state-DB lock on this host); docs-i18n.test.ts Go-toolchain guard (from canonical upstream-test-fixes) crashes on this Go-less host; corner-shape.browser.test.ts known local browser-env artifact; tui-pty e2e not collected outside its gated project. CI is authoritative for these.
  • oxfmt --check clean on every touched file.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2782f05b9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/plugins/bundled-plugin-metadata.test.ts Outdated
Comment thread test/release-check.test.ts Outdated
Comment thread test/scripts/write-unified-entry-dts.test.ts Outdated
@ryan-dyer-sp

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-08T15:37:27.168681Z 7eaa794 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7eaa794dd6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/official-channel-catalog.test.ts Outdated
Comment thread test/scripts/runtime-postbuild.test.ts Outdated
Comment thread test/scripts/release-plan-producer.test.ts Outdated
Comment thread src/plugins/bundled-plugin-metadata.test.ts Outdated
…dled tree

Rework of PR #182 per codex review: tests that assert contracts which
survive bundle-all-plugins are active again with expectations updated to
the patched tree; patch-carried dead tests are removed.

- bundled-plugin-metadata: startup activation inventory now includes
  imap, openai, visitor-access (34 onStartup plugins); empty-config
  Gateway startup inventory updated (cua-computer/openai in,
  diagnostics-otel out after its enabledByDefault removal); Slack
  configured-state specifier/export expectations restored; manifest-edit
  lifecycle test restored to the upstream shape (explicit
  clearPluginMetadataLifecycleCaches).
- release-check: slack runtime-api.js is now a required bundled sidecar
  (negative expectation flipped positive).
- runtime-postbuild: diffs/diffs-language-pack viewer runtime assets are
  expected in the static asset inventory (bundledDist: true).
- release-plan-producer: publisher inventory is 4 npm (core only) and 0
  ClawHub packages - bundled plugins defer external publication.
- openclaw-npm-postpublish-verify: acpx is now a required bundled
  extension root on installed packages.
- official-channel-catalog: docs index regenerated via
  pnpm channels:catalog:gen (bundled classification for slack,
  googlechat, discord, matrix, signal, whatsapp, ...).
- write-unified-entry-dts.test.ts + tsdown-declaration-fixture.ts
  removed: carried by this patch, absent at the 999239d base, and
  scripts/write-unified-entry-dts.ts does not exist anywhere in the
  tree.

Kept skips (verified failing for policy-reversal, not stale
expectations): build-entries exclusion tests, tsdown dist-graph
exclusion, Docker-selected external plugin trees, official external
provider endpoint mirroring, dual-published plugin selectability, and
the runtime sidecar path baseline (needs a product-code update to
src/plugins/runtime-sidecar-paths.ts, out of scope for this test-only
branch).
@vashbrightfire
vashbrightfire Bot force-pushed the skip-bundle-policy-tests branch from 7eaa794 to 29753e7 Compare September 8, 2026 16:26
@vashbrightfire vashbrightfire Bot changed the title skip tests asserting upstream external-plugin bundling policy test: rework bundle-policy skips into expectation updates for the bundled tree Sep 8, 2026
bundle-all-plugins is an intentional contract change: all plugins ship in
the core tarball. Tests are evaluated per-assertion, not per-test:

- bundled-plugin-build-entries: six skipped cases reworked per Rule 2 —
  the bundled build-entry structure assertions (entry presence, Docker
  selection parsing/mapping/ordering, pack-artifact selection
  independence) are kept active; only the superseded pack-artifact
  exclusion conditions are removed. The seven pure-exclusion cases
  (entire assertion = externalized plugin kept out of artifacts)
  remain skipped per Rule 1.
- runtime sidecar path baseline: Rule 3 — the baseline JSON is
  generated output (pnpm runtime-sidecars:gen derives it from bundled
  plugin metadata and the root package files list, both changed by the
  bundle-all patch), so it is regenerated to the patched-tree truth and
  the baseline check is active again.
@ryan-dyer-sp
ryan-dyer-sp merged commit 51e69dc into brightfire/999239d745d/bundle-all-plugins Sep 8, 2026
166 of 173 checks passed
@ryan-dyer-sp
ryan-dyer-sp deleted the skip-bundle-policy-tests branch September 8, 2026 17:18
@vashbrightfire

vashbrightfire Bot commented Sep 8, 2026

Copy link
Copy Markdown
Author

Follow-up on the two remaining CI failures: this PR was merged while the fixes were in verification, so they landed on the skip-bundle-policy-tests branch after the post-merge branch deletion recreated it. They still need to reach brightfire/999239d745d/bundle-all-plugins (cherry-pick d784d166905 + a4a5e073a38 onto the squash head 51e69dc968d9, or a follow-up PR). No product code changed in either commit.

1. checks-fast-baseline-ratchets — d784d166905

src/plugins/bundled-plugin-metadata.test.ts carried a grandfathered oxlint-disable max-lines that was never in config/max-lines-baseline.txt, so the ratchet flagged it as a new suppression. Split into:

  • src/plugins/bundled-plugin-metadata.test.ts — repo metadata, sidecar baseline, public-surface, and generated-path contracts (672 effective lines; test budget 1000)
  • src/plugins/bundled-plugin-activation.test.ts — startup/CLI activation declarations and Gateway startup resolution (205 effective lines)
  • src/plugins/bundled-plugin-metadata.test-support.ts — shared repo-level fixtures and expectation helpers (192 effective lines; follows the existing *-test-support.ts convention — knip's **/*test-support.ts ignores apply, and all exports are consumed by the two test files)

All 37 test cases preserved byte-for-byte (30 + 7), zero skips before and after. The runtime sidecar baseline owner tests stay in the original filename so the check-changed/test-projects mappings keep working unchanged. The activation file routes to the unit-fast-isolated project via the repo's stateful-test-support classifier — the standard routing for test-support consumers. The max-lines suppression is deleted; no baseline entry added.

2. check-dependencies — a4a5e073a38

get-east-asian-width is consumed only by tsdown.config.ts's shouldAlwaysBundleDependency() (bundle-all-plugins), which knip cannot see because the consumption is a string-based bundler policy. Added to the root workspace ignoreDependencies in config/knip.config.ts with a comment naming the consumer — same pattern as the existing highlight.js createRequire entry.

Verification

Fresh six-patch stack rebuilt from 999239d745d (upstream-test-fixes 5cb15f9a0958 + slack-mrkdwn + cli-http-fallback 2966a901bec + webhook-sessiontarget-support + otel-improvements + bundle-all-plugins 43c8b3b545 + this branch), plus the exact pre-merge PR-head tree:

  • pnpm test on both files via repo project routing: 37/37 green (30 in plugins, 7 in unit-fast-isolated)
  • pnpm check:max-lines-ratchet: OK (874 grandfathered, the file's suppression removed)
  • oxfmt --check on all four touched files: clean; targeted oxlint on them: 0 warnings / 0 errors
  • knip production scan with the exact CI command (knip@6.32.2, --production --exclude duplicates): exit 0, no unused-dependency finding
  • knip all-exports scan + deadcode:unused-files at the PR-head tree: exit 0 / pass
  • tsgo core-test shards: pass

One stack-level note (pre-existing, outside this PR): merging the new upstream-test-fixes head (5cb15f9a0958) into the full stack makes the all-exports scan flag src/tui/tui-pty-harness-assertion-test-support.ts: exerciseTerminalOutputSafety as an unused export. Verified present on the six-patch stack without this branch, and absent on the PR-head tree — it first appears when the patches combine, so it will surface in the next full stack build (bf-build-stable), not in per-PR CI.

@vashbrightfire

vashbrightfire Bot commented Sep 8, 2026

Copy link
Copy Markdown
Author

Correction to the commit refs above: to make this follow-up landable after the squash merge, the branch was rebased onto 51e69dc968d9 (the #182 squash head, content-equal through 19ee23d8ff2), so the two fix commits now live at ebca7ca5056 (test split) and 258a0535a2 (knip ignore) — the earlier d784d166905/a4a5e073a38 refs no longer exist on origin. The fixes are delivered as follow-up PR #186 (base brightfire/999239d745d/bundle-all-plugins), which shows only those two commits as its diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant