Skip to content

[ENHANCEMENT] Add Electron visual smoke tests and migrate Playwright components #1358

Description

@zoomote

Problem (one or two sentences)

Zoo Code visual tests provide broad component coverage but do not currently verify that those components render identically inside the real VS Code Extension Development Host. The component harness also still uses @playwright/experimental-ct-react, while current Playwright guidance uses the stable story-gallery component approach.

Context (who is affected and when)

Maintainers reviewing theme, spacing, sizing, and shell-integration changes need both fast component-level diffs and a small production-parity signal from the actual VS Code Electron host. This follow-up was identified while reviewing #1344.

A feasibility probe successfully launched the pinned official VS Code 1.100.0 binary through Playwright Electron, opened the real Zoo Code sidebar, discovered the nested vscode-webview:// frame, queried the live Welcome view, and captured the actual webview pixels.

Desired behavior (conceptual, not technical)

Keep broad, deterministic component visual coverage while adding a small set of real Extension Host snapshots that prove Zoo Code renders correctly inside VS Code. Move the component harness to Playwright current stable component-testing architecture without losing existing theme/state coverage.

Constraints / preferences

  • Keep component visuals as the broad state/theme matrix; do not replace them with a large Electron matrix.
  • Start with 2-4 Electron integration snapshots covering Welcome, one populated chat state, representative theme propagation, and at most one shell-dependent overlay.
  • Capture the webview/sidebar region instead of the entire workbench unless VS Code chrome is under test.
  • Pin VS Code, Playwright, Linux image, fonts, locale, scale factor, window/sidebar geometry, workspace, and user profile.
  • Use semantic extension/webview readiness signals instead of arbitrary sleeps.
  • Keep the Electron job separate from existing behavioral E2E and component visual jobs.
  • Treat Playwright Electron as experimental and verify compatibility whenever the pinned VS Code version changes.

Request checklist

  • I searched existing Issues and Discussions for duplicates.
  • This describes a specific problem with clear context and impact.

Zoo Code Task Links

Acceptance criteria

  • The component suite is migrated from @playwright/experimental-ct-react to the current stable Playwright story-gallery approach without reducing existing visual coverage.
  • A dedicated serial Electron visual runner launches the same VS Code version pinned by apps/vscode-e2e with isolated workspace, user-data, and extensions directories.
  • The runner uses deterministic scene/theme setup and finds the Zoo Code nested webview semantically.
  • Dark and high-contrast snapshots cover the real Welcome view at webview and/or sidebar scope.
  • At least one representative populated chat state is captured in the real Extension Host.
  • Repeated Electron snapshots are stable before the CI job becomes required.
  • CI failures retain expected, actual, diff, trace, and diagnostic workbench artifacts.
  • Existing component visuals, theme-fixture drift checks, behavioral E2E, and full repository tests remain green.

Proposed approach

Use downloadAndUnzipVSCode() for the pinned binary and Playwright _electron.launch() for process ownership. Add a lightweight extension-host scene controller that uses the existing exported Zoo Code test API and filesystem or local IPC readiness handshakes. Scope snapshots to the real iframe/sidebar, fix workbench and sidebar dimensions, and keep the initial Electron matrix intentionally small.

Migrate component tests separately within the same work item to Playwright stable story files and gallery mounting, preserving production CSS imports, providers, mocks, snapshot naming, and the pinned visual-regression container.

Trade-offs / risks

Electron snapshots are slower and noisier than component snapshots because VS Code chrome, focus, notifications, fonts, process startup, and upgrades affect pixels. Playwright Electron is experimental and depends on packaged Electron fuse compatibility. These risks are why Electron should remain a small integration layer rather than replacing component visual coverage.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions