Skip to content

feat(desktop): verify preview open and page load end to end - #6447

Open
echonesis wants to merge 1 commit into
apache:mainfrom
echonesis:feat/5235-preview-verification
Open

echonesis wants to merge 1 commit into
apache:mainfrom
echonesis:feat/5235-preview-verification

Conversation

@echonesis

Copy link
Copy Markdown

Summary

A reachable preview URL does not establish that a browser loaded the document, that it contains the requested content, or that an interaction worked. ArtifactPreview checks HTTP reachability from Desktop; its loaded: false deliberately leaves browser verification to navigation and observation.

Following the scope agreed in #5235, this change adds structured verification results and regression coverage to the existing ArtifactPreview → navigate → snapshot → click → snapshot workflow.

browser_navigate could previously report Loaded after reading window.location.href failed, using the requested URL as a fallback. Chromium could also expose a readable, committed HTTP 200 document before its transfer failed or was stopped. Observations could survive while another navigation was starting.

The existing navigate, snapshot and click tools now return structured evidence through Desktop’s existing projectToolResult → ClientCapabilityCallResult.structuredContent → Host/Runtime path. Unsuccessful verification produces an error outcome.

Result What is reported
status completed, failed, timeout, unsupported, unknown or invalidated
opened, loaded Current-document browser verification state
url, title Actual document observations; a requested URL never substitutes for an observation
documentId Identity used to associate observations with the same document
matchesRequestedUrl Whether navigation observed the requested URL
evidence, error Observations supporting the result, or why verification could not complete
snapshot Content and references for checking the requested document and subsequent change
click.interaction not_verified; a subsequent observation must establish the requested effect

Load completion and evidence lifetime

The controller derives load facts from its existing Electron events. A new navigation invalidates previous completion evidence before commit, including a same-URL reload. A committed current document requires a positive did-finish-load observation.

A main-frame transfer failure returns failed. Stopping an unfinished load returns invalidated. A document still loading after the existing action budget returns timeout; a document that is no longer loading but has no completion evidence remains unknown.

Observations are checked against the existing lease, view identity, committed epoch and navigation identity. Closing or replacing the view, losing admission, or navigating during an observation invalidates stale evidence. Existing partial-page display, retry and iframe behavior are preserved.

Loading alone does not verify the requested content. The acceptance flow checks the title and marker, then verifies the requested interaction through a later snapshot showing the real JavaScript counter changing from 0 to 1 with the same documentId.

Relationship to #5234 / #5254

Navigation consumes the existing preview_preflight classifications and reasons. url_schemes: unsupported for file/data URLs does not make HTTP preview unsupported. A temporarily unobserved view remains unknown.

A verified loopback endpoint proves a listener on Desktop. It does not establish the identity of a service started by a remote SSH Runtime Host or prove document loading.

Scope and remaining boundary

The agreed verification chain starts with an existing registered HTML Artifact. This change preserves ArtifactPreview’s reachability contract and reuses Origin admission, leases, cancellation and the Browser lifecycle.

Artifact creation and ID discovery remain an existing integration boundary. Bash results are published before background Artifact registration, and that path does not return the newly registered artifactId to the model. The Dev acceptance therefore used an explicitly supplied, verified artifactId. It does not establish autonomous discovery of a newly created Artifact.

No new tool, capability probe, loader, generic interaction verifier, product shell server, Host wire protocol or additional compatibility epoch change is introduced by this feature. The merged main’s protocol version is retained. The file://, CSP, sandbox and authorization policies remain unchanged. Generated Files’ inline about:srcdoc CSP behavior is outside scope.

Observed result URLs omit credentials, query and fragment. Managed preview bearer paths are redacted in browser results and saved verification summaries.

Fixes #5235

Verification

Verified the merged revision b91b6bad4, integrating upstream/main at 0db5ffb7e, on 2026-10-28. Used Node 24.22.1, npm 11.19.0 and Python 3.12.12. A root lockfile installation applied all 14 existing dependency patches.

npm ci --no-audit --no-fund
npm run build
npm test
npm --workspace @maka/runtime-host run test:assistant
npm run typecheck
npm run lint
npm run format:check
npx knip --workspace apps/desktop
npx knip --workspace packages/ui
npm run check:renderer-architecture
npm run check:e2e-budget
npm run check:asf-headers
npm run check:locale-hygiene

npm --workspace @maka/desktop run smoke:preview:run
npm --workspace @maka/desktop run smoke:browser-load-failure:run
  • Desktop: 4,042 passed, 2 skipped, 0 failed. Focused browser/preview/admission suites: 143/143 passed.
  • Production build, including assistant-web, root typecheck, lint, format check, Desktop/UI Knip, ASF headers, locale hygiene and whitespace checks passed.
  • Renderer architecture: 168/168 passed. E2E budget: 5/5 passed, retaining 37 cases across 20 files.
  • Real Electron/Chromium preview smoke passed ArtifactPreview → navigate → snapshot → click → snapshot, checking title Preview 5235, marker PREVIEW_READY_5235, the real inline JavaScript counter 0 → 1, and the same documentId.
  • Negative coverage passed for expired/revoked URLs, refused connections, HTTP 404, committed truncated HTTP 200 responses, stopped loads, bounded unfinished-load timeout and closed/recreated views. Tool tests cover unsupported/unknown preflight classifications, cancellation and navigation-invalidated observations.
  • The existing partial-page, retry and iframe load-failure smoke passed.
  • Implementation-time regression runs failed before the fixes: failed location observation still reported Loaded, navigation during a snapshot retained stale evidence, and real committed partial/stopped loads reported loaded:true. Their corrected coverage passed again on the merged revision.

Maka Dev acceptance

Post-merge GPT-6-Astra acceptance passed using an isolated Dev profile. The user submitted the prompts after CUA input proved unreliable; Codex independently checked the original Runtime tool events, model usage records and source hashes.

The HTML was registered through the existing Bash stdout-redirection recorder. Its actual artifactId was looked up read-only in Runtime records and supplied explicitly because the preceding tool result did not expose it.

All six tools completed successfully in one turn:

Step Actual result
preview_preflight surface unknown, endpoint not_checked; classifications preserved
ArtifactPreview reachable:true, loaded:false
browser_navigate completed, opened:true, loaded:true, title correct, matchesRequestedUrl:true, controller_load_complete; navigation-time preflight verified
Initial browser_snapshot title and marker correct, counter 0
browser_click one exact match, interaction:not_verified
Final browser_snapshot counter 1, unchanged documentId

Runtime usage records confirmed openai-codex / gpt-6-astra. The merged commit and all 12 changed-file hashes were unchanged. This verifies the existing registered-Artifact preview chain with a supplied ID; it does not claim automatic Artifact ID discovery or that the CUA input issue was fixed.

After Dev, Desktop main was cleaned and rebuilt before rerunning the focused suites: 52/52 passed.

Whole-repository results

Whole-repository npm test completed with 16,928 passed, 4 failed and 41 skipped across 16,973 Node tests.

The four failures were in Runtime Host:

Test Full-run failure Diagnostic rerun
assistant Host fallback registers a task workspace usable by ordinary Session creation and binding chat.cwd.startsWith(home) assertion Reproduced with the default macOS temporary path; passed with canonical TMPDIR
production Host publishes and retires an implementation child patch Timed out waiting for terminal state Passed
durable wait consumption invalidates the collection with 66 parallel waits Timed out waiting for the collection condition Passed
an invocation-owned detached Host retires after launcher crash Timed out waiting for Host exit Passed

The path diagnostic changed only TMPDIR, without changing source or assertions. The other three full-run failure causes remain unresolved. Individual diagnostic passes do not establish a passing full suite.

Host/Runtime/Core/Storage sources and the lockfile have no branch-specific changes relative to the merged main. No unrelated product fixes were added. This comparison is not a full unmodified-baseline test run.

The Host assistant phase skipped by the earlier failure was run separately: 84/84 passed. Python Eval completed 94 tests, 13 skipped.

Not run: full Playwright E2E, Windows/Linux GUI acceptance, packaging/release qualification or CI.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: GPT-6.1-Sol (high) via OpenAI Codex — contributed to the implementation, regression tests and this description under my direction, ran the local checks, and inspected the original Runtime tool events. GPT-6-Astra via Maka Dev executed the preview acceptance using an explicitly supplied registered artifactId. The commit carries a Generated-by trailer.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

Generated with OpenAI Codex

@github-actions github-actions Bot added the effort/XL Under 2500 readable lines label Oct 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XL Under 2500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(harness): verify preview open and page load end to end

1 participant