Add example: self-healing Playwright tests with multi-model fallback

README.md

@@ -241,3 +241,4 @@ Read more about E2B on the [E2B website](https://e2b.dev) and the official [E2B
 - Next.js app with LLM + Code Interpreter and streaming - [TypeScript](./examples/nextjs-code-interpreter)
 - How to run a Docker container in E2B - [Python/TypeScript](./examples/docker-in-e2b)
 - How to run Playwright in E2B - [TypeScript](./examples/playwright-in-e2b)
+- Self-healing Playwright tests with multi-model LLM fallback - [TypeScript](./examples/self-healing-playwright-tests)

examples/self-healing-playwright-tests/.env.template

@@ -0,0 +1,21 @@
+# E2B sandbox API key — get one at https://e2b.dev
+E2B_API_KEY=
+
+# At least one model provider key is required.
+# The router will try them in the order they are configured (see ROUTER_ORDER).
+ANTHROPIC_API_KEY=
+OPENAI_API_KEY=
+GOOGLE_GENERATIVE_AI_API_KEY=
+
+# Comma-separated provider order. Default: anthropic,openai,google
+# Providers without a key are skipped automatically.
+ROUTER_ORDER=anthropic,openai,google
+
+# How many self-healing attempts before giving up. Default: 3
+HEAL_MAX_ATTEMPTS=3
+
+# Optional: pin specific models per provider.
+# Defaults are sensible for early-2026 (Opus 4.7, GPT-5, Gemini 2.5 Pro).
+ANTHROPIC_MODEL=claude-opus-4-7
+OPENAI_MODEL=gpt-5
+GOOGLE_MODEL=gemini-2.5-pro

examples/self-healing-playwright-tests/README.md

@@ -0,0 +1,173 @@
+# Self-Healing Playwright Tests with Multi-Model Fallback
+
+Generate a Playwright test from a natural-language spec, run it inside an isolated E2B sandbox, and — when it fails — feed the page snapshot back to the LLM so it rewrites the test with more robust selectors. Multi-model fallback (Claude → GPT → Gemini) via the Vercel AI SDK keeps the pipeline alive if one provider is down.
+
+The pattern is inspired by the self-healing behaviour used in production at [qualitymax.io](https://qualitymax.io); this is a standalone TypeScript example contributed to the E2B Cookbook.
+
+## Why
+
+Browser tests break the moment a button moves or a class name changes. Two common responses:
+
+1. Brittle CSS selectors + a human on call — the status quo.
+2. Self-healing tests — capture the page state at failure, hand it to an LLM with the original spec, get back a fixed test.
+
+(2) only works if you can run the regenerated test in a clean, isolated environment without contaminating CI. E2B sandboxes give you exactly that: throwaway Firecracker microVMs, ~200 ms cold-start on the pre-baked `playwright-chromium` template.
+
+## Architecture
+
+```
+  TestSpec (NL description + URL)
+        │
+        ▼
+  ┌──────────────┐  fallback chain (Claude → GPT → Gemini)
+  │   router.ts  │  via Vercel AI SDK
+  └──────┬───────┘
+         │ generated Playwright TS
+         ▼
+  ┌──────────────┐  fresh sandbox per attempt
+  │   runner.ts  │  runs test in `playwright-chromium` template
+  └──────┬───────┘
+         │ pass / fail + stdout / stderr + page snapshot
+         ▼
+  ┌──────────────┐  on fail: rewrite prompt with snapshot + errors
+  │   healer.ts  │  back through router → runner, up to N attempts
+  └──────────────┘
+```
+
+Three small modules, ~250 lines total:
+
+- **`src/router.ts`** — multi-model router. Reads `ROUTER_ORDER` from env, drops providers without keys, falls back on any error.
+- **`src/runner.ts`** — E2B execution. Spins up `Sandbox.create('playwright-chromium')`, writes the generated test, installs `@playwright/test` in a user-writable work dir, executes, captures result.
+- **`src/healer.ts`** — failure → snapshot → prompt → retry loop.
+
+## Setup
+
+```bash
+npm install
+cp .env.template .env    # then fill in keys
+npm run typecheck        # sanity check
+npm run example:basic    # run the simplest example
+```
+
+You need:
+
+- An [E2B](https://e2b.dev) API key (free tier is enough).
+- At least one of: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_GENERATIVE_AI_API_KEY`. The router silently drops providers without keys, so a partial setup still works.
+
+Per attempt wall clock: **~30–80 s** (mostly LLM latency + test run). Full healing demo with 2–3 attempts: **~2–4 min**. For CI where you want the ~5–10 s `@playwright/test` install cost gone too, build your own E2B template on top of `playwright-chromium` with it pre-baked.
+
+## Examples
+
+| Script | What it shows |
+| --- | --- |
+| `examples/01-basic-test.ts` | Simplest path: spec → generated test → one sandboxed run, no healing. |
+| `examples/02-self-healing.ts` | The headline. A vague spec produces a brittle test, the failure snapshot is fed back to the LLM, the retry passes. |
+| `examples/03-multi-model-fallback.ts` | Simulate a provider outage by poisoning the Anthropic key. The router falls back to OpenAI / Google. Needs at least two provider keys configured to demonstrate anything meaningful. |
+| `examples/04-parallel-suite.ts` | Run multiple healing tests concurrently in isolated sandboxes. |
+
+Run any of them with `npm run example:<name>` (see `package.json`).
+
+## How healing actually works
+
+The non-obvious bit is what we feed back to the LLM after a failure:
+
+1. **The runner injects a `test.afterEach` capture hook** into the generated test before each attempt (sentinel-bracketed, stripped and reattached on every heal). On any failed test it writes `await page.content()` to a known path inside the sandbox. The LLM never has to remember to do this — see "Battle-tested patterns" below for why that matters.
+2. **The runner** reads that snapshot off the sandbox filesystem after the failed run.
+3. **The healer classifies the failure** (`strict_mode_violation`, `locator_not_found`, `timeout`, `assertion_failed`, `unknown`) and steers the prompt accordingly. Strict-mode violations in particular get a dedicated hint pointing at three concrete fix patterns instead of letting the LLM re-emit equivalent fragile locators.
+4. **The heal prompt** includes the previous code, the truncated stdout/stderr, the page snapshot, and the failure-type-specific hint.
+5. **Each retry** spins up a fresh sandbox, so failures don't compound.
+
+`HEAL_MAX_ATTEMPTS` caps the loop (default `3`).
+
+### What a real run looks like
+
+```
+$ npm run example:healing
+
+→ attempt 1: LLM writes getByRole('link', { name: /Docs|Documentation/i })
+  ✘ strict mode violation: locator resolved to 4 elements
+  → healer reads /app/failure.html, feeds DOM back to the LLM
+→ attempt 2: LLM rewrites with a more specific selector
+  ✓ test passes
+
+Final: ✓ passed
+Attempts: 2
+  attempt 1 -- fail (generated by google)
+  attempt 2 -- pass (generated by google)
+```
+
+Each attempt spins up a fresh sandbox — failures from attempt 1 can't contaminate attempt 2.
+
+## Battle-tested patterns
+
+The first cut of this example trusted the LLM to do more of the wiring. Three problems showed up in production at [qualitymax.io](https://qualitymax.io) and got rolled back into the code here. If you copy this pattern into your own agent, these are the ones worth keeping:
+
+### 1. Inject the page-snapshot hook yourself; don't ask the LLM to write it
+
+Original approach: tell the LLM to call `fs.writeFileSync(somePath, await page.content())` in its own `afterEach`. Three failure modes, none loud:
+
+- LLM omits the hook in the heal pass → no snapshot → healing degrades to guessing from `stderr`.
+- LLM writes to a root-owned path (`/app/...`) → `EACCES` → silent, the test still fails for the original reason but you have no DOM.
+- LLM duplicates `import fs from 'fs'` on heal → `SyntaxError`, every retry now fails before the test runs.
+
+Fix: `src/runner.ts` injects a controlled `test.afterEach` block between sentinels (`// --- begin self-healing capture hook ---` / `// --- end ---`) and strip-and-reattaches it on every heal. The hook uses `require('fs')` inline instead of a top-level import so a duplicated injection can't break the file.
+
+### 2. Classify the failure before prompting
+
+`src/runner.ts → classifyFailure()` buckets failures into one of five types. The healer only injects the strict-mode hint when `strict_mode_violation` matched. Without classification, the strict-mode guidance pollutes prompts for unrelated failures (timeouts, assertion mismatches) and pushes the LLM toward "fixing" non-issues.
+
+Strict-mode-violation handling is the highest-leverage instance: without the explicit hint, the LLM kept swapping `.first()` in or trying a different `[class*=…]` matcher and burning the whole heal budget on the same failure. With the hint pointing at "anchor by unique text and walk up" / "scope by parent role" / "use the exact `data-test` from the snapshot", it usually heals in one extra attempt.
+
+### 3. Sentinel-based progress, not log scraping
+
+`src/runner.ts` emits `QMAX_PHASE:<phase>` markers at known points in the in-sandbox command and forwards them via the optional `onProgress` callback. A grep for a sentinel survives the npm/Playwright version bump that would break a regex over their normal log output. The pattern reads from `commands.run({ onStdout, onStderr })` rather than waiting for the run to finish, so UI consumers see progress in real time.
+
+```ts
+const result = await runInSandbox(generated, {
+  onProgress: (phase) => console.log(`[${phase}]`),
+});
+```
+
+If you want a *live browser feed* on top of progress markers (watch the test drive Chromium while it runs), see the [playwright-live-vnc-feed](../playwright-live-vnc-feed) example.
+
+## Why multi-model fallback matters
+
+LLM provider availability is not 100%. On 15 April 2026 an Anthropic incident took down Claude API access for several hours. Single-provider agents went dark; anything with a router survived.
+
+`examples/03-multi-model-fallback.ts` reproduces the failure mode locally by poisoning the Anthropic key. You need at least two provider keys configured for this to show anything meaningful — with only one key the router has nowhere to fall back to. Output on a two-key setup:
+
+```
+→ Anthropic key is poisoned. Router should fall back.
+✓ test generated -- the router landed on openai
+✓ test passed
+```
+
+Production code will want to:
+- Distinguish transient (5xx, rate limit) from permanent (auth) errors and retry transients on the same provider before falling back.
+- Track provider-level success rates and reorder dynamically.
+- Cap total wall-clock so a slow fallback doesn't block the loop.
+
+The pattern here is the floor, not the ceiling.
+
+## Notes on the `playwright-chromium` template
+
+The template ships `playwright` + browser binaries pre-installed at `/app`, but not `@playwright/test` (the test runner). The runner handles this:
+
+- `/app` is root-owned, so test files and additional npm installs go to `/home/user/work` instead.
+- `@playwright/test` is installed there, pinned to `1.51.1` to match the template's bundled Playwright version (browser/runner version skew causes "chrome-headless-shell-<rev>: cannot execute binary" failures).
+- `PLAYWRIGHT_BROWSERS_PATH` points at `/app/node_modules/playwright-core/.local-browsers` so the test runner re-uses the template's Chromium instead of downloading its own.
+
+All three bits are documented inline in `src/runner.ts`.
+
+## Adapting it
+
+- **Use a different test framework.** Replace the `ensureTestRunner` step in `src/runner.ts` and the `SYSTEM_RULES` in `src/healer.ts`. Cypress, Vitest browser mode, Pytest + Selenium — anything that runs in a Linux sandbox works.
+- **Add more providers.** `src/router.ts` maps provider names to AI SDK clients. Add a case to `modelFor()` and an entry to `ENV_KEY_BY_PROVIDER`.
+- **Build a custom E2B template.** Pre-bake `@playwright/test` on top of `playwright-chromium` to drop each attempt back toward the 200 ms cold-start floor.
+- **Wire it into CI.** The whole entry point is `runHealingTest(spec, maxAttempts)`. Call it from a GitHub Action and post the trace to your build summary.
+
+## Credits
+
+- Self-healing pattern inspired by [qualitymax.io](https://qualitymax.io).
+- Sandbox runtime: [E2B](https://e2b.dev).
+- Model orchestration: [Vercel AI SDK](https://sdk.vercel.ai).

examples/self-healing-playwright-tests/examples/01-basic-test.ts

@@ -0,0 +1,53 @@
+/**
+ * Example 1 -- the simplest path through the system.
+ *
+ * Generate a Playwright test from a one-line natural-language spec
+ * and run it once in a fresh E2B sandbox. No healing.
+ *
+ * Run:  pnpm tsx examples/01-basic-test.ts
+ */
+
+import 'dotenv/config';
+
+import { route } from '../src/router.js';
+import { runInSandbox } from '../src/runner.js';
+import type { GeneratedTest, TestSpec } from '../src/types.js';
+
+const spec: TestSpec = {
+  url: 'https://e2b.dev',
+  description:
+    'The page should mention "AI agents" somewhere on it within 10 seconds of loading.',
+};
+
+const PROMPT = `You are an expert Playwright test author.
+Output a single TypeScript file -- no markdown fences, no commentary.
+Use @playwright/test, import { test, expect }.
+
+Spec:
+URL: ${spec.url}
+What to verify: ${spec.description}
+
+Return only the TypeScript file content.`;
+
+async function main(): Promise<void> {
+  console.log('→ generating test...');
+  const { text, provider } = await route(PROMPT);
+  const generated: GeneratedTest = { code: text.trim(), provider };
+  console.log(`✓ generated by ${provider} (${generated.code.length} chars)`);
+
+  console.log('→ running in E2B playwright-chromium sandbox (~30-50s)...');
+  const result = await runInSandbox(generated);
+  console.log(result.passed ? '✓ test passed' : '✗ test failed');
+  if (!result.passed) {
+    console.log('--- stdout ---');
+    console.log(result.stdout);
+    console.log('--- stderr ---');
+    console.log(result.stderr);
+    process.exitCode = 1;
+  }
+}
+
+main().catch((err) => {
+  console.error(err);
+  process.exit(1);
+});

examples/self-healing-playwright-tests/examples/02-self-healing.ts

@@ -0,0 +1,51 @@
+/**
+ * Example 2 -- the headline example.
+ *
+ * Run a test that is *deliberately likely to fail on the first try*
+ * (description is vague enough that the LLM may pick brittle selectors).
+ * The healing loop kicks in: failure stdout + a captured page snapshot
+ * are fed back into the LLM, which rewrites the test with more robust
+ * selectors. We retry up to HEAL_MAX_ATTEMPTS times.
+ *
+ * This is the qualitymax.io self-healing pattern, ported to E2B.
+ *
+ * Run:  pnpm tsx examples/02-self-healing.ts
+ */
+
+import 'dotenv/config';
+
+import { runHealingTest } from '../src/healer.js';
+import type { TestSpec } from '../src/types.js';
+
+const spec: TestSpec = {
+  url: 'https://e2b.dev',
+  description:
+    'Click whatever link or button appears to lead to the documentation, ' +
+    'then verify the destination page contains the word "sandbox".',
+  context:
+    'The site nav may have changed since the LLM was trained. Prefer ' +
+    'role / accessible-name / data-testid based selectors.',
+};
+
+async function main(): Promise<void> {
+  const trace = await runHealingTest(spec, 3);
+
+  console.log(`\nFinal: ${trace.finalPassed ? '✓ passed' : '✗ failed'}`);
+  console.log(`Attempts: ${trace.attempts.length}`);
+
+  trace.attempts.forEach((a) => {
+    console.log(
+      `  attempt ${a.attempt} -- ${a.result.passed ? 'pass' : 'fail'} ` +
+        `(generated by ${a.generated.provider})`,
+    );
+  });
+
+  if (!trace.finalPassed) {
+    process.exitCode = 1;
+  }
+}
+
+main().catch((err) => {
+  console.error(err);
+  process.exit(1);
+});

examples/self-healing-playwright-tests/examples/03-multi-model-fallback.ts

@@ -0,0 +1,52 @@
+/**
+ * Example 3 -- show the router fall back from a "broken" provider.
+ *
+ * We simulate the Anthropic outage of 15 April 2026 by overriding the
+ * Anthropic key with a guaranteed-bad value. The router should detect
+ * the auth failure and move on to OpenAI (or Google). The test still
+ * gets generated and run.
+ *
+ * This is the production behavior that kept qualitymax.io serving
+ * customers during that incident -- the architecture is the same here.
+ *
+ * Run:  pnpm tsx examples/03-multi-model-fallback.ts
+ */
+
+import 'dotenv/config';
+
+import { route } from '../src/router.js';
+import { runInSandbox } from '../src/runner.js';
+import type { GeneratedTest } from '../src/types.js';
+
+// Simulate the outage by poisoning Anthropic auth before the SDK reads it.
+// We keep OpenAI / Google keys real so the fallback chain has somewhere
+// to go.
+process.env.ANTHROPIC_API_KEY = 'sk-ant-deliberately-invalid-for-demo';
+
+const PROMPT = `You are an expert Playwright test author.
+Output a single TypeScript file -- no markdown fences, no commentary.
+Use @playwright/test, import { test, expect }.
+
+Spec:
+URL: https://e2b.dev
+What to verify: page <title> is non-empty.
+
+Return only the TypeScript file content.`;
+
+async function main(): Promise<void> {
+  console.log('→ Anthropic key is poisoned. Router should fall back.');
+  const { text, provider } = await route(PROMPT);
+  console.log(`✓ test generated -- the router landed on ${provider}`);
+
+  const generated: GeneratedTest = { code: text.trim(), provider };
+  const result = await runInSandbox(generated);
+  console.log(result.passed ? '✓ test passed' : '✗ test failed');
+  if (!result.passed) {
+    process.exitCode = 1;
+  }
+}
+
+main().catch((err) => {
+  console.error(err);
+  process.exit(1);
+});