Add promptfoo for testing agent development setup and skills

[jeryj]

What?

Adds a Promptfoo-based evaluation harness for the repository's agent skills so we can evaluate if changes to agent instructions, skills, result in improvement or regression.

Also adds a baseline test as an example that checks to see if our e2e reference is read when asking the agent to write an e2e test, and a second suite that proves the agents really are confined to their workspace.

Why?

We ship guidance for coding agents — AGENTS.md files and .agents/skills/ — with no way to tell whether it is discovered or followed. This makes that measurable rather than assumed.

How?

• Adds promptfoo as a devDependency. The package is deliberately not a root workspace: Promptfoo's dependency tree is large and only needed by people running evals, so it keeps its own lockfile and install step, and needs a newer Node than the repository (.nvmrc). Promptfoo is pinned to 0.122.1 (0.122.2 adds a dependency affected by GHSA-jmr9-qjv8-65gv), with overrides for two optional dependencies; CI audits the tree at --audit-level=high.
• One disposable Git workspace is built per run from your working tree, with test/ai-development/ removed so the agent cannot read the assertions grading it. Rows run serially and the workspace is rolled back with git reset --hard between them. This is Promptfoo's documented shape for an agent with side effects — see its claude-agent-sdk/advanced example. Rolling back rather than rebuilding is what makes one workspace practical: archiving Gutenberg once per run is affordable, once per row is not. We can follow-up with parallel test runs.
• Because .claude/ is .gitignored, the workspace regenerates .claude/skills from .agents/skills by calling the repository's own setup-skills tool, as npm run agents:setup does at postinstall. Before it removes settings or generates skills, the harness replaces a repository-supplied .claude symlink with a local directory so host cleanup cannot follow it outside the workspace.
• Confinement is two layers, because neither covers the other's ground. Promptfoo implements no sandboxing of its own — it passes the sandbox block verbatim to the Claude Agent SDK — so lib/sandbox.js declares both:
  • sandbox.filesystem is OS-enforced and covers Bash and every process it starts. Writes need no rules (a sandboxed command can only write to its working directory and session temporary directory); reads are denied by region — the home directory, this checkout, and the temp directory — with the workspace re-allowed inside it. (Denying / instead does not work: it takes the system libraries with it, and a profile no command can run under either fails everything or is discarded.)
  • Read() / Edit() permission rules cover every other tool. These resolve deny before allow and ignore specificity, so the workspace lives in the system temp directory rather than inside the checkout — otherwise the rules keeping the agent out of the source would lock it out of its own working directory.
  • The network is unreachable — the empty allowlist is made a deterministic denial with strictAllowlist, since an unlisted host otherwise prompts and a headless run resolves that as an allow — and Docker is pointed at a dead socket, so the agent cannot look up answers or start containers the sandbox can't clean up.
  • Hooks and project settings go around both layers: hooks are disabled programmatically (disableAllHooks), and the workspace build strips .claude/settings*.json because filesystem.allowRead and network.allowedDomains merge from every settings source — a branch under evaluation could otherwise re-open reads and the network for its own run.
  • The inherited environment is blanked to a minimal keep-list. Promptfoo restores ANTHROPIC_API_KEY after that step, so the sandbox's credential rules also deny it by name.
• Assertions check what tools returned, not what the agent said. Promptfoo records each call's real output on metadata.toolCalls, which is the provider metadata its guide points at for asserting the path an agent took. So "did it read the reference?" requires successful output containing the file's contents. Path matching remains diagnostic only, which lets a successful relative-path read recover from an earlier failed full-path attempt. The sandbox suite proves denial from the recorded error rather than trusting the agent to quote it.
• utils/ holds what specs import, leaving lib/ to the harness, the way test/e2e separates its configuration from the fixtures its specs pull in. assertRead / assertNotRead work on any text file: landmarks are sampled evenly through it, so output that stops early is missing one, and they are read from the file at load time. Reads can get scored by a full read (1) partial read (1 > score > 0) or no read (0).
• Alongside skill-used and command-trajectory checks, an agent-rubric reviews the change itself.
• The harness captures the diff, rather than the grader going to find it. Promptfoo's coding-agent guide notes that an agent's output is "its final text response describing what it did, not the file contents", and that file-level verification means reading the files after the eval. A transform stages the workspace and appends git status and the diff to the response before any assertion runs. The grader then keeps Promptfoo's documented configuration — working_dir and nothing else, which gives it read-only Read/Grep/Glob/LS already confined to the workspace — so it needs no shell and no sandbox of its own, and can still open .agents/skills/ to check the change against the reference rather than a copy of it pasted into the rubric. Otherwise, we'd need to give the grading agent more access, which would undo our attempts to sandbox it. Before host-side Git stages, diffs, or resets the workspace, the agent-controlled .git is replaced with a trusted copy and global/system Git configuration is disabled, so a repository-local clean filter cannot escape the sandbox. The grader also gets no settings sources, no hooks, and the same blanked environment as the subject.
• Configuration is JavaScript. A spec spreads lib/base.js and adds only its own prompt and assertions; one directory per suite under specs/.
• A static-checks CI job installs the package and runs the unit tests and configuration validation on Node 22 (Linux, macOS, Windows) and Node 24 (Linux). No model calls in CI.
• To simplify this initial PR, Claude is the only agent added for testing. Codex is added in Agent evals: Add Codex as a second agent #82001.

Testing Instructions

1. cd test/ai-development && nvm use && npm install
2. npm run test:utils — unit-tests the read assertions against recorded tool calls, and the workspace lifecycle against a real disposable checkout. No model calls; about half a minute. Two live sandbox tests are skipped unless AI_EVAL_LIVE=1 is set.
3. npm run test:agent-evals — runs every spec. Add --config specs/testing-skill-routing/test-skill-routing.eval.js to run just one.
4. npm --prefix test/ai-development run view opens the local viewer against results/.promptfoo, the same storage directory used by evaluation runs.
5. Confirm the skill-routing result shows separate metrics for invoking the testing skill, reading the e2e reference, skipping the Jest and PHPUnit references, and the agent review.
6. The skill-routing test might fail! That's fine. We're setting up the framework, not trying to solidify the e2e reference reading right now — that's a follow-up. It currently routes through the skill roughly 50% of the time. Notably the rubric passes either way: the agent writes a decent test whether or not it invokes the skill, so it's the routing that's unreliable.

Verifying the sandbox

specs/sandbox proves the boundary by behaviour rather than by inspecting configuration. Each probe has to show the attempt in the trajectory and the canary absent from the output, so an agent that simply declines cannot pass it, plus a control read of the workspace that must succeed — otherwise a wholly broken workspace would pass by failing everything.

To confirm it is testing something real, set sandbox.enabled to false in lib/sandbox.js and empty permissions.deny, then re-run. The suite must go red, reporting the reads that succeeded.

AI_EVAL_LIVE=1 npm run test:utils proves the filesystem shape itself through the raw Agent SDK, without Promptfoo in between: denying / never yields a working boundary (it either fails every command, the workspace read included, or is discarded and leaves reads open), while the shipped region denies read the workspace and refuse canaries in the home directory, the checkout, and the temp directory.

The read assertions have the same property without costing anything: npm run test:utils covers a full read, a chunked read, a read that stopped early, a grep, an ls, a denied read, an unopened file, and the inverses.

Notes before running:

• Each run makes real model calls against your own Claude quota and takes 1–2 minutes.
• Results under results/ are gitignored and may contain source code and tool output.

Possible follow ups:

• Agent evals: Compare guidance changes across two Git refs #82000 compares two Git refs in one table, so a guidance change can be measured against the commit before it.
• Agent evals: Add a pull-requests skill and a suite that checks it #82028 adds a pull-request description suite.
• Agent evals: Add Codex as a second agent #82001 adds Codex as a second agent.
• Make the testing skill route reliably.
• Per-row parallel workspaces, if the serial matrix becomes the bottleneck. Only the lifecycle hook would change.

Testing Instructions for Keyboard

Not applicable; this PR does not change the user interface.

Screenshots or screencast

Not applicable.

Use of AI Tools

Codex (GPT-5) and Claude Code (Opus) were used to implement, review, revise, and document this. All output was reviewed and verified by running the harness.

Summary by CodeRabbit

• New Features

  • Added an AI development evaluation package for assessing agent behavior, sandbox boundaries, workspace isolation, and testing-skill routing.
  • Added commands to validate, run, and review evaluation results.
  • Added cross-platform automated checks across supported Node.js versions.

• Documentation

  • Added setup, usage, security, sandbox, grading, and evaluation-authoring guidance.

• Tests

  • Added coverage for Node.js compatibility, filesystem and network restrictions, environment protection, workspace cleanup, Git isolation, and file-read verification.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: jeryj <jeryj@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: manzoorwanijk <manzoorwanijk@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[github-actions]

Size Change: 0 B

Total Size: 7.92 MB

_{compressed-size-action}

[github-actions]

Flaky tests detected in a6b8365.
Some tests passed with failed attempts. The failures may not be related to this commit but are still reported for visibility. See the documentation for more information.

🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/33421468408
📝 Reported tests:

inserts synced patterns by dragging and dropping from the global inserter in /test/e2e/specs/editor/various/inserting-blocks.spec.js, passed after 1 failed attempt.

Error: apiRequestContext.fetch: socket hang up
Call log:
  - → GET http://localhost:8889/wp-json/wp/v2/blocks?per_page=100&status=publish%2Cfuture%2Cdraft%2Cpending%2Cprivate%2Ctrash
    - user-agent: Playwright/1.62.1 (x64; ubuntu 24.04) node/20.20 CI/1
    - accept: */*
    - accept-encoding: gzip,deflate,br
    - X-WP-Nonce: 81a975a6ae
    - cookie: wordpress_test_cookie=WP%20Cookie%20check; wordpress_logged_in_23778236db82f19306f247e20a353a99=admin%7C1788371609%7CLrW3EMW8MrHFGuKRBIn7nD10GH2rqn2H5oG0qQvjafR%7Ca6b6956c434435a5146e6fe42c51e3e68945a9af5c7d3fbef945f1ec77ef8e1c; wp-settings-time-1=1788199224

    at RequestUtils.rest (/home/runner/work/gutenberg/gutenberg/packages/e2e-test-utils-playwright/src/request-utils/rest.ts:112:39)
    at RequestUtils.deleteAllBlocks (/home/runner/work/gutenberg/gutenberg/packages/e2e-test-utils-playwright/src/request-utils/blocks.ts:23:28)
    at /home/runner/work/gutenberg/gutenberg/test/e2e/specs/editor/various/inserting-blocks.spec.js:11:22

[jeryj]

I'm picking up this PR again this week.

[jeryj]

@ciampo I think this is ready for a review. Hopefully it helps our AI skill project work :)

[ciampo]

Thank you for the work here!

left a few initial comments

[ciampo]

[Also flagged by an AI agent]

The grader gets auto-approved Bash, but withWorkspace() adds only working_dir to its provider. The sandbox below is attached to the subject test options and is not propagated to the grader. The grader can therefore modify the artifact it is judging, access the original checkout and host, call the network or Docker through shell commands, and create a direct false-positive result.

Please use constrained read/search tools, or apply a separate fail-closed grader sandbox that disables hooks, denies writes/network/Docker, and exposes only a minimal environment. A regression case should ask the rubric to create the expected file and prove that the grader cannot do so. This matches Promptfoo's warning that shell, write, and network tools enabled for an agent-rubric are actions performed by the grader itself.

[jeryj]

I think my agent has been over-focused on the deterministic file read and trying to do all kinds of extra work with tools. I'm going to strip this back to the default promptfoo set-up and see what the shape looks like.

[ciampo]

This assertion passes for any shell command containing the path, ie. it doesn't seem to require a successful read or evidence that the file contents were returned.

Should we assert a successful Read invocation for the exact file, or assert successful tool-result evidence containing a canary from the reference?

Potentially we could also add negative fixtures for "path listed", "read failed", and "contents read" so this central metric proves the behavior named in the README?

Not sure if we want to somehow abstract this pattern, since I assume that reading specific docs will be a frequent aspect that we'll want to check in our evals

[jeryj]

Good catch — same flaw was in specs/sandbox, which asserted on the agent's own report. Both fixed.

Added an abstracted assertRead / assertNotRead. A read has to have succeeded and returned the file's contents — landmark lines sampled through the file, so ls, a grep, a denied read, or a head all fail, while a chunked read passes.

Reads are scored by how much came back, so skimming the guidance (0.8) is distinguishable from ignoring it (0). assertNotRead got stronger too — skipping now means none of its contents came back, not just that no command mentioned the path.

The assertRead and assertNotRead also have test coverage: specs/utils/test-grader.mjs.

As for the Read invocation, Bash-only is a deliberate cross-provider choice so Claude's Bash and Codex's exec_command share one assertion; adding Read wouldn't replace Bash and would make the "Skipped" assertions pass vacuously. Better revisited with Codex in #82001.

[ciampo]

Adding a few more comments.

Also, a bit nitty, but the PR description is now out of date under a few aspects, it could be confusing for someone approaching this PR.

[ciampo]

Same as previous comment

[ciampo]

(can't verify because I don't have an anthropic sub, flagged by Codex)

Promptfoo passes the host environment to Claude, including ANTHROPIC_API_KEY when set. Bash can expose these values in model requests and saved results. Please give both providers a minimal environment, remove credentials, and add a test proving that a secret value cannot appear in the output.

[jeryj]

Both providers now have a minimal environment and overwrite the credentials with null values. We can't remove credentials that already exist for the environment, but we can blank them.

The test for this is in specs/sandbox: a marker goes into the environment before the config loads, the agent is asked to echo $EVAL_SANDBOX_PROBE, and the assertion reads the command's recorded output rather than the agent's account of it. It fails if the marker comes back.

[ciampo]

Same as previous comment

[ciampo]

Same as previous point

[jeryj]

Yesterday I was caught up in the RC1 Release. Picking this up again today and will update the PR description. Thanks for the review, @ciampo!

[ciampo]

Of course! In the meantime, I'll wait until you explicitly tag me again for a new round of review.

[jeryj]

Made some big changes to align it closer to the default promptfoo recommendations. We can diverge when absolutely necessary, but we should lean on their recommendations for now. Here's what has changed:

Agent summary:
The eval harness gave every test row its own temp workspace, so no path could be written in a config file — the extension rebuilt both agents' working_dir and sandbox blocks at runtime instead. That was 253 lines, plus a 136-line unit test asserting the rebuild produced the right shape.

It now builds one workspace per run, rolls it back between rows, and runs rows serially — promptfoo's documented pattern for agents with side effects. Paths are known up front, so the rules are just written down.

That also fixed a boundary that didn't hold. Promptfoo implements no sandboxing; it passes sandbox verbatim to the Claude Agent SDK, where allowRead is a re-allow within denied regions, not a whitelist. The old allowRead: [workspace] was close to a no-op. Confinement is now two layers: the OS sandbox for Bash, permission rules for everything else. specs/sandbox proves it behaviourally, and goes red if you disable either.

Separately, the grader no longer holds Bash. It had one only to run git diff, which is what forced it to have a sandbox at all. The harness now captures the diff in a transform and hands it over; the grader keeps promptfoo's default read-only tools.

[jeryj]

@ciampo This is worth another review when you get a chance.

[jeryj]

@ciampo Should we bring this in and iterate?

[ciampo]

A couple of extra updates:

• fixed Windows permission handling;
• protected trusted Git metadata from sandbox writes;
• removed unused Chromium downloads from CI.

Regression coverage and documentation were updated, and all non-live checks pass.

I'll give it one last deep review check and approve + merge

[ciampo]

LGTM 🚀

Let's merge, test, iterate.

[manzoorwanijk]

The package is deliberately not a root workspace: Promptfoo's dependency tree is large and only needed by people running evals, so it keeps its own lockfile and install step, and needs a newer Node than the repository (.nvmrc).

We should avoid this because such isolated islands can be a security nightmare, as tools like Dependabot and other security audits do not usually find such lockfiles.

Now that we have Node 24 in the repo, we could probably clean this up. Also, the workflows here still run on Node 22. We should update those as well. Also, we can switch to ./.github/setup-node instead of actions/setup-node

[manzoorwanijk]

I have created #82665 to update it to use Node 24.

[jeryj]

We should avoid this because such isolated islands can be a security nightmare, as tools like Dependabot and other security audits do not usually find such lockfiles.

@manzoorwanijk I was on the fence about this originally. The big kicker was that the root npm version was behind. Now that that's cleared up, very happy to go with it being a root workspace.

[t-hamano]

This audit seems excessive, as the results vary depending on what is in the npm advisory DB when the CI runs, and it impacts every PR. See #83170