feat: Vercel AI Gateway provider and fx agent

[AdnanElAssadi]

Adds the Vercel AI Gateway as a model provider. Benchmarks can now use models served through the gateway by writing model names like vercel/anthropic/claude-sonnet-4.5, authenticated with a single AI_GATEWAY_API_KEY.

The gateway speaks both the OpenAI and Anthropic APIs. Since all agent traffic routes through the LiteLLM proxy, this covers every registered harness: claude, codex, openhands, plus opencode, openclaw, and goose.

Also registers fx (Vercel's own coding agent, ACP-native via fx acp, pinned v0.0.7). fx speaks the AI SDK gateway wire protocol, which the LiteLLM proxy does not expose, so it declares native_provider="vercel": the proxy is bypassed only for vercel/ models, other models fail fast, and other providers' raw keys are scrubbed from the agent env. Under the no-web policy, web_fetch/web_search are hard-denied via fx's settings.json. The ACP client normalizes fx's non-spec stop reasons and usage keys; no released fx reports ACP usage yet (the feature is merged upstream), so fx runs report usage_source="unavailable" and required usage tracking fails closed until the next fx release.

Part of FrontierPhysics#53

Rebased on current main. All tests pass (5986), ty and ruff clean. Live gateway E2E still pending an AI_GATEWAY_API_KEY.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

[AdnanElAssadi]

@xdotli @bingran-you can you give a vercel api key to just do a sanity check run?

[Galius5136]

Thanks for the implementation. I reviewed the current head locally and ran the regression/test gates plus a few runtime probes.

I found two issues that I think should be addressed before merging:

1. fx currently bypasses the LiteLLM proxy based only on the agent, so the bypass also happens with non-Vercel providers. In those cases raw provider credentials can reach the agent environment instead of being isolated by the proxy. I was able to reproduce this locally. I think the safer approach is to scope the native protocol requirement to the Vercel AI Gateway provider rather than to fx globally.

2. fx v0.0.5 returns stopReason: "refused" on a refusal, while the ACP client expects refusal. This currently surfaces as a Pydantic ValidationError instead of a normal refusal/error path.

Other than that, the provider routing, endpoint configuration, key isolation on the proxied path, and the existing regression gates looked good in my checks.

I haven't done the real provider-backed E2E yet because I still need to run it with a valid AI_GATEWAY_API_KEY. Once these two points are addressed, I'm happy to rerun the checks and do the full Vercel E2E as well.

[ElegantLin]

Thanks for the follow-up fixes. I reviewed the current head (a8e996a) against the latest main. The two earlier issues (cross-provider proxy bypass and refused normalization) are addressed, but I found three remaining blockers before merge:

1. No-web isolation is not enforced for fx.

   For tasks with allow_internet = false, BenchFlow preserves outbound agent networking so model calls and installation still work, then relies on each agent's disallow_web_tools_* configuration for the hard Web-tool disable. The new fx config has neither a setup command nor a launch suffix, and fx v0.0.5 does not consume BENCHFLOW_DISALLOW_WEB_TOOLS. Its ACP runtime admits both web_fetch and web_search in the default auto policy. Consequently, an fx rollout for a no-internet task can still browse the Web, which invalidates benchmark isolation.

   Please add an fx-supported hard deny for both tools (for example through the appropriate .fx/settings.json permission rules), plus an internet-policy regression test.

2. The claimed native ACP token-usage path does not work.

   ensure_litellm_runtime(..., usage_tracking="required") currently raises before an fx session starts because native_provider bypasses LiteLLM. In the default auto mode, fx v0.0.5's ACP session/prompt response contains only stopReason; it does not return a usage object. Thus fx produces usage_source="unavailable", contrary to the PR description that usage comes from ACP-reported tokens, and cannot produce a healthy trajectory with complete token metadata.

   This needs either a real native usage capture path (and required-mode support), an fx version that reports compatible ACP usage, or another supported telemetry mechanism. Please cover the actual fx response shape rather than only a synthetic PromptResponse containing usage.

3. acpx/fx drops native_provider and routes incorrectly.

   _acpx_wrap() copies the existing routing fields but omits the new native_provider field. Minimal repro on this head:

   resolve_agent("acpx/fx").native_provider == ""
   needs_litellm_runtime("acpx:fx", "vercel/anthropic/claude-sonnet-4.5") == True
   

   That sends the wrapped fx agent through LiteLLM even though fx requires the Gateway-native wire protocol. Please copy native_provider=config.native_provider in the wrapper and add an acpx/fx routing regression test.

Validation notes:

• The PR merges cleanly with current main (it is currently 4 commits behind).
• uv run ty check src/ and uv run ruff check . pass on the synthetic merge.
• Full pytest result: 5881 passed, 3 failed, 86 skipped, 7 deselected. The same three failures reproduce unchanged on current main (two host Claude-credential assumptions and one checkout/umask executable-bit assertion), so they are not regressions from this PR.
• The Vercel OpenAI/Responses/Anthropic endpoint configuration matches the official Gateway documentation.
• I could not run the provider-backed E2E because no AI_GATEWAY_API_KEY is available in this environment.
• The GitHub workflows are still awaiting maintainer approval and there is no human approval yet.

Please also make the new regression-test docstrings name PR #1052 or the guarded commit, per this repository's AGENTS.md convention.

[AdnanElAssadi]

Thanks for the review @ElegantLin

1. No-web isolation for fx. The fx entry now has a disallow_web_tools_setup_cmd that writes hard deny rules into fx's ~/.fx/settings.json: "permission": {"web_fetch": {"": "deny"}, "web_search": {"": "deny"}}. Checked against fx v0.0.7 source: a rule-deny resolves to a denial before the prompt path, so it can't surface as an ACP permission request (which our client would auto-approve), and web_fetch's permission target is the URL domain, so "*" matches every invocation. One nuance from their source: fx still advertises web_fetch to the model even when denied, so no-web trajectories may show refused attempts: nothing executes. Internet-policy regression test extended to fx.

2. ACP usage. You are right. I've verified the actual state in fx's source: writePromptResponseWithUsage exists on their main (landed Sept 1–2) but is absent at the v0.0.7 tag, so no released fx reports ACP usage yet. What's in the PR now: pin bumped to v0.0.7; the ACP client normalizes fx's dialect at the parse seam: all three non-spec stop reasons (refused, max_output_tokens, max_model_turns) plus the usage shape their main will ship (cacheReadTokens/cacheWriteTokens/reasoningTokens, no totalTokens), tested against those exact payloads. Until fx releases it, fx runs correctly report usage_source="unavailable" and required mode fails closed. Given their release cadence, v0.0.8 may land before merge: then it's a one-line pin bump.

3. acpx/fx routing. native_provider is now copied through _acpx_wrap. Your repro on the new head: resolve_agent("acpx/fx").native_provider == "vercel" and needs_litellm_runtime("acpx:fx", "vercel/anthropic/claude-sonnet-4.5") == False. Regression test added next to the existing acpx web-policy preservation test.

Also done: regression-test docstrings name PR #1052 per AGENTS.md, and the branch is rebased onto current main (5986 passed locally, ty + ruff clean; I don't hit your three environment-dependent failures here).

Still pending on our side too: no AI_GATEWAY_API_KEY available, so the provider-backed E2E remains open. if the team has a key, happy to run the full matrix (claude/codex/openhands/opencode/openclaw/goose via proxy, fx direct).