Skip to content

feat(srbench): bring your own assistant agent (BYOA) - #40

Draft
Tyler Payne (tylerpayne) wants to merge 14 commits into
mainfrom
feat/srbench-byoa-assistant
Draft

feat(srbench): bring your own assistant agent (BYOA)#40
Tyler Payne (tylerpayne) wants to merge 14 commits into
mainfrom
feat/srbench-byoa-assistant

Conversation

@tylerpayne

@tylerpayne Tyler Payne (tylerpayne) commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #39. Closes the loop on bring your own agent for the assistant side (the agent under evaluation). The counterpart agent (requestor/seller, with the forced first turn) is intentionally not user-swappable yet.

Protocol hierarchy

BaseAgent                     run(invoke_tool, tools) and nothing else
├── BaseAssistantAgent        user-swappable, adds nothing beyond run
│   ├── CalendarAssistantAgent (via LLMAgent)
│   └── BuyerAgent             (via LLMAgent)
└── BaseCounterpartAgent      adds generate_text_response + add_forced_action for the forced opening
    ├── CalendarRequestorAgent (via LLMAgent)
    └── SellerAgent            (via LLMAgent)

The LLM machinery formerly named BaseAgent is renamed LLMAgent. Built-in agents subclass both LLMAgent and their role protocol.

Everything flows through tools

  • run now receives the tool space (list[ChatCompletionFunctionToolParam]) from the environment, which owns it per role (ASSISTANT_TOOL_SPACE, BUYER_TOOL_SPACE, ...). A BYOA agent learns what it may invoke without importing anything.
  • The executors no longer push the forced opening into the assistant's context (add_new_messages is gone). The opening stays unread with the wake signal pending, so the assistant's first Wait returns it. Same wall-clock behavior, one less side channel.

Evaluation reads the environment, not agent internals

  • The calendar environment now records an action trace in AgentResources.execute (mirroring marketplace's existing action_trace), exposed on CalendarExecutionResult.action_trace.
  • CalendarReasonableAssistant (the wired-in due diligence evaluator) replays that trace in execution order instead of parsing agent transcripts. This also fixes its GetEmails turn-splitting, which stopped matching real transcripts once the agent-owned run loop landed in refactor(srbench): agents own their run loop; executors no longer orchestrate turns #39 (no injected GetEmails calls exist anymore). Marketplace due diligence already used environment state (result.offers).
  • assistant_context/buyer_context remain on the execution results as debugging artifacts, captured via getattr(agent, "messages", []), so BYOA agents need no transcript reporting.

Bring your own agent

srbench benchmark calendar --assistant-agent my_pkg.my_mod:MyAgent ...
srbench benchmark marketplace --buyer-agent my_pkg.my_mod:MyBuyer ...

load_agent_class resolves the import string and validates the class subclasses BaseAssistantAgent. The calendar factory is called with assistant, allowed_contacts, max_actions. The marketplace factory is called with instruction_message, max_actions. When a factory is set, the built-in agent and its model flags are unused (assistant/buyer model no longer required).

Tests

tests/test_byoa.py covers the hierarchy contracts, the loader (happy path from a real temp package plus all error cases), a full calendar task driven by a custom assistant implementing only run (asserts the opening arrives via Wait, the granted tool space, the factory kwargs, and the recorded action trace), a full marketplace task with a custom buyer, and trace-based reasonable-agent scoring including the invalid-action filter.

tests/ is 231 passed. The 8 failures and 3 errors are pre-existing on the base branch (verified byte-identical on 3fa3d8d14). ruff and ty pass.

🤖 Generated with Claude Code

@tylerpayne @claude
feat(srbench): bring your own assistant agent (BYOA)
7636426 
BaseAgent is now a minimal protocol specifying only run(invoke_tool, tools).
Two role protocols sit under it. BaseAssistantAgent is the user-swappable
side and adds nothing beyond run. BaseCounterpartAgent adds the two hooks
the harness needs to force the opening action. The LLM machinery formerly
named BaseAgent is now LLMAgent, and the built-in benchmark agents subclass
both LLMAgent and their role protocol.

Everything flows through tools. The executors no longer push the forced
opening into the assistant's context. The opening stays unread with the
wake signal pending, so the assistant's first Wait returns it. run() now
receives the tool space from the environment, which owns it per role.

Evaluation no longer depends on agent internals. The calendar environment
records an action trace in AgentResources.execute (mirroring marketplace),
and the reasonable-agent due diligence evaluator replays that trace in
execution order instead of parsing transcripts. This also fixes its stale
GetEmails turn-splitting, which stopped matching real transcripts once the
agent-owned run loop landed. Agent transcripts are still captured on the
execution result as debugging artifacts when an agent exposes messages.

Users select a custom assistant with import-string syntax via
--assistant-agent my_pkg.my_mod:MyClass (calendar) or --buyer-agent
(marketplace), loaded and validated by srbench.shared.load_agent_class.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@tylerpayne
Tyler Payne (tylerpayne) marked this pull request as draft July 10, 2026 15:42
@tylerpayne
Tyler Payne (tylerpayne) changed the base branch from refactor/srbench-agent-run-loop to main July 23, 2026 14:37
Tyler Payne (tylerpayne) and others added 13 commits July 23, 2026 17:16
@tylerpayne
feat(srbench): BYOA assistant contract, MCP bridge, and per-variant a…
db1ae03 
…gent kwargs

Rework the core srbench package so bring-your-own-agent (BYOA) assistants are
first-class and all tool logic/validation lives in the environment:

- Agent contract: BaseAgent.run(invoke_tool, tools) with task via constructor;
  BaseAssistantAgent / BaseCounterpartAgent split. invoke_tool returns a result
  string for every expected outcome (only real bugs raise). Environment owns all
  tool validation, including allowed-contacts enforcement.
- Executors/benchmarks drive user-provided agents through *_agent_factory and
  record actions from the environment trace (no agent transcript required).
- Add reusable srbench.mcp bridge (optional `mcp` extra): build_server plus
  stdio/HTTP/ASGI transports turn (tools, invoke_tool) into an MCP server, with
  validate_input=False so every call reaches the environment.
- Add per-variant BYOA constructor kwargs: assistant_agent_kwargs /
  buyer_agent_kwargs (CLI: --assistant-agent-kwargs / --buyer-agent-kwargs JSON),
  bound into the factory via functools.partial; None coerces to {}.
- Tests: BYOA protocol hierarchy, import-string loader, full-task execution from
  the action trace, and agent-kwargs forwarding for both benchmarks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fb981dd8-822d-49e9-9ea6-926fec9d9511
@tylerpayne
feat(srbench-agents): example BYOA agents (Claude SDK + OpenClaw CLI)
c379aee 
Add a new workspace package `srbench-agents` with ready-to-run, LLM-driven
bring-your-own-agent examples that implement the srbench assistant contract and
are generic over the task (calendar and marketplace, unchanged):

- ClaudeAgent (`srbench_agents.claude_agent`): built on the Claude Agent SDK,
  mounts the srbench MCP server in-process. Optional `claude` extra.
- OpenClawAgent (`srbench_agents.openclaw_agent`): drives the OpenClaw CLI
  (pinned v2026.5.28 / e93216080aa1f425d3ab127014603eba8e365b2d) as a
  subprocess, exposing the environment's tools over streamable-HTTP MCP; asserts
  the installed CLI version at runtime.

Both agents accept `model` and `reasoning_effort` constructor kwargs (also read
from SRBENCH_* env vars): Claude effort -> SDK `effort`, OpenClaw effort -> CLI
`agent --thinking`. All tool logic and validation stay in the environment.

Wire the package into the uv workspace (members + sources) and lock.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fb981dd8-822d-49e9-9ea6-926fec9d9511
@tylerpayne
feat(experiments): add byoa sweep (model × reasoning effort per agent)
ee902d7 
Clone the v0.1.0 experiment but drive the assistant side with the srbench-agents
BYOA implementations (ClaudeAgent, OpenClawAgent) instead of the built-in
model-driven assistant. The counterparty, judge, attacks, styles, defenses, and
data mirror v0.1.0 so results are directly comparable.

Model and reasoning effort are supplied per variant via
assistant_agent_kwargs / buyer_agent_kwargs, defined by an editable AGENTS grid
(models × efforts per agent). Defaults to 2 agents × 1 model × 3 efforts across
both benchmarks (120 variants) on the small split.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fb981dd8-822d-49e9-9ea6-926fec9d9511
@tylerpayne
feat(srbench): mirror BYOA agent model/effort onto reporting fields
04ce123 
BYOA runs left assistant_model / assistant_reasoning_effort (and buyer
equivalents) null because the model lived only inside the *_agent_kwargs
blob, so results.json and the dashboard showed no assistant model.

Add an after model_validator on CalendarRunConfig and MarketplaceRunConfig
that mirrors the "model" / "reasoning_effort" kwargs onto the existing
reporting fields when a BYOA agent is set and those fields aren't already
provided. Kwargs stay intact so they still forward to the agent constructor.
The dashboard reuses its existing model-label logic unchanged.

Document the model/reasoning_effort constructor convention on the
BaseAssistantAgent protocol docstring and cover the hoisting behavior with
tests (default, explicit-wins, and no-model cases).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fb981dd8-822d-49e9-9ea6-926fec9d9511
@tylerpayne
Add shared BYOA system prompt, robust OpenClaw port binding, and clea…
37e9559 
…rer eval errors

Consolidate the BYOA operating prompt into a shared, dependency-free
DEFAULT_ASSISTANT_SYSTEM_PROMPT and expose it as a `system_prompt`
constructor param on the Claude and OpenClaw example agents (with env
fallbacks). Switch the OpenClaw sweep model to openai/gpt-5.4 and pass the
shared prompt to each agent.

Fix an intermittent, concurrency-sensitive OpenClaw failure: bind the MCP
server's ephemeral socket once and hand it directly to uvicorn, closing the
TOCTOU window where a concurrent agent could steal the port between
allocation and bind.

Surface the real cause of failed calendar tasks: skip evaluation when the
execution already errored (returning the true execution error) instead of
letting the deterministic due-diligence scorer raise a misleading
"failed to score" error on the empty trace, and reword the residual
action-less-trace error.

Include the BYOA agent's underlying model in run directory names via
agent_model_label(), and add an `assistantAgent` dashboard pivot dimension.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fb981dd8-822d-49e9-9ea6-926fec9d9511
@tylerpayne
Harden OpenClawAgent failure handling with retries and loud errors
d6e0b22 
Previously _drive() swallowed non-zero exits, timeouts, and no-op runs
into the transcript, so run() "succeeded" with an empty action trace.
This surfaced downstream as confusing "made no scorable decisions"
errors that hid the true cause.

- _drive() now returns a _RunOutcome (returncode/stdout/stderr/
  timed_out/tool_calls) instead of swallowing failures.
- _run_agent_with_retries() retries only no-op failures (no tool calls
  yet) with fresh session keys and backoff; raises a detailed error
  otherwise. Never retries after the agent has mutated state.
- _failure_message() distinguishes timeout, non-zero exit (with stderr
  tails), and clean-exit-but-no-tool-calls.
- _terminate() kills the whole process group (start_new_session=True +
  os.killpg) to reap OpenClaw helper children over long sweeps.
- New SRBENCH_OPENCLAW_MAX_RETRIES env (default 2).
- Added unit tests covering retry-then-succeed, persistent-no-op raise,
  and no-retry-after-engaging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fb981dd8-822d-49e9-9ea6-926fec9d9511
@tylerpayne
Drive OpenClawAgent over the OpenClaw Gateway instead of a subprocess
8969023 
`openclaw agent --local` gave us no usable failure channel: a bad model id
surfaced only as `openclaw exited with code 1` with the real cause
(`FailoverError: Unknown model: openai/claude-sonnet-4-6`) buried in a stderr
tail. Replace the per-task subprocess with OpenClaw's Gateway, a long-lived
process driven over WebSocket RPC.

New `openclaw_gateway` module owns the transport: `GatewayClient` (protocol 4
handshake, token auth), `GatewayProcess` (throwaway profile, two reserved
ports), `GatewayWorker` (config patching, model preflight, turn execution) and
a `GatewayPool` sized by SRBENCH_OPENCLAW_POOL_SIZE.

Notes on the design, all established against openclaw@2026.5.28:

- Control-plane writes (`config.patch`, `gateway.restart.*`, ...) are rate
  limited to 3 per 60s, keyed by device+IP, and reconnecting does not reset it.
  So each Gateway registers its MCP server exactly once, at a port it owns for
  its lifetime; per-task model and thinking level ride on `sessions.create` and
  `sessions.patch`, which are not limited. Each task serves its own tools
  behind that fixed URL and tears them down afterwards, so tool isolation is
  structural rather than negotiated.
- `agent.wait` reports `status: "ok"` even for a failed run; failures appear
  only in `chat.history`, either as `stopReason: "error"` or as assistant text
  prefixed "Agent failed before reply". `is_error_message` detects both.
- Request frames reject unknown top-level keys, so deadlines are client-side.
- With `auth: none` a first read-only call permanently pins the device to
  `operator.read`, which then blocks approving itself; we use token auth and
  request all scopes up front.

Model ids are validated before the sweep starts, with a suggestion when a bare
id matches a known provider-qualified one, and the byoa experiment now asks for
`openai/gpt-5.4` rather than an unresolvable bare `claude-sonnet-4-6`.

Two teardown fixes, both regression tested:

- `uvicorn.Server.serve` has no `finally`, so cancelling it skipped uvicorn's
  shutdown and left a reader registered on a listening socket we then closed.
  That flooded the event loop with `OSError: [Errno 9] Bad file descriptor` on
  every accept and wedged the next task on the same port. Shut the server down
  cooperatively and close its socket explicitly.
- Nothing calls `shutdown_pool`, so a finished sweep leaked Node processes and
  temp profiles; an atexit hook now stops any live Gateways.

Verified with a 21 task calendar sweep across 4 concurrent Gateways.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 797e8dc1-675f-4edb-8b25-ba5474a23479
@tylerpayne
Narrow the byoa sweep to a fast openclaw-only iteration loop
576f2de 
Scoped down to what is actually being iterated on right now: openclaw only,
calendar only, no attacks, and both swept openai models. Widened to 10 tasks in
flight, which needs SRBENCH_OPENCLAW_POOL_SIZE=10 to avoid queueing behind a
single Gateway.

The claude agent, the other two attacks and the marketplace benchmark are
disabled in place rather than deleted, so restoring the full grid is a matter of
uncommenting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 797e8dc1-675f-4edb-8b25-ba5474a23479
@luizvalle
Make OpenClaw respect the OPENAI_BASE_URL env var and use chat comple…
a68bae2 
…tions instead of responses api to avoid encryption issue. See microsoft/aifsdk#2915
@tylerpayne
Add phyagi OpenClaw provider and an explicit BYOA constructor contract
8a6c1ee 
Register `phyagi/<model>` as a first-class OpenClaw provider running on the
Responses API, and give BYOA agents a declared constructor contract so the
harness can configure them.

The gateway load-balances across upstreams, so the Responses API is only usable
if a session's requests are pinned to one of them: the adapter replays encrypted
reasoning blobs across turns, and an unpinned follow-up lands elsewhere and is
rejected with `invalid_encrypted_content`. Pinning needs two gateway-owned
request-body parameters, `session_id` and `strict_session`, which OpenClaw
cannot emit from config -- its only body passthrough (`params.extra_body`) is
gated on `api: "openai-completions"` and never reaches `/responses`, and
unknown `params.*` keys are dropped by an allowlist. A bundled provider plugin
supplies them through the `wrapStreamFn` hook, which is API agnostic.

The plugin also declares a `thinkingLevelMap` via `normalizeResolvedModel`.
Without it, `getSupportedThinkingLevels` treats `xhigh` as unavailable and
silently degrades it to `high`; `compat.supportedReasoningEfforts` does not
help because the Responses adapter never reads it.

Leave the built-in `openai` provider alone. Overlaying its baseUrl made every
`openai/*` id mean "whatever endpoint the environment pointed at", which is
both surprising and impossible to report accurately. That overlay also ran on
Chat Completions, where OpenClaw emits no reasoning effort at all on proxy
routes -- so `--assistant-reasoning-effort` was silently a no-op for gateway
agents. Moving to Responses fixes that.

Separately, `--assistant-reasoning-effort` and `--system-prompt` never reached
BYOA agents, which only received `*_agent_kwargs`. `BaseAssistantAgent` now
declares a concrete `__init__(*, task, model, reasoning_effort, system_prompt)`
and both benchmarks pass those explicitly, so an agent participates just by
subclassing. `*_agent_kwargs` remains the escape hatch and still wins on
conflict. Numeric efforts are now rejected loudly by the two named-level
backends rather than dropped.

Convert experiments/byoa to the per-agent config fields. It had also pinned
`system_prompt` in kwargs, which overrode every defense preset and made the
`none`/`all` sweep two identical variants.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 44eb02a1-4cd0-4617-a60e-1530fc6279ef
@tylerpayne
Overlay the built-in openai provider instead of adding a phyagi one
58fe846 
Replaces the bespoke `phyagi/{model}` provider with a plain override of
OpenClaw's built-in `openai` provider, activated by `OPENAI_BASE_URL`.
`openai/*` now means whatever endpoint the environment is configured for,
which is what every other OpenAI-compatible client already does with that
variable, so a gateway needs no srbench-specific model prefix or base-URL
setting. With `OPENAI_BASE_URL` unset, OpenClaw talks to real OpenAI
unchanged.

The session-affinity plugin moves to `openclaw_plugins/openai_affinity` and
registers against the bundled `openai` provider; verified that provider
plugins do hook bundled providers, so both the `session_id`/`strict_session`
injection and the `thinkingLevelMap` that unlocks `xhigh` still apply.

Because the plugin now augments a provider that can point at real OpenAI --
which rejects `session_id` as an unknown parameter -- the affinity key is
exported only alongside the overlay, and an empty key leaves the stream path
untouched.

The catalog stays declared rather than discovered: a gateway generally serves
no `/models` endpoint, and the bundled `openai` catalog is frozen to the
models that OpenClaw release shipped with (it has no `gpt-5.4` at all) with
metadata describing real OpenAI.

`SRBENCH_PHYAGI_*` is gone; the contract is now `OPENAI_BASE_URL`,
`OPENAI_API_KEY`, `SRBENCH_OPENAI_MODELS` and `SRBENCH_OPENAI_STRICT_SESSION`,
documented in the srbench-agents README.

Verified with a loopback capture probe (affinity keys and
`reasoning.effort: "xhigh"` present on `/responses`) and a live two-turn run
against a real gateway, where turn two's reasoning references turn one --
proving encrypted-reasoning replay survives the pin.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 44eb02a1-4cd0-4617-a60e-1530fc6279ef
@tylerpayne
Revert "Overlay the built-in openai provider instead of adding a phya…
324c612 
…gi one"

This reverts commit 58fe846.
@tylerpayne
Hardcode the phyagi gateway endpoint
1cdd1dc 
The endpoint was configured through `SRBENCH_PHYAGI_BASE_URL`, falling back to
`OPENAI_BASE_URL`, and the provider was registered only when one of them was
set. That made `phyagi/*` silently unresolvable on a machine with neither, and
the failure surfaced late and unhelpfully as an unknown model id.

There is only one endpoint this provider exists to talk to, and its address is
routable rather than secret, so it is now the constant `PHYAGI_BASE_URL` and
the provider always registers. Access is still gated by
`SRBENCH_PHYAGI_API_KEY` (or `OPENAI_API_KEY`).

Dropping the `OPENAI_BASE_URL` fallback also decouples two unrelated routes:
that variable is read by `srbench_llm` to point the built-in OpenAI client at
an endpoint for unprefixed models, so having it also steer this provider meant
one variable quietly reconfigured both.

`SRBENCH_PHYAGI_MODELS` and `SRBENCH_PHYAGI_STRICT_SESSION` are unchanged.

Verified with a live two-turn run with `OPENAI_BASE_URL` unset: both turns
resolved `provider: "phyagi"` on `openai-responses`, and turn two reported
`cacheRead: 17408`, confirming the session-affinity pin still landed both
turns on the same upstream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 44eb02a1-4cd0-4617-a60e-1530fc6279ef
@luizvalle Luiz do Valle (luizvalle) mentioned this pull request Jul 31, 2026
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tylerpayne @luizvalle