From 9119ef0bac49b0f21e218eb5526b2fb6953adee8 Mon Sep 17 00:00:00 2001 From: emmanuelgjr Date: Fri, 9 Oct 2026 16:12:20 -0400 Subject: [PATCH] Triage: catalogue EvoRiskBench, PAGE and STEER-Bench Three benchmark rows from the watcher triage of #196, #194 and #198. None of the three has released its benchmark, so each row records it as not released, quoting the paper or noting the arXiv record links none. PAGE and STEER-Bench are introduced by defense papers; each row names the authors' own defense evaluated on it, as the APort Vault row does for its disclosure. The OWASP entry column is DRAFT, as for every row in the catalogue. docs/TRIAGE_RULES.md rule 3 records the maintainer ruling that a defense paper introducing a named benchmark gets a row for the benchmark, and one with no named benchmark is noted and closed. Co-Authored-By: Claude Opus 5.5 --- docs/TRIAGE_RULES.md | 7 ++++++- evals/EXTERNAL_BENCHMARKS.md | 6 +++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/TRIAGE_RULES.md b/docs/TRIAGE_RULES.md index 9886f24..43423dc 100644 --- a/docs/TRIAGE_RULES.md +++ b/docs/TRIAGE_RULES.md @@ -2,7 +2,7 @@ OWASP GenAI Crosswalk File : docs/TRIAGE_RULES.md Purpose : Where an incoming item goes — incident, benchmark catalogue, or noted and closed - Version : 1.0.0 — 2026-09-18 + Version : 1.1.0 — 2026-10-09 License : CC BY-SA 4.0 --> @@ -51,6 +51,11 @@ this repository sets, and not a reproduction of anyone's results. Benchmarks are published with their own harness and licence, and a pass mark for someone else's benchmark is theirs to set. +A defense paper that also introduces a **named** benchmark gets a row for the benchmark; +the defense itself is noted. The row names the authors' own defense evaluated on it. +A defense paper with no named benchmark is noted and closed. (Maintainer ruling, +2026-10-09, on #194 and #198.) + ## CVEs A published CVE in GenAI tooling is an incident record when it names an affected version diff --git a/evals/EXTERNAL_BENCHMARKS.md b/evals/EXTERNAL_BENCHMARKS.md index dba1f7d..54f4bb8 100644 --- a/evals/EXTERNAL_BENCHMARKS.md +++ b/evals/EXTERNAL_BENCHMARKS.md @@ -1,7 +1,7 @@ @@ -45,6 +45,9 @@ as a profile in this repository.** | **AgentLSD** | **Adversarial task contamination** of AI security agents — deceptive artifacts in the environment, including non-instructional evidence "such as fake results and decoy endpoints", rather than injected instructions alone | Paired clean and trap-augmented runs on CTF challenges with deterministic trap generation; 6 models on 11 web CTF challenges | LLM01 Prompt Injection · ASI01 Agent Goal Hijack | [arXiv:2609.19140](https://arxiv.org/abs/2609.19140) · release: [Golim/agent-lsd](https://github.com/Golim/agent-lsd) (MIT) | | **AgentXploit-Bench** | **End-to-end exploitability of AI-agent systems** in authorized white-box pre-deployment auditing: attacks must act through the task-defined attacker interface and be confirmed by an external verifier | 72 reproducible vulnerabilities across 12 open-source AI-agent systems and frameworks | ASI02 Tool Misuse and Exploitation · ASI05 Unexpected Code Execution · LLM01 Prompt Injection | [arXiv:2609.31318](https://arxiv.org/abs/2609.31318) · release: [lwd17/AgentXploit](https://github.com/lwd17/AgentXploit) (README states Apache-2.0; no licence file in the repository) | | **PrivDrift** | **User-secret leakage under topic drift**: whether secrets a user disclosed earlier in an active conversation remain recoverable after the dialogue moves on, under persuasion-based probing | 1,000 controlled multi-turn dialogues with seeded secrets, content-dense drift turns and standardized extraction probes; 3 LLMs with extended context windows | LLM02 Sensitive Information Disclosure · DSGAI11 Cross-Context Conversation Bleed | [arXiv:2609.30094](https://arxiv.org/abs/2609.30094) · not released: the paper states the authors "plan to release" the generation code, probes, scripts and a sanitized subset | +| **EvoRiskBench** | **Runtime security risks in workspace agents** — models combined with execution harnesses that perform stateful, multi-step tasks on external resources — organised around the authors' EP-Path-EF framework, which "links an initial risk entry point to a one-hop technical effect through an agent-mediated risk path"; outcomes are verified independently from runtime traces and environment states | 450 adversarial tasks across six scenarios; nine entry-point and five effect categories; 9 model-harness configurations (3 models × Claude Code, Codex and OpenClaw) | ASI02 Tool Misuse and Exploitation · ASI05 Unexpected Code Execution | [arXiv:2610.03153](https://arxiv.org/abs/2610.03153) · not released: the paper states the cases and platform "will be released after completion of artifact safety and reproducibility checks" | +| **PAGE** (Persona-Aware Guardrail Evaluation) | **Function-specific guardrails** for customer-facing agents: whether a guardrail keeps an agent within its intended functionality, across "benign, adversarial, and out-of-domain interactions on both user and agent turns" | Not stated in the abstract | LLM01 Prompt Injection · LLM03 Excessive Agency | [arXiv:2610.03434](https://arxiv.org/abs/2610.03434) · not released: the arXiv record links no benchmark release. The benchmark is introduced alongside, and used to evaluate, the authors' own guardrail (Persona Guardrail), which the paper states is deployed in production. | +| **STEER-Bench** | **Branch steering** against computer-use agents: whether crafted untrusted data can "coerce a CUA down a hazardous, pre-approved branch without injecting explicit instructions", including agents built on the Dual-LLM planner/quarantine pattern | 101 tasks across 9 domains; the authors report 94.4% attack success against standard and 89.5% against vanilla Dual-LLM agents | LLM01 Prompt Injection · ASI01 Agent Goal Hijack | [arXiv:2610.03089](https://arxiv.org/abs/2610.03089) · not released: the arXiv record links no benchmark release. The benchmark is introduced alongside the authors' own defense (COBRA), which they report reduces attack success on it to 0%. | ## Why these and not others @@ -56,3 +59,4 @@ Routing — incident, catalogue, or noted and closed — is defined once in [`.. |---|---| | 2026-09-18 | Created with LongPIBench, GenIaC-SecBench and TIER, from the watcher triage of issues #47, #55 and #70. | | 2026-09-30 | Added APort Vault, ClashBench, AgentLSD, AgentXploit-Bench and PrivDrift, from the watcher triage of issues #125, #137, #144, #160 and #171; added the release and licence note. | +| 2026-10-09 | Added EvoRiskBench, PAGE and STEER-Bench, from the watcher triage of issues #196, #194 and #198. PAGE and STEER-Bench are introduced by defense papers; each row names the authors' own defense evaluated on it. |