diff --git a/docs/TRIAGE_RULES.md b/docs/TRIAGE_RULES.md index 9886f24..43423dc 100644 --- a/docs/TRIAGE_RULES.md +++ b/docs/TRIAGE_RULES.md @@ -2,7 +2,7 @@ OWASP GenAI Crosswalk File : docs/TRIAGE_RULES.md Purpose : Where an incoming item goes — incident, benchmark catalogue, or noted and closed - Version : 1.0.0 — 2026-09-18 + Version : 1.1.0 — 2026-10-09 License : CC BY-SA 4.0 --> @@ -51,6 +51,11 @@ this repository sets, and not a reproduction of anyone's results. Benchmarks are published with their own harness and licence, and a pass mark for someone else's benchmark is theirs to set. +A defense paper that also introduces a **named** benchmark gets a row for the benchmark; +the defense itself is noted. The row names the authors' own defense evaluated on it. +A defense paper with no named benchmark is noted and closed. (Maintainer ruling, +2026-10-09, on #194 and #198.) + ## CVEs A published CVE in GenAI tooling is an incident record when it names an affected version diff --git a/evals/EXTERNAL_BENCHMARKS.md b/evals/EXTERNAL_BENCHMARKS.md index dba1f7d..54f4bb8 100644 --- a/evals/EXTERNAL_BENCHMARKS.md +++ b/evals/EXTERNAL_BENCHMARKS.md @@ -1,7 +1,7 @@ @@ -45,6 +45,9 @@ as a profile in this repository.** | **AgentLSD** | **Adversarial task contamination** of AI security agents — deceptive artifacts in the environment, including non-instructional evidence "such as fake results and decoy endpoints", rather than injected instructions alone | Paired clean and trap-augmented runs on CTF challenges with deterministic trap generation; 6 models on 11 web CTF challenges | LLM01 Prompt Injection · ASI01 Agent Goal Hijack | [arXiv:2609.19140](https://arxiv.org/abs/2609.19140) · release: [Golim/agent-lsd](https://github.com/Golim/agent-lsd) (MIT) | | **AgentXploit-Bench** | **End-to-end exploitability of AI-agent systems** in authorized white-box pre-deployment auditing: attacks must act through the task-defined attacker interface and be confirmed by an external verifier | 72 reproducible vulnerabilities across 12 open-source AI-agent systems and frameworks | ASI02 Tool Misuse and Exploitation · ASI05 Unexpected Code Execution · LLM01 Prompt Injection | [arXiv:2609.31318](https://arxiv.org/abs/2609.31318) · release: [lwd17/AgentXploit](https://github.com/lwd17/AgentXploit) (README states Apache-2.0; no licence file in the repository) | | **PrivDrift** | **User-secret leakage under topic drift**: whether secrets a user disclosed earlier in an active conversation remain recoverable after the dialogue moves on, under persuasion-based probing | 1,000 controlled multi-turn dialogues with seeded secrets, content-dense drift turns and standardized extraction probes; 3 LLMs with extended context windows | LLM02 Sensitive Information Disclosure · DSGAI11 Cross-Context Conversation Bleed | [arXiv:2609.30094](https://arxiv.org/abs/2609.30094) · not released: the paper states the authors "plan to release" the generation code, probes, scripts and a sanitized subset | +| **EvoRiskBench** | **Runtime security risks in workspace agents** — models combined with execution harnesses that perform stateful, multi-step tasks on external resources — organised around the authors' EP-Path-EF framework, which "links an initial risk entry point to a one-hop technical effect through an agent-mediated risk path"; outcomes are verified independently from runtime traces and environment states | 450 adversarial tasks across six scenarios; nine entry-point and five effect categories; 9 model-harness configurations (3 models × Claude Code, Codex and OpenClaw) | ASI02 Tool Misuse and Exploitation · ASI05 Unexpected Code Execution | [arXiv:2610.03153](https://arxiv.org/abs/2610.03153) · not released: the paper states the cases and platform "will be released after completion of artifact safety and reproducibility checks" | +| **PAGE** (Persona-Aware Guardrail Evaluation) | **Function-specific guardrails** for customer-facing agents: whether a guardrail keeps an agent within its intended functionality, across "benign, adversarial, and out-of-domain interactions on both user and agent turns" | Not stated in the abstract | LLM01 Prompt Injection · LLM03 Excessive Agency | [arXiv:2610.03434](https://arxiv.org/abs/2610.03434) · not released: the arXiv record links no benchmark release. The benchmark is introduced alongside, and used to evaluate, the authors' own guardrail (Persona Guardrail), which the paper states is deployed in production. | +| **STEER-Bench** | **Branch steering** against computer-use agents: whether crafted untrusted data can "coerce a CUA down a hazardous, pre-approved branch without injecting explicit instructions", including agents built on the Dual-LLM planner/quarantine pattern | 101 tasks across 9 domains; the authors report 94.4% attack success against standard and 89.5% against vanilla Dual-LLM agents | LLM01 Prompt Injection · ASI01 Agent Goal Hijack | [arXiv:2610.03089](https://arxiv.org/abs/2610.03089) · not released: the arXiv record links no benchmark release. The benchmark is introduced alongside the authors' own defense (COBRA), which they report reduces attack success on it to 0%. | ## Why these and not others @@ -56,3 +59,4 @@ Routing — incident, catalogue, or noted and closed — is defined once in [`.. |---|---| | 2026-09-18 | Created with LongPIBench, GenIaC-SecBench and TIER, from the watcher triage of issues #47, #55 and #70. | | 2026-09-30 | Added APort Vault, ClashBench, AgentLSD, AgentXploit-Bench and PrivDrift, from the watcher triage of issues #125, #137, #144, #160 and #171; added the release and licence note. | +| 2026-10-09 | Added EvoRiskBench, PAGE and STEER-Bench, from the watcher triage of issues #196, #194 and #198. PAGE and STEER-Bench are introduced by defense papers; each row names the authors' own defense evaluated on it. |