FEAT add CodeAttackConverter and CodeAttackAttack (closes #1945) by u7k4rs6 · Pull Request #1960 · microsoft/PyRIT

Utkarsh Bahuguna (u7k4rs6) · 2026-06-09T12:00:30Z

Closes #1945.

Summary

Implements CodeAttack (Ren et al., ACL 2024, arXiv:2403.07865), which reformulates a harmful query as a code-completion task. The query is encoded into a data-structure initialization sequence inside a partial code template with a decode() stub, and the target is asked to complete the code. Because the intent is expressed as a programming task rather than a natural-language request, safety training keyed to natural language triggers less reliably. Black-box, no compute requirements.

This ships as a converter plus a technique registration. There is no attack class: an earlier revision had one, and it was dropped since the behaviour is fully expressible by wiring the converter onto PromptSendingAttack through the technique registry.

Three notes for review (deltas from the issue)

Encoding is word-by-word, not character-by-character. The issue described it as char-by-char (from the paper abstract), but the reference implementation (renqibing/CodeAttack) splits on whitespace and hyphens via regex, with character-level only as a fallback for single-token inputs. I matched the reference code. One consequence: separators are normalized on encode (hyphens and runs of whitespace are consumed as delimiters), which is documented in the converter docstring.
Eight templates, not five. The issue scoped five (one per language), but the reference ships eight: the three Python types each have a base and a verbose variant, and cpp and go have no verbose variant upstream. I included all eight to match the reference. Happy to drop the four verbose files if you would rather keep it to five.
No attack class. Per the above, CodeAttackAttack is not part of this PR. The technique factory does the wiring instead, which keeps the surface to one new class.

Design

CodeAttackConverter (pyrit/converter/code_attack_converter.py): encodes the prompt into the chosen data-structure operations and renders the code template. Single parameter template: Template | pathlib.Path, defaulting to the verbose Python-stack template; the enum covers the eight reference templates and a Path allows a custom one. Standalone Converter, composes through the normal pipeline.
Registered as a code_attack factory in pyrit/setup/initializers/techniques/core.py, which attaches the converter to PromptSendingAttack. That is where the behaviour previously carried by a dedicated attack class now lives.
Seed prompt templates live under pyrit/datasets/converters/, matching the CodeChameleon convention.

Tests

31 unit tests in tests/unit/converter/test_code_attack_converter.py, covering per-language template rendering, verbose vs base variants, word-recovery round-trips, empty / special-character / long prompts, and custom-template paths. tests/unit/setup/test_technique_initializer.py gains code_attack in CORE_TECHNIQUE_NAMES. Converter, docs and setup suites all green; pre-commit clean.

Files

New (9):

pyrit/converter/code_attack_converter.py
8 seed-prompt templates under pyrit/datasets/converters/
tests/unit/converter/test_code_attack_converter.py

Modified (9):

pyrit/converter/__init__.py
pyrit/setup/initializers/techniques/core.py
tests/unit/setup/test_technique_initializer.py
doc/references.bib
doc/code/executor/1_single_turn.py and its .ipynb
doc/code/converters/1_text_to_text_converters.py and its .ipynb

Checklist

Utkarsh Bahuguna (u7k4rs6)

PR Risk Summary

Quality Score: 9/10
Risk Level: low
Merge Recommendation: Safe to merge
Rationale: The changes appear to be focused on adding and refining code attack functionalities and their associated prompt converters. The review found no issues, and the changes are well-contained within the relevant modules. The addition of unit tests further strengthens the quality of this pull request.

Roman Lutz (romanlutz) · 2026-06-22T13:45:11Z

                - file: code/executor/attack/4_sequential_attack.ipynb
                - file: code/executor/attack/chunked_request_attack.ipynb
                - file: code/executor/attack/context_compliance_attack.ipynb
+                - file: code/executor/attack/code_attack.ipynb


we just restructured the attack docs and almost certainly don't want a separate file for it. Can you see if it fits into one of the existing ones (after pulling in latest main)?

e.g., flip attack is in https://microsoft.github.io/PyRIT/latest/code/executor/single-turn/

Good catch. Rebased on main, and since the attack docs restructure removed code/executor/attack/, I dropped the standalone code_attack.ipynb and the myst.yml entry. Added a ## Code section to 1_single_turn right after Flip instead, plus a row in the attack table.

Roman Lutz (romanlutz) · 2026-06-22T13:46:38Z

We need a references.bib update to include the paper.

Roman Lutz (romanlutz) · 2026-06-22T13:49:40Z

    "SingleTurnAttackStrategy",
    "SingleTurnAttackContext",
    "PromptSendingAttack",
+    "CodeAttackAttack",


That is not an ideal name 😆 CodeAttack is definitely better

Roman Lutz (romanlutz) · 2026-06-22T13:51:25Z

+        attack_scoring_config: AttackScoringConfig | None = None,
+        prompt_normalizer: PromptNormalizer | None = None,
+        max_attempts_on_failure: int = 0,
+        language: Literal["python_stack", "python_list", "python_string", "cpp", "go"] = "python_stack",


We've primarily been using enums for this. See style guide.

Roman Lutz (romanlutz) · 2026-06-22T13:52:56Z

+        language: Literal["python_stack", "python_list", "python_string", "cpp", "go"] = "python_stack",
+        verbose: bool = True,


maybe this should just be 1 param that's the template_path and you have an enum with the few provided ones? That way, people can choose from those or provide their own AND it's only 1 param.

Roman Lutz (romanlutz) · 2026-06-22T13:55:53Z

+        code_converter = PromptConverterConfiguration.from_converters(
+            converters=[CodeAttackConverter(language=language, verbose=verbose)]
+        )
+        self._request_converters = code_converter + self._request_converters
+
+        system_prompt_path = pathlib.Path(EXECUTOR_SEED_PROMPT_PATH) / "code_attack.yaml"
+        system_prompt = SeedPrompt.from_yaml_file(system_prompt_path).value
+        self._system_prompt = Message.from_system_prompt(system_prompt=system_prompt)


Maybe we don't need a class for this if it's just PromptSendingAttack with a converter? Doesn't seem simpler than just doing that directly? The converter is very useful by itself, of course.

…te enum, add bib entry, docs - Rename CodeAttackAttack -> CodeAttack (Task 1) - Collapse language + verbose into a single CodeAttackConverter.Template enum modelled on BinaryConverter.BitsPerChar; custom pathlib.Path still accepted for caller-supplied YAML templates (Task 2) - CodeAttack.__init__ now accepts template: CodeAttackConverter.Template | Path and forwards it to the converter; language/verbose params removed (Task 3) - Add @ren2024codeattack entry to doc/references.bib after liu2024flipattack (Task 4) - Add Code row to the attack table in 1_single_turn.py, add ## Code section after ## Flip mirroring the FlipAttack shape, regenerate notebook (Task 5) - Rebase onto upstream/main (doc/code/executor/attack/ directory was removed upstream; old standalone code_attack.ipynb/.py deleted, content moved into 1_single_turn.py) - Update all unit tests for the new Template-based API; add custom-Path cases

Utkarsh Bahuguna (u7k4rs6) · 2026-06-22T18:40:54Z

Roman Lutz (@romanlutz) Thanks for the thorough pass. Addressed everything:

Rebased on main; folded the docs into 1_single_turn next to Flip, removed the separate file and myst.yml entry
Added the ren2024codeattack reference and cited it in the docstrings
Renamed CodeAttackAttack to CodeAttack
Collapsed language + verbose into a single enum-typed template param (CodeAttackConverter.Template) that also accepts a custom Path
Kept the attack class but structured it like FlipAttack (system prompt via _setup_async); left the converter independently usable

Open to dropping the class for converter-only if you'd rather. Re-requesting review.

Roman Lutz (romanlutz) · 2026-06-22T20:28:10Z

+  - Qibing Ren
+  - Chang Gao
+  - Jing Liu
+  - Wenqi Fan
+  - Li Chen
+  - Ruizhe Zhong
+  - Chaochao Lu
+  - Qingsong Wen


this list doesn't match the list in the references.bib file. how come?

I've fixed the metadata across all the code_attack yamls to match, and corrected the affiliation in the same block while I was at it.

Utkarsh Bahuguna (u7k4rs6) · 2026-06-26T21:03:18Z

Tracked the framing variant in #2088. Resolving.

…te enum, add bib entry, docs - Rename CodeAttackAttack -> CodeAttack (Task 1) - Collapse language + verbose into a single CodeAttackConverter.Template enum modelled on BinaryConverter.BitsPerChar; custom pathlib.Path still accepted for caller-supplied YAML templates (Task 2) - CodeAttack.__init__ now accepts template: CodeAttackConverter.Template | Path and forwards it to the converter; language/verbose params removed (Task 3) - Add @ren2024codeattack entry to doc/references.bib after liu2024flipattack (Task 4) - Add Code row to the attack table in 1_single_turn.py, add ## Code section after ## Flip mirroring the FlipAttack shape, regenerate notebook (Task 5) - Rebase onto upstream/main (doc/code/executor/attack/ directory was removed upstream; old standalone code_attack.ipynb/.py deleted, content moved into 1_single_turn.py) - Update all unit tests for the new Template-based API; add custom-Path cases

Implement CodeAttack (Ren et al., ACL 2024) as a standalone converter and a PromptSendingAttack subclass following the FlipAttack pattern. CodeAttackConverter encodes a natural-language prompt word-by-word into a data-structure initialisation sequence (deque appends, list appends, or a string assignment) and embeds it in a partial code template that asks the model to complete the code. Five language variants are supported: python_stack, python_list, python_string, cpp, go. The verbose flag selects the _plus template (detailed paragraphs) for the three Python variants; cpp and go have no plus variant upstream. CodeAttackAttack wraps the converter in a PromptSendingAttack, prepends a system prompt that frames the session as code completion, and forwards language and verbose to the converter. Callers supply a scorer via AttackScoringConfig as usual. Files added: pyrit/prompt_converter/code_attack_converter.py pyrit/executor/attack/single_turn/code_attack.py pyrit/datasets/executors/code_attack.yaml pyrit/datasets/prompt_converters/code_attack_python_stack{,_plus}.yaml pyrit/datasets/prompt_converters/code_attack_python_list{,_plus}.yaml pyrit/datasets/prompt_converters/code_attack_python_string{,_plus}.yaml pyrit/datasets/prompt_converters/code_attack_cpp.yaml pyrit/datasets/prompt_converters/code_attack_go.yaml tests/unit/prompt_converter/test_code_attack_converter.py (23 tests) tests/unit/executor/attack/single_turn/test_code_attack.py (16 tests) doc/code/executor/attack/code_attack.py doc/code/executor/attack/code_attack.ipynb Files modified: pyrit/prompt_converter/__init__.py pyrit/executor/attack/single_turn/__init__.py pyrit/executor/attack/__init__.py doc/myst.yml

…te enum, add bib entry, docs - Rename CodeAttackAttack -> CodeAttack (Task 1) - Collapse language + verbose into a single CodeAttackConverter.Template enum modelled on BinaryConverter.BitsPerChar; custom pathlib.Path still accepted for caller-supplied YAML templates (Task 2) - CodeAttack.__init__ now accepts template: CodeAttackConverter.Template | Path and forwards it to the converter; language/verbose params removed (Task 3) - Add @ren2024codeattack entry to doc/references.bib after liu2024flipattack (Task 4) - Add Code row to the attack table in 1_single_turn.py, add ## Code section after ## Flip mirroring the FlipAttack shape, regenerate notebook (Task 5) - Rebase onto upstream/main (doc/code/executor/attack/ directory was removed upstream; old standalone code_attack.ipynb/.py deleted, content moved into 1_single_turn.py) - Update all unit tests for the new Template-based API; add custom-Path cases

Correct the authors to match the ren2024codeattack bib entry: Ren, Gao, Shao, Yan, Tan, Lam, Ma (SJTU / Shanghai AI Lab / CUHK). Remove incorrect names (Liu, Fan, Chen, Zhong, Lu, Wen) and replace Nanyang Technological University with Shanghai Jiao Tong University.

Remove CodeAttack attack class and its test file; wire code_attack as a PromptSendingAttack + CodeAttackConverter entry in scenario_techniques.py. Delete the now-orphaned executor system-prompt seed YAML. Update the single-turn executor doc and regenerate the notebook to show the converter-based usage pattern.

All five encoders (_encode_python_stack, _encode_python_list, _encode_python_string, _encode_cpp, _encode_go) now use json.dumps() to escape embedded double quotes and backslashes before interpolating into string literals. A prompt containing a double quote no longer produces malformed code. Also fix the class docstring: separator normalisation on [\s\-]+ applies only to python_stack; python_list uses str.split() and preserves hyphens. Add tests for embedded double quotes in all five encoder paths.

…ues/core.py Upstream (microsoft#2155) replaced initializers/components/scenario_techniques.py with initializers/techniques/core.py and renamed strategy_tags to technique_tags. The core group tag is now injected by build_technique_factories rather than stored in the factory. Port the code_attack entry and its test list update to the new layout.

Satisfies tests/unit/docs/test_converter_documentation.py, which requires every converter in pyrit.converter.__all__ to appear in a notebook under doc/code/converters/. The example lives in its own cell so the existing section 1.2 cell (which contains non-deterministic converters) keeps its stored output untouched. The converter is template-based and runs offline, so the new cell carries real executed output like every other cell on the page.

Every other factory in techniques/core.py carries a description; code_attack was the only one without. Matches the sibling register: single sentence, third-person present, states the mechanism.

Utkarsh Bahuguna (u7k4rs6) · 2026-08-09T19:50:31Z

Roman Lutz (@romanlutz) Rebased onto current main. The branch was ~150 commits behind, so this
absorbs the PromptConverter → Converter rename from #2161 across all
three directory moves and both symbol renames, and drops the FlipAttack
and ContextComplianceAttack hunks now that those are gone upstream.

Also fixed a gap that predates the rebase: test_all_converters_are_documented
was failing because CodeAttackConverter had no entry under
doc/code/converters/. Added an example to 1_text_to_text_converters,
plus a description= on the code_attack factory so it matches the others.

Converter, docs and setup suites green, pre-commit clean. Description
updated for the new names.

Utkarsh Bahuguna (u7k4rs6) force-pushed the feat/code-attack branch from 2f306cc to 88631e8 Compare June 9, 2026 12:13

Utkarsh Bahuguna (u7k4rs6) commented Jun 14, 2026

View reviewed changes

Roman Lutz (romanlutz) reviewed Jun 22, 2026

View reviewed changes

Roman Lutz (romanlutz) self-assigned this Jun 22, 2026

Utkarsh Bahuguna (u7k4rs6) force-pushed the feat/code-attack branch from 88631e8 to 76b74d6 Compare June 22, 2026 18:30

Roman Lutz (romanlutz) reviewed Jun 22, 2026

View reviewed changes

Richard Lundeen (rlundeen2) reviewed Jun 22, 2026

View reviewed changes

Comment thread pyrit/executor/attack/single_turn/code_attack.py Outdated

Utkarsh Bahuguna (u7k4rs6) mentioned this pull request Jun 26, 2026

Add a code_attack technique variant that applies the code-completion system prompt #2088

Open

Utkarsh Bahuguna (u7k4rs6) force-pushed the feat/code-attack branch from 9fc7c3c to b500382 Compare July 10, 2026 06:31

Utkarsh Bahuguna (u7k4rs6) requested review from Richard Lundeen (rlundeen2) and Roman Lutz (romanlutz) August 4, 2026 02:43

Utkarsh Bahuguna (u7k4rs6) added 8 commits August 10, 2026 00:12

MAINT add description to the code_attack technique factory

5b5876e

Every other factory in techniques/core.py carries a description; code_attack was the only one without. Matches the sibling register: single sentence, third-person present, states the mechanism.

Utkarsh Bahuguna (u7k4rs6) force-pushed the feat/code-attack branch from b500382 to 5b5876e Compare August 9, 2026 19:42

		language: Literal["python_stack", "python_list", "python_string", "cpp", "go"] = "python_stack",
		verbose: bool = True,

Uh oh!

Conversation

Utkarsh Bahuguna (u7k4rs6) commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Three notes for review (deltas from the issue)

Design

Tests

Files

Checklist

Uh oh!

Utkarsh Bahuguna (u7k4rs6) left a comment

Choose a reason for hiding this comment

PR Risk Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Utkarsh Bahuguna (u7k4rs6) commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Utkarsh Bahuguna (u7k4rs6) commented Jun 26, 2026

Uh oh!

Utkarsh Bahuguna (u7k4rs6) commented Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Utkarsh Bahuguna (u7k4rs6) commented Jun 9, 2026 •

edited

Loading