Skip to content

capture: share image history strings across turns to avoid quadratic memory - #1534

Open
caoxiaoyuyuyuyuyu wants to merge 1 commit into
openenv:mainfrom
caoxiaoyuyuyuyuyu:fix/capture-shared-image-history
Open

caoxiaoyuyuyuyuyu wants to merge 1 commit into
openenv:mainfrom
caoxiaoyuyuyuyuyu:fix/capture-shared-image-history

Conversation

@caoxiaoyuyuyuyuyu

@caoxiaoyuyuyuyuyu caoxiaoyuyuyuyuyu commented Oct 30, 2026 •

Copy link
Copy Markdown

Summary

RolloutGraph keeps every turn's request_messages. A chat request re-sends the whole conversation and each request body is JSON-decoded on its own, so turn k holds a fresh copy of every image (and other long text) attached in turns 1..k. For multimodal rollouts graph memory therefore grows quadratically with the number of turns: n turns with one base64 image each retain n(n+1)/2 image copies instead of n.

This PR makes add_turn replace long strings (>= 1024 chars) in request_messages, request_tools and response_message with a single graph-local copy.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • New environment
  • Refactoring

Reproduction (synthetic)

Build a graph of 12 turns where turn k re-sends k+1 user messages, each with a distinct 64 KiB data:image/png;base64,... URL, and every request body is passed through json.loads(json.dumps(...)) as the capture server does. Before this change the graph retains 78 distinct image string objects (~5 MiB); after it, 12 (~0.8 MiB).

Why the output is unchanged

  • Only immutable str objects are shared; equal strings are swapped for an equal string, so every value is identical.
  • Dicts and lists are rebuilt per turn, so message structure is unchanged and mutating one turn's messages cannot affect another turn.
  • The table is per RolloutGraph (not process-global), so it is released together with the graph.

RFC Status

  • Not required (bug fix, docs, minor refactoring)
  • RFC exists: #___
  • RFC needed (will create before merge)

Test Plan

New tests/envs/test_capture_graph_shared_text.py (fully synthetic data):

  • each image is retained once across all turns (object identity), and all turns reference the first copy;
  • editing a nested message field in one turn does not change any other turn;
  • export_session(..., include_messages=True) serialised with json.dumps is byte-identical to the export from the same graph with sharing disabled.

Two of the three tests fail on main and all pass with this change.

PYTHONPATH=src:envs uv run pytest tests/envs/test_capture_graph_shared_text.py -q   # 3 passed
PYTHONPATH=src:envs uv run pytest tests/envs -q -k capture --ignore=tests/envs/test_coding_codeact_env.py   # 191 passed, 26 skipped
uv run usort check <changed files> && uv run ruff format --check <changed files> && uv run ruff check src/ tests/   # clean

(test_coding_codeact_env.py fails to collect locally for an unrelated missing optional dependency.)

Claude Code Review

N/A

🤖 Generated with Claude Code


Note

Low Risk
Memory-only deduplication of immutable strings at graph ingest; behavior and exports are covered by tests, with no auth or training-token logic changes.

Overview
RolloutGraph now interns strings ≥1024 characters when ingesting turns, so repeated base64 image URLs and other long text from re-sent chat history are stored once per graph instead of once per turn.

On add_turn, request_messages, request_tools, and response_message are passed through a recursive _share_text helper that replaces matching long strings with a single graph-local copy while rebuilding lists and dicts per turn (only immutable strings are shared).

Adds tests/envs/test_capture_graph_shared_text.py: multimodal multi-turn graphs retain one object per distinct image, edits in one turn do not leak to others, and export_session JSON is byte-identical with sharing disabled.

Reviewed by Cursor Bugbot for commit 3265518. Bugbot is set up for automated code reviews on this repo. Configure here.

Each chat request re-sends the whole conversation and is JSON-decoded on
its own, so turn k holds a fresh copy of every image and long text from
turns 1..k. RolloutGraph kept all of them, which makes memory grow
quadratically with the number of turns in multimodal rollouts.

add_turn now replaces long strings (>= 1024 chars) in request_messages,
request_tools and response_message with one graph-local copy. Only
immutable str objects are shared; dicts and lists are rebuilt per turn,
so values and message structure are unchanged and exports are
byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant