[Bug]: Mid-conversation role: system messages in Claude inbound are aggregated into top-level instructions, destroying prompt cache prefix

[richardfeiliu-a11y]

Client or integration

Claude Code

Area

Proxy and routing

Summary

In src/claude/inbound.ts:334-348, OpenCodex translates incoming Claude Messages by iterating through raw.messages. When it encounters msg.role === "system", it appends the message text to systemParts, which is then joined and assigned to body.instructions:

// src/claude/inbound.ts:334-348
else if (msg.role === "system") {
  const text = systemMessageText(msg.content);
  if (text.length > 0) systemParts.push(text);
}
...
if (systemParts.length > 0) body.instructions = systemParts.join("\n\n");

When clients like Claude Desktop or Claude Code inject dynamic system messages during a multi-turn conversation (e.g., style reminders, date changes, or subagent notices):

• Turn 1: instructions = [topLevelSystem + reminder_1]
• Turn 2: instructions = [topLevelSystem + reminder_1 + reminder_2]

Because instructions is placed at the very head of the LLM prompt sequence, mutating instructions on every turn completely invalidates the prompt cache prefix (KV cache) for all subsequent user/assistant/tool message history.

Real-world Impact & Production Evidence

In real-world long-running Claude Desktop sessions (e.g. 100+ turns), this defect causes prompt cache reuse to freeze at the initial system prompt boundary:

• Observed historical degradation: cachedInputTokens remained strictly frozen at 24,576 tokens across dozens of turns while total input expanded from 60,000 to 137,000 tokens. The cache hit rate collapsed from 40.7% down to 19.4%, resulting in >110,000 un-cached tokens billed at full price on every single turn.
• Controlled mitigation proof: By suppressing dynamic mid-conversation reminders on the client side (outputStyle: default and totalTokensReminder: off), instructions length growth across turns became exactly 0. In the exact same Claude Desktop 3P harness over 10 turns (70k-92k tokens), cachedInputTokens successfully grew with the conversation (51k -> 70k -> 79k), achieving 96.24% ~ 99.93% cache hit rate.

Reproduction

1. Start OpenCodex on port 10100.
2. Send a multi-turn Claude Messages request where Turn 1 contains messages: [{ role: "user", content: "..." }, { role: "system", content: "reminder 1" }].
3. Send Turn 2 with messages: [{ role: "user", ... }, { role: "system", ... }, { role: "assistant", ... }, { role: "user", ... }, { role: "system", content: "reminder 2" }].
4. Inspect the translated Responses request body.instructions:
   • Observe that Turn 1 instructions has length L1;
   • Turn 2 instructions has length L2 = L1 + delta;
   • The top-level instruction header has changed, breaking the prefix match for all preceding conversation history on upstream providers (Anthropic, OpenCode Go, DeepSeek, etc.).

Version

2.48.0

Operating system

macOS 15 (Darwin arm64)

Provider and model

All providers routed through Claude inbound (opencode-go/muse-spark-1.3-contributor, sensenova/deepseek-v4-flash, etc.)

Logs or error output

Turn 1: input=60,153, cached=0 (0.0%)
Turn 2: input=60,444, cached=24,576 (40.7%)
Turn 6: input=65,020, cached=24,576 (37.8%)
Turn 11: input=75,423, cached=24,576 (32.6%)
Turn 21: input=84,487, cached=24,576 (29.1%)
Turn 51: input=105,858, cached=24,576 (23.2%)
Turn 101: input=137,458, cached=26,624 (19.4%)
// Note: cachedInputTokens frozen at 24,576 while input expanded to 137k.

Expected Behavior & Design Consideration

Mid-conversation role: system messages should either:

1. Be mapped into the message sequence as chronological input items (e.g. role: "developer" in Responses input) rather than hoisted into top-level instructions, preserving the immutable prefix; OR
2. Provide a deduplication/filter policy so static/identical recurring reminders do not continuously append to instructions.

Note on downstream adapters: Care must be taken with src/adapters/openai-chat.ts:724, which currently hoists developer items back into systemParts for non-native OpenAI routes.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[lidge-jun]

리뷰 · 우선순위 70 / 80

지금 dev 끝은 #4155로 들어온 post-2.49 round-2 이선(two-lane) 계획이 잡혀 있고, 이 이슈는 그 계획의 Lane A 2번(#4129 다음)으로 이미 박혀 있습니다. 문제는 Claude inbound가 대화 중간에 들어온 role: system 메시지를 전부 모아서 맨 위 instructions에 붙인다는 점입니다. 매 턴마다 프롬프트 맨 앞이 바뀌니, 그 뒤에 쌓인 user/assistant/tool 히스토리의 프롬프트 캐시 접두사가 매번 깨집니다. 이슈에 적힌 Desktop 실측(캐시가 24,576에 고정되고 히트율이 40%대에서 19%대로 떨어짐)은 그 증상과 정확히 맞습니다.

현재 체크아웃의 src/claude/inbound.ts를 보면 systemParts를 만들고(322행), 최상위 system을 넣은 뒤(324행), 메시지 루프에서 msg.role === "system"이면 또 systemParts에 넣고(334-336행), 마지막에 body.instructions = systemParts.join(...)(348행)으로 붙입니다. prompt_cache_key 폴백도 같은 systemParts를 해시합니다(373-394행 근처). 예전에 ChatGPT가 input의 system 아이템을 거부해서 접어 넣은 선택이었고, tests/claude-integration/claude-inbound.test.ts 313 근처가 그 모양을 잠가 둔 상태입니다. round-2 유닛 문서(devlog/_plan/260910_post249_round2/030_4148_claude_system_hoist.md)는 Responses가 시간순 role: "developer" 아이템을 받는다고 정리했고, role: "system"을 input에 넣는 길은 ChatGPT Codex 거부·parser 재호이스트·responses forward 재접기 때문에 피하라고 적혀 있습니다.

고치려면 in-messages system은 systemParts에 넣지 말고, 시간 순서대로 { type: "message", role: "developer", content: [{ type: "input_text", text }] } 형태로 input에 넣어야 합니다. 최상위 Anthropic system만 지금처럼 instructions로 두는 쪽이 계획과 맞습니다. 다만 비-native OpenAI Chat 경로의 src/adapters/openai-chat.ts 722-736 근처는 developer를 다시 systemParts로 모아 system 한 줄로 올립니다. Claude inbound만 고쳐도 Chat 계열 업스트림에서는 접두사 깨짐이 일부 남을 수 있으니, 그 경로까지 같이 볼지 Lane A PR 범위에 밝히면 좋습니다.

라인 334-336 - 대화 중간의 system까지 전부 systemParts에 넣어 instructions 접두사를 매 턴 바꿉니다
라인 348 - systemParts 전체를 body.instructions에 써서 KV 캐시 접두사가 깨집니다
라인 373-394 - Desktop prompt_cache_key 폴백도 같은 systemParts를 해시해 캐시 키가 같이 돌아갑니다
경로 src/adapters/openai-chat.ts (대략 722-736) - 비-native 경로가 developer를 다시 system으로 호이스트해서 inbound만 고쳐도 Chat 업스트림에서 같은 증상이 남을 수 있습니다
경로 tests/claude-integration/claude-inbound.test.ts (대략 313) - 예전 fold-into-instructions 모양을 잠근 테스트라 수정 시 기대값을 developer 아이템 쪽으로 바꿔야 합니다

메인테이너의 판단이 필요한 지점

• 계획 문서가 택한 범위(모든 in-messages system을 developer로)를 그대로 갈지, leading-only만 바꿀지. leading-only는 보고된 실패(매 턴 새 leading system)를 못 닫습니다.
• openai-chat의 developer→system 재호이스트를 이번 Lane A PR에 넣을지, 후속으로 미룰지.
• 캐시 키 폴백을 instructions-only 해시로 둘지, developer 타임라인까지 넣을지.

너의 추천
Lane A에서 #4129 다음에 이 이슈를 바로 집어라. 계획 문서의 선택(전부 developer 아이템, top-level system만 instructions)을 PR 본문에 명시하고, inbound 테스트 기대를 그에 맞게 고친 뒤 머지 후보로 올려라. openai-chat 재호이스트는 같은 PR에 넣거나, 넣지 않으면 후속 이슈로 링크만 남겨라.

이 댓글은 grok-bot이 작성했습니다

[lidge-jun]

Fixed on dev by #4161, merged as 5b8f1fcfa.

Mid-conversation role: "system" messages are no longer folded into instructions. Each one becomes a chronological role: "developer" input item, so the prompt head stops changing every turn and the provider's prefix cache survives. Top-level Anthropic system still flows into instructions exactly as before.

The choice worth naming: every in-messages system message moves, not only the ones after the first user turn. Hoisting just the leading one would have kept the old test green while still mutating instructions whenever a client injects a fresh leading system message each turn — which is the failure this report is about.

developer rather than system in input is deliberate. The schema accepts system there, but ChatGPT Codex rejects it, parseRequest re-hoists it, and the canonical Responses forward folds text-only system items back into instructions. developer is first-class and keeps timeline order.

One consequence to be aware of: Anthropic and Google outbound present developer items as chronological user messages. That is semantic drift from privileged system text, but it is prefix-stable, which is the point of the fix.

Still open and deliberately out of scope: src/adapters/openai-chat.ts re-hoists text developer messages into a leading system message for non-api.openai.com Chat Completions, so a DeepSeek or SenseNova route still loses the prefix. Reversing that is a separate compatibility tradeoff — please open a follow-up if you need it.

Evidence: exact-head CI at 799330bcfcfd44dd22b3dd253adaab87f584bf9c concluded success — gh run view 34415035528 --exit-status returns 0 for Cross-platform CI, with React Doctor, Enforce PR target branch and PR hygiene also 0. Twenty-eight checks succeeded and none were cancelled. git merge-base --is-ancestor 799330bcf origin/dev passes.

Closing by hand because pull requests here target dev, so GitHub does not auto-close a linked issue.

Plan and root-cause record: devlog/_plan/260910_post249_round2/030_4148_claude_system_hoist.md.