Skip to content

Load inline screenshots from Claude Code sessions - #271

Open
mostafaseyedan wants to merge 2 commits into
microsoft:mainfrom
mostafaseyedan:fix/claude-image-extraction
Open

mostafaseyedan wants to merge 2 commits into
microsoft:mainfrom
mostafaseyedan:fix/claude-image-extraction

Conversation

@mostafaseyedan

Copy link
Copy Markdown

Problem

The Coding Moments gallery counts image-bearing Claude Code requests, but cannot display a single one. Every Claude session lands on the empty state:

These sessions referenced images, but the raw screenshots could not be loaded.

extractSessionImages in src/core/parser-vscode-files.ts dispatches on the .jsonl extension and hands off to extractImagesFromJsonl, which assumes the VS Code chat-session shape — kind/k/v envelopes and requests[].variableData.variables. Claude Code also writes .jsonl, so its sessions take that branch, match nothing, and return [].

Claude stores images inline on the user turn instead:

{
  "uuid": "",
  "message": {
    "role": "user",
    "content": [
      { "type": "text", "text": "what is wrong here?" },
      { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "" } }
    ]
  }
}

Note the moments themselves are counted correctly — parser-claude.ts sets variableKinds.image via countClaudeImages — so the gallery advertises screenshots it can never render.

Fix

Add extractClaudeImagesFromJsonl to parser-claude.ts and fall back to it when the VS Code shape yields nothing.

Matching is exact rather than heuristic: parseClaudeRequest already uses the entry uuid as the request id (parser-claude.ts), so the originating line is located directly by id rather than by scanning for image-shaped data. The existing four-image cap per request is preserved, as is the quick includes(requestId) reject before parsing each line.

Testing

Three unit tests added alongside the existing parser-claude tests: extraction by request id, the no-images and unknown-id cases, and the four-image cap. Verified they fail against a stubbed-out extractor (2 failed / 22 passed) and pass with it (24 passed).

Also verified end to end against a real log set spanning 17 project directories: 247 of 247 image-bearing requests now resolve, averaging 169 KB for the first image, where previously none did.

npm run typecheck and npm run lint are clean (0 errors).

The Coding Moments gallery counts image-bearing Claude requests but can
never display them: every session shows "These sessions referenced
images, but the raw screenshots could not be loaded".

extractSessionImages dispatches on the .jsonl extension and assumes the
VS Code chat-session shape (kind/k/v envelopes, requests[].variableData
.variables). Claude Code also writes .jsonl, so its sessions take that
branch, match nothing and return an empty array. Claude stores images
inline on the user turn instead, as
message.content[].source = { type: 'base64', media_type, data }.

Add extractClaudeImagesFromJsonl and fall back to it when the VS Code
shape yields nothing. Matching is exact rather than heuristic: the
Claude parser already uses the entry uuid as the request id, so the
originating line can be located directly. The existing four-image cap
per request is preserved.

Verified against a real log set: 247 of 247 image-bearing requests now
resolve, where none did before.
@mostafaseyedan

Copy link
Copy Markdown
Author

Added a second commit: the extractor alone was not sufficient.

getSessionImages looks up sessionSourceIndex and returns { images: [] } when a session has no entry. That index was populated only by parser-vscode.ts — the Claude collector never registered a source, and parseClaudeSessions did not return the file each session was parsed from. So the RPC bailed before reaching the extractor and the gallery stayed empty.

parseClaudeProjectSessions now returns a sessionFiles map (sessionId → source path) and the Claude collector records each entry into sessionSourceIndex. Threading it through added an optional parameter to collectExternalHarnessesSync/Async; it is optional, so Codex and OpenCode are unaffected.

Verified through the RPC path rather than the extractor in isolation:

moments: 245 | sessionSourceIndex size: 131
RPC path -> resolved 245 | missing source 0 | source but no images 0

Three assertions in parser-main.test.ts covered the old call signatures and were updated. Suite is at 1344 passing; the 7 github-app-analytics failures are pre-existing on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant