feat(indexing): index many markdown documents in one call - #203
Closed
adityamparikh wants to merge 3 commits into
Closed
adityamparikh wants to merge 3 commits into
adityamparikh wants to merge 3 commits into
Conversation
MarkdownDocumentCreator treated the whole input as one document, so a client with sixty documents needed sixty tool calls, each a full model round trip. A new document now starts at every YAML front matter block; text before the first block is its own document, a thematic break followed by prose does not split, and input with at most one block is parsed exactly as before, with the same content-derived id. The tool description says so, and the parameter description no longer calls the input a single document. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
Replaces the front-matter splitter with a signature change. There is no standard multi-document Markdown format, so the splitter was a convention of our own with a heuristic that could misfire; a typed array makes the document boundary part of the tool schema instead. The client sends one string per document, each goes through the unchanged single-document creator, and the model emits a JSON array natively, which it does reliably. MarkdownDocumentCreator is back to its main version; the splitter tests go with it, and IndexingServiceTest covers the per-element path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
This was referenced Sep 14, 2026
The mock returns null without it, and a strict stub on the one-argument commit would be flagged unnecessary once apache#196's soft commit lands. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
This was referenced Sep 14, 2026
Open
Contributor
Author
|
Superseded by #207, which bundles the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
index-markdown-documentstook one string and produced exactly one document (#144), so a client with sixty documents needed sixty tool calls, each a full model round trip. That is what made the Markdown path slow: 61 documents took over two minutes end to end while Solr's share was under a second.The parameter is now
List<String> documents, one Markdown string per document, each optionally starting with YAML front matter. A whole dataset is one call. Each element goes through the unchanged single-document creator, so front matter, title, headings, body and the content-derived id behave exactly as before.Why a signature change and not server-side splitting
There is no standard multi-document Markdown format — Jekyll, Hugo and every parser assume one file per document, and commonmark's front-matter extension only reads a block at the start — so splitting one string into many would be a convention of our own, resting on a heuristic that can misfire on a horizontal rule followed by a
key: value-looking line. Making the boundary part of the tool schema needs no parsing at all, and the model emits JSON arrays natively. Same approach as #202 for JSON.What changed
IndexingService.indexMarkdownDocuments(collection, List<String> documents); the description says one array element per document and to send many per call.index-dataprompt names thedocumentsparameter for the Markdown path.IndexingServiceTestcovers the per-element path; the MCP-level test sends an array.MarkdownDocumentCreatoris untouched.Independent of #202 and #205; any merge order.
Tests
./gradlew build(Java 25) and./gradlew nativeTest -Pnative(GraalVM CE 25.0.2) are green.🤖 Generated with Claude Code