Skip to content

fix: treat only spaces and tabs as blank when picking the streaming parser's stable boundary - #235

Merged
Peter Gromov (donnerpeter) merged 1 commit into
JetBrains:masterfrom
rvandermeulen:streaming-boundary-blank-line-matches-block-parser
Oct 7, 2026
Merged

Peter Gromov (donnerpeter) merged 1 commit into
JetBrains:masterfrom
rvandermeulen:streaming-boundary-blank-line-matches-block-parser

Conversation

@rvandermeulen

Copy link
Copy Markdown
Contributor

Problem

The block parser treats a line as blank only when it holds nothing but spaces and tabs (MarkdownParserUtil.isEmptyOrSpaces), but lastBlankLineEndOrNull, which picks the streaming parser's stable boundary, tests isWhitespace(), which also accepts \r, a non-breaking space and other whitespace. With CRLF line endings, line one\r\nline two\r\n\r\nnext\r\n is a single paragraph to a full parse, since \r\n\r\n is not a blank line to the block parser. Streamed one character at a time, the boundary settles after \r\n\r\n and next becomes a second paragraph, so the streamed tree depends on how the text was chunked. A line holding only a non-breaking space does the same with LF line endings.

Approach

Use the block parser's definition of a blank line in lastBlankLineEndOrNull. The range also drops the two \n endpoints, which were only there because isWhitespace() accepts them.

With CRLF line endings the boundary now never settles on a blank line, so the whole document stays in the unstable tail and is reparsed on every append, as it would be for a document without blank lines. That is consistent with what a full parse produces; recognizing \r\n as a line ending in the block parser would restore incremental parsing but is a much larger change.

Tests

A CRLF document and an LF document with a line holding only a non-breaking space, each streamed one character at a time, produce the same top-level children as a full parse. Each input fails on its own without the fix.

…arser's stable boundary

The block parser treats a line as blank only when it holds nothing but spaces and tabs, while the streaming
parser looked for the last blank line with `isWhitespace()`, which also accepts `\r`, a non-breaking space
and other whitespace. With CRLF line endings `\r\n\r\n` is no blank line to the block parser, so a full parse
gives one paragraph, but once a chunk ends right after it the streaming parser settles a boundary there and
the text after it starts a second paragraph. The streamed tree then depends on how the text was chunked. Use
the block parser's definition of a blank line.
@donnerpeter
Peter Gromov (donnerpeter) merged commit d28fbea into JetBrains:master Oct 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants