Repository navigation
✨ feat(markdown): add native Markdown parser - #2489
Merged
Merged
Conversation
harehare
force-pushed
the
feat/native-markdown-parser
branch
from
October 2, 2026 12:31
c4df612 to
f55dc60
Compare
Merging this PR will not alter performance
|
harehare
marked this pull request as ready for review
October 3, 2026 15:19
harehare
force-pushed
the
feat/native-markdown-parser
branch
7 times, most recently
from
October 11, 2026 15:12
6d3ceb2 to
5a9756f
Compare
harehare
force-pushed
the
feat/native-markdown-parser
branch
from
October 13, 2026 13:01
c4f4fd0 to
4e1adea
Compare
Add a built-in two-phase parser (block, then inline) that builds Node directly, behind the `native-parser` feature (enabled by default). Covers CommonMark, GFM, frontmatter, math and MDX. Output, including positions, is differential-tested against markdown-rs and the CommonMark/GFM spec examples. Disable the feature to fall back to markdown-rs.
The entity names in the native parser are defined by the HTML spec and include short non-words such as `ges` and `curren`.
Parse and render HTML with the parser of this crate only. The `native-parser` feature, the mdast conversion and the markdown dependency are gone, and `to_html` is rendered from the block tree by a new renderer. Differences from markdown-rs found by comparing both on the CommonMark and GFM spec, 83k real documents and generated input are fixed or documented: - a panic on `www` before a multibyte character - a byte order mark, multiline inline HTML, and tabs that a container consumes in part - tables: trailing whitespace, single column, and rows that end a table - control character references, escaped footnote labels, NUL in destinations - MDX: lone CR, `~` in emphasis, Unicode names of JSX tags `unicode-id` is now a direct dependency for the names of JSX tags.
Cover every construct, URLs and protocols, footnotes, line endings and the final line ending with the exact output that is expected.
Check kinds, nesting, attributes and positions, including columns that count bytes of multibyte characters, line endings and a byte order mark.
State what holds for any document: blocks parse as their parts, CRLF and containers change nothing else, other characters in place of letters keep the structure, positions are on character boundaries, and the HTML is well-formed with safe URLs.
…t and performance Check the parser against the CommonMark 0.31.2 spec, the GFM spec and its extensions, and reference implementations (commonmark.js, cmark-gfm, micromark), on generated input and about 99k real documents, and fix what differed. Source style: - keep the marker of a list item (-, +, *, . or )) and the character of a thematic break, and write them back - RenderOptions.list_style is now optional, and None keeps the original bullet. --list-style no longer defaults to dash Performance (about 30% faster on real documents): - cache whether a line has a tab, check blank lines by bytes, skip the tab scan for inline positions, and skip plain text in the inline scanner with a table - add benchmarks for real documents and for many short lines CommonMark: - emphasis opens and closes by the flanking rules, and the rule of three goes by the whole run - a heading keeps a # that starts its text, and the lines of a code span lose their leading whitespace - a paragraph is continued lazily by a line of backticks, tildes, underscores or an angle bracket at the end of the document - drop the restrictions on the first line of a container, on markers on the second line of a paragraph, on dashes after a setext heading, on empty nested items in loose lists, and on the first line of indented code - tabs that a container consumes in part stay in indented code as spaces, and the alt text of an image includes the alt text of an image in it - reference labels have their whitespace collapsed, and links in link text are unwrapped GFM: - write the language of a code block as lang, the checkbox attributes in the order of the spec, and filter the tags script, style, textarea and a few more - read the checkbox of a task as the start of its text, so an empty task and what follows the checkbox read as they do in GitHub - link an email address after a slash, allow emoji in a domain, and do not link an email address followed by @ - a footnote reference can follow a !, and a footnote definition goes on after a blank line that follows its marker - test the examples of the CommonMark and GFM extensions specs Obsidian: - read a wikilink and an embed when the text is scanned, so emphasis cannot split them, an escape stops them, and their position is their own - a callout has the fold marker as its own field, a title with markup, and a type of letters, digits, - and _ MDX: - read import and export blocks as ESM Rendering: - do not panic on nodes with overlapping positions - keep tables in quotes and list items, nested items in order, the checkbox, links, references, emphasis in strong, footnotes and empty fenced code intact - write a rule of dashes at the start as *** when another one would read as frontmatter, and escape pipes in table cells
…markers - Render the fence language as `class="language-*"` on `<code>` and compare spec HTML directly - End raw HTML blocks only at the closing tag of the opening element - Detect a task marker on the first non-blank line after an empty list marker - Use the tab-expanded column for the position of text after a task marker
- Import `ListStyle` in the `markdown.rs` tests - Require `html-to-markdown` for the `html_convert` example - Gate the benchmark documents behind the features of their benchmarks
A callout title followed by a blank line absorbed the next paragraph, and a title made of an embed was dropped. The header line is now split off the first paragraph before its inline content is parsed, so the title is the line as written and the body starts on the next line. Add exhaustive and property-based tests for callout titles and bodies.
…e parser Obsidian syntax in HTML output - Render wikilinks as `<a class="internal-link">`, embeds as img, audio, video, embed or a link by extension, and callouts as `div.callout` or `details.callout` with the classes Obsidian uses. Robustness and performance - Cap inline nesting at 128 levels and report deeper MDX elements as an error, so deeply nested input no longer overflows the stack. - Match emphasis in linear time with a delimiter stack and a linked list, stop scanning for `]]` at the first bracket or line ending, and cap reference labels at 999 characters. - Stop opening containers on a line at the depth limit and decide thematic breaks in constant time from per-line precomputed offsets. Lines of repeated list markers no longer take minutes. Fixes - Keep a dashed rule apart from the paragraph text above it, so it is not read back as a setext heading. - Update the known GFM round trip failures: 16 of them now pass. Types - Add `ListMarker` and `HorizontalRuleMarker` in place of `char` markers. Their JSON form is unchanged. `ListMarker::Continuation` replaces the NUL sentinel used to render the rest of an item after nested items. Tests cover the limits, the Obsidian HTML, the markers and the rules.
- Treat wikilinks and embeds as paragraph text so a dash rule after them keeps its blank line and does not read back as a setext heading. - Use the display text as the label of a note embed. - Split an image embed display into alt text and a trailing size, so `![[img.png|photo|200]]` keeps both.
…hed rules Move each construct tried by parse_blocks into a BlockRule type chained in Rules, so the order is the precedence and the dispatch is static. The state handed between constructs is returned as a Step instead of local variables.
…inear Raw HTML comments, instructions, CDATA and declarations, and MDX expressions no longer scan to the end of the input from every opener.
Children were joined to the tags on one line, which dropped headings and turned flow elements into text elements when read back.
Newtype variants cannot carry an internal tag, so literal and expression attributes failed to serialize. They now use adjacent tagging.
…em back Full references write the label as it was, and the check that decides between shortcut and full form uses the label normalization of the parser.
Escape character references, a bang before a link and email-like text, write line endings at the ends of a heading as references, decode autolink literals in link text, and drop trailing whitespace-only lines from indented code. The three GFM spec examples that changed meaning on a round trip no longer do.
…ns and errors extensible The continuation of an item is a flag next to the reordered nodes, so ListMarker has the five markers that a document can write and as_char returns a char. MdxFound and MdxPlace are non_exhaustive, and ParseOptions too, with with_frontmatter to set it.
…ument the HTML, list style and frontmatter behavior The comments that named the old parser now say what CommonMark, GFM and mdast do. A new book page covers callouts, wikilinks and the tag filter in HTML output, the list style default, and --no-frontmatter. listStyle is no longer defaulted to dash in mq-web and mq-nodejs, so they keep the markers of the document as the CLI does. Em dashes are gone from the comments of this change.
… lazy lines LeafState kept one counter of container markers, so a quote could not tell what the lines it held were inside. It now follows a stack of the quotes, items and footnotes that are open, with the indentation of each item, and the text of a paragraph that starts like a definition, and MDX flow content that spans lines. A line under definitions only is text, an item keeps the lines it is indented for after a blank line, and a fence inside a multi-line expression is text. The property test no longer leaves any of these out.
Without the dash default, listStyle was left out and the generated Options type, which wants null, no longer accepted it.
…tables and fences in lazy lines apart - End a `<script>`, `<pre>`, `<style>` or `<textarea>` block at any of their closing tags, as the spec says, and not only at the one that opened it - Read `[ ]` as a task only when the next line goes on as the paragraph, so that a fence, heading, table or setext underline after it stays a block of its own - Decide whether a line is a table header for the lazy line checks the way the table rule does: a header without pipes, and a lone pipe, which has no cells - Keep the blank lines that end an unclosed fence in an item as code, so that the list stays tight
…s when writing Markdown `Markdown::to_html` and `-F html` write the nodes and parse them again, so these came out as other nodes or lost their text: - Emphasis next to emphasis or strong, and nested at an edge, takes the delimiter that does not fuse with its neighbour into one run, alternating by depth - Strong right after emphasis that ends in `*` uses `__` - A reference followed by a colon escapes it, which would read back as a definition - A list that ends in an unclosed fence is not on the line of the block that follows it
…de units Columns of positions were bytes, except that a tab advanced to the next multiple of four. A tab is now one column and a character is as many columns as it has UTF-16 code units, which is what micromark and the LSP use. The byte order mark is not a column. The indentation that decides the structure still expands tabs. Columns are counted where they are asked for, so that a document of ASCII pays nothing and no index of the characters is built. On the parse benchmarks, ASCII prose is unchanged, Japanese prose is about 5% slower and one very long Japanese line about 10%. BREAKING CHANGE: the `column` of a position that follows a tab or a multibyte character is lower than before.
… of the text it starts Text that starts with an escape began after its backslash, while the same text after other characters began at them, so the position of `\*a` was not the source it was read from. It now starts at the backslash, as it does in micromark. BREAKING CHANGE: the start column of a text that starts with an escape is one lower than before.
…ver up front The vectors of the items, the slots of the delimiters, the nodes of a paragraph and the nodes of a block were grown one push at a time, which was about a fifth of the time of parsing. They are now sized from the text and the delimiters, and the nodes of a block are moved in one step. On the parse benchmarks that is about 17% faster for ASCII prose and 15 to 20% for Japanese.
…ow follows it The line of a paragraph was split into cells for every line of text to remember how many it had. It is kept as it is now, and counted when a line that can be a delimiter row comes after it.
…ems small An item was as large as a node, 144 bytes, and was moved into and out of the list of slots while the delimiters were matched. Boxing the finished nodes makes an item half of that, 72 bytes. The parse benchmarks are 3 to 10% faster.
…er between quote, item and footnote The three loops took lines for the container, held blank lines back and took lazy continuation lines with the same steps. They are one `Collector` now.
…he block rules `block.rs` held the line rules, the quote, the list and its items, the footnote, the HTML block and the paragraph. The lists, footnotes and HTML blocks are in files of their own next to `leaf.rs` and `rule.rs`, with what each of them needs named at the top. The behavior is the same.
harehare
force-pushed
the
feat/native-markdown-parser
branch
from
October 13, 2026 13:20
4e1adea to
ad08112
Compare
…inator of the line before The terminator that follows a line belongs between it and the next line, not before the line it ends.
…line HTML or MDX text Rendered without it, the rule reads back as a setext heading underline.
`markdown` reads generated Markdown and MDX and renders it as Markdown, HTML, text and JSON. `markdown_scaling` fails when the time for a document repeated to four times its size grows by more than 12 times.
to_json writes about 100 times the source, so a 1 MiB source made a run build hundreds of megabytes of output and pushed the resident size of the fuzzer up to the libFuzzer limit.
…e and i64 Code that used the depth of a heading as a u8 keeps working for comparisons, and `usize::from(depth)` replaces `depth as usize`.
The delimiter of emphasis or strong after other emphasis is picked from how the node before it was written, which rendered every node before it again. A run of 16 thousand `*_` took 3 seconds to write as Markdown and as HTML. The caller now passes what it already wrote for the node before.
The JSON is pretty printed, so nested nodes make it a few thousand times the source: 165 MB for 64 KiB of `[^1]: `.
…s unfenced The parser knows an indented block is not fenced, so set fence to false instead of inferring it from the line count. A trailing whitespace-only line no longer turns the block into a fenced one when written back.
…agraph Emphasis or strong after other emphasis switched delimiters even when a blank line separated them, so a second paragraph of `*b*` was written as `_b_`. Switch only when the two nodes touch.
…e wikilinks and embeds `[[x\|y]]` in a cell had the target `x\` and an href ending in `%5C`. Unescape `\|` as code spans already do.
A `!` at the end of a paragraph was escaped when the next paragraph began with a link, and a `:` that began a paragraph was escaped after a paragraph that ended in a reference. Apply the touching check that emphasis uses to these rules and to the word-character check too.
The last row of a table ends with its own line ending, and the gap of two lines to the next block added two more, so a blank line in the source became two in the output. A block that followed a table without a gap had one blank line, so the output changed again when read back.
Inside a quote or callout, an item that held a nested item before the rest of its content was written out of order, so `para` after a nested `- b` came before it. Items on one line took a padded marker, which made the third level an indented code block, and a bullet under an ordered item was indented by two columns and left the item. Share the ordering and the indent of the content with the top level.
… container A block after a nested quote or after a footnote in an item or quote was written on the next line, so it read back as lazy continuation. Share the rules for the blank line with the top level. Write an empty math block without a blank line, which was read back as a line of content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a built-in two-phase parser (block, then inline) that builds Node directly, behind the
native-parserfeature (enabled by default). Covers CommonMark, GFM, frontmatter, math and MDX. Output, including positions, is differential-tested against markdown-rs and the CommonMark/GFM spec examples. Disable the feature to fall back to markdown-rs.Type of Change
Checklist
cargo fmtandcargo clippyand addressed any warningsjust test-alland all tests pass/docs, crateREADME.md) if neededAdditional Context