From cf392dfa43b46fcec7194eec946fc2e60da6a6d6 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Mon, 12 Oct 2026 10:06:04 +0200 Subject: [PATCH] Reading it as a document, and what a table actually is go-pdfkit/document v0.1.0 reads a PDF back as headings, paragraphs, lists and tables, and extract v0.9.0 gained the table detector under it. Neither was documented here. Both pages say what the inference will NOT do, because that is the part a reader needs before piping the output anywhere: ruled lines are not read, multi-column layouts are not read, and a numbered list comes back as a two-column table. And both carry the measurements that set their thresholds rather than the thresholds alone -- 588 tables down to 40 on one 213-page document, and the share of characters that separates a heading size (up to 5.76%) from a second body size (from 32.6%). Co-Authored-By: Claude Opus 5 --- content/api.md | 3 +- content/working-with-a-file/as-a-document.md | 103 +++++++++++++++++++ content/working-with-a-file/extraction.md | 35 +++++++ 3 files changed, 140 insertions(+), 1 deletion(-) create mode 100644 content/working-with-a-file/as-a-document.md diff --git a/content/api.md b/content/api.md index 5f499d1..063a856 100644 --- a/content/api.md +++ b/content/api.md @@ -12,7 +12,8 @@ The rest of the family has its reference there too: [`ops`](https://pkg.go.dev/github.com/go-pdfkit/ops), [`render`](https://pkg.go.dev/github.com/go-pdfkit/render), [`pdffont`](https://pkg.go.dev/github.com/go-pdfkit/pdffont), -[`extract`](https://pkg.go.dev/github.com/go-pdfkit/extract) and +[`extract`](https://pkg.go.dev/github.com/go-pdfkit/extract), +[`document`](https://pkg.go.dev/github.com/go-pdfkit/document) and [`coedit`](https://pkg.go.dev/github.com/go-pdfkit/coedit). ## Document diff --git a/content/working-with-a-file/as-a-document.md b/content/working-with-a-file/as-a-document.md new file mode 100644 index 0000000..7369563 --- /dev/null +++ b/content/working-with-a-file/as-a-document.md @@ -0,0 +1,103 @@ +--- +title: "Reading it as a document" +weight: 55 +--- + +[`document`](https://github.com/go-pdfkit/document) reads a PDF back as +headings, paragraphs, lists and tables, into the +[richdoc](https://github.com/go-richdoc/richdoc) model — which gets Markdown, +reST and HTML out of a PDF through code the fleet already has. + +```go +doc := document.Parse(src) // every page, in order +md, _ := markdown.Write(doc) // and out as Markdown + +for _, p := range document.Pages(src) { + // p.Number, p.Blocks — a page at a time, for indexing +} +``` + +From the command line, that is `pdfops markdown`: + +```sh +pdfops markdown report.pdf > report.md +pdfops markdown -json report.pdf # one record per page, for a retrieval pipeline +``` + +## A PDF has no headings + +It has instructions for drawing glyphs at places. Everything a reader sees +above that — this line is a heading, those three are a list, that block is a +table — is an **inference** from where the places are and how big the glyphs +were. + +Worth saying plainly, because "convert a PDF to Markdown" suggests the +structure was in there and merely needed a different spelling. It was not. A +PDF is the output of a typesetter, and this reads the typesetter's work +backwards. + +There is also **no error to return**. A page that cannot be *read* is an empty +page, which is an answer rather than a failure, and an error that is always +nil would teach every caller to write a branch that never runs. + +## What a heading is + +Text set larger than the body, where the body size is the one the document +draws the most *characters* at, over every page. + +⛔ **But larger is not enough.** A nine-page report set in 10, 12 and 14 point +came back with **131 headings, 130 of which were paragraphs of prose**: the +report has two body sizes, and "most popular" can only pick one. + +So a heading size also has to be **rare**, which is what a heading is — a line +or two per section against paragraphs per page. Measured over six real +documents, as the share of characters at each size *above* that document's own +body size: + +| | | +|:--|:--| +| heading sizes | 0.01% – **5.76%** | +| second body sizes | **32.6%** – 39.4% | + +An order of magnitude apart, and the threshold sits between them at 15%. + +## The headings are found before the tables + +A numbered heading leaves a gap after its number. The table below it leaves +that same gap empty on every row. So asking the page for its tables returns +**one two-column table that is the heading and the grid together** — and the +grid's real columns are gone, because the heading's own text reaches across all +of them: + +```text +| 2. | Vue d’ensemble technique | +| --- | --- | +| | HGX B300 GH200 NVL32 GB200 NVL36 GB300 NVL72 | +``` + +Nothing in the alignment says that line is a heading. Its **size** does, and +`document` is the only package that knows which sizes those are, so the tables +are looked for among the lines that are not headings. + +## Joining a line to the one above + +⛔ **No space after a hyphen**, because that is where a typesetter broke a +word: `cadre man-` and `ager` joined with a space is `man- ager`, which is not +what the page says. + +⛔ **And the hyphen stays.** Dropping it reads `manager` — right there, wrong +for `well-known` broken at its own hyphen, and the two cannot be told apart +without a dictionary of the document's language. Keeping it loses nothing a +reader cannot undo. + +## What it does not read + +Multi-column page layouts, footnotes, ruled lines, and the reading order of a +page whose text was drawn out of order. Two tables side by side on one page are +read as one, because a table's box says where its words are and not which +words. + +A numbered list laid out as "number here, text there" is read as a two-column +**table**, which keeps both the number and the text in cells of their own. +Taking `1.` as a list marker would mean deciding that `1990` at the start of a +line is one too, and a table of dates is nothing but those. diff --git a/content/working-with-a-file/extraction.md b/content/working-with-a-file/extraction.md index a3570fc..8791d56 100644 --- a/content/working-with-a-file/extraction.md +++ b/content/working-with-a-file/extraction.md @@ -10,8 +10,43 @@ pictures. text, _ := extract.Text(d, 1) // the page as words, lines put together runs, _ := extract.Runs(d, 1) // every piece, with its place and size imgs, _ := extract.Images(d, 1) // every picture, and where it lands +tabs, _ := extract.Tables(d, 1) // the grids its text lines up into ``` +`Lines` and `Join` are exported too, because the rule has to be the *same* +rule: `Text` uses them to put a page's text back together, and anything reading +structure out of a page — as [`document`](../as-a-document/) does — has to +agree with it line for line. + +## A table is a stretch of page every row leaves empty + +A column boundary is a stretch no row of the table draws anything in. The first +line of a block proposes its own gaps, each line after it takes away whatever +it draws in them, and a line that leaves nothing closes the block. + +⛔ **Subtraction, not the intersection of the lines' gaps.** A cell whose text +wraps leaves a line with one piece of text on it and so no gaps of its own, so +intersecting ended the table there: a four-row table with one wrapped cell came +back as two tables of two rows. + +Three things are not columns, each found by running this over real documents +rather than over a fixture, and measured on one 213-page document as each went +in — 588 tables, then 213, then 83, then 40: + +| | | +|:--|:--| +| a **bullet** | a bulleted list *is* a two-column grid. What tells them apart is a column carrying no information: one or two characters with no letter and no digit in them. `A`/`B` are labels and `1990` is a date, so those stay. | +| a column **nothing was drawn in** | that is a stretch of page nothing was drawn in, which is what a boundary already is. | +| a line **far below** the row above | a page number sits in the left-hand column and draws nothing in any boundary, which is exactly what a row of a table does. | + +The forty are that document's real `Code`/`Intitulé` pairs. A 3-page prose +report in the same run reports **none**, which is the control: a detector that +cannot say no to a paragraph says yes to every page there is. + +⛔ **Ruled lines are not read.** This is the *text* strategy — the one a +bordered table does not need — so a table held apart by rules rather than by +whitespace is not found. + ## A page does not hold text It holds instructions for drawing glyphs, and the text is what those glyphs