Skip to content

A picture 3.5% past the per-picture ceiling is dropped in SILENCE, so a page yields no pictures and no error #108

Open

Description

@tannevaled

What happens

/Users/Shared/pdfscans/ia-americana/sim_unitarian-register-and-the-universalist-leader_christian-register_1825-06-25_4_25.pdf,
page 1, at render v0.67.0:

render.Images(d, 1)  ->  0 images, err == nil
render.Page(d, 1, …) ->  1269x1775, 154 055 dark pixels   (it draws)
pdfimages -list      ->  3 pictures
   1  0 image  7048 9856  gray 1 8 jpx     object 18
   1  1 image  7048 9856  gray 1 8 jpx     object 20
   1  2 smask  7048 9856  gray 1 1 jbig2   object 20

An empty slice and no error. A caller cannot tell "this page draws no
picture" from "every picture on it was too big to hand back".

Why

decodeBase:

bpp := int64(4)
if !r.bounded && r.mayPackOneByte(dict) {
	bpp = 1
}
if w <= 0 || h <= 0 || int64(w)*int64(h)*bpp > maxImageBytes {
	return nil
}
7 048 × 9 856 69 465 088 pixels
maxImagePixels (= maxImageBytes/4) 67 108 864
over by 2 356 224 — 3.5%
at four bytes a pixel 277 860 352 bytes
maxImageBytes 268 435 456

r.bounded is true exactly on the Images path, so the one-byte packing a
one-component grey would get is disabled and the picture is charged four bytes.
277.9 MB against a 268.4 MB ceiling: nil, and nil is not an error here.

The page budget does not catch it either — afford charges pixels against
maxImagesPixels, and 3 × 69.5M = 208M passes — so the picture is charged to
the page and then silently dropped.

Why it matters more than its size

ErrTooMuchToDecode exists for the page budget precisely because "nothing
comes back with it: half the pictures of a page would be read as the whole of
them by anything counting"
. That reasoning applies here and there is no
equivalent error: this is the half-a-page case, reached one picture at a time.

Measured consequence: go-pdfkit/conformance's compare puts this page at
19.23% of pixels differing from poppler — the worst of 3 214 pages in the
corpus
— and its images harness measured it not at all, because a
document that yields no picture produced no row. The two instruments disagreed
about whether the page exists, and the silent one was the one with the records.

What a fix looks like

Either of two, and they are not the same decision:

  • Say so. A picture over maxImageBytes returns an error rather than nil,
    and Images refuses the page the way it refuses a page over the budget. One
    line of honesty, no change in what is admitted.
  • Charge what it costs. The comment on maxImageBytes already argues this
    for the drawing path — "a 400 dpi bilevel newspaper scan is 7 779 by 10 699,
    past the old ceiling, and in the one-byte form it holds in 83 MB, a third of
    what the ceiling allows"
    — and r.bounded turns that argument off for
    Images because the output is raster.Image at four bytes a pixel. That is
    honest about the OUTPUT and is the same root as
    #100.

The first is a bug fix. The second is the API question #100 holds.

How to see it

The conformance side of this is fixed in
conformance#92: a page we
produce no picture for now asks the judge and counts what it took out, so this
document reports its three pictures instead of vanishing.

🤖 Generated with Claude Code

Activity

  1. tannevaled commented on Oct 4, 2026

    @tannevaled
    ContributorAuthor

    It is not one document: the corpus's two worst pages are both this

    page size of each of its three pictures over the per-picture ceiling compare says
    sim_unitarian-…_1825-06-25_4_25.pdf 7 048 × 9 856 = 69 465 088 px +3.5% 19.23% of pixels differ — worst of 3 214
    sim_unitarian-…_1847-04-17_26_16.pdf 7 528 × 9 408 = 70 823 424 px +5.5% 14.70% — second worst

    Both are the Internet Archive's standard layout for a 400 dpi newspaper scan:
    two JPEG 2000 greys and a JBIG2 soft mask, all three the same size. Both are
    drawn by render.Page. Both return 0 pictures and no error from
    render.Images.

    So the two pages at the top of this corpus's page-level disagreement are the two
    pages the picture-level instrument could not see, and the ceiling they exceed it
    by is three and a half, and five and a half, percent.

    And a negative control: the forms corpus has none

    Asking our side only — render.Images against the image XObjects a page's own
    resources name, no poppler, 2 240 documents of pdfforms:

    2240 documents read, 808 name an image XObject on page 1, 1 of those return NOTHING
    SILENT PDFBOX-3053-reduced.pdf   1 image XObject(s) named, largest 165726 pixels
    

    That one is not this issue. 165 726 pixels is four hundred times under the
    ceiling; it is the legitimate case the imagesDrawn comment describes — a
    picture named in the resources that the content stream never draws, where the
    resources are "a catalogue of what a page MAY draw and not a record of what it
    does"
    .

    So the ceiling bites only where the pictures are big: mass-digitisation scans,
    which is the population this whole corpus exists to measure.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions