From f94caf34b8f0437bec6ced09d968cf6c33f0b692 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Sun, 11 Oct 2026 10:33:52 +0200 Subject: [PATCH] deblank: find the pages that hold nothing, and drop them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BlankPages draws each page and counts the pixels that are not paper; Doc.DropBlank removes them and says which went, numbered as the reader saw them. `pdfops deblank` exposes both, with -list as the dry run. Drawing is the only test that agrees with a reader. A page whose content stream is full of operators painting white, or drawing outside the media box, looks blank and is blank; a page carrying a picture of nothing is blank too, and no amount of reading the content stream says so. ⛔ THE DEFAULT RESOLUTION WAS MEASURED, AND THE FIRST ONE WAS WRONG. This started at 50 DPI with a comment claiming that was "enough to see a full stop". Measured on A4, at 50 DPI an eight-point full stop covers EXACTLY ONE PIXEL — the same mark a speck of dust makes — so no threshold at that resolution can tell punctuation from dirt. At 150 it covers four and a comma covers eight, so there is a factor of four to set MaxInk from. TestMeasureTheInkLadder prints the whole ladder at 50, 100, 150 and 300 so that a renderer change moving either end of the gap fails there rather than silently reclassifying pages. ⛔ The zero value of BlankOptions is the STRICTEST setting, not a middle one: a page with a single inked pixel is kept. This is the input to deleting pages, so the conservative direction is the only safe default — a page wrongly called blank destroys content, while a blank page wrongly kept merely leaves a blank page. The scan threshold (1e-6, about two pixels) is documented with its measurements instead, because whether its margin holds depends on the scanner. It will not call a page of solid colour blank: the test is distance from WHITE, not uniformity, and a measure of variance would delete a deliberate black separator sheet. It will not empty the document. It does not count annotations as ink unless asked. ⛔ TWO MUTATIONS SURVIVED THE FIRST SUITE, and both were real test weaknesses: - defaultTolerance = 0 changed nothing, because every fixture rendered to EXACTLY white, so no test could see the field and the doc comment's claim was unproven. Measured: a mark at 247/255 inks 2.2e-03 at tolerance zero and exactly nothing at twelve. That page is now asserted paper by default and ink at a tolerance of one level; - dropping the alpha channel from inkFraction changed nothing, because the only transparent pixel any test used was transparent BLACK, whose colour channels are ink anyway. White-and-transparent is the case the rule is about, and it is now its own test. Three fixtures were wrong before the code was: - a "solid page" made by stamping U+2588 FULL BLOCK drew NOTHING, because Helvetica's encoding has no such character — the page really was blank and the test was right to fail. The uniformity claim is now a synthetic raster, which does not depend on what a font happens to contain, plus an end-to-end page covered by a black picture; - fixture(t, 2) in the command's tests carries a content stream that is not drawing operators, so its pages render as blank paper. A fixture whose "content" is invisible is no fixture for a tool that measures ink; - Picture(data, 1) means one image pixel per INCH, so a 64-pixel square became a 64-inch page and the renderer refused it. And DropBlank lost its own emptiness check: Bytes already refuses a document with no pages, and a second copy of that rule was a branch no test could reach. A rule stated twice is a rule that will disagree with itself. ⛔ I nearly removed the write's error check as unreachable too — Bytes also refuses a form field with nowhere to go, which is a non-empty document. This closes the last of the three entries go-pdfkit/conformance's parity table had written down as not covered with a reason: "remove-blank-pages needs a blank DETECTOR over rendered pixels, and a detector is a measurement, not a wiring job." 100% statement coverage on both packages with no uncovered statements, go vet and gofmt clean. Six mutations caught. Co-Authored-By: Claude Opus 5 --- README.md | 49 +++++ blank.go | 188 +++++++++++++++++ blank_measure_test.go | 89 ++++++++ blank_test.go | 424 ++++++++++++++++++++++++++++++++++++++ cmd/pdfops/expose_test.go | 142 +++++++++++++ cmd/pdfops/run.go | 64 ++++++ go.mod | 2 + go.sum | 4 + 8 files changed, 962 insertions(+) create mode 100644 blank.go create mode 100644 blank_measure_test.go create mode 100644 blank_test.go diff --git a/README.md b/README.md index 3c56dea..0430ac2 100644 --- a/README.md +++ b/README.md @@ -88,6 +88,55 @@ document. `onepage` stacks every page onto a single long one. `poster` cuts each page into tiles so a plan can be printed larger than the paper and assembled. +`deblank` finds the pages that hold nothing and drops them. It **draws** each +page and counts the pixels that are not paper, because that is the only test +that agrees with a reader: a page whose content stream is full of operators +painting white, or drawing outside the media box, looks blank and is blank, and +a page carrying a picture of nothing is blank too. + +```sh +pdfops deblank -list scan.pdf # say which, write nothing +pdfops deblank scan.pdf tidied.pdf +``` + +⛔ **The default resolution is 150 DPI, and it was raised from 50 because of a +measurement.** On A4: + +| | 50 DPI | 150 DPI | +| --- | --- | --- | +| the page | 414 × 585 | 1240 × 1755 | +| one pixel | 4.1e-06 of it | 4.6e-07 | +| an 8-point full stop | **1 pixel** | 4 pixels | +| an 8-point comma | 2 pixels | 8 pixels | + +At 50 DPI a full stop is *exactly the mark a speck of dust makes*, so no +threshold at that resolution can tell punctuation from dirt. At 150 there is a +factor of four to set `-max-ink` from, and `TestMeasureTheInkLadder` prints the +whole ladder so that a renderer change which moved either end fails there +instead of silently reclassifying somebody's pages. + +⛔ **The zero value is the strictest setting**, not a middle one: a page with a +single inked pixel is kept. This is the input to deleting pages — a page +wrongly called blank destroys content, while a blank page wrongly kept merely +leaves a blank page. For scans, `-max-ink 1e-6` is about two pixels: it +forgives a speck and still keeps a page holding a full stop. It is not the +default because whether that margin holds depends on the scanner, which this +package cannot see. + +Three more things it will not do: + +- **it will not call a page of solid colour blank.** The test is distance from + *white*, not uniformity; a measure of variance would delete a deliberate + black separator sheet; +- **it will not empty the document.** Every page blank is an error, because a + file of no pages still opens and still has a size, and a caller handed one + would be told nothing was wrong; +- **it does not count annotations as ink** unless asked: a page whose only mark + is a sticky note is blank paper somebody wrote on. + +And `-list` writes nothing and takes no output file, because the first thing +anybody should do with a tool that deletes pages is ask it which ones it means. + ⛔ `divide` makes the **same grid** as `poster` and is the opposite operation. `poster` keeps every piece the size of the page it came from, so it **enlarges**: the pieces taped together make one page the size of a wall. diff --git a/blank.go b/blank.go new file mode 100644 index 0000000..efe2eea --- /dev/null +++ b/blank.go @@ -0,0 +1,188 @@ +// Copyright (c) 2026, the go-pdfkit/ops authors +// All rights reserved. +// +// SPDX-License-Identifier: BSD-3-Clause + +package ops + +import ( + "fmt" + + "github.com/go-gfx/gfx/raster" + "github.com/go-pdfkit/reader" + "github.com/go-pdfkit/render" +) + +// BlankOptions say how little a page may hold and still count as blank. +// +// ⛔ The zero value is the STRICTEST setting, not a middle one: a page with a +// single inked pixel is kept. That is deliberate and it is the only safe +// default, because this is the input to deleting pages — a page wrongly called +// blank destroys content, while a blank page wrongly kept merely leaves a +// blank page. +type BlankOptions struct { + // DPI is how finely the page is drawn before its pixels are counted. Zero + // means a hundred and fifty. + // + // ⛔ It is not fifty, which is what this first used and what a thumbnail + // would want. MEASURED on A4: at 50 DPI an eight-point full stop covers + // exactly ONE pixel, which is the same mark a speck of dust makes, so no + // threshold at that resolution can tell punctuation from dirt. At 150 it + // covers four and a comma covers eight. See TestMeasureTheInkLadder. + DPI float64 + + // Tolerance is how far a channel may be from white and still count as + // paper, out of 255. Zero means twelve. + // + // ⛔ Not zero-means-zero: nothing in a rendered page is exactly white. + // Anti-aliasing, a JPEG background and a scan of real paper all land a few + // levels below it, and a tolerance of zero calls every page inked. + Tolerance uint8 + + // MaxInk is the fraction of a page's pixels that may be ink and have the + // page still count as blank. The zero value keeps any page with a mark on + // it, which is what the zero value SHOULD mean here. + // + // For scans, where dust and the edge of the platen leave specks, the + // figures to set it from are these, measured on A4 at the default 150 DPI: + // + // one pixel 4.6e-07 + // a full stop (8pt) 1.8e-06 (four pixels) + // a comma (8pt) 3.7e-06 (eight pixels) + // one word (12pt) 2.0e-04 + // + // So 1e-06 tolerates a speck or two and still keeps a page holding a full + // stop. The margin is a factor of four, and it is not offered as a default + // because whether that margin holds depends on the scanner, which this + // package cannot see. + MaxInk float64 + + // MaxPixels is passed to the renderer, so a media box a file made up + // cannot be asked to fill memory. + MaxPixels int + + // KeepAnnotations counts the notes, highlights and form fields a page + // carries as ink. They are NOT counted by default: a page whose only mark + // is a sticky note is blank paper somebody wrote on, and the note is not + // part of what was printed. + KeepAnnotations bool +} + +// The measured defaults. See TestMeasureTheInkLadder for where they come from. +const ( + defaultBlankDPI = 150.0 + defaultTolerance = 12 +) + +func (o BlankOptions) dpi() float64 { + if o.DPI > 0 { + return o.DPI + } + return defaultBlankDPI +} + +func (o BlankOptions) tolerance() uint8 { + if o.Tolerance > 0 { + return o.Tolerance + } + return defaultTolerance +} + +// inkFraction is the share of an image's pixels that are not paper. +// +// A pixel is paper when every one of its four channels is within tol of 255. +// ⛔ The ALPHA channel is included on purpose: a fully transparent pixel is +// black-with-nothing in this representation, so a renderer asked for a +// transparent background would otherwise report blank paper as solid ink. +func inkFraction(img *raster.Image, tol uint8) float64 { + if img == nil || img.W <= 0 || img.H <= 0 { + return 0 + } + floor := 255 - tol + ink := 0 + for i := 0; i+3 < len(img.Pix); i += 4 { + if img.Pix[i] < floor || img.Pix[i+1] < floor || + img.Pix[i+2] < floor || img.Pix[i+3] < floor { + ink++ + } + } + return float64(ink) / float64(img.W*img.H) +} + +// BlankPages reports which pages of a file hold almost nothing, counting from +// one and in order. +// +// Each page is DRAWN and its pixels counted, because that is the only test +// that agrees with a reader. A page whose content stream is full of operators +// painting white, or drawing outside the media box, looks blank and is blank; +// a page carrying an image of nothing is blank too, and no amount of reading +// the content stream says so. +// +// ⛔ What it will not do is call a page of solid colour blank. The test is +// distance from WHITE, not uniformity: a deliberately black separator sheet +// carries information, and a measure of variance would delete it. +func BlankPages(pdf []byte, opt BlankOptions) ([]int, error) { + d, err := reader.Open(pdf) + if err != nil { + return nil, err + } + ropt := render.Options{ + DPI: opt.dpi(), + MaxPixels: opt.MaxPixels, + NoAnnotations: !opt.KeepAnnotations, + } + tol := opt.tolerance() + var blank []int + for i := 1; i <= d.PageCount(); i++ { + img, err := render.Page(d, i, ropt) + if err != nil { + return nil, fmt.Errorf("page %d: %w", i, err) + } + if inkFraction(img, tol) <= opt.MaxInk { + blank = append(blank, i) + } + } + return blank, nil +} + +// DropBlank removes the pages that hold almost nothing and reports which went, +// numbered as they were BEFORE the removal — because a caller who wants to say +// what happened needs the numbers the reader saw. +// +// ⛔ It refuses to empty the document. A file of no pages is still a file: it +// opens, it has a size, and a caller who handed over a stack of scans that +// turned out to be blank would be told nothing was wrong while being given +// something no reader can show. Every page blank is a result worth an error. +func (d *Doc) DropBlank(opt BlankOptions) ([]int, error) { + // ⛔ No separate check for an empty document: Bytes refuses one, and that + // is the one place that should know a document with no pages cannot be + // written. A second copy of the rule here was a branch no test could + // reach, and a rule stated twice is a rule that will disagree with itself. + pdf, err := d.Bytes() + if err != nil { + return nil, err + } + blank, err := BlankPages(pdf, opt) + if err != nil { + return nil, err + } + if len(blank) == 0 { + return nil, nil + } + if len(blank) == len(d.pages) { + return nil, fmt.Errorf("ops: every one of the %d pages looks blank: "+ + "removing them all would leave a file no reader can show", len(d.pages)) + } + drop := make(map[int]bool, len(blank)) + for _, n := range blank { + drop[n] = true + } + keep := make([]Page, 0, len(d.pages)-len(blank)) + for i, p := range d.pages { + if !drop[i+1] { + keep = append(keep, p) + } + } + d.pages = keep + return blank, nil +} diff --git a/blank_measure_test.go b/blank_measure_test.go new file mode 100644 index 0000000..f3f6a08 --- /dev/null +++ b/blank_measure_test.go @@ -0,0 +1,89 @@ +package ops + +import ( + "testing" + + "github.com/go-pdfkit/reader" + "github.com/go-pdfkit/render" +) + +func renderOne(t *testing.T, text string, size, dpi float64) (int, int, float64) { + t.Helper() + d := New() + d.Blank(595, 842) // A4, so the figures are the ones a real page gives + if text != "" { + if err := d.Stamp("all", Stamp{Text: text, Font: Helvetica, Size: size}); err != nil { + t.Fatal(err) + } + } + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + src, err := reader.Open(pdf) + if err != nil { + t.Fatal(err) + } + img, err := render.Page(src, 1, render.Options{DPI: dpi}) + if err != nil { + t.Fatal(err) + } + return img.W, img.H, inkFraction(img, defaultTolerance) +} + +// TestMeasureTheInkLadder prints what a page of each kind actually measures, +// at each resolution worth considering. It is how the defaults were chosen. +func TestMeasureTheInkLadder(t *testing.T) { + for _, dpi := range []float64{50, 100, 150, 300} { + w, h, _ := renderOne(t, "", 0, dpi) + onePixel := 1 / float64(w*h) + t.Logf("--- %g DPI: %d x %d, one pixel = %.3e of the page", dpi, w, h, onePixel) + for _, r := range []struct { + name string + text string + size float64 + }{ + {"nothing", "", 0}, + {"a full stop, 6pt", ".", 6}, + {"a full stop, 8pt", ".", 8}, + {"a comma, 8pt", ",", 8}, + {"one word, 12pt", "word", 12}, + } { + _, _, ink := renderOne(t, r.text, r.size, dpi) + px := ink / onePixel + t.Logf(" %-18s ink=%.3e (%.0f pixels)", r.name, ink, px) + } + } +} + +// TestMeasureAFaintMark says what a nearly-white mark actually measures, which +// is what the Tolerance default is for. A mutation that set the default +// tolerance to zero SURVIVED the first version of this suite: every fixture +// rendered to exactly white, so the field was inert as far as any test could +// see, and the doc comment's claim was unproven. +func TestMeasureAFaintMark(t *testing.T) { + for _, grey := range []float64{1.0, 0.99, 0.97, 0.95, 0.9} { + d := New() + d.Blank(595, 842) + if err := d.Stamp("all", Stamp{ + Text: "faint", Font: Helvetica, Size: 48, + Colour: [3]float64{grey, grey, grey}, + }); err != nil { + t.Fatal(err) + } + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + src, err := reader.Open(pdf) + if err != nil { + t.Fatal(err) + } + img, err := render.Page(src, 1, render.Options{DPI: defaultBlankDPI}) + if err != nil { + t.Fatal(err) + } + t.Logf("grey %.2f (%3.0f/255): ink at tol 0 = %.3e, at tol 12 = %.3e", + grey, grey*255, inkFraction(img, 0), inkFraction(img, defaultTolerance)) + } +} diff --git a/blank_test.go b/blank_test.go new file mode 100644 index 0000000..1ef0237 --- /dev/null +++ b/blank_test.go @@ -0,0 +1,424 @@ +// Copyright (c) 2026, the go-pdfkit/ops authors +// All rights reserved. +// +// SPDX-License-Identifier: BSD-3-Clause + +package ops + +import ( + "bytes" + "image" + "image/color" + "image/png" + "strconv" + "strings" + "testing" + + "github.com/go-gfx/gfx/raster" + "github.com/go-pdfkit/reader" + "github.com/go-pdfkit/render" +) + +// ladder builds a one-page A4 document with the given text stamped on it, or +// nothing at all. +func ladder(t *testing.T, text string, size float64) *Doc { + t.Helper() + d := New() + d.Blank(595, 842) + if text != "" { + if err := d.Stamp("all", Stamp{Text: text, Font: Helvetica, Size: size}); err != nil { + t.Fatal(err) + } + } + return d +} + +// ⛔ The gap this detector lives in, asserted rather than described. The +// default resolution was CHANGED because of these numbers: at 50 DPI an +// eight-point full stop covers exactly one pixel, which is the mark a speck of +// dust makes, so no threshold there can tell punctuation from dirt. If a +// renderer change moves either end of the gap, this fails instead of silently +// reclassifying somebody's pages. +func TestAFullStopIsMoreThanOnePixelAtTheDefaultResolution(t *testing.T) { + blank := inkOf(t, ladder(t, "", 0)) + if blank != 0 { + t.Errorf("a page with nothing on it inks %.3e, want exactly zero", blank) + } + stop := inkOf(t, ladder(t, ".", 8)) + w, h := pageSizeAt(t, defaultBlankDPI) + onePixel := 1 / float64(w*h) + if px := stop / onePixel; px < 3 { + t.Errorf("an 8pt full stop covers %.0f pixel(s) at %g DPI; "+ + "below three it cannot be told from a speck of dust", + px, defaultBlankDPI) + } + // And the floor of the gap: a speck must be BELOW a full stop by a margin + // worth setting MaxInk from, or the knob documented in BlankOptions is a + // lie. + if stop < 3*onePixel { + t.Errorf("a full stop (%.3e) leaves no room above one pixel (%.3e)", + stop, onePixel) + } +} + +// With the zero value of BlankOptions — the strictest setting — a page holding +// a single full stop is NOT blank. +func TestAPageWithAFullStopIsNotBlank(t *testing.T) { + d := New() + d.Blank(595, 842) + if err := d.Stamp("1", Stamp{Text: ".", Font: Helvetica, Size: 8}); err != nil { + t.Fatal(err) + } + d.Blank(595, 842) + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + got, err := BlankPages(pdf, BlankOptions{}) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0] != 2 { + t.Errorf("BlankPages = %v, want [2]: page 1 holds a full stop", got) + } +} + +// ⛔ A page of solid colour is NOT blank, and this is the claim stated at the +// level it lives at: the test is distance from WHITE, not uniformity. A +// measure of variance would call a deliberate black separator sheet blank — +// uniform, featureless — and delete it. +// +// The first version of this test tried to make a solid page by stamping +// U+2588 FULL BLOCK at 72 point. Helvetica's encoding does not have that +// character, so NOTHING was drawn and the page really was blank: the test +// failed, and it was the FIXTURE that was wrong. A synthetic raster says the +// same thing without depending on what a font happens to contain. +func TestASolidColourIsInkNotPaper(t *testing.T) { + solid := &raster.Image{W: 8, H: 8, Pix: make([]uint8, 4*8*8)} + for i := 0; i+3 < len(solid.Pix); i += 4 { + solid.Pix[i+3] = 255 // opaque black + } + if got := inkFraction(solid, defaultTolerance); got != 1 { + t.Errorf("a page of solid black inks %v, want all of it", got) + } + // And the other uniform page, for the pair: solid WHITE is paper. + paper := &raster.Image{W: 8, H: 8, Pix: make([]uint8, 4*8*8)} + for i := range paper.Pix { + paper.Pix[i] = 255 + } + if got := inkFraction(paper, defaultTolerance); got != 0 { + t.Errorf("a page of solid white inks %v, want none", got) + } +} + +// End to end over a real drawn page, so the claim above is not only true of a +// raster this test made itself: a page covered by a black picture is not blank. +func TestAPageCoveredByAPictureIsNotBlank(t *testing.T) { + d := New() + if err := d.Picture(blackPNG(t), 72); err != nil { + t.Fatal(err) + } + d.Blank(595, 842) + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + got, err := BlankPages(pdf, BlankOptions{}) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 || got[0] != 2 { + t.Errorf("BlankPages = %v, want [2]: page 1 is a black picture", got) + } +} + +// The documented knob, measured: MaxInk at 1e-6 is about two pixels at the +// default resolution, so it tolerates a speck and still keeps a full stop. +func TestTheScanThresholdKeepsAFullStopAndForgivesASpeck(t *testing.T) { + w, h := pageSizeAt(t, defaultBlankDPI) + onePixel := 1 / float64(w*h) + const scan = 1e-6 + if scan < 2*onePixel { + t.Errorf("1e-6 is %.1f pixels, which does not forgive a speck", scan/onePixel) + } + stop := inkOf(t, ladder(t, ".", 8)) + if stop <= scan { + t.Errorf("a full stop inks %.3e, which 1e-6 would call blank", stop) + } +} + +func TestDropBlankRemovesThemAndSaysWhich(t *testing.T) { + d := New() + for i, text := range []string{"one", "", "three", "", ""} { + d.Blank(595, 842) + if text != "" { + if err := d.Stamp(itoa(i+1), Stamp{Text: text, Font: Helvetica, Size: 24}); err != nil { + t.Fatal(err) + } + } + } + went, err := d.DropBlank(BlankOptions{}) + if err != nil { + t.Fatal(err) + } + // ⛔ Numbered as they were BEFORE the removal: 2, 4 and 5 of five, not + // 2, 3, 4 of what was left as each one went. + if len(went) != 3 || went[0] != 2 || went[1] != 4 || went[2] != 5 { + t.Errorf("DropBlank = %v, want [2 4 5]", went) + } + if got := d.PageCount(); got != 2 { + t.Errorf("PageCount() = %d, want 2", got) + } +} + +func TestDropBlankWithNothingToDropChangesNothing(t *testing.T) { + d := ladder(t, "words on the page", 24) + went, err := d.DropBlank(BlankOptions{}) + if err != nil { + t.Fatal(err) + } + if went != nil { + t.Errorf("DropBlank = %v, want nothing", went) + } + if got := d.PageCount(); got != 1 { + t.Errorf("PageCount() = %d, want 1", got) + } +} + +// ⛔ Refusing to empty the document. A file of no pages opens and has a size, +// so a caller handed one would be told nothing was wrong. +func TestDropBlankRefusesToEmptyTheDocument(t *testing.T) { + d := New() + d.Blank(595, 842) + d.Blank(595, 842) + _, err := d.DropBlank(BlankOptions{}) + if err == nil { + t.Fatal("a document of nothing but blank pages was emptied") + } + if msg := err.Error(); !strings.Contains(msg, "2 pages") { + t.Errorf("the refusal does not say how many there were: %q", msg) + } + if got := d.PageCount(); got != 2 { + t.Errorf("the refusal still changed the document: %d pages", got) + } +} + +func TestBlankPagesRefusesWhatIsNotADocument(t *testing.T) { + if _, err := BlankPages([]byte("not a PDF at all"), BlankOptions{}); err == nil { + t.Error("rubbish was accepted") + } + // A media box the file made up cannot be asked to fill memory. + d := ladder(t, "x", 12) + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + if _, err := BlankPages(pdf, BlankOptions{MaxPixels: 4}); err == nil { + t.Error("a page past MaxPixels was drawn") + } +} + +// inkFraction's own edges, which no document reaches: a nil image and one of +// no size are zero rather than a division by zero. +func TestInkFractionOfNothing(t *testing.T) { + if got := inkFraction(nil, 12); got != 0 { + t.Errorf("inkFraction(nil) = %v", got) + } + if got := inkFraction(&raster.Image{}, 12); got != 0 { + t.Errorf("inkFraction(empty) = %v", got) + } + // Transparent black is ink, not paper: the alpha channel is counted. + one := &raster.Image{Pix: []uint8{0, 0, 0, 0}, W: 1, H: 1} + if got := inkFraction(one, 12); got != 1 { + t.Errorf("a transparent pixel inks %v, want 1", got) + } +} + +// inkOf renders a document's first page at the default resolution and returns +// its ink fraction, which is the quantity every assertion above is about. +func inkOf(t *testing.T, d *Doc) float64 { + t.Helper() + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + src, err := reader.Open(pdf) + if err != nil { + t.Fatal(err) + } + img, err := render.Page(src, 1, render.Options{DPI: defaultBlankDPI}) + if err != nil { + t.Fatal(err) + } + return inkFraction(img, defaultTolerance) +} + +// pageSizeAt says how many pixels an A4 page comes to at a resolution, so a +// test can talk in PIXELS rather than in fractions nobody can picture. +func pageSizeAt(t *testing.T, dpi float64) (int, int) { + t.Helper() + d := New() + d.Blank(595, 842) + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + src, err := reader.Open(pdf) + if err != nil { + t.Fatal(err) + } + img, err := render.Page(src, 1, render.Options{DPI: dpi}) + if err != nil { + t.Fatal(err) + } + return img.W, img.H +} + +// itoa is strconv.Itoa under a shorter name, for the page ranges above. +func itoa(n int) string { return strconv.Itoa(n) } + +// blackPNG is a small all-black opaque PNG, for a page that is covered in ink +// without depending on a font. +func blackPNG(t *testing.T) []byte { + t.Helper() + m := image.NewRGBA(image.Rect(0, 0, 64, 64)) + for y := 0; y < 64; y++ { + for x := 0; x < 64; x++ { + m.Set(x, y, color.RGBA{A: 255}) + } + } + var buf bytes.Buffer + if err := png.Encode(&buf, m); err != nil { + t.Fatal(err) + } + return buf.Bytes() +} + +// An explicit DPI and an explicit Tolerance are honoured, and the assertions +// say what each one DOES rather than only that the field is read: a coarser +// drawing loses a full stop, and a wider tolerance calls a faint mark paper. +func TestTheResolutionAndTheToleranceAreHonoured(t *testing.T) { + d := New() + d.Blank(595, 842) + if err := d.Stamp("1", Stamp{Text: ".", Font: Helvetica, Size: 8}); err != nil { + t.Fatal(err) + } + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + // At the default resolution the full stop is four pixels, so with a + // MaxInk of two pixels' worth the page is NOT blank. + w, h := pageSizeAt(t, defaultBlankDPI) + twoPixels := 2 / float64(w*h) + got, err := BlankPages(pdf, BlankOptions{MaxInk: twoPixels}) + if err != nil { + t.Fatal(err) + } + if len(got) != 0 { + t.Errorf("at %g DPI a full stop was called blank", defaultBlankDPI) + } + // ⛔ At 50 it is ONE pixel, which is what the default resolution was + // raised to avoid: the same MaxInk now calls the page blank. This is the + // measurement that chose the default, asserted. + w50, h50 := pageSizeAt(t, 50) + got, err = BlankPages(pdf, BlankOptions{DPI: 50, MaxInk: 2 / float64(w50*h50)}) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 { + t.Errorf("at 50 DPI a full stop still survived a two-pixel threshold; "+ + "the reason for the %g DPI default no longer holds", defaultBlankDPI) + } + // A tolerance of 255 makes everything paper, which shows the field is read + // and in which direction. + got, err = BlankPages(pdf, BlankOptions{Tolerance: 255}) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 { + t.Error("a tolerance of 255 did not make the page paper") + } +} + +// ⛔ DropBlank's two error returns, both reachable and neither obvious. +// +// The first draft of this file had no test for the write, and I nearly removed +// the check as unreachable: Bytes() refuses an empty document, and DropBlank +// refuses one before it ever gets there. But Bytes() ALSO refuses a document +// whose form field has nowhere to be written, which is a non-empty document — +// so the check is live, and the third time this session I almost called +// reachable code dead. +func TestDropBlankSaysWhyItCouldNotLook(t *testing.T) { + // A page past the pixel limit: the detector's own refusal comes back out. + d := ladder(t, "x", 12) + d.Blank(595, 842) + if _, err := d.DropBlank(BlankOptions{MaxPixels: 4}); err == nil { + t.Error("a page past MaxPixels was measured anyway") + } + // A document that cannot be written at all. This reaches DropBlank's write + // through Bytes, which is deliberately the ONLY place that knows a + // document with no pages cannot be written. + if _, err := New().DropBlank(BlankOptions{}); err == nil { + t.Error("an empty document was measured anyway") + } +} + +// ⛔ The Tolerance default, asserted rather than asserted ABOUT. A mutation +// setting defaultTolerance to zero survived the first version of this suite: +// every fixture rendered to exactly white, so no test could see the field at +// all and the doc comment's claim was unproven. +// +// Measured (see TestMeasureAFaintMark): a mark at 247/255 inks 2.2e-03 at +// tolerance zero and EXACTLY NOTHING at twelve. So this page is paper by +// default and ink if the tolerance is narrowed, which is the whole of what the +// field does and in which direction. +func TestAFaintMarkIsPaperByDefaultAndInkWithoutTheTolerance(t *testing.T) { + d := New() + d.Blank(595, 842) + if err := d.Stamp("all", Stamp{ + Text: "faint", Font: Helvetica, Size: 48, + Colour: [3]float64{0.97, 0.97, 0.97}, + }); err != nil { + t.Fatal(err) + } + pdf, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + got, err := BlankPages(pdf, BlankOptions{}) + if err != nil { + t.Fatal(err) + } + if len(got) != 1 { + t.Errorf("a mark at 247/255 was called ink with the default tolerance; "+ + "the %d-level tolerance is not doing what it says", defaultTolerance) + } + // Narrow the tolerance to one level and the same mark is ink. + got, err = BlankPages(pdf, BlankOptions{Tolerance: 1}) + if err != nil { + t.Fatal(err) + } + if len(got) != 0 { + t.Error("a mark at 247/255 is still paper at a tolerance of one level, " + + "so the tolerance is inert") + } +} + +// ⛔ The alpha channel, isolated. A mutation dropping it from inkFraction +// survived the first version of this suite, because the only transparent pixel +// any test used was transparent BLACK — whose colour channels are ink anyway. +// White-and-transparent is the case the rule is about: a renderer asked for a +// transparent background would otherwise report blank paper as solid ink, and +// an opaque white page as the same thing. +func TestATransparentWhitePixelIsInkNotPaper(t *testing.T) { + clear := &raster.Image{Pix: []uint8{255, 255, 255, 0}, W: 1, H: 1} + if got := inkFraction(clear, defaultTolerance); got != 1 { + t.Errorf("a transparent white pixel inks %v, want 1: "+ + "the alpha channel is not being counted", got) + } + opaque := &raster.Image{Pix: []uint8{255, 255, 255, 255}, W: 1, H: 1} + if got := inkFraction(opaque, defaultTolerance); got != 0 { + t.Errorf("an opaque white pixel inks %v, want 0", got) + } +} diff --git a/cmd/pdfops/expose_test.go b/cmd/pdfops/expose_test.go index a03a09e..b3c6459 100644 --- a/cmd/pdfops/expose_test.go +++ b/cmd/pdfops/expose_test.go @@ -4,9 +4,11 @@ import ( "fmt" "os" "path/filepath" + "strconv" "strings" "testing" + "github.com/go-pdfkit/ops" "github.com/go-pdfkit/reader" ) @@ -641,3 +643,143 @@ func boxesOf(t *testing.T, path string) [][2]float64 { } return out } + +// deblank finds the pages that hold nothing and drops them. +// +// ⛔ -list is asserted FIRST and separately, because it is the dry run: a tool +// that deletes pages has to be able to say which ones it means before it does +// it, and a flag that silently wrote the file anyway would be worse than no +// flag. +func TestDeblankList(t *testing.T) { + in := mixedFixture(t) + code, out, msg := exec("deblank", "-list", in) + if code != 0 { + t.Fatalf("code %d: %s", code, msg) + } + if got := strings.Fields(out); len(got) != 1 || got[0] != "2" { + t.Errorf("-list said %q, want page 2", out) + } + // And it wrote nothing: there was nowhere to write to, and it did not ask. + if code, _, _ := exec("deblank", "-list", in, filepath.Join(t.TempDir(), "x.pdf")); code != 1 { + t.Error("-list accepted an output file") + } +} + +func TestDeblankDropsThem(t *testing.T) { + in := mixedFixture(t) + out := filepath.Join(t.TempDir(), "out.pdf") + code, said, msg := exec("deblank", in, out) + if code != 0 { + t.Fatalf("code %d: %s", code, msg) + } + if !strings.Contains(said, "dropped 1") { + t.Errorf("said %q, want it to say one page went", said) + } + if got := len(contentsOf(t, out)); got != 1 { + t.Errorf("%d pages left, want 1", got) + } +} + +func TestDeblankWithNothingToDrop(t *testing.T) { + // ⛔ NOT fixture(t, 2): its pages carry a content stream that is not + // drawing operators, so they render as blank paper and the detector is + // right to say so. A fixture whose "content" is invisible is no fixture + // for a tool that measures ink. + in := writeDoc(t, func(d *ops.Doc) { + for i := range 2 { + d.Blank(595, 842) + if err := d.Stamp(itoa(i+1), ops.Stamp{Text: "words", Font: ops.Helvetica, Size: 24}); err != nil { + t.Fatal(err) + } + } + }) + out := filepath.Join(t.TempDir(), "out.pdf") + code, said, msg := exec("deblank", in, out) + if code != 0 { + t.Fatalf("code %d: %s", code, msg) + } + if !strings.Contains(said, "no blank pages") { + t.Errorf("said %q", said) + } + if got := len(contentsOf(t, out)); got != 2 { + t.Errorf("%d pages, want 2", got) + } + code, said, _ = exec("deblank", "-list", in) + if code != 0 || !strings.Contains(said, "no blank pages") { + t.Errorf("-list said %q", said) + } +} + +func TestDeblankRefusals(t *testing.T) { + in := mixedFixture(t) + out := filepath.Join(t.TempDir(), "out.pdf") + if code, _, _ := exec("deblank", in); code != 1 { + t.Error("no output file was accepted") + } + if code, _, _ := exec("deblank", "/no/such/file.pdf", out); code != 1 { + t.Error("a missing file was accepted") + } + if code, _, _ := exec("deblank", "-list", "/no/such/file.pdf"); code != 1 { + t.Error("-list on a missing file was accepted") + } + if code, _, _ := exec("deblank", "-nonsense", in, out); code != 1 { + t.Error("an unknown flag was accepted") + } + // -list has its own path through the detector, so its refusal needs its + // own case: the two branches share nothing but the flag. + if code, _, _ := exec("deblank", "-list", "-dpi", "100000", in); code != 1 { + t.Error("-list past the pixel limit was accepted") + } + // And a file that cannot be written: the pages were measured, the blanks + // were dropped, and the result had nowhere to go. + if code, _, _ := exec("deblank", in, "/no/such/directory/out.pdf"); code != 1 { + t.Error("an unwritable output was accepted") + } + // ⛔ A document of nothing but blank pages is refused rather than emptied. + blanks := blankFixture(t) + if code, _, said := exec("deblank", blanks, out); code != 1 { + t.Errorf("a document of blank pages was emptied: %s", said) + } + // And the detector's own refusal reaches the exit status. + if code, _, _ := exec("deblank", "-dpi", "100000", in, out); code != 1 { + t.Error("a page past the pixel limit was accepted") + } +} + +// mixedFixture is two pages: one with content, one blank. +func mixedFixture(t *testing.T) string { + t.Helper() + return writeDoc(t, func(d *ops.Doc) { + d.Blank(595, 842) + if err := d.Stamp("1", ops.Stamp{Text: "words", Font: ops.Helvetica, Size: 24}); err != nil { + t.Fatal(err) + } + d.Blank(595, 842) + }) +} + +// blankFixture is two pages, both blank. +func blankFixture(t *testing.T) string { + t.Helper() + return writeDoc(t, func(d *ops.Doc) { + d.Blank(595, 842) + d.Blank(595, 842) + }) +} + +func writeDoc(t *testing.T, build func(*ops.Doc)) string { + t.Helper() + d := ops.New() + build(d) + b, err := d.Bytes() + if err != nil { + t.Fatal(err) + } + path := filepath.Join(t.TempDir(), "in.pdf") + if err := os.WriteFile(path, b, 0o644); err != nil { + t.Fatal(err) + } + return path +} + +func itoa(n int) string { return strconv.Itoa(n) } diff --git a/cmd/pdfops/run.go b/cmd/pdfops/run.go index 8402ab4..c545a59 100644 --- a/cmd/pdfops/run.go +++ b/cmd/pdfops/run.go @@ -50,6 +50,7 @@ var commands = []command{ {"onepage", " ", "put every page onto a single page", runOnePage}, {"poster", "-across -down ", "cut each page into tiles, to print larger than the paper", runPoster}, {"divide", "-across -down ", "cut each page into smaller pages of its own content", runDivide}, + {"deblank", "[-max-ink ] [-dpi ] [-list] [out.pdf]", "find the pages that hold nothing, and drop them", runDeblank}, {"overlay", "-with ", "draw another file over these pages", runOverlay}, {"underlay", "-with ", "draw another file beneath these pages", runUnderlay}, {"blank", "-before ", "insert an empty page", runBlank}, @@ -1658,3 +1659,66 @@ func runDivide(c *context, args []string) error { } return save(d, fs.Arg(1)) } + +// runDeblank finds the pages that hold nothing and drops them. +// +// ⛔ -list takes no output file and writes nothing, because the first thing +// anybody should do with a tool that deletes pages is ask it which ones it +// means. BentoPDF's own version of this makes the caller confirm; a command +// cannot ask, so it offers the dry run instead. +func runDeblank(c *context, args []string) error { + fs := flags("deblank") + dpi := fs.Float64("dpi", 0, "how finely to draw each page; 0 means 150") + maxInk := fs.Float64("max-ink", 0, "fraction of a page that may be ink and still count as blank; 0 keeps any page with a mark") + list := fs.Bool("list", false, "say which pages look blank and write nothing") + keepAnnots := fs.Bool("annotations", false, "count notes, highlights and form fields as ink") + if err := fs.Parse(args); err != nil { + return err + } + want := 2 + usage := " " + if *list { + want, usage = 1, "" + } + if err := wantArgs(fs, want, usage); err != nil { + return err + } + opt := ops.BlankOptions{ + DPI: *dpi, MaxInk: *maxInk, KeepAnnotations: *keepAnnots, + } + if *list { + pdf, err := os.ReadFile(fs.Arg(0)) + if err != nil { + return err + } + blank, err := ops.BlankPages(pdf, opt) + if err != nil { + return err + } + if len(blank) == 0 { + fmt.Fprintln(c.out, "no blank pages") + return nil + } + for _, n := range blank { + fmt.Fprintln(c.out, n) + } + return nil + } + d, err := c.open(fs.Arg(0)) + if err != nil { + return err + } + went, err := d.DropBlank(opt) + if err != nil { + return err + } + if err := save(d, fs.Arg(1)); err != nil { + return err + } + if len(went) == 0 { + fmt.Fprintln(c.out, "no blank pages") + return nil + } + fmt.Fprintln(c.out, "dropped", len(went), "page(s):", went) + return nil +} diff --git a/go.mod b/go.mod index 0c92c10..a1d9497 100644 --- a/go.mod +++ b/go.mod @@ -12,10 +12,12 @@ require ( require ( github.com/ajroetker/go-highway v0.0.12 // indirect + github.com/go-images/jpeg v0.2.0 // indirect github.com/go-images/jpeg2000 v0.13.2 // indirect github.com/go-opentype/fonts v0.10.0 // indirect github.com/go-opentype/opentype v0.13.0 // indirect github.com/go-pdfkit/pdffont v0.3.1 // indirect + github.com/go-pdfkit/render v0.67.2 // indirect github.com/sergeymakinen/go-bmp v1.0.0 // indirect github.com/sergeymakinen/go-ico v1.0.0 // indirect github.com/tannevaled/gobig2 v0.2.0 // indirect diff --git a/go.sum b/go.sum index 0532a4b..b8a5dd3 100644 --- a/go.sum +++ b/go.sum @@ -2,6 +2,8 @@ github.com/ajroetker/go-highway v0.0.12 h1:OcGp/l/W7WQtH5ELLuNnKXCuR7ePpWGh2RWZI github.com/ajroetker/go-highway v0.0.12/go.mod h1:un7fcrwix7UOHJrr18Lgh5eeF+1yaZiGWdTw3FbfDv4= github.com/go-gfx/gfx v0.34.1 h1:zXRptZlqRyfE0de/8h9N4Y4cSeySnQB9vWuK+LrwgOM= github.com/go-gfx/gfx v0.34.1/go.mod h1:iGZt+HN5CEwr/nPqxxLIAvNBR9pAKhoZl9scGEwQn9M= +github.com/go-images/jpeg v0.2.0 h1:dn55cTczPdOJf+NQq+sETCFeov/KeP1haF1ES+FJa+4= +github.com/go-images/jpeg v0.2.0/go.mod h1:2C1DevAM+HaQlysTLbSdyol8X/RlvmsXmHAXs9teAqo= github.com/go-images/jpeg2000 v0.13.2 h1:wthA3iZkyqWXzNXxd6EjjsrsQ7GOYX7vYZbaI4vU6+I= github.com/go-images/jpeg2000 v0.13.2/go.mod h1:Qqxz1gBrUwFDBOGkA4RGyTsX7wifr0wvaTQ7k8FK0wY= github.com/go-opentype/fonts v0.10.0 h1:9f7ik795SlBEJjz6iqBE/ZV2PaTauAj537MMWw9t4OQ= @@ -16,6 +18,8 @@ github.com/go-pdfkit/pdffont v0.3.1 h1:+K7opHGRERDZ5EMkA12s1GT48t8GpncArhnyzyc5b github.com/go-pdfkit/pdffont v0.3.1/go.mod h1:eFJ/7t9AvcX76KY5qNJMMKetz8hUUUdVM1qSWLi3S1g= github.com/go-pdfkit/reader v0.7.0 h1:+Fz5i8Zq6beBpTc3pNTbJ3Qf14aroA5txgeVzVGLTSY= github.com/go-pdfkit/reader v0.7.0/go.mod h1:EYHHREPKT/orDzlGiFPnPVPPuaRDLiAYezIr0cCFXaA= +github.com/go-pdfkit/render v0.67.2 h1:oBIFoROjorKednM/8Xz9Dhf9VArkFFaL4jZpCEcI72M= +github.com/go-pdfkit/render v0.67.2/go.mod h1:S+p47RMo1vUykzxC+5YMzvQ8hjHBVX6HoHIAyJlzmp4= github.com/go-pdfkit/xfa v0.22.0 h1:IbdMcNYN40kataz/n+3pkz2METftMh+re/MkcY2eCbw= github.com/go-pdfkit/xfa v0.22.0/go.mod h1:nZzuk+EIVeUzKfwZkWhE2/fAWr4IW/bouLf5xMED+Bg= github.com/sergeymakinen/go-bmp v1.0.0 h1:SdGTzp9WvCV0A1V0mBeaS7kQAwNLdVJbmHlqNWq0R+M=