Repository navigation
Foundation: image-check hardening follow-ups (pin test, zlib end check, inflate cap) (#329) - #334
Conversation
…IDAT inflate cap (#329) Refs #329. Pin test uses plain attribute access to vision.IMAGE_LIMITS; IDAT is inflated at most one byte past the length the header implies (with a backstop for absurd headers); tests kill the removal of the zlib end check; docs/vision.md points committed fixtures at the stricter 1 MiB asset rule. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
PR #334 — independent Opus review 1 of 2
VerdictApprove. The mechanism is correct: I re-derived the expected-length arithmetic independently, probed zlib's What I verified about the new cap, independently
FindingsNo must-change findings. S1. The memory bound is still
|
case, as a file under recipes/*/assets/ |
compressed | traced peak | max RSS |
|---|---|---|---|
| 320×320 header, 100 MiB zero bomb (the new test's case) | 101,929 B | 962,161 B (3.14) / 970,461 B (3.10) | +2 MB |
| 320×320 header, 1 GiB zero bomb | 1,043,644 B | 3,786,986 B | +2 MB |
| 9000×9000 header, 300 MiB zero bomb | 305,767 B | 486,740,812 B | 494 MB |
The last row is a 299 KB file that passes the 1 MiB rule and makes the Hygiene job — which runs on every pull
request, forks included, on Python 3.10 — allocate about half a gigabyte; the peak is ~2× expected because
CPython grows the output buffer by copying. This is not a regression (on main any ~1 MiB bomb reaches the
same peak with any header), it is disclosed in the PR body, and it is harmless on a 16 GB runner inside a
5-minute job, so I am not blocking on it. But the issue's own wording is "so a small compressed bomb cannot
use large memory in the Hygiene job", and that holds only for an honest header.
The cheap fix, here or in a follow-up: lower _MAX_INFLATED. The largest asset the dimension rules allow
needs 19,671,932 bytes and the largest embedded figure in the repository today needs 5,030,535, so 32 MiB
keeps 1.6× headroom over any legal asset and 6× over today's figures while cutting the worst case from
~490 MB to ~64 MB. (Passing the asset path a tighter cap than the embedded path would be stricter still, at
the cost of a parameter on inspect_png.)
S2. A test comment — and one row of the PR body — states something that is not true
tests/test_check_recipe_assets.py:866-867:
One byte too many is refused and the exact length is accepted, so neither a limit that is one too small
(which would reject every good file) nor one that is too large is possible.
Both halves are false, and I checked each by mutating the tool myself:
- A limit one too small does not reject good files, and this test does not detect it. With
decompress(bytes(idat), expected),test_the_inflate_limit_is_exactly_one_byte_past_the_expected_length
passes (1 passed). zlib returnsZ_STREAM_ENDwhen it reads the final block and the trailer even though
the output buffer filled exactly, soeofisTrueand a good file is accepted. Verified directly on zlib
1.2.13 and 1.3.2:maxlen=1000on a 1000-byte payload giveslen(out)=1000, eof=True, unconsumed=0. The
mutant is killed, but bytest_idat_is_inflated_no_further_than_the_length_the_header_impliesand
test_the_inflated_idat_length_must_match_the_image_header— i.e. because an over-long stream then reports
"does not end cleanly" instead of "inflates past", by the message, not by a rejected good file. The body's
row
"limitexpected(no+ 1) | killed (exact-length file rejected)" has the right verdict and the wrong
reason. - A limit that is too large is possible.
expected + 2survives the whole file (432 passed). Only margins
of order a megabyte are caught, and only through the 2 MiB peak bound — which the body does say honestly.
What the + 1 actually buys is worth writing down instead, because it is a real argument: it keeps the accept
path off the output limit entirely, so a valid file never depends on zlib's exact-fill behaviour, and one
spare byte is what makes "too long" distinguishable from "does not end cleanly". Please reword the comment
and that table row, and consider renaming the test: what it pins is "at least expected, and over-length is
reported as over-length", not "exactly one".
S3. _MAX_INFLATED is pinned by nothing, and the new refusal path can become a false positive
tools/check_recipe_assets.py:156, 297-302. Lowering the constant to 16 MiB passes all 432 tests, yet it
then rejects the largest asset the dimension rules allow — 1568×1568 RGBA 16-bit interlaced, 19,671,932 bytes
raw in a 19 KB file — with "the image header implies 19,671,932 bytes of pixel data; the most inflated is
16,777,216". I built that file and ran it. Before this PR a wrong value there could only widen a limit; now
it can reject a legal image, so it deserves a test: four lines next to the backstop test that build that
19 KB file and assert it is accepted. That also records why the constant has to stay above ~20 MB — useful if
anyone acts on S1.
S4. The peak measurement is sound; two lines would make it robust
tests/test_check_recipe_assets.py:853-862. Choosing tracemalloc over RSS is right: it is deterministic, it
needs no resource module (so it works on Windows, which contributors here use), and the two interpreters
agree to within 1% (962,161 vs 970,461 bytes) while the uncapped code peaks at 219,758,182 — a 200× margin, so
the 2 MiB bound is in no danger of flaking on its own. The hole is the surrounding bookkeeping:
tracemalloc.start() is a no-op when tracing is already on, get_traced_memory() then returns the peak of
the whole session, and the finally: tracemalloc.stop() switches off tracing that something else turned on.
Under PYTHONTRACEMALLOC=1 the test fails for that reason alone — I reproduced it, "peak 4,851,256 bytes".
Guard it:
started = not tracemalloc.is_tracing()
tracemalloc.start()
tracemalloc.reset_peak()
try:
...
finally:
if started:
tracemalloc.stop()S5 (minor). "without inflating" is not observed
tests/test_check_recipe_assets.py:874-879 proves the rule and the message, not the absence of the inflate.
If the name is to carry that claim, monkeypatch.setattr(assets.zlib, "decompressobj", boom) would make it
literal.
S6 (minor). png_with_idat replaces only the first IDAT
tests/test_check_recipe_assets.py:814-819. Correct today — Pillow 13 writes one IDAT for the flat 320×320
png(); I checked the chunk list is [('IHDR', 13), ('IDAT', 954), ('IEND', 0)] — but on a base image Pillow
splits, the helper would leave a second IDAT after the spliced body and the tests built on it would be
measuring something else. Dropping every other IDAT in the same pass makes it unconditional.
S7 (minor). The PR body overstates the docs/vision.md item
The body says that edit "is listed as an acceptance item on the issue". Issue #329 has exactly three
acceptance criteria and none mentions docs/vision.md; neither linked review comment asks for it either (the
#321 comment only observes that the 1 MiB and 16 MiB caps coexist on purpose). The body also says the doc
now "says the loader's 16 MiB limit applies to the loader", which is slightly ahead of the text: the diff adds
a parenthetical pointing committed fixtures at the 1 MiB rule. The edit itself is accurate, one sentence long,
in the spirit of the follow-up and consistent with docs/fixtures.md:351, and the relative link resolves — I
would keep it and fix the sentence in the body.
Status of the prior findings this issue tracks
| Prior finding | Status |
|---|---|
#321 comment 6305791542, suggestion 1 — "make the cross-pin unconditional … replace the getattr with a direct attribute access and delete the comment that says 'until it does'" |
Resolved. tests/test_check_recipe_assets.py:1124-1131 uses plain vision.IMAGE_LIMITS[k] and the comment is gone. It bites on the checker side alone: renaming IMAGE_LIMITS throughout vision.py fails exactly one test in that file with AttributeError, and max_area = 2_458_625 in vision.py fails exactly one test in that file. |
#322 comment 6305113842, S9 — "removing if not inflater.eof or inflater.unconsumed_tail: now survives … a test with zlib.compress(rows)[:-4] would pin it" |
Resolved. The new test uses both the missing-Adler stream and a Z_SYNC_FLUSH stream, each asserted (in the test itself, so a zlib version difference would fail loudly rather than weaken it) to inflate to exactly the right rows. if not inflater.eof: → if False: is killed by this test and by nothing else. |
#322 comment 6305113842, S10 — "inflater.decompress(idat, expected + 1) bounds memory to about 20 MB for the largest legal asset" |
Resolved as to the code; the stated bound holds only for an honest header. The call is exactly what was asked for and is tested, but see S1: "about 20 MB" is not the worst case, and no test shows the residual. |
Acceptance criteria of #329
- Compare the checker's limits with
vision.IMAGE_LIMITSby plain attribute access, not a skip-if-missing
lookup, so a rename cannot turn it into a silent pass — met
(tests/test_check_recipe_assets.py:1124-1131). The old form was a silent pass by construction
(getattr(..., None)thenif limits is not None); the new form raisesAttributeErroron a rename, which
I demonstrated, and also catches a value divergence. - A test exercises the "zlib stream ended cleanly" check on its own (a complete IDAT stream whose final
block or checksum is missing); the mutant that removes the check is killed — met. Two independent
streams, exact rule and message asserted, mutant killed by this test alone. - IDAT decompression stops at the expected decompressed length (plus a small margin) instead of allowing up
to 256 MiB, so a small compressed bomb cannot use large memory in the Hygiene job; tested — met, with
the caveat in S1. The stop is implemented and tested (962,161 bytes peak against 219,758,182 without it),
but a header claiming large dimensions still raises the bound to the backstop (measured peak
486,740,812 bytes). The PR body discloses this and asks reviewers to choose; I am content to see it
tightened in a follow-up.
The docs/vision.md item the body claims is not an acceptance criterion (S7); the edit itself is accurate.
The PR body against the diff
Accurate except where S2 and S7 say otherwise — in particular every row of the mutation table has the right
verdict, and the "worst case remains bounded by the header, not by a fixed small number" paragraph is an
honest statement of S1. Numbers I reproduced:
- "Full
pyteston 3.14: 5896 passed, 49 skipped, 50 deselected, 31 xfailed" — I get 5897 passed, 49
skipped, 50 deselected, 31 xfailed, 0 failed, in 195 s. One more passing test than the body says; the
likeliest explanation is that the body's run predates the merge oforigin/maininto the branch. Nothing
fails either way. - "On 3.10:
tests/test_check_recipe_assets.py tests/test_vision.py432 passed" — reproduced exactly, 432
passed on 3.10.21;tests/test_check_recipe_assets.pyalone is 165 passed on 3.10.21. ruff check .andruff format --check .— both pass (401 files formatted).- "
check_hygiene.py: passed (611 files)", "check_recipe_assets.py: passed (461 files)",
"render_catalog.py --check: exit 0" — all three reproduced exactly. - "Red before: with
tools/check_recipe_assets.pyfromorigin/main, the new test file gives 2 failed (the
bomb test and the backstop test), 163 passed; the zlib end test passes on old code" — reproduced exactly:
withtools/check_recipe_assets.pyfrom39ddfce,2 failed, 163 passed, and the two are those tests.
One detail the body does not mention: red-before, the bomb test fails on its message assertion (the old
code inflates the lot and says "IDAT inflates to … bytes; the image header implies 307,520"), not on the
peak, because the message is asserted first — the 2 MiB peak bound
is what kills the_MAX_INFLATEDmutant, not what makes the test red againstmain. Both are covered, but
they are different assertions.
Mutants I applied myself
Each applied alone to the worktree and run against tests/test_check_recipe_assets.py +
tests/test_vision.py (432 tests), then the file restored byte for byte; git status --porcelain empty
afterwards.
| Mutant | Result |
|---|---|
if not inflater.eof: → if False: |
killed — test_a_complete_deflate_stream_must_still_end_with_its_final_block_and_checksum |
limit → _MAX_INFLATED |
killed — bomb test, "peak 219,758,182 bytes" |
limit → expected (no + 1) |
killed (2 tests) — but not by the off-by-one test; see S2 |
limit → expected + 2**20 |
killed — bomb test, "peak 3,057,912 bytes" |
limit → expected + 2 |
survived (see S2) |
if len(inflated) > expected: → if False: |
killed (2 tests) |
if len(inflated) > expected: → >= |
killed (42 tests) |
if expected > _MAX_INFLATED: → if False: |
killed — the new backstop test |
if expected > _MAX_INFLATED: → >= |
survived (unreachable boundary, harmless) |
if len(inflated) != expected: → if False: |
killed — the short-stream case |
if inflater.unused_data: → if False: |
killed — test_idat_must_be_one_clean_zlib_stream |
interlace → 0 in the expected computation |
killed — test_sixteen_bit_and_interlaced_pngs_are_accepted |
restore or inflater.unconsumed_tail |
survived — confirms the removal is behaviour-preserving |
_MAX_INFLATED → 16 MiB |
survived (see S3) |
vision.IMAGE_LIMITS["max_area"] → 2,458,625 |
killed, including one test inside the checker's own file |
IMAGE_LIMITS renamed throughout vision.py |
killed with AttributeError, including one test in the checker's file |
Checks I ran
Fresh venvs under /home/agent/r1-pr334/ (uv venv --python 3.14 and --python 3.10,
uv pip install --link-mode symlink -e ".[dev]"), every heavy command under
flock .claude/heavy.lock nice -n 10, pytest -p no:cacheprovider, no nested flock, long runs backgrounded
with log polling, no network, no key.
ruff check .— All checks passed.ruff format --check .— 401 files already formatted.pytest(full, Python 3.14.7) — 5897 passed, 49 skipped, 50 deselected, 31 xfailed, 0 failed, 195 s.pytest tests/test_check_recipe_assets.py tests/test_vision.py— 432 passed on 3.14.7;
432 passed on 3.10.21.tests/test_check_recipe_assets.pyalone — 165 passed on 3.10.21, the floor the Hygiene
job runs on.python tools/check_hygiene.py— passed, 611 files.python tools/check_recipe_assets.py— passed, 461 files underrecipes/.python tools/render_catalog.py --check— exit 0.- My own probes (standard library only, synthetic in-memory PNGs, run on 3.10.21 and 3.14.7): the
25,245-combinationpng_raw_lengthcross-check; the exact/±1 boundary per colour type and bit depth,
interlaced and not; zlibmax_lengthsemantics on zlib 1.2.13 and 1.3.2; the three bomb measurements in S1;
inspect_embeddedover all 134 images in the 51 committed notebooks; the largest legal asset; 16 mutants. - No notebook executed. The diff changes no notebook, no fixture and nothing a notebook imports: no recipe
importsjev_cookbook.vision, andtools/check_recipe_assets.pyis loaded only by the checkers
(tools/check_notebook_fresh.py:172-176). The notebook-facing effect of this change isinspect_embedded,
so I exercised that over every image in every committed notebook instead, with no finding; CI's
Notebook (…)jobs were green on this head where they had finished when I looked. git status --porcelainempty at the end. The only files I created are under/home/agent/r1-pr334/
(deleted when I finished) and this review.
Outside this PR, noticed but not to be fixed here
check_imagerecordsfile-too-largeand then parses the file anyway, andtracked_entrieshas already
read every tracked byte into memory, so the per-file 1 MiB rule does not by itself bound the checker's
memory. Pre-existing, unrelated to this diff.- A PNG whose only IDAT chunk has zero length reports "no IDAT chunk" rather than an empty-stream error.
Pre-existing, cosmetic.
OPUS-REVIEW: APPROVE ba797d6
PR #334 — Opus review 2 of 2 (independent)
VerdictNo must-change findings. Approve. The change is a strict improvement, it is verdict-preserving on everything I ScopeThree files, all
Branch name The central question: does the new checker ever disagree with the old one?This file is the gate in the Hygiene job on every PR, so a verdict change on a real or plausible image is the thing Reasoned first. The accept condition is unchanged in substance. OLD accepts iff Then measured. 33 hand-built cases (valid RGB8 / RGB16 / interlaced / palette / 1-bit grey / split-IDAT; short
One rule-class change exists, outside that corpus. A PNG whose IDAT both inflates past
Still a failure either way, and arguably the better message: the over-length is the real defect and the trailing byte The backstop boundary is exact and inclusive. I solved for a header whose implied length is exactly The whole real tree. Both checkers print Real notebook figures. I decoded every image embedded in every committed notebook — 51 notebooks, 134 embedded Memory and wall timePeak traced allocation while checking a 305,767-byte deflate bomb that inflates to 300 MiB (identical on 3.10.21 and
So the realistic case improves by a factor of ~342, and the absurd-header case by ~876. The middle row is the Wall time is neutral to better. Over the whole tree, in-process, 7 runs each: OLD median 1.6 ms, NEW median 1.6 ms
The Hygiene job's 5-minute timeout is in no danger from this change. DocsThe
The constants in FindingsMust changeNone. SuggestionsS1 (medium, for the record and for a follow-up). The PR body understates the memory numbers by about 2×, and the The builder asked for a decision on this ("Say if you want either"). My preference, for a follow-up rather than this Per CONTRIBUTING.md ("A pull-request description is written once, at open … and never edited after the final push"), S2 (low). The backstop constant's value is unpinned in both directions, and lowering it would be a silent
The reason is that the largest image anywhere in the suite is S3 (low). S4 (informational, PR body). The stated reason for dropping S5 (low). Link the section, not the page. S6 (informational). One test-count digit in the PR body. The body reports "5896 passed, 49 skipped, 50 The PR body against the diffEverything I could check holds.
Acceptance criteria of #329
Status of the prior findings the issue lists
Checks runFresh venvs under Python 3.14.7 (
Python 3.10.21 (
Behavioural comparison (old vs new)
Mutation runs (each applied alone, worktree restored byte-for-byte,
Red-before
Notebooks. I executed none. No notebook can change: the diff touches the checker, its tests and one sentence of Repository state. Outside my task, reported not fixed
OPUS-REVIEW: APPROVE ba797d6 |
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
PR #334 — delta re-review (reviewer 1 of 2)
I accept the deltaThe delta is the merge commit alone and it carries no content:
So nothing I reviewed has changed, and every finding and measurement in The body's "Corrections after review": one of the four is wrong
The one thing to fix before merging (body only — no code change, no new commit, so no further re-review): Checks I ran at this head (fresh venv under
|
PR #334 — Opus delta re-review (reviewer 2 of 2)
I accept the code deltaThe merge itself is clean and I accept it without reservation. My approval of the diff at
One blocking problem, in the new body text onlyThe first bullet of "Corrections after review" is false. It says:
The So the original table row was right ("limit Consequently "Only the The third clause needs a smaller fix. The comment at Its parenthetical "(which would reject every good file)" is the wrong part, for the What to change (body text only, no code): rewrite that bullet to say that I note two consequences of editing the description again: CONTRIBUTING.md says a description "is written once, at open The other three corrections are exactly trueEach matches my own measurements to the figure:
The rest of the body is unchanged from the head I approved, and my CI
Checks run locally at
|
PR #334 — Opus addendum (reviewer 2 of 2): body corrections verified
My sole blocker is resolved, and I checked every sentence of the rewritten "Corrections after review" section and the OPUS-REVIEW: APPROVE cfb4fb0 |
Closes #329
What this adds
Three changes to the image-asset checker, plus the doc cross-reference item on the issue. Files:
tools/check_recipe_assets.py,tests/test_check_recipe_assets.py,docs/vision.md.test_the_checker_limits_match_the_shared_image_limits(renamed; the "when the package exports them" qualifier is gone) readsvision.IMAGE_LIMITSby plain attribute access and always compares. A rename now errors instead of passing with nothing compared.test_a_complete_deflate_stream_must_still_end_with_its_final_block_and_checksumbuilds two IDAT streams that inflate to exactly the right rows with nothing left over:zlib.compress(rows)[:-4](no Adler-32) and aZ_SYNC_FLUSH-ended stream (no final block). Only the "does not end cleanly" check can reject them, and the test asserts that rule and message.inspect_pngnow computes the expected raw length first and callsdecompress(idat, expected + 1), so a bomb allocates about one image, not up to 256 MiB. A stream that reachesexpected + 1is reported asIDAT inflates past the N bytes the image header implies. Theor inflater.unconsumed_tailterm is dropped from the end check because the over-length check now covers it._MAX_INFLATED(256 MiB) stays only as a backstop: a header that implies more than that is refused before any inflating, because otherwise a header with absurd dimensions would raise the cap with it.docs/vision.mdsays the loader's 16 MiB limit applies to the loader, and that committed fixtures are held to the stricter 1 MiB, no-metadata rule indocs/fixtures.md.New tests:
test_idat_is_inflated_no_further_than_the_length_the_header_implies(a 100 MiB zero bomb, about 100 KiB compressed, peak traced memory under 2 MiB),test_the_inflate_limit_is_exactly_one_byte_past_the_expected_length,test_a_header_implying_more_than_the_backstop_is_refused_without_inflating.Decisions for the reviewers
+ 2**20) is also killed, but only because of the 2 MiB peak bound; a margin of a few hundred KiB would not be. I judged that acceptable under "a small margin".recipes/*/assets/the dimension limits are checked afterinspect_png, so I did not reorder them (that would change which finding a too-large image gets). Lowering_MAX_INFLATEDwould risk false positives on embedded notebook figures, which have no dimension limit. Say if you want either.docs/vision.mdedit is outside the "checker and its tests" paths but is listed as an acceptance item on the issue.How it was verified
Python 3.14.7 venv, and 3.10.21 for the checker and vision tests, under the heavy lock.
ruff check .passes;ruff format --check .passes (3.14 and 3.10 venvs).pyteston 3.14: 5896 passed, 49 skipped, 50 deselected, 31 xfailed. On 3.10:tests/test_check_recipe_assets.py tests/test_vision.py432 passed.python tools/check_hygiene.py: passed (611 files).python tools/check_recipe_assets.py: passed (461 files).python tools/render_catalog.py --check: exit 0.python tools/notebook_ci.py fixtures: 51 recipe folders, 0 failed.tools/check_recipe_assets.pyfromorigin/main, the new test file gives 2 failed (the bomb test and the backstop test), 163 passed; the zlib end test passes on old code, as the check already existed. The old pin test withIMAGE_LIMITSrenamed invision.pystill passes (silent pass); the new one fails withAttributeError.tests/test_check_recipe_assets.pyandtests/test_vision.py, then restored:if not inflater.eof:replaced withif False:decompress(..., expected + 1)back to_MAX_INFLATEDexpected(no+ 1)test_idat_is_inflated_no_further_than_the_length_the_header_impliesandtest_the_inflated_idat_length_must_match_the_image_header(not because an exact-length file is rejected, as an earlier version of this row said)expected + 2**20expected + 2is not killedMAX_PIXELS2,458,625IMAGE_LIMITSrenamed invision.pyAttributeError)Corrections after review (reviewers 1 and 2 of this PR)
expected + 2. Only the+ 2**20margin is killed, and only by the 2 MiB peak bound. The exact+ 1is a design choice, not a pinned property. The comment neartests/test_check_recipe_assets.py:866gives a wrong reason for why theexpected-without-+1mutant is killed (it is killed by two other tests, named in the table)._MAX_INFLATEDis pinned by no test: mutants setting it to 16 MiB or 4 GiB, or changing>to>=, survive the suite, even though 16 MiB would reject the largest legal asset (1568x1568 RGBA 16-bit, implies 19,670,560 bytes non-interlaced or 19,671,932 interlaced).png-structurerather thantrailing-data.These are follow-up items, not defects in the verdicts the checker gives.