fix(#1856): bound profile and config JSON at one configurable limit - #1981
Conversation
The two JSON readers carried different fixed caps, and the smaller one sat inside the range iccDEV's own tools produce. CIccProfileJson::LoadJson refused any document over 64 MiB, on the stated assumption that "real JSON dumps of even big iccMAX spectral profiles top out around 5 MB". The tracked corpus does not support that estimate. The profile built from Testing/hybrid/CMYK-W_Overprint_Profile.xml emits 58,159,328 bytes at the default -indent=2 and 97,444,884 bytes at -indent=4, a setting iccToJson documents and accepts. So iccToJson could write, at a documented setting, a file iccFromJson then refused. The refusal itself was clean - specific diagnostic, exit 1, no output file - so this was a threshold defect, not a silent failure. loadJsonFrom in IccJsonUtil.cpp, which reads the -cfg documents, applied an unrelated 100 MiB to the same kind of input. Both readers now share one bound, ICC_JSON_MAX_FILE_BYTES, derived at configure time from a new ICC_JSON_MAX_FILE_MB cache variable. It is rejected if it is not a positive integer or exceeds 1024 MiB, since a bound large enough to be no bound defeats the purpose. Each translation unit keeps a finite fallback so builds that do not go through Build/Cmake still compile with a cap, and the diagnostic now reports the limit that was actually compiled in rather than a hard-coded number. The default is 128 MiB. That clears every tracked fixture at the documented -indent=4 (97,444,884 bytes) with headroom through -indent=5, which is the range the defect was actually about. It does not remove the asymmetry entirely, since -indent accepts up to 20 and emits 411,729,332 bytes on the same profile. It is deliberately not raised to icXmlMaxTextFileBytes' 256 MiB, even though this issue asks to align the XML and JSON limits, because a byte count is a weak proxy for the memory the cap is meant to bound. Indentation inflates the file without inflating the parsed document: peak RSS was flat at 79 MB across the same profile emitted at -indent=0, 1 and 2, spanning 18.9 MB to 58.2 MB of input. Cost per admitted byte is governed by document shape instead, measured from 1.4x on an indented profile to 16.3x on a dense array of small values, so 128 MiB already admits roughly 2 GiB of peak resident memory and 256 MiB would admit twice that. docs/iccjson.md records both tables so the bound can be revisited against data rather than an estimate. The case added to iccdev.json-parser-regressions builds the fixture from its tracked XML, converts at -indent=4 and requires the round trip to succeed. It fails if the emitted document ever falls back under the former 64 MiB cap, because at that point it would no longer be exercising the threshold, and skips rather than fails when a build configures a bound below the fixture size, where refusal is the correct behaviour. The script now also resolves ICCDEV_TOOLS_DIR and ICCDEV_TESTING_DIR to absolute paths: this test runs iccFromXml with Testing/hybrid as its working directory, and the script's own documented relative defaults would not resolve the binary from there. Scope: this is the JSON half of #1856 only. The XML_PARSE_HUGE half is unchanged - IccProfileXml.cpp:1038 records a deliberate decision to drop that flag because it disables libxml2's depth, name-length, text-length and entity-expansion guards, so restoring it is a security trade-off for a maintainer to rule on rather than something to settle here.
CI Status2026-08-05 14:29:09 UTC Review in Process: Error: Value cannot be null. (Parameter 'ContainerId') |
PR Status2026-08-05 14:33:59 UTC
|
|
Part of #1856 — the JSON half. The
XML_PARSE_HUGEhalf is deliberately notaddressed here; see Scope at the end.
Developed from the QA branch
ci-qa-issue-1856(@xsscx,f81ff047), which iswhere the CMake-knob approach comes from. The code here is written fresh, and
the review of that branch turned up four things worth correcting before this
lands — those are listed below rather than folded in silently.
The defect
The two JSON readers carried different fixed caps, and the smaller one sat
inside the range iccDEV's own tools produce.
CIccProfileJson::LoadJsonrefused any document over 64 MiB, on the statedassumption that "real JSON dumps of even big iccMAX spectral profiles top out
around 5 MB". The tracked corpus does not support that estimate:
.icciccToJsonoutputTesting/hybrid/CMYK-W_Overprint_Profile.xml-indent=2, the default)-indent=4, documented)So
iccToJsoncould write, at a documented setting, a fileiccFromJsonthenrefused. The refusal itself was clean — specific diagnostic, exit 1, no output
file — so this is a threshold defect, not a silent-failure one.
Separately,
loadJsonFrominIccJsonUtil.cpp, which reads the-cfgdocuments, applied an unrelated 100 MiB to the same kind of input.
The fix
Both readers now share one bound,
ICC_JSON_MAX_FILE_BYTES, derived atconfigure time from a new
ICC_JSON_MAX_FILE_MBcache variable.The default is 128 MiB, which clears every tracked fixture at the documented
-indent=4with headroom through-indent=5— the range the defect was about.-indentWhy not 256 MiB, to match the XML side
This issue asks to align the XML and JSON limits, and
icXmlMaxTextFileBytes(
IccXML/IccLibXML/IccUtilXml.h) is 256 MiB — so matching it is the obviousmove. I measured before taking it, and the measurement argues against.
A byte count is a weak proxy for the memory this cap exists to bound.
Indentation inflates the file without inflating the parsed document: peak RSS
was flat at 79 MB for the same profile emitted at
-indent=0,1and2,spanning 18.9 MB to 58.2 MB of input. What actually drives memory is document
shape:
-indent=2(58.2 MB)-indent=0(18.9 MB)So 128 MiB already admits roughly 2 GiB of peak resident memory on hostile
input, and 256 MiB would admit about 4 GiB — more than a CI runner has. The
alignment argument is about consistency, not about memory, and I did not want it
to read as though the larger cap were the safer one.
Two follow-ups this suggests, not done here:
parser_callback_t) wouldbound memory directly instead of by proxy. That is the control this issue
probably wants in the long run.
loadJsonFromholdsstd::string buf(flen)and the DOM simultaneously, whereLoadJsonstreams via
f >> rootand never holds the raw text.A third 64 MiB cap, left alone deliberately
IccJSON/IccLibJSON/IccTagJson.cpp:2248(kMaxClutFileBytes) also refuses at64 MiB. It is not included in this change, and that is a judgement call
worth stating rather than leaving to look like an oversight: it bounds an
external CLUT data file referenced from a JSON document, not the JSON document
itself, so it is a different resource with a different threat model and no
reason to move in lockstep. Its sibling
kMaxSparseMatrixArrayBytes(
IccTagBasic.cpp:5599) is 64 MiB for the same class of reason. Happy to bringeither in scope if you would rather all four moved together.
This also does not remove the emit/read asymmetry entirely, and
docs/iccjson.mdsays so rather than implying it is solved.Also: the value is rejected at configure time if it is not a positive integer or
exceeds 1024 MiB; each translation unit keeps a finite fallback so non-CMake
builds still compile with a cap; and the diagnostic now reports the limit that
was actually compiled in (
JSON file exceeds 256 MiB limit) instead of ahard-coded number.
Findings from the QA branch, corrected here
iccFromXmlinside
cd Testing/hybrid, but the script's own documented defaultICCDEV_TOOLS_DIR=Build/Toolsis relative, so the binary is not found fromthere. CI never saw it because CTest injects an absolute path. Red-tested:
relative → exit 127, absolute → exit 0. The script now resolves
ICCDEV_TOOLS_DIR/ICCDEV_TESTING_DIRto absolute paths up front.build configuring a lower bound would fail on correct behaviour. It now fails
only if the fixture drops back under the former 64 MiB cap (at which point
it has stopped exercising the threshold) and skips when the configured
bound is below the fixture, where refusal is right.
exit 1at top level).Fixture construction moved inside the test: one
[FAIL], not a suite abort.reports the compiled-in value.
Testing
iccdev.json-parser-regressions, red/green proven:[FAIL] iccFromJson rejected 97444884 bytes ... (exit=1)[PASS] issue-1856-indent4 (97444884 byte -indent=4 document round-tripped)-DICC_JSON_MAX_FILE_MB=64the readerrefuses and prints
exceeds 64 MiB limit; at the default it accepts.Configure-time validation rejects
0,abc,07,2048; accepts1024.ctest: 149/150. The one failure,iccdev.spectral-tiff-preview, is a localenvironment gap (
ModuleNotFoundError: No module named 'imagecodecs') and isunrelated to this change.
released immediately; the whole script runs 28.8s against its 300s budget.
grep -cE 'warning:'= 0): strict clangASan+UBSan Debug; GCC 15.2.0 in the pinned CI regression container with
strict warnings ENABLED.
shellcheck0.9.0 (CI's pin) clean.Scope
This is the JSON half only.
IccProfileXml.cpp:1038records a deliberatedecision to drop
XML_PARSE_HUGE, because it disables libxml2's depth,name-length, text-length and entity-expansion guards. Restoring it is a security
trade-off, and the issue still lists that item as pending confirmation from
@ChrisCoxArt — so it is left untouched here and the issue should stay open for
it.