Skip to content

fix(#1856): bound profile and config JSON at one configurable limit - #1981

Merged
xsscx merged 2 commits into
masterfrom
issue-1856-json-file-cap
Aug 5, 2026
Merged

fix(#1856): bound profile and config JSON at one configurable limit#1981
xsscx merged 2 commits into
masterfrom
issue-1856-json-file-cap

Conversation

@colourbill-ctrl

Copy link
Copy Markdown
Contributor

Part of #1856 — the JSON half. The XML_PARSE_HUGE half is deliberately not
addressed here; see Scope at the end.

Developed from the QA branch ci-qa-issue-1856 (@xsscx, f81ff047), which is
where the CMake-knob approach comes from. The code here is written fresh, and
the review of that branch turned up four things worth correcting before this
lands — those are listed below rather than folded in silently.

The defect

The two JSON readers carried different fixed caps, and the smaller one sat
inside the range iccDEV's own tools produce.

CIccProfileJson::LoadJson refused any document over 64 MiB, on the stated
assumption that "real JSON dumps of even big iccMAX spectral profiles top out
around 5 MB"
. The tracked corpus does not support that estimate:

source .icc iccToJson output
Testing/hybrid/CMYK-W_Overprint_Profile.xml 3,721,864 58,159,328 (-indent=2, the default) under the old cap
same 3,721,864 97,444,884 (-indent=4, documented) refused by the old cap

So iccToJson could write, at a documented setting, a file iccFromJson then
refused. The refusal itself was clean — specific diagnostic, exit 1, no output
file — so this is a threshold defect, not a silent-failure one.

Separately, loadJsonFrom in IccJsonUtil.cpp, which reads the -cfg
documents, applied an unrelated 100 MiB to the same kind of input.

The fix

Both readers now share one bound, ICC_JSON_MAX_FILE_BYTES, derived at
configure time from a new ICC_JSON_MAX_FILE_MB cache variable.

The default is 128 MiB, which clears every tracked fixture at the documented
-indent=4 with headroom through -indent=5 — the range the defect was about.

-indent emitted JSON within 128 MiB
2 (default) 58,159,328 (55.5 MiB) yes
4 97,444,884 (92.9 MiB) yes
5 117,087,662 (111.7 MiB) yes
6 136,730,440 (130.4 MiB) no
20 (max) 411,729,332 (392.7 MiB) no

Why not 256 MiB, to match the XML side

This issue asks to align the XML and JSON limits, and icXmlMaxTextFileBytes
(IccXML/IccLibXML/IccUtilXml.h) is 256 MiB — so matching it is the obvious
move. I measured before taking it, and the measurement argues against.

A byte count is a weak proxy for the memory this cap exists to bound.
Indentation inflates the file without inflating the parsed document: peak RSS
was flat at 79 MB for the same profile emitted at -indent=0, 1 and 2,
spanning 18.9 MB to 58.2 MB of input. What actually drives memory is document
shape:

input peak RSS per input byte
profile at -indent=2 (58.2 MB) 79 MB 1.4x
same profile at -indent=0 (18.9 MB) 79 MB 4.3x
dense object of small pairs (10.9 MB) 109 MiB 10.5x
dense array of small values (20.0 MB) 312 MiB 16.3x

So 128 MiB already admits roughly 2 GiB of peak resident memory on hostile
input, and 256 MiB would admit about 4 GiB — more than a CI runner has. The
alignment argument is about consistency, not about memory, and I did not want it
to read as though the larger cap were the safer one.

Two follow-ups this suggests, not done here:

  • A node or depth budget (nlohmann supports one via parser_callback_t) would
    bound memory directly instead of by proxy. That is the control this issue
    probably wants in the long run.
  • The two paths are not symmetric at equal byte counts: loadJsonFrom holds
    std::string buf(flen) and the DOM simultaneously, where LoadJson
    streams via f >> root and never holds the raw text.

A third 64 MiB cap, left alone deliberately

IccJSON/IccLibJSON/IccTagJson.cpp:2248 (kMaxClutFileBytes) also refuses at
64 MiB. It is not included in this change, and that is a judgement call
worth stating rather than leaving to look like an oversight: it bounds an
external CLUT data file referenced from a JSON document, not the JSON document
itself, so it is a different resource with a different threat model and no
reason to move in lockstep. Its sibling kMaxSparseMatrixArrayBytes
(IccTagBasic.cpp:5599) is 64 MiB for the same class of reason. Happy to bring
either in scope if you would rather all four moved together.

This also does not remove the emit/read asymmetry entirely, and
docs/iccjson.md says so rather than implying it is solved.

Also: the value is rejected at configure time if it is not a positive integer or
exceeds 1024 MiB; each translation unit keeps a finite fallback so non-CMake
builds still compile with a cap; and the diagnostic now reports the limit that
was actually compiled in (JSON file exceeds 256 MiB limit) instead of a
hard-coded number.

Findings from the QA branch, corrected here

  1. The new test hard-failed on a bare developer run. It runs iccFromXml
    inside cd Testing/hybrid, but the script's own documented default
    ICCDEV_TOOLS_DIR=Build/Tools is relative, so the binary is not found from
    there. CI never saw it because CTest injects an absolute path. Red-tested:
    relative → exit 127, absolute → exit 0. The script now resolves
    ICCDEV_TOOLS_DIR/ICCDEV_TESTING_DIR to absolute paths up front.
  2. The test asserted a hard-coded 64 MiB while the cap is configurable, so a
    build configuring a lower bound would fail on correct behaviour. It now fails
    only if the fixture drops back under the former 64 MiB cap (at which point
    it has stopped exercising the threshold) and skips when the configured
    bound is below the fixture, where refusal is right.
  3. A missing hybrid fixture aborted the whole suite (exit 1 at top level).
    Fixture construction moved inside the test: one [FAIL], not a suite abort.
  4. The diagnostic lost its number ("exceeds configured size limit"). It now
    reports the compiled-in value.

Testing

  • New case in iccdev.json-parser-regressions, red/green proven:
    • red (unpatched): [FAIL] iccFromJson rejected 97444884 bytes ... (exit=1)
    • green: [PASS] issue-1856-indent4 (97444884 byte -indent=4 document round-tripped)
    • skip path exercised by configuring a bound below the fixture
  • Knob proven end-to-end: rebuilt at -DICC_JSON_MAX_FILE_MB=64 the reader
    refuses and prints exceeds 64 MiB limit; at the default it accepts.
    Configure-time validation rejects 0, abc, 07, 2048; accepts 1024.
  • ctest: 149/150. The one failure, iccdev.spectral-tiff-preview, is a local
    environment gap (ModuleNotFoundError: No module named 'imagecodecs') and is
    unrelated to this change.
  • Cost: the new case adds ~19s and ~97 MiB of transient disk under ASan, both
    released immediately; the whole script runs 28.8s against its 300s budget.
  • Builds, CI's own zero-warning gate (grep -cE 'warning:' = 0): strict clang
    ASan+UBSan Debug; GCC 15.2.0 in the pinned CI regression container with
    strict warnings ENABLED. shellcheck 0.9.0 (CI's pin) clean.

Scope

This is the JSON half only. IccProfileXml.cpp:1038 records a deliberate
decision to drop XML_PARSE_HUGE, because it disables libxml2's depth,
name-length, text-length and entity-expansion guards. Restoring it is a security
trade-off, and the issue still lists that item as pending confirmation from
@ChrisCoxArt — so it is left untouched here and the issue should stay open for
it.

The two JSON readers carried different fixed caps, and the smaller one sat
inside the range iccDEV's own tools produce.

CIccProfileJson::LoadJson refused any document over 64 MiB, on the stated
assumption that "real JSON dumps of even big iccMAX spectral profiles top out
around 5 MB". The tracked corpus does not support that estimate. The profile
built from Testing/hybrid/CMYK-W_Overprint_Profile.xml emits 58,159,328 bytes
at the default -indent=2 and 97,444,884 bytes at -indent=4, a setting iccToJson
documents and accepts. So iccToJson could write, at a documented setting, a file
iccFromJson then refused. The refusal itself was clean - specific diagnostic,
exit 1, no output file - so this was a threshold defect, not a silent failure.
loadJsonFrom in IccJsonUtil.cpp, which reads the -cfg documents, applied an
unrelated 100 MiB to the same kind of input.

Both readers now share one bound, ICC_JSON_MAX_FILE_BYTES, derived at configure
time from a new ICC_JSON_MAX_FILE_MB cache variable. It is rejected if it is not
a positive integer or exceeds 1024 MiB, since a bound large enough to be no
bound defeats the purpose. Each translation unit keeps a finite fallback so
builds that do not go through Build/Cmake still compile with a cap, and the
diagnostic now reports the limit that was actually compiled in rather than a
hard-coded number.

The default is 128 MiB. That clears every tracked fixture at the documented
-indent=4 (97,444,884 bytes) with headroom through -indent=5, which is the range
the defect was actually about. It does not remove the asymmetry entirely, since
-indent accepts up to 20 and emits 411,729,332 bytes on the same profile.

It is deliberately not raised to icXmlMaxTextFileBytes' 256 MiB, even though
this issue asks to align the XML and JSON limits, because a byte count is a weak
proxy for the memory the cap is meant to bound. Indentation inflates the file
without inflating the parsed document: peak RSS was flat at 79 MB across the
same profile emitted at -indent=0, 1 and 2, spanning 18.9 MB to 58.2 MB of
input. Cost per admitted byte is governed by document shape instead, measured
from 1.4x on an indented profile to 16.3x on a dense array of small values, so
128 MiB already admits roughly 2 GiB of peak resident memory and 256 MiB would
admit twice that. docs/iccjson.md records both tables so the bound can be
revisited against data rather than an estimate.

The case added to iccdev.json-parser-regressions builds the fixture from its
tracked XML, converts at -indent=4 and requires the round trip to succeed. It
fails if the emitted document ever falls back under the former 64 MiB cap,
because at that point it would no longer be exercising the threshold, and skips
rather than fails when a build configures a bound below the fixture size, where
refusal is the correct behaviour. The script now also resolves ICCDEV_TOOLS_DIR
and ICCDEV_TESTING_DIR to absolute paths: this test runs iccFromXml with
Testing/hybrid as its working directory, and the script's own documented
relative defaults would not resolve the binary from there.

Scope: this is the JSON half of #1856 only. The XML_PARSE_HUGE half is
unchanged - IccProfileXml.cpp:1038 records a deliberate decision to drop that
flag because it disables libxml2's depth, name-length, text-length and
entity-expansion guards, so restoring it is a security trade-off for a
maintainer to rule on rather than something to settle here.
@github-actions github-actions Bot added Documentation Documentation-only or documentation-related change Testing CTest, regression, or test coverage Source C or C++ source code changes Scripts Shell, PowerShell, or repository automation scripts Configuration Repository, CMake, YAML, JSON, or tool configuration Build Build system, CMake, compiler, or packaging JSON ICC JSON parser, config, or round-trip behavior Governance Maintainer-owned policy, CODEOWNERS, prompts, or skills Unix Linux, macOS, Bash, or POSIX shell scope labels Aug 5, 2026
@xsscx
xsscx enabled auto-merge (squash) August 5, 2026 14:19
@github-actions github-actions Bot added the pending CI checks still running label Aug 5, 2026
@xsscx

xsscx commented Aug 5, 2026

Copy link
Copy Markdown
Member

CI Status

2026-08-05 14:29:09 UTC

Review in Process: Error: Value cannot be null. (Parameter 'ContainerId')
https://github.com/InternationalColorConsortium/iccDEV/actions/runs/31014523300

@xsscx xsscx self-assigned this Aug 5, 2026
@xsscx xsscx added Resolving Merge Conflict Maintainer indicates in process Resolution of Merge Conflict. HOLD CHANGES Hold for Maintainer Hold for Maintainer Action - Monitoring and removed Documentation Documentation-only or documentation-related change Testing CTest, regression, or test coverage Source C or C++ source code changes Scripts Shell, PowerShell, or repository automation scripts Configuration Repository, CMake, YAML, JSON, or tool configuration Build Build system, CMake, compiler, or packaging pending CI checks still running JSON ICC JSON parser, config, or round-trip behavior Governance Maintainer-owned policy, CODEOWNERS, prompts, or skills Unix Linux, macOS, Bash, or POSIX shell scope labels Aug 5, 2026
@github-actions github-actions Bot added Documentation Documentation-only or documentation-related change Testing CTest, regression, or test coverage Source C or C++ source code changes Scripts Shell, PowerShell, or repository automation scripts Configuration Repository, CMake, YAML, JSON, or tool configuration Build Build system, CMake, compiler, or packaging ci Continuous integration workflow changes github-actions GitHub Actions workflow or action configuration JSON ICC JSON parser, config, or round-trip behavior Governance Maintainer-owned policy, CODEOWNERS, prompts, or skills Unix Linux, macOS, Bash, or POSIX shell scope labels Aug 5, 2026
@xsscx

xsscx commented Aug 5, 2026

Copy link
Copy Markdown
Member

PR Status

2026-08-05 14:33:59 UTC

  • Locally tested the Docker & ci-pr-action fix for the QA Tag
  • Monitoring CI Status

@xsscx xsscx removed Resolving Merge Conflict Maintainer indicates in process Resolution of Merge Conflict. HOLD CHANGES Hold for Maintainer Hold for Maintainer Action - Monitoring labels Aug 5, 2026
@xsscx xsscx added this to the v2.3.2.3 milestone Aug 5, 2026
@github-actions github-actions Bot added the failed One or more CI checks failed label Aug 5, 2026
@xsscx
xsscx merged commit 11fe435 into master Aug 5, 2026
42 of 46 checks passed
@xsscx
xsscx deleted the issue-1856-json-file-cap branch August 5, 2026 15:00
@xsscx

xsscx commented Aug 6, 2026

Copy link
Copy Markdown
Member

Two follow-ups this suggests, not done here:

A node or depth budget (nlohmann supports one via parser_callback_t) would
bound memory directly instead of by proxy. That is the control this issue
probably wants in the long run.
The two paths are not symmetric at equal byte counts: loadJsonFrom holds
std::string buf(flen) and the DOM simultaneously, where LoadJson
streams via f >> root and never holds the raw text.
A third 64 MiB cap, left alone deliberately
IccJSON/IccLibJSON/IccTagJson.cpp:2248 (kMaxClutFileBytes) also refuses at
64 MiB. It is not included in this change, and that is a judgement call
worth stating rather than leaving to look like an oversight: it bounds an
external CLUT data file referenced from a JSON document, not the JSON document
itself, so it is a different resource with a different threat model and no
reason to move in lockstep. Its sibling kMaxSparseMatrixArrayBytes
(IccTagBasic.cpp:5599) is 64 MiB for the same class of reason. Happy to bring
either in scope if you would rather all four moved together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Build Build system, CMake, compiler, or packaging ci Continuous integration workflow changes Configuration Repository, CMake, YAML, JSON, or tool configuration Documentation Documentation-only or documentation-related change failed One or more CI checks failed github-actions GitHub Actions workflow or action configuration Governance Maintainer-owned policy, CODEOWNERS, prompts, or skills JSON ICC JSON parser, config, or round-trip behavior Scripts Shell, PowerShell, or repository automation scripts Source C or C++ source code changes Testing CTest, regression, or test coverage Unix Linux, macOS, Bash, or POSIX shell scope

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants