Spread CD sectors in place, and decode LZMA into the caller's buffer

[rtissera]

Two memory savings for targets where RAM is the binding constraint, behind a config knob that leaves everything else on the code path it has today.

What changes

Each CD codec kept a hunk-sized scratch (19584 B), decoded the sector data into it, then copied every byte across to the caller's hunk. With CHDR_CD_SCRATCH_BUFFER=0 the codec decodes into the caller's hunk and spreads the sectors to the 2448-byte frame stride in place, needing only a frame of bounce plus subcode staging — 3120 B.

Separately, LzmaDec's dictionary now is the destination buffer, which is what the SDK's own one-call LzmaDecode() does. That removes a second hunk-sized allocation and a copy per hunk. It has no trade to make, so it is unconditional.

Numbers

Per-codec peak heap, rv32imafc under qemu with LOWRAM_TARGET=ON — the figure tests/rv32/check_budget.py already measures in CI:

codec	before	after
cd_cdlz	99014 B	57906 B	−41.5%
cd_cdzl	66249 B	49785 B	−24.9%
cd_cdzs	145257 B	128793 B	−11.3%
hd_lzma	32022 B	27858 B	−13.0%

On a microcontroller reading from an SD card, every hunk of 14 images (7.7 GB, 128 minutes): 1.002× on time, peak heap down 33–74 KB per CD image, and the smallest largest-free-block roughly doubled on each — on one image 16 KB to 78 KB, which is the difference between a later allocation of that size failing and succeeding.

Defaults

CHDR_CD_SCRATCH_BUFFER follows LOWRAM_TARGET. A build that is not memory constrained keeps the previous CD behaviour and the previous speed exactly; only the LZMA change applies there.

Set it explicitly to pin it either way. 1 is also what a caller wants whose hunk buffer is uncached, write-combining or a FIFO, since the buffer is then written once, linearly, and never read back.

Where the in-place path is on it costs a few percent of decode CPU and roughly halves what an open CD image needs.

Two notes for reviewers

The spread bounces each sector through the frame-sized scratch rather than using memmove. The moves overlap by 96 bytes, and memmove on overlapping regions degrades to a byte-at-a-time loop in some C libraries — measured at up to 1.19× a whole hunk's decode CPU. Two non-overlapping memcpys are far cheaper there and no worse elsewhere.

chd_read's buffer must be ordinary readable RAM, and 2-byte aligned. With the scratch off a CD hunk is decoded into that buffer and spread in place, so it is read back during the call. LZMA reads it back in every build, scratch or not, since its dictionary now is that buffer. And the FLAC codecs write int16_t into it directly — which the plain FLAC codec has always done, so the alignment requirement is older than this branch. In the CD case an odd pointer returns CHDERR_INVALID_PARAMETER rather than trapping on a target without unaligned stores. All of this is documented on chd_read; CHDR_CD_SCRATCH_BUFFER=1 lifts only the CD read-back.

Checked against the projects that embed libchdr

RetroArch/libretro-common, PCSX2, DuckStation and Flycast were read to see what they actually do, and then a harness reproducing each one's call sequence was run against both sides.

None of them sets any of this project's CMake options, so LOWRAM_TARGET is 0 and the in-place path is unreachable for all four; only the LZMA change applies. All four pass a plain heap buffer of exactly hunkbytes, at least 2-byte aligned — RetroArch already casts it to uint16_t* to byte-swap in place, so it relies on that guarantee today. The harness covers chd_open, chd_open_file, chd_open_core_file with caller-supplied callbacks, chd_precache, parent chains, resident and per-call buffers, and sector-granular reads, with guard bytes around every buffer.

Against origin/master, no flags on either side:

• 4253 CHDs on hand, 400 hunks sampled each: byte-identical, 0 diffs
• the four consumer shapes over 600 of those, and over the seeds plus a parent chain: 0 diffs
• 1020 mutants (byte flips, truncations) of every seed, comparing the full per-hunk chd_error map: identical, so a corrupt file is still rejected in exactly the same place
• ASan + UBSan over 220 real images and over 255 seed × access-pattern × alignment combinations: 0 behaviour differences, identical sanitizer output
• nm -D on the shared library: same 18 exported symbols, same undefined set

Two things that came out of it, in the last commit. #if CHDR_CD_SCRATCH_BUFFER on an undefined macro is 0, the in-place path — harmless in this tree, where every file reaches chdconfig.h, but a project that keeps its own chdconfig.h and syncs sources into it would switch decode paths silently, or size one codec's scratch for a path the decoder is not on. That is now an #error, verified against a downstream-shaped chdconfig.h. And chd_read's comment was corrected as above.

The only configuration where the two scratch settings disagree at all is WANT_SUBCODE=0, and the difference is entirely inside the 96-byte subcode gap that neither setting writes — sector data is identical. Noted in chdconfig.h.

Test gaps closed on the way

The test targets compiled against chdconfig.h's defaults rather than the values the library was built with, so kat.c failed to link with WANT_RAW_DATA_SECTOR=OFF. They now get CHDR_DEFINES directly; chdr-static keeps them PRIVATE so unnamespaced macros are not pushed onto consumers.

And no image in the corpus had a parent, so COMPRESSION_PARENT was decoded by no test at all — which matters here, because a parent-referenced hunk reaches the caller's buffer by a different route than an in-place spread. generate.sh now builds a parent with two children (one identical, so every hunk is a parent reference; one perturbed, so references and real compressed hunks share a map) and tests/parent_decode.c checks both. They live outside seeds/, since a child cannot be opened standalone and the workflows that walk seeds/ do exactly that.

Verification

• 287 images byte-identical against master, in both CD modes and both shipping configurations
• known-answer, AVHuff and the new parent/delta tests
• big-endian powerpc and s390x, including the in-place path
• ASan/UBSan across 80 images in both configurations; 3.8M libFuzzer executions
• every combination of LOWRAM_TARGET, WANT_SUBCODE and WANT_RAW_DATA_SECTOR builds and passes
• rv32 RAM-budget job run locally

---

Also in this PR: CI consolidation

Folded in rather than split out, to stop the waste immediately.

Every job was running twice. All 14 workflows were on: [push, pull_request], so for a branch in this repo the push and the pull request each triggered the full set on the same commit — 76 checks where there are 38 distinct ones. push is now restricted to master; pull_request keeps full coverage.

Superseded runs now cancel. A concurrency group per workflow, excluding master so a merge is never cancelled by the next one. Previously a force-push left the old commit's entire matrix running to completion.

esp32p4-ram-budget removed. Its own header described it as byte-identical to rv32-ram-budget — same toolchain file, same RV32IMAFC/ilp32f ABI, same qemu machine, same probe, same expected numbers — existing "for CI visibility". tests/esp32p4/'s three files went with it; they differed from tests/rv32/'s only in comments. esp32p4-build stays: it compiles the real integration surface and duplicates nothing.

Path filters per workflow. A quarter of the last forty commits on master touched only documentation, contrib or markdown, and each ran the whole matrix — twelve BSD/Haiku VMs included — to prove a README edit does not break DragonFlyBSD. Scoped to what each workflow actually builds:

change	workflows triggered
docs / markdown only	0 of 13
library source	13 of 13 — unchanged
one contrib/ target	6 of 13

Nothing that could affect Android, the BSDs, MSYS2, Switch or Vita skips them. master has no required status checks, so a skipped workflow cannot wedge a merge.

Net: 76 checks → 37 on a code change, → 0 on a docs change.