Add 'pread'-backed method for thread-safe hunk reading - #162
Conversation
9a1137d to
160f6af
Compare
dd4c6c4 to
4159fe7
Compare
4159fe7 to
234ce82
Compare
…WRAM cdzs_codec_init() built two ZSTD_DCtx, one per stream. A DCtx is ~94KB - the single largest allocation left in libchdr - so a cdzs-coded CHD spent 187KB on decompression contexts alone, most of a 320KB part's memory before anything else. Sharing one is safe. The two streams are decoded strictly in sequence by cd_codec_decompress(): base to completion, error returns early, then subcode, into disjoint halves of the same buffer, never nested and never concurrently. zstd_codec_decompress() calls ZSTD_initDStream() on entry, so nothing carries between them. And zstd_codec_init() ignores its size argument, so the two contexts were identical objects to begin with. It is not free, which is why it is conditional. Alternating two differently-shaped streams through one context rebuilds its working set each way: measured on x86-64 over a whole file, +3.9%. 94KB for 3.9% is a good trade on a memory-constrained part and a bad one on a desktop, so it is made only under LOWRAM_TARGET - the switch that exists to make exactly this choice. A default build keeps two contexts and its previous speed. Ikaruga (92.9% cdzs), peak heap under LOWRAM_TARGET=1: 251.1 -> 157.4KB. The same sharing was tried for cdzl and reverted. Once the unused 32KB miniz dictionary is gone an inflate context is only ~8KB, and it measured +2.6% for that - not worth it. Also worth recording what did not work, since the reasoning looked sound: switching the zstd codec from the streaming API to one-shot ZSTD_decompressDCtx() saves nothing at all. ZSTD_DStream is a typedef for ZSTD_DCtx, so the memory is the context itself and not streaming staging buffers; measured identical peak and marginally slower. The equivalent change for miniz worked only because miniz allocates a genuinely separate and, in libchdr's usage, entirely unused 32KB dictionary. NOTE for future threading work (see PR #162): codec state is already shared across concurrent chd_read() calls, but this removes even the accidental separation between base and subcode. Per-thread codec state has to mean per-thread codec instances. Verified: 14/14 sample files decode identically to separate contexts under LOWRAM_TARGET=1, block CRCs verified throughout; both LOWRAM_TARGET=1 and =0 build and run; cdzl speed confirmed back at its baseline after the revert. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KMYbZzB8mioFmotWGFnAXG
|
@xakep666 Sorry this sat for months without a review. The goal is right and I want it. "Leave concurrency to the caller" is the correct shape for this library, and driving it from Go without pthread is a constraint worth designing for — libchdr should not grow a threading dependency. Two things stop me merging it as it stands, and I would rather ask than decide alone. The state sharing is wider than the two things this covers. Besides the compressed buffer and the file position, a read on current master also mutates the codec instances themselves (zlib and LZMA inflate state, cdfl's FLAC decoder and its buffer), the self-reference cache, the read-ahead window, and under Did you hit this in Adding That part looks avoidable without giving anything up. Since the callbacks are reached through a pointer, the new entry point does not have to live in the struct: typedef struct chd_core_file_pread_callbacks {
size_t (*fpread)(void *buf, size_t len, uint64_t offset, void *argp);
} core_file_pread_callbacks;
CHD_EXPORT chd_error chd_set_pread_callbacks(chd_file *chd,
const core_file_pread_callbacks *cb);A new symbol, nothing existing grows, and your fallback stays as written: nothing registered, Questions I do not have an answer to:
The |
Addressing proposal in #34
This PR adds method
chd_read_threadsafe- a thread-safe version ofchd_read. This method will be helpful for implementing threaded read-aheads purely by consumers without adding external dependencies (likepthread) .My main programming language is Go and there's a popular approach "leave concurrency to a caller" used when concurrent code is written. And I think it fits good in this case.
Some implementation details:
chd_read_threadsafemethod has extra argumentcompressed_bufferwhich is used to store raw compressed data read from file. This is the one of two things required for thread-safety.chd_readuses same buffer stored inchd_filestructure.fpreadadded tochd_core_file_callbacksto allow thread-safe reading from file. Used only inchd_read_threadsafe.fpreadwas not providedchd_read_threadsafewill return errorCHDERR_NOT_SUPPORTED.hunk_read_into_memoryhas been tweaked a bit to accept arbitrary compressed data buffer pointer and different reading methods:seek_and_read(old one) orread_at(new one).