[zsltg]

Summary

Fixes #75.

The decoder read a length or a count from the file and allocated a buffer of that size before it read the data. A malformed file of 22 bytes could declare a 2^40-byte string. The Go runtime then stopped the process with fatal error: out of memory, and Decoder.Parse cannot recover from that error. With this change, memory follows the data that actually arrives, and a malformed file returns an error.

Changes

• readString and readLZF read through a new helper, readBytes. A length up to 1 MiB is allocated at once, the same as before. For a longer length, the buffer starts at 1 MiB and doubles only as data arrives. A truncated value returns the same io.EOF or io.ErrUnexpectedEOF as before.
• lzf.Decompress rejects an output length that is more than 88 times the input length. One back reference of 3 bytes produces at most 264 bytes, and a literal run produces fewer bytes than it uses, so valid data never exceeds that ratio.
• Counts from a buffer that is already in memory are checked against the size of that buffer. This applies to the intset cardinality and to the stream listpack counts. The intset reader also rejects a buffer shorter than its 8-byte header.
• Counts read from the file only set the initial capacity, at most 1024, through capHint. append grows the slice, and the loops still run for the real count. This applies to the list, set, zset, stream consumer group, and Valkey slot import readers. The slot import branch now uses append instead of indexed writes.

Tests

• TestParseDeclaredLengthExceedsData builds small malformed files for the string, LZF, list, set, zset, and intset types. Each file must return an error from Parse.
• TestParseValkeySlotImportHugeRanges and TestReadStreamEntryContentHugeCount cover the slot import branch and the stream counts.
• TestReadBytesAboveChunk covers a value longer than 1 MiB: a complete value, a value that is cut short, and a value that ends exactly at the 1 MiB boundary.
• TestDecompressRejectsImpossibleOutLen and TestDecompressEmptyInput cover the LZF bound and the empty input.
• TestCheckCount and TestCapHint cover the two helpers.

Before the fix, the string and LZF tests stopped with fatal error: out of memory when run under ulimit -v 8000000. After the fix, all new tests pass.

Validation

• The CI command go test -coverpkg=./core,./ ./... passes on Go 1.18.10 and Go 1.27.
• A parse of every file in cases/ makes the same number of allocations before and after the change (70,749). The time difference is within measurement noise.
• GOARCH=386 go test ./lzf passes.

TestToJsonWithRegex and TestFlameGraph in helper can fail now and then. The cause is in the tests, not in this change. ToJsons writes lines from several workers in the order that they finish, but the test expects a fixed order. d3flame.Web returns before the server listens, and the test sends a request at once. This change does not touch either test. If one of them fails in CI, run the job again.

---

Generated by Claude Code