fix: read HPACK string literals with readNBytes - #1231
Open
pjfanning wants to merge 1 commit into
Open
Conversation
Motivation: Decoder.readStringLiteral asked for a whole string literal with a single `in.read(buf)` and treated any shorter result as a decompression failure. InputStream.read(byte[]) is explicitly allowed to return fewer bytes than requested even when more are available, so this relies on a contract the API does not offer. It is not an observable bug today: the only caller wraps a compacted ByteString (HeaderDecompression compacts because the decoder needs mark/reset and meaningful available()), and the decoder waits for `in.available() >= length` before reading, so the single read always fills the buffer. It is a trap for any future change to how that stream is produced. Modification: Use InputStream.readNBytes, which loops until the requested number of bytes has been read or the stream ends, and compare the returned length. Available since JDK 11 and this branch requires JDK 17. Result: The decoder no longer depends on a single read filling the buffer. Behaviour is unchanged for a stream that does fill it, and truncated input is still reported as a decompression failure. Tests: - New HpackDecoderSpec round-trips header blocks through the shaded Encoder and Decoder over a stream that has all its data available but returns 1, then 7, bytes per read. Both cases fail before this change with a decompression failure and pass after it - sbt "http-core / Test / testOnly org.apache.pekko.http.impl.engine.http2.hpack.HpackDecoderSpec" - 3 passed - sbt "http2-tests / Test / testOnly org.apache.pekko.http.impl.engine.http2.RequestParsingSpec" - 25 passed, 3 pending - sbt http-core/javafmtCheck and scalafmt - clean References: None - found while reviewing the code base against the JDK 17 baseline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Decoder.readStringLiteralasks for a whole HPACK string literal with a singlereadand treats anything shorter as a decompression failure:InputStream.read(byte[])is explicitly allowed to return fewer bytes than requested even when more are available, so this depends on a guarantee the API does not give.This is not an observable bug today, and I want to be clear about that rather than oversell it. The only caller wraps a compacted
ByteString-HeaderDecompression.scala:99doespayload.compact.asInputStream, because the decoder needsmark/resetand a meaningfulavailable()- and the decoder waits forin.available() >= lengthbefore callingreadStringLiteral, so the single read always fills the buffer. What this change removes is a trap for anyone who later changes how that stream is produced.Modification
readNBytesloops until the requested count is read or the stream ends. Available since JDK 11; this branch requires JDK 17.Truncated input is still reported as a decompression failure, and a zero-length literal still works (
readNBytes(0)returns an empty array, matching the oldreadof an empty buffer).This file is already adapted from github.com/twitter/hpack rather than kept byte-identical to upstream - it imports
org.apache.pekko.http.impl.util.StringTools- so a local change here is in keeping with how it is maintained.Result
The decoder no longer depends on a single read filling the buffer, with no behaviour change for a stream that does.
Note on a tempting follow-up
Fixing this does not on its own make it safe to drop the
.compactinHeaderDecompressionand save a copy per header block. The decoder also relies onin.available()reporting the full remaining length (available() < nameLengthguards atDecoder.java:280and:373) and onmark/reset, neither of which a rope-backedByteStringInputStream provides. Worth a separate look, not a freebie from this change.Tests
HpackDecoderSpecround-trips header blocks through the shadedEncoderandDecoderover a stream that has all of its data available but hands it out 1 byte, and then 7 bytes, perread. Both cases fail before this change withjava.io.IOException: decompression failure, and pass after it.sbt "http-core / Test / testOnly org.apache.pekko.http.impl.engine.http2.hpack.HpackDecoderSpec"- 3 passed.sbt "http2-tests / Test / testOnly org.apache.pekko.http.impl.engine.http2.RequestParsingSpec"- 25 passed, 3 pending.sbt http-core/javafmtCheckandscalafmt- clean. Note the Java formatter was run on JDK 21 rather than the JDK 17 the contributing guide suggests, since that is what was available here.References
None - found while reviewing the code base against the JDK 17 baseline