Skip to content

fix: read HPACK string literals with readNBytes - #1231

Open
pjfanning wants to merge 1 commit into
apache:mainfrom
pjfanning:fix/hpack-short-read
Open

fix: read HPACK string literals with readNBytes#1231
pjfanning wants to merge 1 commit into
apache:mainfrom
pjfanning:fix/hpack-short-read

Conversation

@pjfanning

Copy link
Copy Markdown
Member

Motivation

Decoder.readStringLiteral asks for a whole HPACK string literal with a single read and treats anything shorter as a decompression failure:

byte[] buf = new byte[length];
if (in.read(buf) != length) {
  throw DECOMPRESSION_EXCEPTION;
}

InputStream.read(byte[]) is explicitly allowed to return fewer bytes than requested even when more are available, so this depends on a guarantee the API does not give.

This is not an observable bug today, and I want to be clear about that rather than oversell it. The only caller wraps a compacted ByteString - HeaderDecompression.scala:99 does payload.compact.asInputStream, because the decoder needs mark/reset and a meaningful available() - and the decoder waits for in.available() >= length before calling readStringLiteral, so the single read always fills the buffer. What this change removes is a trap for anyone who later changes how that stream is produced.

Modification

byte[] buf = in.readNBytes(length);
if (buf.length != length) {
  throw DECOMPRESSION_EXCEPTION;
}

readNBytes loops until the requested count is read or the stream ends. Available since JDK 11; this branch requires JDK 17.

Truncated input is still reported as a decompression failure, and a zero-length literal still works (readNBytes(0) returns an empty array, matching the old read of an empty buffer).

This file is already adapted from github.com/twitter/hpack rather than kept byte-identical to upstream - it imports org.apache.pekko.http.impl.util.StringTools - so a local change here is in keeping with how it is maintained.

Result

The decoder no longer depends on a single read filling the buffer, with no behaviour change for a stream that does.

Note on a tempting follow-up

Fixing this does not on its own make it safe to drop the .compact in HeaderDecompression and save a copy per header block. The decoder also relies on in.available() reporting the full remaining length (available() < nameLength guards at Decoder.java:280 and :373) and on mark/reset, neither of which a rope-backed ByteString InputStream provides. Worth a separate look, not a freebie from this change.

Tests

  • New HpackDecoderSpec round-trips header blocks through the shaded Encoder and Decoder over a stream that has all of its data available but hands it out 1 byte, and then 7 bytes, per read. Both cases fail before this change with java.io.IOException: decompression failure, and pass after it.
  • sbt "http-core / Test / testOnly org.apache.pekko.http.impl.engine.http2.hpack.HpackDecoderSpec" - 3 passed.
  • sbt "http2-tests / Test / testOnly org.apache.pekko.http.impl.engine.http2.RequestParsingSpec" - 25 passed, 3 pending.
  • sbt http-core/javafmtCheck and scalafmt - clean. Note the Java formatter was run on JDK 21 rather than the JDK 17 the contributing guide suggests, since that is what was available here.

References

None - found while reviewing the code base against the JDK 17 baseline

Motivation:
Decoder.readStringLiteral asked for a whole string literal with a single
`in.read(buf)` and treated any shorter result as a decompression failure.
InputStream.read(byte[]) is explicitly allowed to return fewer bytes than
requested even when more are available, so this relies on a contract the API
does not offer.

It is not an observable bug today: the only caller wraps a compacted ByteString
(HeaderDecompression compacts because the decoder needs mark/reset and
meaningful available()), and the decoder waits for `in.available() >= length`
before reading, so the single read always fills the buffer. It is a trap for
any future change to how that stream is produced.

Modification:
Use InputStream.readNBytes, which loops until the requested number of bytes has
been read or the stream ends, and compare the returned length. Available since
JDK 11 and this branch requires JDK 17.

Result:
The decoder no longer depends on a single read filling the buffer. Behaviour is
unchanged for a stream that does fill it, and truncated input is still reported
as a decompression failure.

Tests:
- New HpackDecoderSpec round-trips header blocks through the shaded Encoder and
  Decoder over a stream that has all its data available but returns 1, then 7,
  bytes per read. Both cases fail before this change with a decompression
  failure and pass after it
- sbt "http-core / Test / testOnly org.apache.pekko.http.impl.engine.http2.hpack.HpackDecoderSpec" - 3 passed
- sbt "http2-tests / Test / testOnly org.apache.pekko.http.impl.engine.http2.RequestParsingSpec" - 25 passed, 3 pending
- sbt http-core/javafmtCheck and scalafmt - clean

References:
None - found while reviewing the code base against the JDK 17 baseline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant