Repository navigation
Avoid splitting a surrogate pair when truncating to a byte budget - #1044
Open
xianshijing-lk wants to merge 2 commits into
Open
xianshijing-lk wants to merge 2 commits into
xianshijing-lk wants to merge 2 commits into
Conversation
truncateBytes binary searches for the longest prefix fitting a byte budget, counting in UTF-16 code units. When the budget fell between the two halves of a surrogate pair it cut the pair in half, returning a string ending in an unpaired high surrogate. That encodes to UTF-8 as a replacement character, so an RPC error message or data payload trimmed at such a boundary reached the remote peer corrupted. Step the cut point back one code unit when it lands on a high surrogate. Port of livekit/client-sdk-js#2126. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
xianshijing-lk
requested review from
MaxHeimbrock and
davidliu
as code owners
October 23, 2026 22:17
🦋 Changeset detectedLatest commit: 9d5bdab The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Diffuse output: AARJAR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port of livekit/client-sdk-js#2126.
The bug
String.truncateBytesbinary searches for the longest prefix that fits a byte budget, but it counts in UTF-16 code units and cuts withsubstring. When the budget falls between the two halves of a surrogate pair, it splits the pair and returns a string ending in an unpaired high surrogate.That is not representable in UTF-8, so encoding it emits a replacement character — meaning an
RpcErrormessage (capped atMAX_MESSAGE_BYTES, 256) or data payload trimmed at such a boundary reaches the remote peer corrupted and no longer round trips.Note the binary search actively steers into this case: an unpaired surrogate encodes to a single
?byte, so a split prefix measures shorter than the intact one and looks like a better fit.Repro
Added
StringByteUtilsTest. Against the unfixed code:"a😀".truncateBytes(3)returned"a\uD83D"instead of"a".The fix
Step the cut point back one code unit when it lands on a high surrogate, matching the upstream JS change. The new test asserts the result stays within budget and round trips losslessly through UTF-8.
Verification
:livekit-android-test:testReleaseUnitTest— the two new assertions fail before the change and pass after; the full module suite is green.Not needed elsewhere
Swift's
truncate(maxBytes:)usesprefix/count, which operate on graphemeCharacters, and Rust'struncate_bytesiteratescharscalar values — neither can split a pair. The Flutter SDK has the same class of bug and is fixed separately.🤖 Generated with Claude Code