Skip to content

Avoid splitting a surrogate pair when truncating to a byte budget - #1044

Open
xianshijing-lk wants to merge 2 commits into
mainfrom
port/js-2126-truncate-surrogate-pair
Open

xianshijing-lk wants to merge 2 commits into
mainfrom
port/js-2126-truncate-surrogate-pair

Conversation

@xianshijing-lk

Copy link
Copy Markdown
Contributor

Port of livekit/client-sdk-js#2126.

The bug

String.truncateBytes binary searches for the longest prefix that fits a byte budget, but it counts in UTF-16 code units and cuts with substring. When the budget falls between the two halves of a surrogate pair, it splits the pair and returns a string ending in an unpaired high surrogate.

That is not representable in UTF-8, so encoding it emits a replacement character — meaning an RpcError message (capped at MAX_MESSAGE_BYTES, 256) or data payload trimmed at such a boundary reaches the remote peer corrupted and no longer round trips.

Note the binary search actively steers into this case: an unpaired surrogate encodes to a single ? byte, so a split prefix measures shorter than the intact one and looks like a better fit.

Repro

Added StringByteUtilsTest. Against the unfixed code:

StringByteUtilsTest > does not split a surrogate pair when the budget falls mid-pair FAILED
    org.junit.ComparisonFailure: expected:<a[]> but was:<a[?]>
StringByteUtilsTest > never emits an unpaired surrogate FAILED
    java.lang.AssertionError: truncateBytes(2) ended on a high surrogate

"a😀".truncateBytes(3) returned "a\uD83D" instead of "a".

The fix

Step the cut point back one code unit when it lands on a high surrogate, matching the upstream JS change. The new test asserts the result stays within budget and round trips losslessly through UTF-8.

Verification

:livekit-android-test:testReleaseUnitTest — the two new assertions fail before the change and pass after; the full module suite is green.

Not needed elsewhere

Swift's truncate(maxBytes:) uses prefix/count, which operate on grapheme Characters, and Rust's truncate_bytes iterates char scalar values — neither can split a pair. The Flutter SDK has the same class of bug and is fixed separately.

🤖 Generated with Claude Code

truncateBytes binary searches for the longest prefix fitting a byte
budget, counting in UTF-16 code units. When the budget fell between the
two halves of a surrogate pair it cut the pair in half, returning a
string ending in an unpaired high surrogate. That encodes to UTF-8 as a
replacement character, so an RPC error message or data payload trimmed
at such a boundary reached the remote peer corrupted.

Step the cut point back one code unit when it lands on a high surrogate.

Port of livekit/client-sdk-js#2126.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 9d5bdab

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
client-sdk-android Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Diffuse output:

OLD: diffuse-source-file
NEW: livekit-android-sdk-release.aar

 AAR      │ old      │ new      │ diff  
──────────┼──────────┼──────────┼───────
      jar │    3 MiB │    3 MiB │ +83 B 
 manifest │  1.5 KiB │  1.5 KiB │   0 B 
 lint-jar │ 12.7 KiB │ 12.7 KiB │   0 B 
    other │  2.4 KiB │  2.4 KiB │   0 B 
──────────┼──────────┼──────────┼───────
    total │    3 MiB │    3 MiB │ +83 B 

 JAR     │ old   │ new   │ diff       
─────────┼───────┼───────┼────────────
 classes │  1707 │  1707 │  0 (+0 -0) 
 methods │ 21992 │ 21994 │ +2 (+2 -0) 
  fields │  5708 │  5708 │  0 (+0 -0)
AAR
 size  │ diff  │ path          
───────┼───────┼───────────────
 3 MiB │ +83 B │ ∆ classes.jar 
───────┼───────┼───────────────
 3 MiB │ +83 B │ (total)
JAR
METHODS:

   old   │ new   │ diff       
  ───────┼───────┼────────────
   21992 │ 21994 │ +2 (+2 -0) 
  
  + java.lang.Character isHighSurrogate(char) → boolean
  + java.lang.String charAt(int) → char

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant