Skip to content

json: the two loads doors diverge on RAW lone surrogates in the document — untyped accepts (CPython-matching), typed refuses at the UTF-16→UTF-8 transcode #1597

Description

@antonsynd

Found while executing the #1487 close-out (plan-48fa86 verification). The #1487 fix covers unpaired surrogate escapes (\ud800 as six characters of escape text) — both doors now refuse those with one shared message. A raw lone-surrogate char (the actual UTF-16 code unit embedded in the document string) is a different cell:

  • untyped json.loads: accepts and preserves the char — matches CPython 3.12 (json.loads('"\ud800"') returns the 1-char string; measured 2026-08-20).
  • typed json.loads[T]: Err(JSONDecodeError: Cannot transcode invalid UTF-16 string to UTF-8 JSON text.) — System.Text.Json refuses at the input transcode, before any parsing.

Divergence pinned in JsonLoadsAgreementTests.RawLoneSurrogateChar_UntypedAcceptsWhileTypedRefuses (cites this issue). Note the agreement-corpus row form cannot carry this cell: a lone surrogate in an xUnit theory argument does not survive the test-case pipeline intact, so a corpus row silently measures a replaced character and passes vacuously — the named fact with an inline const is the working instrument.

Resolution direction (per the #1487 owner ruling's Axiom-1 logic): the CPython-matching acceptance is a deferred trap — the accepted lone-surrogate string explodes on its first UTF-8 encode. Aligning both doors on refusal (extending the pre-scan or the untyped parser to raw code units) would match the escapes ruling; aligning on acceptance is blocked by STJ's transcode. Owner call needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions