TextString: what the document says in a human language

[tannevaled]

pdfops info printed a document's creator as

creator    \xfe\xffL\x00a\x00T\x00e\x00X\x00 \x00v\x00i\x00a\x00 \x00p\x00a\x00n\x00d\x00o\x00c

and had done for as long as the verb existed. ToString hands back the
bytes, and the bytes of a text string are not text: a title, an author, a
bookmark's label and a form field's value are PDFDocEncoding, or UTF-16BE
behind FE FF, or — since PDF 2.0 — UTF-8 behind EF BB BF.

s, ok := reader.TextString(info["Creator"])   // "LaTeX via pandoc"

⛔ Only those two marks, and no guessing beyond them

A string that merely looks like UTF-16LE is PDFDocEncoding that happens to
contain a zero byte, and decoding it as UTF-16 turns a readable title into
mojibake. That is the opposite failure to the one above, and just as silent.

⛔ PDFDocEncoding is not Latin 1

Which is the other half of the defect, and the half that a decoder written from
memory gets wrong:

0x18–0x1F	the spacing accents ˘ˇˆ˙˝˛˚˜, where Latin 1 has control characters
0x80–0x9E	the typographic marks, where Latin 1 has nothing at all
0xA0	the Euro, where Latin 1 has a non-breaking space
0x7F, 0x9F, 0xAD	undefined — said to mean nothing rather than quietly becoming a soft hyphen

The table was checked against Annex D.2 of ISO 32000-1 entry by entry rather
than remembered.

How it is checked

100% of statements, no uncovered blocks, 12 mutations caught. One of them
needed the test to assert the whole string: accepting FF FE as a mark and
then decoding big-endian anyway gives mojibake too, so "it does not say La"
could not tell the two apart.

Surrogate pairs, an odd trailing byte (half a code unit is not a character, and
a truncated string should look truncated), a bare BOM, and invalid UTF-8 behind
a UTF-8 mark all have tests of their own.

🤖 Generated with Claude Code