fix: return heredoc bodies that match what Terraform evaluates (#326)

[livingstaccato]

Fixes #326.
Fixes #330.

What

preserve_heredocs=False returned heredoc bodies that differ from the values Terraform and OpenTofu evaluate the same source to. Three separate causes, all checked against OpenTofu v1.12.5 rather than read off the spec:

1. The newline before the closing marker was dropped. The spec ends the template where the delimiter "subsequently appears again on a line of its own", so every content line, the last one included, is terminated by its own newline: <<EOT\nline\nEOT is "line\n". The code stripped that newline along with the marker's indentation.
2. <<- measured its indent in spaces only, so a tab-indented body was not dedented at all. OpenTofu dedents it by one tab per level; measuring whitespace characters matches it, and is identical to counting spaces on space-indented input.
3. A whitespace-only line was excluded from the margin measurement but dedented anyway.
4. The closing marker's own indent was stripped as spaces and tabs only, so a body indented with a non-breaking space, a vertical tab, a form feed or an ideographic space came back with that character appended to the value. The dedent already measured whitespace, matching OpenTofu; the marker strip now uses the same rule.

This is not a regression -- 7.2.1 returned the same values -- so it changes long-standing behaviour rather than restoring anything.

Write side

strings_to_heredocs added a newline before the closing marker that the value already carried, emitting a body one line longer than the string it came from. The two errors cancelled inside this library's own round trip, which is why no test caught either. They do not cancel against Terraform: running test/integration/specialized/heredocs.tf through flatten -> strings_to_heredocs -> restore, 5 of the 11 locals came back with different values. A value that does not end in a newline cannot be written as a heredoc without gaining one, so it is now left quoted.

Carriage returns

The flattened form is quoted-string source, and a quoted string cannot hold a literal carriage return -- OpenTofu rejects one with "No closing marker was found for the string". A heredoc read out of a CRLF file therefore flattened to source that would not parse again. \r is now escaped like \n; both process_escape_sequences and OpenTofu resolve it back, so the value survives. The value form (strip_string_quotes=True) still hands back real characters.

A carriage return that does not end a line is a different case, and cannot be written as a heredoc at all. OpenTofu rejects <<EOF\nx\ry\nEOF with "No closing marker was found for the string", while the quoted "x\ry\n" it came from is valid and evaluates to that character. Such a value therefore stays quoted, for the same reason a value that does not end in a newline does — writing the heredoc would trade a wrong value for a file the reference implementation cannot read.

The delimiter search counts CRLF lines too. The body is split on \n, so a CRLF line carries its own \r, and OpenTofu ends a heredoc on EOF\r as readily as on EOF .

The writer had to learn the same escape. _unescape_heredoc_body resolved \n, \" and \\ but not \r, so flattening a CRLF heredoc and restoring it produced a body holding a literal backslash and an r -- a heredoc interprets no escape, so that is a different value. Both halves are now inverses, and a test runs the whole path rather than each half separately: OpenTofu evaluates a CRLF file containing <<EOF\r\nx\r\ny\r\nEOF\r\n to "x\r\ny\r\n", and that is what survives the round trip.

Tests

test/unit/test_heredoc_matches_terraform.py carries a 16-case table whose expectations come from OpenTofu, not from this implementation. bin/heredoc_ground_truth re-derives that table by evaluating every case with tofu console and exits non-zero on any mismatch, so the expectations can be rechecked against a future Terraform rather than trusted.

Delimiter selection

strings_to_heredocs wrote <<EOF over every value without looking at it, so a string holding a line reading EOF closed its own heredoc there and produced a file that no longer parsed -- and log excerpts, shell scripts and embedded configs, the payloads heredocs exist for, are exactly the values that contain the word. The delimiter is now chosen against the body: EOF when nothing in it could close the heredoc, a numbered variant otherwise, so ordinary output is byte-for-byte what it was.

Which lines count as markers is Terraform's rule rather than this grammar's, which is stricter. OpenTofu v1.12.5 ends a heredoc on EOF and evaluates <<EOF\nbody\nEOF \n to "body\n"; HEREDOC_TEMPLATE here requires the newline to follow the word and rejects that file outright. Choosing against the looser reading is what keeps the written file readable by both. That parser-side divergence is a separate defect and is not addressed here.

Not fixed here

Escapes other than \n, \r, \" and \\ are still not decoded when a value is written as a heredoc -- pre-existing, filed as #329.

Merging

It touches the same code as #350 (hcl2/rules/strings.py). Whichever of those lands first, this one needs a rebase rather than a merge — the overlaps are real edits to the same methods, not adjacent lines, so resolving them by hand risks losing one of the two fixes. Say the word and I will rebase and re-run the suite.

---

This pull request, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the author. Please review with that provenance in mind.

[livingstaccato]

Please hold off on merging this one for now — I want to do another review pass over it before it goes in. Opened as a draft for that reason; I will mark it ready and say so here once I am done.

[livingstaccato]

Review pass done, so the hold above no longer applies — this is ready for review now.

Rebased on current main; GitHub reports it mergeable as it stands.

_{🤖 Drafted with Claude Code.}