fix: decline to flatten a heredoc whose interpolation spans lines (#347)

CHANGELOG.md

@@ -7,7 +7,18 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## \[Unreleased\]
 
-- Nothing yet.
+### Fixed
+
+- A heredoc whose interpolation spans lines is not flattened. The quoted form cannot hold one: the newlines inside `${...}` are expression source, where OpenTofu rejects an escaped newline and a raw one makes the string span lines, which it also rejects. It used to emit the raw version -- output neither Terraform nor this library could read, written with no error -- and now hands the heredoc back in the form `preserve_heredocs=True` produces, which reads back as that heredoc. Declining is the only answer that does not change what the document means. ([#347](https://github.com/amplify-education/python-hcl2/issues/347))
+- `$${` and `%%{` resolve to `${` and `%{` in the value form, in both quoted strings and heredocs. They are HCL's escapes for a literal sigil, exactly as `\"` is for a quote, and OpenTofu evaluates `"$${esc}"` to the six characters `${esc}`; returning them doubled made the value differ from the one Terraform reads, in the one mode that promises the value. ([#336](https://github.com/amplify-education/python-hcl2/issues/336))
+- `strings_to_heredocs` resolves every escape the reader does. It knew `\n`, `\r`, `\"` and `\\`, so `"a\tb\n"` was written into the body as a backslash and a `t` -- two characters where Terraform reads one tab -- and `\uNNNN` fared the same. It now uses `process_escape_sequences`, the package's one implementation of that alphabet. ([#329](https://github.com/amplify-education/python-hcl2/issues/329))
+- Escapes are no longer added or resolved inside `${...}`. The text there is expression source, where a nested `"..."` is a string literal of its own: OpenTofu reads `"${upper("a")}"` as `A` and rejects `"${upper(\"a\")}"` outright, so escaping through an interpolation produced source the reference implementation will not parse, and resolving through one closed a nested literal early. Both directions now work on the spans the text is actually made of, and the scan knows the things inside an expression that can carry a non-structural brace: a string literal, HCL's `#`, `//` and `/* */` comments, and the nested expressions a string literal may itself contain -- OpenTofu evaluates `${1 /* } */ + 2}` to 3 and `"a ${upper("v${ "{" }w")} b"` to `a V{W b`, so counting either brace closed the expression inside itself. ([#339](https://github.com/amplify-education/python-hcl2/issues/339))
+- Flattened heredoc bodies match the values Terraform and OpenTofu evaluate the same source to. Three things differed, all checked against OpenTofu v1.12.5 rather than read off the spec: the newline terminating the last content line was dropped (`<<EOT\nline\nEOT` returned `'line'`, not `'line\n'`); `<<-` measured its indent in spaces alone, so a tab-indented body was not dedented at all; and a whitespace-only line was excluded from the measurement but trimmed anyway. This is not a regression — 7.2.1 returned the same values — so it changes long-standing behaviour rather than restoring anything.
+- A carriage return in a flattened heredoc body is written as `\r` rather than left raw. `preserve_heredocs=False` returns quoted-string *source*, and a quoted string cannot hold a literal carriage return: OpenTofu rejects one with "No closing marker was found for the string". A heredoc read out of a CRLF file therefore flattened to source that would not parse again. The value form (`strip_string_quotes=True`) is unchanged and still hands back real carriage returns. `strings_to_heredocs` resolves `\r` when it writes a body, so the two halves stay each other's inverse: a heredoc interprets no escape, so a body carrying a backslash and an `r` would be those two characters rather than the carriage return the value held.
+- A `<<-` heredoc whose closing marker is indented with something other than spaces or tabs no longer appends that indentation to the value. The dedent already measured whitespace rather than spaces, matching OpenTofu, but the marker's own indent was stripped as `[ \t]*`, so a body indented with a non-breaking space, a vertical tab, a form feed or an ideographic space came back with one of those characters on the end. Four such cases are now in the table that `bin/heredoc_ground_truth` re-derives from OpenTofu.
+- `strings_to_heredocs` leaves a value carrying a lone carriage return quoted. A heredoc body is read literally, so it can hold a `\r` only where one ends a line: OpenTofu rejects `<<EOF\nx\ry\nEOF` with "No closing marker was found for the string", while the quoted `"x\ry\n"` it came from is valid. Such a value stays quoted, for the same reason one that does not end in a newline does.
+- `strings_to_heredocs` picks a delimiter the body cannot close. It wrote `<<EOF` over every value, so a string holding a line reading `EOF` -- a log excerpt, a shell script, an embedded config, the payloads heredocs are for -- ended its own heredoc early and produced a file that no longer parsed. A numbered variant is used when the body occupies `EOF`, and ordinary values are written exactly as before. The lines that count as markers are Terraform's, which are looser than this grammar's: OpenTofu ends a heredoc on `EOF  ` while `HEREDOC_TEMPLATE` here requires the newline to follow the word. A CRLF body counts too -- it is split on `\n`, so its lines carry their own `\r`, and OpenTofu ends a heredoc on `EOF\r` as readily as on `EOF `. ([#330](https://github.com/amplify-education/python-hcl2/issues/330))
+- `strings_to_heredocs` no longer adds a line to the body it writes. The value's own trailing newline is the one that precedes the closing marker, so a heredoc was being emitted one line longer than the string it came from. A value that does not end in a newline is now left as a quoted string, since no heredoc can express it. Flattening a document and restoring it now yields HCL that OpenTofu evaluates identically to the original; five of the eleven values in the round-trip fixture did not survive it before.
 
 ## \[8.1.3\] - 2026-08-26
 

bin/heredoc_ground_truth

@@ -0,0 +1,108 @@
+#!/usr/bin/env python
+"""Check the heredoc expectations in the test suite against Terraform itself.
+
+`test/unit/test_heredoc_matches_terraform.py` asserts what a heredoc body
+evaluates to. Those values did not come from this library or from reading the
+spec -- each one was produced by handing the same source to OpenTofu. That
+provenance is a docstring, which a reader has to take on trust and which
+nothing re-checks if the reference implementation ever moves.
+
+This script re-derives them. It reads the `CASES` table out of that test module,
+evaluates every source with `tofu console` (or `terraform console`), and
+compares. It is not part of the test run: the suite must not depend on a
+Terraform binary, and these values change about as often as the HCL spec does.
+
+Usage:
+    bin/heredoc_ground_truth            # verify; non-zero exit on any mismatch
+    bin/heredoc_ground_truth --print    # print the table as Python, to paste
+
+Requires `tofu` or `terraform` on PATH.
+"""
+
+import argparse
+import json
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
+
+from test.unit.test_heredoc_matches_terraform import CASES  # noqa: E402
+
+BINARIES = ("tofu", "terraform")
+
+
+def find_binary():
+    """Return the first Terraform-compatible binary on PATH, or None."""
+    for name in BINARIES:
+        path = shutil.which(name)
+        if path:
+            return path
+    return None
+
+
+def evaluate(binary, source):
+    """Return the value `binary` evaluates the given heredoc expression to.
+
+    The source is written as a local rather than an output so that nothing has
+    to be applied, and `jsonencode` is what carries the exact string back --
+    the console's own rendering escapes newlines for display.
+    """
+    with tempfile.TemporaryDirectory() as directory:
+        # newline="" so a case testing CRLF is written with the bytes it names.
+        with open(os.path.join(directory, "main.tf"), "w", encoding="utf-8", newline="") as handle:
+            handle.write("locals {\n  x = %s\n}\n" % source)
+        result = subprocess.run(
+            [binary, "console"],
+            cwd=directory,
+            input="jsonencode(local.x)\n",
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+        if result.returncode != 0:
+            raise RuntimeError(result.stderr.strip())
+        return json.loads(json.loads(result.stdout.strip().splitlines()[-1]))
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument(
+        "--print",
+        dest="print_table",
+        action="store_true",
+        help="print the evaluated table as Python instead of verifying",
+    )
+    args = parser.parse_args()
+
+    binary = find_binary()
+    if binary is None:
+        print("neither `tofu` nor `terraform` is on PATH", file=sys.stderr)
+        return 2
+
+    print("using %s\n" % binary, file=sys.stderr)
+    mismatches = 0
+    for source, expected in CASES:
+        actual = evaluate(binary, source)
+        if args.print_table:
+            print("    (%r, %r)," % (source, actual))
+            continue
+        if actual == expected:
+            print("ok   %r" % source)
+        else:
+            mismatches += 1
+            print("BAD  %r\n       expected %r\n       %s says %r" % (source, expected, binary, actual))
+
+    if args.print_table:
+        return 0
+
+    # stdout, so it lands after the per-case lines rather than ahead of them
+    # when the output is piped.
+    print("\n%d of %d cases disagree" % (mismatches, len(CASES)))
+    return 1 if mismatches else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

cli/json_to_hcl.py

@@ -180,7 +180,7 @@ def main():  # pylint: disable=too-many-branches,too-many-statements,too-many-lo
     parser.add_argument(
         "--strings-to-heredocs",
         action="store_true",
-        help="Convert strings containing escaped newlines to heredocs",
+        help="Convert newline-terminated escaped strings to heredocs",
     )
 
     # FormatterOptions flags

docs/01_getting_started.md

@@ -74,7 +74,7 @@ data = loads(text, serialization_options=SerializationOptions(
 | `wrap_objects` | `bool` | `False` | Wrap object values as inline HCL2 strings                                                                                                       |
 | `wrap_tuples` | `bool` | `False` | Wrap tuple values as inline HCL2 strings                                                                                                        |
 | `explicit_blocks` | `bool` | `True` | Add `__is_block__: True` markers to blocks. **Mandatory for JSON->HCL2 deserialization and reconstruction.**                                    |
-| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. |
+| `preserve_heredocs` | `bool` | `True` | Keep heredocs in their original form, markers included. When `False`, a heredoc becomes a quoted string with its newlines escaped (`'"a\nb\n"'`); combine with `strip_string_quotes` to get the body as a plain multi-line value instead. Either way the body keeps the newline that terminates its last line, as Terraform's does. |
 | `force_operation_parentheses` | `bool` | `False` | Force parentheses around all operations                                                                                                         |
 | `preserve_scientific_notation` | `bool` | `True` | Keep scientific notation as-is                                                                                                                  |
 | `strip_string_quotes` | `bool` | `False` | Yield string *values* rather than source text: remove surrounding quotes (e.g. `"hello"` instead of `'"hello"'`) and resolve escape sequences (`"a\nb"` becomes a real newline). String literals inside expressions keep their quotes, so `upper("x")` stays `'${upper("x")}'`. **Breaks JSON->HCL2 deserialization and reconstruction.** |
@@ -127,7 +127,7 @@ text = dumps(data, deserializer_options=DeserializerOptions(
 | Field | Type | Default | Description |
 |---|---|---|---|
 | `heredocs_to_strings` | `bool` | `False` | Convert heredocs to plain strings |
-| `strings_to_heredocs` | `bool` | `False` | Convert strings with `\n` to heredocs |
+| `strings_to_heredocs` | `bool` | `False` | Convert newline-terminated strings to heredocs. A value that does not end in a newline is left as a quoted string, because a heredoc body always ends in one and writing it as a heredoc would change the value. |
 | `object_elements_colon` | `bool` | `False` | Use `:` instead of `=` in object elements |
 | `object_elements_trailing_comma` | `bool` | `True` | Add trailing commas in object elements |
 

docs/06_migrating_to_v8.md

@@ -213,17 +213,18 @@ This restores the v7 dict shape but disables round-trip support and comment pres
 
 ```python
 hcl2.loads('x = <<-EOT\n  line1\n  line2\n  EOT\n', serialization_options=V7_COMPAT)
-# {'x': 'line1\nline2'}
+# {'x': 'line1\nline2\n'}
 ```
 
-Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2"'`), because that output is meant to be reconstructable.
+Note that `preserve_heredocs=False` on its own — without `strip_string_quotes` — produces the quoted *source* form with escaped newlines (`'"line1\\nline2\\n"'`), because that output is meant to be reconstructable.
 
-Two details of heredoc values are easy to trip over, and both match how HCL itself behaves:
+Three details of heredoc values are easy to trip over, and all three match how HCL itself behaves:
 
+- **The body ends with a newline.** Every content line is terminated by its own newline, the last one included, so `<<EOT\nline\nEOT` is `'line\n'` — the same value Terraform evaluates it to. v7 returned `'line'`, and so did 8.1.x; both were wrong. Only an empty body has no trailing newline, because it has no content line.
 - **Backslash escapes are not interpreted in heredocs.** `strip_string_quotes` resolves `\n` inside a *quoted* string, but a heredoc body containing the two characters `\n` keeps them verbatim. HCL only processes escape sequences in quoted templates.
 - **Line endings come through as written.** A heredoc in a CRLF file yields a body with `\r\n`, because a carriage return inside the body is content rather than structure. Normalize on your side if you need `\n`.
 
 ```python
 hcl2.loads('x = <<EOT\na\\nb\nEOT\n', serialization_options=V7_COMPAT)
-# {'x': 'a\\nb'}  — the backslash and the "n" are two literal characters
+# {'x': 'a\\nb\n'}  — the backslash and the "n" are two literal characters
 ```