`strings_to_heredocs` inside a list or object emits `EOF,`, which does not parse

[livingstaccato]

Summary

_deserialize_string_as_heredoc builds a token ending at the closing marker
with no newline after it. The grammar's own HEREDOC_TEMPLATE includes that
newline, and a heredoc ends at a marker on a line of its own, so the token is
one character short of what the grammar accepts. A top-level attribute survives
only because the reconstructor happens to add a newline there; inside a list or
an object, the container's comma lands on the marker's line.

Reproduction

_deserialize_string_as_heredoc is hcl2/deserializer.py:262 at v8.1.3,
and builds f"<<EOF\n{content}\nEOF" -- no newline after the marker.

Reaching it needs a value that ends in a newline, which 8.1.3 never produces,
because it drops that newline when reading the heredoc (the defect PR #335
fixes). The reproduction below was run against
livingstaccato/python-hcl2@71e3b4fab99659c2cc615198f15534bdcdcaf854, the head
of that PR's branch:

flat = hcl2.loads('a = [<<EOT\nline1\nEOT\n, "p"]\n',
                  serialization_options=SerializationOptions(preserve_heredocs=False))
hcl2.dumps(flat, deserializer_options=DeserializerOptions(strings_to_heredocs=True))

a = [
  <<EOF
line1
EOF,
  "p",
]

hcl2.loads of that raises UnexpectedToken; OpenTofu reports "Unterminated
template string / No closing marker was found". The same happens for an object
value, an object key and a block label.

Why the obvious fix is not the fix

Appending \n to the token makes the container case parse, but the
reconstructor already supplies one for a top-level attribute, so every
top-level heredoc then gains a blank line after it (verified: the round-trip
fixture in test/integration/test_specialized.py grows one per heredoc). The
newline belongs wherever the reconstructor decides a heredoc's line ends, not
in the token text.

---

This issue, and the investigation behind it, were produced by an AI assistant (Claude) working on behalf of the author. Please review with that provenance in mind.