Repository navigation
Fix lineage through LATERAL FLATTEN, reused derived aliases, predicate subqueries and star options - #2
Merged
Conversation
sqlparser gives `LATERAL FLATTEN(...)` its own table factor,
`TableFactor::Function`, which the lineage visitor did not match. The
alias was never registered and no issue was raised, so a column read
through it lost its source:
SELECT o.order_id, f.value::STRING AS tag
FROM orders o,
LATERAL FLATTEN(input => o.tags) f
`f` was taken for a table name: tag came out derived from a column
`value` of a table `F` that does not exist, the implied schema listed
`F` beside ORDERS, and orders.tags was read neither for lineage nor for
the implied schema. An unqualified `value` was reported as ambiguous
across the tables in scope, and a FLATTEN over the value of another had
nothing to follow.
Register FLATTEN under its alias, or under its name when it has none,
as a relation with the columns Snowflake documents: SEQ, KEY, PATH,
INDEX, VALUE and THIS. VALUE and THIS carry the element, so they derive
from the columns of the `INPUT =>` argument, or of the first positional
one, with the call as their expression. SEQ, KEY, PATH and INDEX say
where an element sits and read nothing. They are added without the
relation-level edge a source-less projection gets, which would have made
`'n' || f.index` depend on every base relation of the FROM clause. Each
FLATTEN gets its own node, keyed by its scope, since CTEs commonly reuse
one alias such as `f`. A star over the scope now includes the six
columns.
The alias is also recorded as a subquery alias, as a derived table's
is, so that what is read through it stays out of the implied schema,
and the qualified columns of the input are recorded there as projected
columns are: the query above now implies ORDERS(order_id, tags) and
nothing else. With column lineage disabled only that alias is recorded:
at table level the input is a column of a relation already in the FROM
clause, so a FLATTEN node would have no edge, and no column is emitted.
Those six columns are all a FLATTEN has, so the scope records it as a
relation with fixed columns. A bare name it lacks, with no schema to
place it, still resolves to the one other relation of the FROM clause,
as it did while FLATTEN went unregistered: `id` in `SELECT id FROM
orders o, LATERAL FLATTEN(input => o.tags) f` stays orders.id instead of
turning ambiguous, and in `FROM customers, LATERAL FLATTEN(input =>
addresses) a, LATERAL FLATTEN(input => phones) p` the second input is
read from customers. A warning for a name that is ambiguous between
tables no longer lists the FLATTEN beside them. VALUE and THIS read the
same input, so an input that cannot be resolved is reported once.
Any other `LATERAL <function>(...)` now raises UNSUPPORTED_SYNTAX, as
`TABLE(<function>(...))` already does, instead of being dropped
silently.
The snowflake_lateral_flatten snapshot changes accordingly: `value AS
p_id` now derives from b.cool_ids through the FLATTEN node, B's implied
columns gain cool_ids, and the ambiguity warning for `value` is gone.
Tests: the query above with a decoy `value` column on orders, its
implied schema without a schema given, chained flattens back to the
first input, the position columns reading nothing under a CTE joined to
another table, a literal input, one alias reused by two CTEs with named
and positional input, a star over FLATTEN, a bare column and a bare
filter beside a FLATTEN with no schema, a bare input to a second
FLATTEN, an ambiguous input reported once, SPLIT_TO_TABLE reported as
unsupported, and no FLATTEN node or column with column lineage
disabled.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A derived table's node was keyed by statement and alias. Two derived
tables of one statement sharing an alias in different scopes, as a
generated point in time query writes one `(...) AS v` per satellite,
therefore shared one node and one set of column nodes, and each consumer
received every producer's lineage:
WITH a_intervals AS (
SELECT k, ts FROM (SELECT DISTINCT k, ts FROM sat_a) AS v
),
b_intervals AS (
SELECT k, ts FROM (SELECT DISTINCT k, ts FROM sat_b) AS v
)
SELECT a.ts AS a_ts, b.ts AS b_ts
FROM a_intervals a JOIN b_intervals b ON a.k = b.k
a_ts came out derived from sat_a.ts and sat_b.ts both, and so did b_ts,
with no issue raised.
Count the derived tables of each alias per statement. The first keeps
the key it always had, so nothing changes for a statement that does not
repeat an alias, and each later one is keyed `alias#n`, getting a node
and column nodes of its own. The node keeps the alias as its label. Two
derived tables of one alias in the same scope are invalid SQL, and
lookups already go through the scope, so a per statement count is
enough.
A CTE name defined again in a nested WITH merges the same way, but CTEs
are also looked up by name once the statement has been walked, so that
takes more than a key and is left alone here.
Tests: the query above, each output from its own satellite and two
nodes labelled v; the same alias over twelve satellites, each output
from its own. Both fail without the change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… out of the outputs
A subquery a predicate reads was analysed with no target node, so its
select list fell back to whatever the enclosing select targets: the
statement's output at the top, and inside a CTE that CTE's column list,
which a star over it then returned.
SELECT k FROM t WHERE x >= (SELECT MAX(y) AS floor_y FROM u)
returned k and floor_y. The same held for IN, for EXISTS, whose
`SELECT 1` came back as col_0, and for a subquery in HAVING, a join
condition or a grouping expression. No issue said so.
Such a subquery is now analysed against a node of its own, labelled
`(subquery)` and keyed per occurrence, which owns its columns and their
lineage, and what it projects is taken back from the projection buffer
once it has been read. A scalar subquery in the select list is not
visited there and is returned as before.
A derived table without an alias, which Snowflake and others accept,
had no node at all: its projection hung on the enclosing target in the
same way, and the outer select found no table in scope and reported so.
It now gets a node like an aliased one, registered in its scope under a
name no query can write, `(derived)`, so that a star over it expands and
a bare column resolves.
Tests: a scalar comparison, IN, a correlated NOT EXISTS, HAVING and a
join condition each return only k; inside a CTE the subquery's column
stays on its own node and out of the star; a select list scalar
subquery is still returned; an unaliased derived table reads like an
aliased one, with no issue. All but the select list case fail without
the change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
expand_wildcard emitted every column of the relation whatever options
the wildcard carried: `SELECT * EXCLUDE (b) FROM p` returned b, and
`SELECT * RENAME (b AS bee) FROM p` returned b under its old name.
The common idiom of replacing a column fared worse:
SELECT * EXCLUDE (b), UPPER(b) AS b FROM p
The star copy of b and the computed b shared one output node, the copy
came first, and the UPPER(b) derivation was merged away: b came out as
a plain copy of p.b.
The wildcard's options now reach the expansion. A column named by
EXCLUDE, or by EXCEPT where the dialect writes it so, is not expanded; a
column named by RENAME is expanded under its new name, still read from
the old one. Names are compared as the expanded columns' own names are.
REPLACE and ILIKE are still not read. A plain `SELECT *, f(x) AS x`,
which does write two columns named x, is left as it was.
Tests: EXCLUDE with and without parentheses, qualified and inside a CTE;
EXCEPT; RENAME with and without parentheses, read from the old column;
the idiom above, whose b carries UPPER(b). All fail without the change.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
IL-William
added a commit
that referenced
this pull request
Sep 29, 2026
…e subqueries and star options (#2) (#3) * fix(core): read LATERAL FLATTEN as a relation with its own columns sqlparser gives `LATERAL FLATTEN(...)` its own table factor, `TableFactor::Function`, which the lineage visitor did not match. The alias was never registered and no issue was raised, so a column read through it lost its source: SELECT o.order_id, f.value::STRING AS tag FROM orders o, LATERAL FLATTEN(input => o.tags) f `f` was taken for a table name: tag came out derived from a column `value` of a table `F` that does not exist, the implied schema listed `F` beside ORDERS, and orders.tags was read neither for lineage nor for the implied schema. An unqualified `value` was reported as ambiguous across the tables in scope, and a FLATTEN over the value of another had nothing to follow. Register FLATTEN under its alias, or under its name when it has none, as a relation with the columns Snowflake documents: SEQ, KEY, PATH, INDEX, VALUE and THIS. VALUE and THIS carry the element, so they derive from the columns of the `INPUT =>` argument, or of the first positional one, with the call as their expression. SEQ, KEY, PATH and INDEX say where an element sits and read nothing. They are added without the relation-level edge a source-less projection gets, which would have made `'n' || f.index` depend on every base relation of the FROM clause. Each FLATTEN gets its own node, keyed by its scope, since CTEs commonly reuse one alias such as `f`. A star over the scope now includes the six columns. The alias is also recorded as a subquery alias, as a derived table's is, so that what is read through it stays out of the implied schema, and the qualified columns of the input are recorded there as projected columns are: the query above now implies ORDERS(order_id, tags) and nothing else. With column lineage disabled only that alias is recorded: at table level the input is a column of a relation already in the FROM clause, so a FLATTEN node would have no edge, and no column is emitted. Those six columns are all a FLATTEN has, so the scope records it as a relation with fixed columns. A bare name it lacks, with no schema to place it, still resolves to the one other relation of the FROM clause, as it did while FLATTEN went unregistered: `id` in `SELECT id FROM orders o, LATERAL FLATTEN(input => o.tags) f` stays orders.id instead of turning ambiguous, and in `FROM customers, LATERAL FLATTEN(input => addresses) a, LATERAL FLATTEN(input => phones) p` the second input is read from customers. A warning for a name that is ambiguous between tables no longer lists the FLATTEN beside them. VALUE and THIS read the same input, so an input that cannot be resolved is reported once. Any other `LATERAL <function>(...)` now raises UNSUPPORTED_SYNTAX, as `TABLE(<function>(...))` already does, instead of being dropped silently. The snowflake_lateral_flatten snapshot changes accordingly: `value AS p_id` now derives from b.cool_ids through the FLATTEN node, B's implied columns gain cool_ids, and the ambiguity warning for `value` is gone. Tests: the query above with a decoy `value` column on orders, its implied schema without a schema given, chained flattens back to the first input, the position columns reading nothing under a CTE joined to another table, a literal input, one alias reused by two CTEs with named and positional input, a star over FLATTEN, a bare column and a bare filter beside a FLATTEN with no schema, a bare input to a second FLATTEN, an ambiguous input reported once, SPLIT_TO_TABLE reported as unsupported, and no FLATTEN node or column with column lineage disabled. * fix(core): give each derived table its own node when an alias repeats A derived table's node was keyed by statement and alias. Two derived tables of one statement sharing an alias in different scopes, as a generated point in time query writes one `(...) AS v` per satellite, therefore shared one node and one set of column nodes, and each consumer received every producer's lineage: WITH a_intervals AS ( SELECT k, ts FROM (SELECT DISTINCT k, ts FROM sat_a) AS v ), b_intervals AS ( SELECT k, ts FROM (SELECT DISTINCT k, ts FROM sat_b) AS v ) SELECT a.ts AS a_ts, b.ts AS b_ts FROM a_intervals a JOIN b_intervals b ON a.k = b.k a_ts came out derived from sat_a.ts and sat_b.ts both, and so did b_ts, with no issue raised. Count the derived tables of each alias per statement. The first keeps the key it always had, so nothing changes for a statement that does not repeat an alias, and each later one is keyed `alias#n`, getting a node and column nodes of its own. The node keeps the alias as its label. Two derived tables of one alias in the same scope are invalid SQL, and lookups already go through the scope, so a per statement count is enough. A CTE name defined again in a nested WITH merges the same way, but CTEs are also looked up by name once the statement has been walked, so that takes more than a key and is left alone here. Tests: the query above, each output from its own satellite and two nodes labelled v; the same alias over twelve satellites, each output from its own. Both fail without the change. * fix(core): keep a predicate's subquery and an unaliased derived table out of the outputs A subquery a predicate reads was analysed with no target node, so its select list fell back to whatever the enclosing select targets: the statement's output at the top, and inside a CTE that CTE's column list, which a star over it then returned. SELECT k FROM t WHERE x >= (SELECT MAX(y) AS floor_y FROM u) returned k and floor_y. The same held for IN, for EXISTS, whose `SELECT 1` came back as col_0, and for a subquery in HAVING, a join condition or a grouping expression. No issue said so. Such a subquery is now analysed against a node of its own, labelled `(subquery)` and keyed per occurrence, which owns its columns and their lineage, and what it projects is taken back from the projection buffer once it has been read. A scalar subquery in the select list is not visited there and is returned as before. A derived table without an alias, which Snowflake and others accept, had no node at all: its projection hung on the enclosing target in the same way, and the outer select found no table in scope and reported so. It now gets a node like an aliased one, registered in its scope under a name no query can write, `(derived)`, so that a star over it expands and a bare column resolves. Tests: a scalar comparison, IN, a correlated NOT EXISTS, HAVING and a join condition each return only k; inside a CTE the subquery's column stays on its own node and out of the star; a select list scalar subquery is still returned; an unaliased derived table reads like an aliased one, with no issue. All but the select list case fail without the change. * fix(core): honour EXCLUDE, EXCEPT and RENAME in a wildcard's expansion expand_wildcard emitted every column of the relation whatever options the wildcard carried: `SELECT * EXCLUDE (b) FROM p` returned b, and `SELECT * RENAME (b AS bee) FROM p` returned b under its old name. The common idiom of replacing a column fared worse: SELECT * EXCLUDE (b), UPPER(b) AS b FROM p The star copy of b and the computed b shared one output node, the copy came first, and the UPPER(b) derivation was merged away: b came out as a plain copy of p.b. The wildcard's options now reach the expansion. A column named by EXCLUDE, or by EXCEPT where the dialect writes it so, is not expanded; a column named by RENAME is expanded under its new name, still read from the old one. Names are compared as the expanded columns' own names are. REPLACE and ILIKE are still not read. A plain `SELECT *, f(x) AS x`, which does write two columns named x, is left as it was. Tests: EXCLUDE with and without parentheses, qualified and inside a CTE; EXCEPT; RENAME with and without parentheses, read from the old column; the idiom above, whose b carries UPPER(b). All fail without the change. * docs(changelog): note the relation and scope lineage fixes --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1: its base is
fix/expression-column-refs, whose tests add theexpected_sourceshelper used below, so this shows only its own five commits,four fixes and their changelog entry. Merging #1 first retargets it to
master.Summary
Four ways a relation or a scope was misread, each giving wrong column lineage
without raising an issue.
LATERAL FLATTENwas not matched by the lineage visitor, so its alias wastaken for a table name: in
SELECT f.value::STRING AS tag FROM orders o, LATERAL FLATTEN(input => o.tags) f,tagcame from a columnvalueof a tableFthat does not exist, andorders.tagswas never read.generated point in time query writes one
(SELECT DISTINCT k, ts FROM sat_x) AS vper satellite, in separate CTEs, and every consumer then received everysatellite's lineage.
list fell back to whatever the enclosing select targets:
SELECT k FROM t WHERE x >= (SELECT MAX(y) AS floor_y FROM u)returnedkandfloor_y, anEXISTS (SELECT 1 ...)returnedcol_0, and inside a CTE the column joinedthe CTE's list, which a star over it then returned. A derived table without an
alias had no node at all, and the outer select found no table in scope.
* EXCLUDE,* EXCEPTand* RENAMEwere ignored in wildcard expansion.With the common idiom
SELECT * EXCLUDE (b), UPPER(b) AS b FROM p, the starcopy of
band the computedbshared one output node, the copy came first,and the
UPPER(b)derivation was lost.Changes
One commit per fix, each with its tests:
fix(core): read LATERAL FLATTEN as a relation with its own columnsregisters FLATTEN under its alias as a relation with the six columns
Snowflake documents; VALUE and THIS derive from the input, SEQ, KEY, PATH and
INDEX read nothing. Each FLATTEN gets a node keyed by its scope, since CTEs
commonly reuse one alias. Any other
LATERAL <function>(...)now raisesUNSUPPORTED_SYNTAX. Thesnowflake_lateral_flattensnapshot changesaccordingly.
fix(core): give each derived table its own node when an alias repeatscounts derived tables per alias and statement: the first keeps the key it
always had, so statements that do not repeat an alias are unchanged, and each
later one is keyed
alias#n.fix(core): keep a predicate's subquery and an unaliased derived table out of the outputsanalyses a subquery read by a predicate, a join condition or agrouping expression against a node of its own, labelled
(subquery), andtakes what it projects back from the projection buffer. A scalar subquery in
the select list is not visited there and is returned as before. A derived
table without an alias gets a node like an aliased one, registered in scope
under
(derived). This one builds on the previous commit's per occurrence key.fix(core): honour EXCLUDE, EXCEPT and RENAME in a wildcard's expansionpasses the wildcard's options to
expand_wildcard: excluded columns are notexpanded, renamed ones are expanded under their new name, read from the old
one. REPLACE and ILIKE are still not read, and a plain
SELECT *, f(x) AS xis left as it was.
A CTE name defined again in a nested WITH merges the same way the derived
tables did, but CTEs are also looked up by name after the statement has been
walked, so that takes more than a key and is not attempted here.
Each commit message has the details and examples. There is no API or schema
change.
Validation
cargo test --workspace --locked,cargo clippy --workspace --locked -- -D warnings,cargo fmt --all -- --checkand the schema guard pass on thisbranch.
pinning that a select list scalar subquery is still returned, which passes
either way by design.
On a large Snowflake dbt project: 105 of 111 FLATTEN value pairs had been
missing, 8 point in time models had their satellites crossed, 33 models
gained a column their SQL does not return, and 16 edges ran through columns an
EXCLUDEhad removed.The committed browser WASM is not rebuilt here, since CI builds its own.
🤖 Generated with Claude Code