Repository navigation
Fix lineage through LATERAL FLATTEN, reused derived aliases, predicate subqueries and star options - #3
Merged
Conversation
…e subqueries and star options (#2) * fix(core): read LATERAL FLATTEN as a relation with its own columns sqlparser gives `LATERAL FLATTEN(...)` its own table factor, `TableFactor::Function`, which the lineage visitor did not match. The alias was never registered and no issue was raised, so a column read through it lost its source: SELECT o.order_id, f.value::STRING AS tag FROM orders o, LATERAL FLATTEN(input => o.tags) f `f` was taken for a table name: tag came out derived from a column `value` of a table `F` that does not exist, the implied schema listed `F` beside ORDERS, and orders.tags was read neither for lineage nor for the implied schema. An unqualified `value` was reported as ambiguous across the tables in scope, and a FLATTEN over the value of another had nothing to follow. Register FLATTEN under its alias, or under its name when it has none, as a relation with the columns Snowflake documents: SEQ, KEY, PATH, INDEX, VALUE and THIS. VALUE and THIS carry the element, so they derive from the columns of the `INPUT =>` argument, or of the first positional one, with the call as their expression. SEQ, KEY, PATH and INDEX say where an element sits and read nothing. They are added without the relation-level edge a source-less projection gets, which would have made `'n' || f.index` depend on every base relation of the FROM clause. Each FLATTEN gets its own node, keyed by its scope, since CTEs commonly reuse one alias such as `f`. A star over the scope now includes the six columns. The alias is also recorded as a subquery alias, as a derived table's is, so that what is read through it stays out of the implied schema, and the qualified columns of the input are recorded there as projected columns are: the query above now implies ORDERS(order_id, tags) and nothing else. With column lineage disabled only that alias is recorded: at table level the input is a column of a relation already in the FROM clause, so a FLATTEN node would have no edge, and no column is emitted. Those six columns are all a FLATTEN has, so the scope records it as a relation with fixed columns. A bare name it lacks, with no schema to place it, still resolves to the one other relation of the FROM clause, as it did while FLATTEN went unregistered: `id` in `SELECT id FROM orders o, LATERAL FLATTEN(input => o.tags) f` stays orders.id instead of turning ambiguous, and in `FROM customers, LATERAL FLATTEN(input => addresses) a, LATERAL FLATTEN(input => phones) p` the second input is read from customers. A warning for a name that is ambiguous between tables no longer lists the FLATTEN beside them. VALUE and THIS read the same input, so an input that cannot be resolved is reported once. Any other `LATERAL <function>(...)` now raises UNSUPPORTED_SYNTAX, as `TABLE(<function>(...))` already does, instead of being dropped silently. The snowflake_lateral_flatten snapshot changes accordingly: `value AS p_id` now derives from b.cool_ids through the FLATTEN node, B's implied columns gain cool_ids, and the ambiguity warning for `value` is gone. Tests: the query above with a decoy `value` column on orders, its implied schema without a schema given, chained flattens back to the first input, the position columns reading nothing under a CTE joined to another table, a literal input, one alias reused by two CTEs with named and positional input, a star over FLATTEN, a bare column and a bare filter beside a FLATTEN with no schema, a bare input to a second FLATTEN, an ambiguous input reported once, SPLIT_TO_TABLE reported as unsupported, and no FLATTEN node or column with column lineage disabled. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(core): give each derived table its own node when an alias repeats A derived table's node was keyed by statement and alias. Two derived tables of one statement sharing an alias in different scopes, as a generated point in time query writes one `(...) AS v` per satellite, therefore shared one node and one set of column nodes, and each consumer received every producer's lineage: WITH a_intervals AS ( SELECT k, ts FROM (SELECT DISTINCT k, ts FROM sat_a) AS v ), b_intervals AS ( SELECT k, ts FROM (SELECT DISTINCT k, ts FROM sat_b) AS v ) SELECT a.ts AS a_ts, b.ts AS b_ts FROM a_intervals a JOIN b_intervals b ON a.k = b.k a_ts came out derived from sat_a.ts and sat_b.ts both, and so did b_ts, with no issue raised. Count the derived tables of each alias per statement. The first keeps the key it always had, so nothing changes for a statement that does not repeat an alias, and each later one is keyed `alias#n`, getting a node and column nodes of its own. The node keeps the alias as its label. Two derived tables of one alias in the same scope are invalid SQL, and lookups already go through the scope, so a per statement count is enough. A CTE name defined again in a nested WITH merges the same way, but CTEs are also looked up by name once the statement has been walked, so that takes more than a key and is left alone here. Tests: the query above, each output from its own satellite and two nodes labelled v; the same alias over twelve satellites, each output from its own. Both fail without the change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(core): keep a predicate's subquery and an unaliased derived table out of the outputs A subquery a predicate reads was analysed with no target node, so its select list fell back to whatever the enclosing select targets: the statement's output at the top, and inside a CTE that CTE's column list, which a star over it then returned. SELECT k FROM t WHERE x >= (SELECT MAX(y) AS floor_y FROM u) returned k and floor_y. The same held for IN, for EXISTS, whose `SELECT 1` came back as col_0, and for a subquery in HAVING, a join condition or a grouping expression. No issue said so. Such a subquery is now analysed against a node of its own, labelled `(subquery)` and keyed per occurrence, which owns its columns and their lineage, and what it projects is taken back from the projection buffer once it has been read. A scalar subquery in the select list is not visited there and is returned as before. A derived table without an alias, which Snowflake and others accept, had no node at all: its projection hung on the enclosing target in the same way, and the outer select found no table in scope and reported so. It now gets a node like an aliased one, registered in its scope under a name no query can write, `(derived)`, so that a star over it expands and a bare column resolves. Tests: a scalar comparison, IN, a correlated NOT EXISTS, HAVING and a join condition each return only k; inside a CTE the subquery's column stays on its own node and out of the star; a select list scalar subquery is still returned; an unaliased derived table reads like an aliased one, with no issue. All but the select list case fail without the change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(core): honour EXCLUDE, EXCEPT and RENAME in a wildcard's expansion expand_wildcard emitted every column of the relation whatever options the wildcard carried: `SELECT * EXCLUDE (b) FROM p` returned b, and `SELECT * RENAME (b AS bee) FROM p` returned b under its old name. The common idiom of replacing a column fared worse: SELECT * EXCLUDE (b), UPPER(b) AS b FROM p The star copy of b and the computed b shared one output node, the copy came first, and the UPPER(b) derivation was merged away: b came out as a plain copy of p.b. The wildcard's options now reach the expansion. A column named by EXCLUDE, or by EXCEPT where the dialect writes it so, is not expanded; a column named by RENAME is expanded under its new name, still read from the old one. Names are compared as the expanded columns' own names are. REPLACE and ILIKE are still not read. A plain `SELECT *, f(x) AS x`, which does write two columns named x, is left as it was. Tests: EXCLUDE with and without parentheses, qualified and inside a CTE; EXCEPT; RENAME with and without parentheses, read from the old column; the idiom above, whose b carries UPPER(b). All fail without the change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs(changelog): note the relation and scope lineage fixes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#2 was merged into the branch of #1, its base while the two were stacked, so its commit never reached
master. This replays that commit, e0d4110, ontomasteras it is: same change, no other. Its code has not been through CI yet, since the workflow only runs for pull requests intomaster.🤖 Generated with Claude Code