Found while reviewing #13 (#25). #25 fixed the case where a column reaches no source; this is the case where it reaches one through an intermediate column, and loses the classification on the way.
A column is classified by its last hop only
SELECT c FROM (SELECT SUM(x) AS c FROM t) d
t.x is carried across the derived table correctly. The aggregation is not — c is reported as a plain pass-through of t.x. The same holds through a CTE, and whether or not the inner projection has sources:
WITH x AS (SELECT COUNT(*) AS c FROM t) SELECT c FROM x -- Direct
SELECT c FROM (SELECT COUNT(*) AS c FROM t) d -- Direct
SELECT c FROM (SELECT SUM(x) AS c FROM t) d -- Direct, t.x carried
Cause
collect_output_sources walks transitively to the leaves but records only the kind of the immediate incoming edge:
for &edge_idx in &incoming[node_id] {
let edge = &graph.edges[edge_idx];
let (sub_sources, sub_back) =
collect_leaf_origins(edge.from, graph, resolved, incoming, visited);
for _ in &sub_sources {
kinds.push(edge.kind.clone()); // the immediate edge, every time
}
...
}
collect_leaf_origins returns origins and nothing else, so every kind met deeper in the walk is discarded. For SELECT c FROM (SELECT SUM(x) AS c FROM t) d the immediate edge is Ref(c) → Output(c) with kind Direct, because the outer projection really is a plain reference; the inner Ident(x) → Output(c) edge carrying ViaAggregation is never consulted.
The column's own kind, added in #25, does not reach this either. It describes the outer projection, which is correctly Direct.
Consequence: the star form disagrees with the named form
Expanding a star classifies each inner column from that column's own node, taking no second hop, so the same derived table answers differently depending on how the column is selected:
SELECT * FROM (SELECT COUNT(*) AS c FROM t) d -- Aggregation
SELECT c FROM (SELECT COUNT(*) AS c FROM t) d -- Direct
Pinned by only_the_last_hop_classifies_the_column in sqllineage/tests/cte.rs, so whichever way this is resolved, the change is visible.
What a fix needs
collect_leaf_origins has to report the kinds it traversed, and collect_output_sources has to merge them rather than stamping the immediate edge's kind per source. Worth settling at the same time: whether an aggregate two hops down should surface at all, or whether Direct is the honest answer for a projection that is genuinely a plain reference to a column that happens to have been aggregated upstream. The tests assume the former.
This is in the area #14 reworks, so it may be resolved there.
Found while reviewing #13 (#25). #25 fixed the case where a column reaches no source; this is the case where it reaches one through an intermediate column, and loses the classification on the way.
A column is classified by its last hop only
t.xis carried across the derived table correctly. The aggregation is not —cis reported as a plain pass-through oft.x. The same holds through a CTE, and whether or not the inner projection has sources:Cause
collect_output_sourceswalks transitively to the leaves but records only the kind of the immediate incoming edge:collect_leaf_originsreturns origins and nothing else, so every kind met deeper in the walk is discarded. ForSELECT c FROM (SELECT SUM(x) AS c FROM t) dthe immediate edge isRef(c) → Output(c)with kindDirect, because the outer projection really is a plain reference; the innerIdent(x) → Output(c)edge carryingViaAggregationis never consulted.The column's own kind, added in #25, does not reach this either. It describes the outer projection, which is correctly
Direct.Consequence: the star form disagrees with the named form
Expanding a star classifies each inner column from that column's own node, taking no second hop, so the same derived table answers differently depending on how the column is selected:
Pinned by
only_the_last_hop_classifies_the_columninsqllineage/tests/cte.rs, so whichever way this is resolved, the change is visible.What a fix needs
collect_leaf_originshas to report the kinds it traversed, andcollect_output_sourceshas to merge them rather than stamping the immediate edge's kind per source. Worth settling at the same time: whether an aggregate two hops down should surface at all, or whetherDirectis the honest answer for a projection that is genuinely a plain reference to a column that happens to have been aggregated upstream. The tests assume the former.This is in the area #14 reworks, so it may be resolved there.