Skip to content

Column transform keeps only the last hop's classification #26

Description

@funcpp

Found while reviewing #13 (#25). #25 fixed the case where a column reaches no source; this is the case where it reaches one through an intermediate column, and loses the classification on the way.

A column is classified by its last hop only

SELECT c FROM (SELECT SUM(x) AS c FROM t) d
c  ←  t.x    (direct)

t.x is carried across the derived table correctly. The aggregation is not — c is reported as a plain pass-through of t.x. The same holds through a CTE, and whether or not the inner projection has sources:

WITH x AS (SELECT COUNT(*) AS c FROM t) SELECT c FROM x   -- Direct
SELECT c FROM (SELECT COUNT(*) AS c FROM t) d             -- Direct
SELECT c FROM (SELECT SUM(x) AS c FROM t) d               -- Direct, t.x carried

Cause

collect_output_sources walks transitively to the leaves but records only the kind of the immediate incoming edge:

for &edge_idx in &incoming[node_id] {
    let edge = &graph.edges[edge_idx];
    let (sub_sources, sub_back) =
        collect_leaf_origins(edge.from, graph, resolved, incoming, visited);
    for _ in &sub_sources {
        kinds.push(edge.kind.clone());   // the immediate edge, every time
    }
    ...
}

collect_leaf_origins returns origins and nothing else, so every kind met deeper in the walk is discarded. For SELECT c FROM (SELECT SUM(x) AS c FROM t) d the immediate edge is Ref(c) → Output(c) with kind Direct, because the outer projection really is a plain reference; the inner Ident(x) → Output(c) edge carrying ViaAggregation is never consulted.

The column's own kind, added in #25, does not reach this either. It describes the outer projection, which is correctly Direct.

Consequence: the star form disagrees with the named form

Expanding a star classifies each inner column from that column's own node, taking no second hop, so the same derived table answers differently depending on how the column is selected:

SELECT * FROM (SELECT COUNT(*) AS c FROM t) d   -- Aggregation
SELECT c    FROM (SELECT COUNT(*) AS c FROM t) d -- Direct

Pinned by only_the_last_hop_classifies_the_column in sqllineage/tests/cte.rs, so whichever way this is resolved, the change is visible.

What a fix needs

collect_leaf_origins has to report the kinds it traversed, and collect_output_sources has to merge them rather than stamping the immediate edge's kind per source. Worth settling at the same time: whether an aggregate two hops down should surface at all, or whether Direct is the honest answer for a projection that is genuinely a plain reference to a column that happens to have been aggregated upstream. The tests assume the former.

This is in the area #14 reworks, so it may be resolved there.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions