Skip to content

fix(select): a projection sees the projections before it - #620

Merged
singaraiona merged 10 commits into
devfrom
fix/select-projection-visibility-617
Sep 25, 2026
Merged

singaraiona merged 10 commits into
devfrom
fix/select-projection-visibility-617

Conversation

@ser-vasilich

Copy link
Copy Markdown
Collaborator

Closes #617.

Bug

A projection in select could not reference a projection defined before it in the same query. nn: (+ notional 1) after notional: (* price volume) failed with 'notional' undefined; the literal form (+ 'notional 1) was compiled as arithmetic on the symbol's interned id and returned a number. In a grouped select the same reference failed the same way.

Two related defects in the grouped path surfaced while fixing this: an output whose head is if over an aggregate (f: (if (> (count price) 1) 1 0)) was evaluated once over the whole group result, so one branch was picked for every group and, when the output could not be appended, the hidden count was left in the result under a made-up name; and an aggregate of an aggregate (mx: (max (sum price))) folded the whole table into every group.

Fix

Each projection is bound under its alias once its expression has compiled, in an alias store on the graph (sel_alias_syms/ids, separate from the 32-slot env used for lambda/let inlining, so a wide select does not stop lambdas from inlining). A name reference resolves through the env, then the aliases, then the table columns; a literal symbol resolves through the aliases, then the columns, and otherwise stays a constant. The binding is made after the expression, so price: (* price 2) p2: (+ price 1) reads the source in price and the doubled value in p2. where: and by: are compiled outside the window and stay alias-blind. A lambda defined outside the select keeps its own free names: the alias store is hidden while its body compiles.

Projections that are evaluated outside the compiled path (a whole-column verb such as reverse beside them) see earlier aliases too: each row-shaped output is bound into the table the remaining expressions read.

In a grouped select the alias is resolved at the expression level: select_resolve_grouped_aliases substitutes the earlier expression into later outputs before compilation. Inside an aggregate's argument a name that is a source column stays the source column (s: (sum s) mx: (max s) keeps its meaning); an aggregate alias inside another aggregate, and an aggregate of an aggregate written out, raise a domain error. Outputs headed by if (and by the loop and scope forms) are evaluated per group rather than once over the group result, and an output that does not evaluate to one value per group is reported.

Arithmetic on a symbol inside a select, update or group key raises the type error it raises outside: compile_expr_dag records it on the graph and the sites without an evaluation fallback return it.

Tests

test/rfl/regress/issue_617.rfl: the reported query, alias chains, shadowing (bare and literal), the fallback path, a 35-projection select with a five-formal lambda, grouped aliases (bare, literal, chained, if over an alias, try), the in-aggregate rule, nested aggregates, symbol arithmetic in select/update/where, and guards for the behaviour that must not change. Documentation: docs/docs/queries/select.md, section "Projections that build on earlier projections".

ser-vasilich and others added 5 commits September 23, 2026 18:14
The DAG compiler compiled every projection against the source table
alone, so `nn: (+ notional 1)` after `notional: (* price volume)` was a
schema error, and the literal form `(+ 'notional 1)` silently returned
the symbol's interned id plus one.

Ungrouped: once a projection compiles, its alias is bound in the
compile-time env, where a later projection's name — bare or literal —
finds it ahead of the source columns.  The binding is made after the
expression, so an alias that shadows a source column reads the source
in its own definition and the new column in every later one.  where:,
by: and the sort keys compile earlier and stay alias-blind.

Grouped: outputs are aggregates over source rows, so the reference is
resolved at the expression level — each output is rewritten with the
earlier aliases' expressions substituted, and the arith-of-aggs
decomposition evaluates the result over the group result.  Inside an
aggregate's argument a name that is a source column stays the source
column (`s: (sum s) mx: (max s)` is the max of the rows); outside, the
alias wins as in the ungrouped case.  The rewritten dict re-enters the
select with the evaluated table as from: and a flag that stops the
rewrite from running twice.

Arithmetic on a symbol inside a select is a type error, as outside: the
DAG compiler declines `+ - * / % div pow` with a SYM operand and the
per-row fallback raises arith.c's error.  Comparisons and membership on
symbols are untouched.

Test: test/rfl/regress/issue_617.rfl — the reported shapes, alias
chains, shadowing in both positions, where: blindness, literal-symbol
and lambda-formal contracts, grouped aliases including the aggregate-
argument rule, and the symbol-arithmetic errors.

Fixes #617.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…back and per group

The select's output aliases now live in their own store on the graph
(sel_alias_syms/ids) instead of the 32-slot compile-time env shared with
lambda and let inlining: a wide select no longer stops a lambda from
inlining (which sent the whole query to per-row evaluation), and the
33rd alias is bound like the first. A name reference consults the env,
then the aliases, then the table; a literal symbol consults the aliases
before a same-named source column, as the bare name does. A lambda
defined outside the select keeps its own free names: the alias store is
hidden while its body compiles.

Projections evaluated outside the compiled path (a whole-column verb
beside them) see the earlier aliases too: each row-shaped output is
bound into the table the remaining expressions read, replacing a source
column of the same name.

Grouped selects: an output whose head is `if` (or a loop / scope form)
is evaluated per group instead of once over the group result, where
a vector condition picked one branch for every group and the failure
to append the output left a hidden aggregate under a made-up name in
the result; an output that does not evaluate to one value per group is
now reported. An aggregate alias inside another aggregate's argument
(`s: (sum price) mx: (max s)`) is a domain error rather than a
whole-table value.

Arithmetic on a symbol inside a select, update or group key raises the
same `type` error everywhere: compile_expr_dag records it on the graph
and the sites without an evaluation fallback return it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lect

`mx: (max (sum price))` with a by: clause folded the whole table into
every group: the inner aggregate is already one value per group, so
the outer one has no rows to reduce and was evaluated over the source
columns instead. The output is now rejected with a domain error, as
the same shape written through an alias already is. An aggregate over
a whole-column verb such as `(count (distinct price))` is unaffected.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@singaraiona singaraiona left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking — one of these crashes the process and two are silent regressions
against dev. I reproduced the three below myself on bd71c157 (release and
ASan/UBSan), against an origin/dev build of the same tree, so these are not
static-analysis guesses.

The direction of the PR is right and #617 is a real gap. It is the substitution
mechanism that needs rework, not the goal.

1. SIGSEGV on a 476-byte query — src/ops/query.c:6404

select_alias_subst substitutes each earlier expression by value, so an
output naming two earlier aliases doubles the expression at every step. 20
outputs is enough to kill the process:

(set g (table [sym price volume] (list (as 'SYM ["a" "b" "c"]) [10 20 30] [1 2 3])))
(select {from: g by: sym a0: (sum price) a1: (sum volume)
         a2: (+ a1 a0) a3: (+ a2 a1) ... a21: (+ a20 a19)})
=== rayforce fatal SIGSEGV at fault addr 0x00007430f89f4b40 ===
rayforce 2.6.2 (bd71c157)
exit 139

dev answers error: name: 'a1' undefined and exits 0. A pure doubling chain
(a(n): (+ a(n-1) a(n-1))) is reported to crash at 12 outputs.

(+ prev prev2) is ordinary analytics code — a running spread, a two-lag
difference — so this is not an adversarial shape. Note the REPL's 4095-byte input
limit is what kept expressions this large unreachable before, which is why no
existing test finds it.

Underneath the size blow-up there is a second bug worth fixing in its own right:
ray_group_build holds ray_op_t* pointers in key_ops/agg_ins across
agg_ins node insertions that realloc g->nodes, so the stale pointer is what
actually faults (src/ops/graph.c:848). Substituting by reference, or bounding
the expansion, would both help — but the stale pointer will outlive any bound.

2. Correct grouped outputs rejected — src/ops/query.c:11244

The new one-value-per-group check does not materialize a lazy result, so
ray_eval returning RAY_LAZY fails both ray_is_vec and RAY_LIST and the
check fires unconditionally:

(select {from: g by: sym f: (deltas (sum price))})
result
dev 3x2 table
this PR error: domain: select by: output 'f' did not evaluate to one value per group

The value is one per group; the diagnosis is wrong. Same for sums,
reverse, asc, not over an aggregate. (raze (deltas (sum price))) works,
which is the tell. The surrounding code already materializes result — the same
ray_lazy_materialize is needed before this shape test.

3. A clear error becomes silently wrong data — src/ops/query.c:10816

(select {from: g a: (* price 2) r: (reverse volume) b: (+ 'a 1)})
b column
dev error: type: cannot add sym and i64
this PR (list [21 41 61] [21 41 61] [21 41 61])

With a = [20 40 60], b should be either an error or [21 41 61]; it is a
LIST of three identical copies. select_fallback_bind_alias makes the alias a
column, but eval_expr_per_row in the fallback does not resolve literal symbols
per row. The compiled path is correct — it is only the eval fallback, which the
reverse forces us into — and the new test covers only the bare-name alias
there, not the quoted form.

Of the three this is the one I would fix first regardless of the others: a wrong
number is worse than a crash, because nothing tells the user.

Two further findings I have not personally reproduced

Reported by the review pass, listed so they are not lost:

  • src/ops/query.c:1760 — the new arith-on-sym rejection fires on an
    unreachable if branch inside a grouped aggregate
    ((sum (if (> price 0) price (+ sym 1)))), because the DAG compiles both arms
    eagerly and the aggregate-argument site has no eval fallback. Regression vs
    dev.
  • src/ops/query.c:6351 — select_alias_skip_form skips fn wholesale, so
    a lambda written inline sees aliases in an ungrouped query but not a grouped
    one. Same query text, different answer depending on whether by: is present.

Cleared

For what it is worth, several things I expected to be problems are not: no
ASan/UBSan errors or .sys.mem growth on the new error paths (the ~64 bytes per
failed query matches dev); the ref accounting in select_fallback_bind_alias
is correct and the source table is never mutated; the sel_alias_* scratch carve
is correctly sized and cleared on every exit, so where:, by: and the sort
keys stay alias-blind. make test TEST_FILTER=rfl is 550/550 green, including
the new issue_617.rfl — the gaps are all outside what it covers.

Happy to re-review as soon as the substitution is bounded (or by-reference) and
2 and 3 are closed. CI is green on all nine checks, which is worth knowing about
the suite rather than about the patch.

ser-vasilich and others added 4 commits September 24, 2026 22:50
`(count (* price 2))` in a grouped select crashed: the count's input node
is not a scan, so it has no ext, and the result-naming step dereferenced
the missing ext for its symbol. A count reads no input column, so the
input is looked up only when there is one and the placeholder name is
used otherwise (the query names the column afterwards anyway).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…if arms

Grouped aliases are no longer substituted by value. An alias of a per-row
expression (`vals: price`) is still substituted where it is named, under
a size cap. An alias that is one value per group — an aggregate, or an
output built on one — is read by reference: an output that names it
outside an aggregate leaves the engine's dict, its aggregate sub-calls
are computed by the engine under hidden names, and it is evaluated over
the group result once grouping is done, whole-column, or row by row when
it holds a control form or a lambda. Hidden columns are dropped, the
outputs keep the dict's order and sort keys and take: that name such an
output apply afterwards. A chain like `a3: (+ a2 a1)` therefore costs one
column per output where copying the expression doubled it at every step
and, past twenty outputs, exhausted the compiler. Two shapes are
rejected with a domain error instead of a wrong answer or an undefined
name: an aggregate over an alias that is already one value per group,
and an output that reads a source column beside such an alias outside
an aggregate (a group key is one value per group and is allowed).

The grouped compile keeps the node ids of its key and aggregate inputs
and re-resolves the pointers before the group node is built; the node
array is reallocated as expressions compile, so a pointer from an
earlier compile can be stale by then.

The one-value-per-group check of a combined output materialises a lazy
result before testing its shape (`(deltas (sum price))` is one value per
group).

In the per-row evaluation a select falls back to, a literal column name
now reads the row's cell as the bare name does; it used to stand for the
whole column, giving every row the same list. The evaluator's literal
rule reads the value through ray_active_query_literal, which knows the
current row.

Arithmetic on a symbol is still a type error, but not inside a branch of
`if` or `cond`: the branches are evaluated element-wise and a branch is
read only where it is selected, so `(if (> price 0) price (+ sym 1))`
compiles as before.

An inline lambda sees earlier aliases in a grouped select as it does in
an ungrouped one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…-617' into fix/select-projection-visibility-617
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Rayforce targeted audit passed

The required Rayforce audit gate passed on the latest run.

Workflow run: https://github.com/RayforceDB/rayforce/actions/runs/36057768212

The row a per-row evaluation publishes for literal column names applied
to whatever query table was active, so a nested select, a where mask or a
per-group evaluation reached from one of the outer select's outputs read
one cell of its own column — the outer row's — instead of the column,
and past that column's length, out of bounds. The cell is now read only
while the active table is the one the row indexes, the row is within the
column and no query scope has been opened since the row was bound
(ray_env_query_scope_above); every nested query path opens one.

The grouped alias planner returns before allocating anything unless an
output mentions the name of an output before it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@singaraiona
singaraiona merged commit 31f45a7 into dev Sep 25, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

select: a projection can no longer refer to an earlier projection of the same select (worked before the DAG compiler)

2 participants