fix(select): a projection sees the projections before it - #620
Conversation
The DAG compiler compiled every projection against the source table alone, so `nn: (+ notional 1)` after `notional: (* price volume)` was a schema error, and the literal form `(+ 'notional 1)` silently returned the symbol's interned id plus one. Ungrouped: once a projection compiles, its alias is bound in the compile-time env, where a later projection's name — bare or literal — finds it ahead of the source columns. The binding is made after the expression, so an alias that shadows a source column reads the source in its own definition and the new column in every later one. where:, by: and the sort keys compile earlier and stay alias-blind. Grouped: outputs are aggregates over source rows, so the reference is resolved at the expression level — each output is rewritten with the earlier aliases' expressions substituted, and the arith-of-aggs decomposition evaluates the result over the group result. Inside an aggregate's argument a name that is a source column stays the source column (`s: (sum s) mx: (max s)` is the max of the rows); outside, the alias wins as in the ungrouped case. The rewritten dict re-enters the select with the evaluated table as from: and a flag that stops the rewrite from running twice. Arithmetic on a symbol inside a select is a type error, as outside: the DAG compiler declines `+ - * / % div pow` with a SYM operand and the per-row fallback raises arith.c's error. Comparisons and membership on symbols are untouched. Test: test/rfl/regress/issue_617.rfl — the reported shapes, alias chains, shadowing in both positions, where: blindness, literal-symbol and lambda-formal contracts, grouped aliases including the aggregate- argument rule, and the symbol-arithmetic errors. Fixes #617. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…back and per group The select's output aliases now live in their own store on the graph (sel_alias_syms/ids) instead of the 32-slot compile-time env shared with lambda and let inlining: a wide select no longer stops a lambda from inlining (which sent the whole query to per-row evaluation), and the 33rd alias is bound like the first. A name reference consults the env, then the aliases, then the table; a literal symbol consults the aliases before a same-named source column, as the bare name does. A lambda defined outside the select keeps its own free names: the alias store is hidden while its body compiles. Projections evaluated outside the compiled path (a whole-column verb beside them) see the earlier aliases too: each row-shaped output is bound into the table the remaining expressions read, replacing a source column of the same name. Grouped selects: an output whose head is `if` (or a loop / scope form) is evaluated per group instead of once over the group result, where a vector condition picked one branch for every group and the failure to append the output left a hidden aggregate under a made-up name in the result; an output that does not evaluate to one value per group is now reported. An aggregate alias inside another aggregate's argument (`s: (sum price) mx: (max s)`) is a domain error rather than a whole-table value. Arithmetic on a symbol inside a select, update or group key raises the same `type` error everywhere: compile_expr_dag records it on the graph and the sites without an evaluation fallback return it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lect `mx: (max (sum price))` with a by: clause folded the whole table into every group: the inner aggregate is already one value per group, so the outer one has no rows to reduce and was evaluated over the source columns instead. The output is now rejected with a domain error, as the same shape written through an alias already is. An aggregate over a whole-column verb such as `(count (distinct price))` is unaffected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
singaraiona
left a comment
There was a problem hiding this comment.
Blocking — one of these crashes the process and two are silent regressions
against dev. I reproduced the three below myself on bd71c157 (release and
ASan/UBSan), against an origin/dev build of the same tree, so these are not
static-analysis guesses.
The direction of the PR is right and #617 is a real gap. It is the substitution
mechanism that needs rework, not the goal.
1. SIGSEGV on a 476-byte query — src/ops/query.c:6404
select_alias_subst substitutes each earlier expression by value, so an
output naming two earlier aliases doubles the expression at every step. 20
outputs is enough to kill the process:
(set g (table [sym price volume] (list (as 'SYM ["a" "b" "c"]) [10 20 30] [1 2 3])))
(select {from: g by: sym a0: (sum price) a1: (sum volume)
a2: (+ a1 a0) a3: (+ a2 a1) ... a21: (+ a20 a19)})=== rayforce fatal SIGSEGV at fault addr 0x00007430f89f4b40 ===
rayforce 2.6.2 (bd71c157)
exit 139
dev answers error: name: 'a1' undefined and exits 0. A pure doubling chain
(a(n): (+ a(n-1) a(n-1))) is reported to crash at 12 outputs.
(+ prev prev2) is ordinary analytics code — a running spread, a two-lag
difference — so this is not an adversarial shape. Note the REPL's 4095-byte input
limit is what kept expressions this large unreachable before, which is why no
existing test finds it.
Underneath the size blow-up there is a second bug worth fixing in its own right:
ray_group_build holds ray_op_t* pointers in key_ops/agg_ins across
agg_ins node insertions that realloc g->nodes, so the stale pointer is what
actually faults (src/ops/graph.c:848). Substituting by reference, or bounding
the expansion, would both help — but the stale pointer will outlive any bound.
2. Correct grouped outputs rejected — src/ops/query.c:11244
The new one-value-per-group check does not materialize a lazy result, so
ray_eval returning RAY_LAZY fails both ray_is_vec and RAY_LIST and the
check fires unconditionally:
(select {from: g by: sym f: (deltas (sum price))})| result | |
|---|---|
dev |
3x2 table |
| this PR | error: domain: select by: output 'f' did not evaluate to one value per group |
The value is one per group; the diagnosis is wrong. Same for sums,
reverse, asc, not over an aggregate. (raze (deltas (sum price))) works,
which is the tell. The surrounding code already materializes result — the same
ray_lazy_materialize is needed before this shape test.
3. A clear error becomes silently wrong data — src/ops/query.c:10816
(select {from: g a: (* price 2) r: (reverse volume) b: (+ 'a 1)})b column |
|
|---|---|
dev |
error: type: cannot add sym and i64 |
| this PR | (list [21 41 61] [21 41 61] [21 41 61]) |
With a = [20 40 60], b should be either an error or [21 41 61]; it is a
LIST of three identical copies. select_fallback_bind_alias makes the alias a
column, but eval_expr_per_row in the fallback does not resolve literal symbols
per row. The compiled path is correct — it is only the eval fallback, which the
reverse forces us into — and the new test covers only the bare-name alias
there, not the quoted form.
Of the three this is the one I would fix first regardless of the others: a wrong
number is worse than a crash, because nothing tells the user.
Two further findings I have not personally reproduced
Reported by the review pass, listed so they are not lost:
src/ops/query.c:1760— the new arith-on-sym rejection fires on an
unreachableifbranch inside a grouped aggregate
((sum (if (> price 0) price (+ sym 1)))), because the DAG compiles both arms
eagerly and the aggregate-argument site has no eval fallback. Regression vs
dev.src/ops/query.c:6351—select_alias_skip_formskipsfnwholesale, so
a lambda written inline sees aliases in an ungrouped query but not a grouped
one. Same query text, different answer depending on whetherby:is present.
Cleared
For what it is worth, several things I expected to be problems are not: no
ASan/UBSan errors or .sys.mem growth on the new error paths (the ~64 bytes per
failed query matches dev); the ref accounting in select_fallback_bind_alias
is correct and the source table is never mutated; the sel_alias_* scratch carve
is correctly sized and cleared on every exit, so where:, by: and the sort
keys stay alias-blind. make test TEST_FILTER=rfl is 550/550 green, including
the new issue_617.rfl — the gaps are all outside what it covers.
Happy to re-review as soon as the substitution is bounded (or by-reference) and
2 and 3 are closed. CI is green on all nine checks, which is worth knowing about
the suite rather than about the patch.
`(count (* price 2))` in a grouped select crashed: the count's input node is not a scan, so it has no ext, and the result-naming step dereferenced the missing ext for its symbol. A count reads no input column, so the input is looked up only when there is one and the placeholder name is used otherwise (the query names the column afterwards anyway). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…if arms Grouped aliases are no longer substituted by value. An alias of a per-row expression (`vals: price`) is still substituted where it is named, under a size cap. An alias that is one value per group — an aggregate, or an output built on one — is read by reference: an output that names it outside an aggregate leaves the engine's dict, its aggregate sub-calls are computed by the engine under hidden names, and it is evaluated over the group result once grouping is done, whole-column, or row by row when it holds a control form or a lambda. Hidden columns are dropped, the outputs keep the dict's order and sort keys and take: that name such an output apply afterwards. A chain like `a3: (+ a2 a1)` therefore costs one column per output where copying the expression doubled it at every step and, past twenty outputs, exhausted the compiler. Two shapes are rejected with a domain error instead of a wrong answer or an undefined name: an aggregate over an alias that is already one value per group, and an output that reads a source column beside such an alias outside an aggregate (a group key is one value per group and is allowed). The grouped compile keeps the node ids of its key and aggregate inputs and re-resolves the pointers before the group node is built; the node array is reallocated as expressions compile, so a pointer from an earlier compile can be stale by then. The one-value-per-group check of a combined output materialises a lazy result before testing its shape (`(deltas (sum price))` is one value per group). In the per-row evaluation a select falls back to, a literal column name now reads the row's cell as the bare name does; it used to stand for the whole column, giving every row the same list. The evaluator's literal rule reads the value through ray_active_query_literal, which knows the current row. Arithmetic on a symbol is still a type error, but not inside a branch of `if` or `cond`: the branches are evaluated element-wise and a branch is read only where it is selected, so `(if (> price 0) price (+ sym 1))` compiles as before. An inline lambda sees earlier aliases in a grouped select as it does in an ungrouped one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…-617' into fix/select-projection-visibility-617
Rayforce targeted audit passedThe required Rayforce audit gate passed on the latest run. Workflow run: https://github.com/RayforceDB/rayforce/actions/runs/36057768212 |
The row a per-row evaluation publishes for literal column names applied to whatever query table was active, so a nested select, a where mask or a per-group evaluation reached from one of the outer select's outputs read one cell of its own column — the outer row's — instead of the column, and past that column's length, out of bounds. The cell is now read only while the active table is the one the row indexes, the row is within the column and no query scope has been opened since the row was bound (ray_env_query_scope_above); every nested query path opens one. The grouped alias planner returns before allocating anything unless an output mentions the name of an output before it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Closes #617.
Bug
A projection in
selectcould not reference a projection defined before it in the same query.nn: (+ notional 1)afternotional: (* price volume)failed with'notional' undefined; the literal form(+ 'notional 1)was compiled as arithmetic on the symbol's interned id and returned a number. In a grouped select the same reference failed the same way.Two related defects in the grouped path surfaced while fixing this: an output whose head is
ifover an aggregate (f: (if (> (count price) 1) 1 0)) was evaluated once over the whole group result, so one branch was picked for every group and, when the output could not be appended, the hidden count was left in the result under a made-up name; and an aggregate of an aggregate (mx: (max (sum price))) folded the whole table into every group.Fix
Each projection is bound under its alias once its expression has compiled, in an alias store on the graph (
sel_alias_syms/ids, separate from the 32-slot env used for lambda/let inlining, so a wide select does not stop lambdas from inlining). A name reference resolves through the env, then the aliases, then the table columns; a literal symbol resolves through the aliases, then the columns, and otherwise stays a constant. The binding is made after the expression, soprice: (* price 2) p2: (+ price 1)reads the source inpriceand the doubled value inp2.where:andby:are compiled outside the window and stay alias-blind. A lambda defined outside the select keeps its own free names: the alias store is hidden while its body compiles.Projections that are evaluated outside the compiled path (a whole-column verb such as
reversebeside them) see earlier aliases too: each row-shaped output is bound into the table the remaining expressions read.In a grouped select the alias is resolved at the expression level:
select_resolve_grouped_aliasessubstitutes the earlier expression into later outputs before compilation. Inside an aggregate's argument a name that is a source column stays the source column (s: (sum s) mx: (max s)keeps its meaning); an aggregate alias inside another aggregate, and an aggregate of an aggregate written out, raise adomainerror. Outputs headed byif(and by the loop and scope forms) are evaluated per group rather than once over the group result, and an output that does not evaluate to one value per group is reported.Arithmetic on a symbol inside a select, update or group key raises the
typeerror it raises outside:compile_expr_dagrecords it on the graph and the sites without an evaluation fallback return it.Tests
test/rfl/regress/issue_617.rfl: the reported query, alias chains, shadowing (bare and literal), the fallback path, a 35-projection select with a five-formal lambda, grouped aliases (bare, literal, chained,ifover an alias,try), the in-aggregate rule, nested aggregates, symbol arithmetic in select/update/where, and guards for the behaviour that must not change. Documentation:docs/docs/queries/select.md, section "Projections that build on earlier projections".