Repository navigation
Row-level expressions in every column position: time dimensions, partition keys, ranking keys, parameters - #512
Open
Conversation
…-parser-sql-forms' into egor/dev-2119-row-level-expressions-in-every-column-position-time
…-parser-sql-forms' into egor/dev-2119-row-level-expressions-in-every-column-position-time
…position New tests: expression time dimensions (forms, names, keys, every time position, per-leaf re-bucketing), granularity calls over expressions, the spine refusal, expression partition keys, expression ranking keys / values / parameters, the ranked kernel on Tier-1 dialects, and the stored object-to-text upgrade. Retargeted tests pinning refusals this change lifts: - test_time_points_exec.py: month(upper(code)) -> date-operand type error - test_dev1883_errors.py: month(upper(x)) -> binding-time temporal error - test_dev2015_time_dimensions.py: fiscal_year(upper(id)) -> temporal error - test_dev1903_first_last_dispatch.py: first(expr) / first(1) bind to the ranked aggregate - test_dev1832_cross_model_exec.py: first over an expression emits a ranked aggregate - test_dev1832_transform_source.py, test_dev1859_plan_structure.py: first over a mixed source (explicit ranking key id) emits a ranked aggregate - test_dev1953_partition_alias.py: expression partition key equal to an alias binds to it; aggregate keys name the computed-dimension remedy
Move the exact-repeat extension-join dedupe from the query-backed-only planner branch into apply_extension_overlay, keyed on a semantic edge identity (target, name, join type, cardinality, pair set), and mirror it in the schema-drift stage graph. Attach saved joins after the stage-model rename with an explicit key check, route the saved-joins lookup through one bundle helper, and shorten the models-guide addition to one sentence.
Replace design decision 5's saved-relation-only workaround with the one cross-model rule, and add the models/join-traversal delta for it.
DuckDB reports an ETag change across CDN edges as 'this likely means the remote file has changed'; classify it as a transient outage of the named remote host so the notebook skips instead of failing.
Preserve declared joins on saved query-backed models
chore: bump version to 1.4.1
…e-bucketing, spine refusal
…and aggregation parameters; stored time dimensions migrate to text; docs
…erred to DEV-2136
…expression refusals
…pression is never an axis
…-custom-parser-sql-forms' into egor/dev-2119-row-level-expressions-in-every-column-position-time # Conflicts: # slayer/core/refs.py # slayer/engine/binding.py
…n-every-column-position-time
Contributor
|
Important Review skippedToo many files! This PR contains 222 files, which is 72 over the limit of 150. To get a review, reduce the PR to 150 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to Team to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
 |
…like a computed dimension's; unnamed ones are named from the template
…n date range and label dedupe at binding
…/ join-path helpers out of two over-complex functions (Sonar S3776)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves DEV-2119. Stacked on #509 (DEV-2101, the Mode-B grammar); GitHub retargets this to
mainonce that merges.What and why
Several query positions took only a column, so bucketing, partitioning, ranking or weighting by a computed value needed a query-time
ModelExtensioncolumn. DEV-2021 is going to remove those extension columns, so this PR first makes every column position take any row-level Mode-B expression of the right type. A column is just the one-leaf case:semantics.arc42.mdAxiom 13 now says so, and Axiom 15 that an expression is never a spine axis.What users can now write
Time dimensions over an expression, also in the functional form:
{"source_model": "orders", "time_dimensions": [{"dimension": "coalesce(shipped_at, created_at)", "granularity": "month", "date_range": ["2025-01-01", "2025-03-31"], "name": "effective_at"}], "measures": ["count(*)", "sum(amount, window='3m')", "change(sum(amount))"]}returns one row per month of the per-order coalesced timestamp, keyed
orders.effective_at(unnamed:orders.coalesce_shipped_at_created_at).date_range,whole_periods_only(date bounds for a DATE expression, instant bounds for a TIMESTAMP one),window=, time-ordered transforms andfirst/lastall read the per-row value, and the results equal those of a derived column holding the same expression."month(CAST(created_txt AS DATE))"works indimensions,time_dimensionsandorder, and a granularity call takes any row-level argument in a filter too.Also accepted now:
sum(amount, partition_by=upper(city))— a key equal to a query computed dimension or a time dimension's bucket is that dimension; any other aggregate-bearing key points at the computed-dimension remedy.last(balance, coalesce(updated_at, created_at))andfirst(amount - customers.discount)— expression ranking keys and values, mixed sources included, rendered by the ranked kernel on every Tier-1 dialect.weighted_avg(price, weight=quantity * 2),corr(amount, quantity + 1), and custom-aggregation parameters as expressions.{var}literals inside a time-dimension expression, e.g.month(coalesce(shipped_at, created_at + INTERVAL '{days}' DAY)); an unnamed one is keyed from the template, so the result key never depends on the value.coalesce(shipped_at, created_at)andCOALESCE(shipped_at,created_at)) that agree in name, date range and label project once.Fails closed, with typed errors raised before any SQL is generated:
CAST(…)remedy);Breaking and result changes
TimeDimension.dimensionis Mode-B text, sotd.dimension.name/.modelare gone. It dumps as a string, and the JSON schema advertises a string. The legacy{"name", "model"}object is still accepted on input, and stored queries, models and memories migrate on load (SlayerQuery v8, SlayerModel v17, Memory v6, stored-only).first/lastwithout a ranking key now ranks by the query's first DATE/TIMESTAMP dimension, a temporal computed dimension included (previously only a column qualified). In a producer with a coarserpartition_by=the default stays producer-local; DEV-2136 tracks reading the query's dimensions there.Also fixed
The source-model prefix strip used to rewrite an inner hop: rooted at
orders,count(customers.orders.id)silently becamecount(customers.id). It now strips only a leading qualifier, so the revisit is refused with the circular-path error.The full spec surface is in
openspec/changes/dev-2119-row-level-expressions-in-every-column-position-time/.