Support for Indexes over `joinedRecordType`s in Relational by g31pranjal · Pull Request #4653 · FoundationDB/fdb-record-layer

g31pranjal · 2026-09-22T22:00:42Z

Stacked on #4737, which adds RecordLayerJoinedSyntheticTable and its metadata serialization. That in turn is stacked on #4706. Review and merge in that order.

This PR adds support for indexes with shapes that warrant the need of JoinedRecordType.

What is supported

Syntax. Six spellings, converging on identical metadata: a comma join with the equality in WHERE, an explicit INNER JOIN ... ON, or INNER JOIN ... USING, each declared with CREATE INDEX ... AS SELECT or CREATE INDEX ... ON <view>. They converge because the join condition is conjoined into the WHERE predicate while the query graph is built.

Shapes. N-way joins, self-joins (the same table under two constituents), columns of one table at non-adjacent key positions, and keyWithValue split points.

Predicates. The equalities relating two constituents become the synthetic table's join conditions. Anything else the definition asks for stays a filter, re-rooted at its constituent and stored as the index predicate, so only the joined records asked for are indexed.

Limitations

Rejected: outer joins; cross product; join with condition not an equality; a filter comparing two of the joined tables (as an unsupported predicate); a join of subqueries; aggregate indexes; version columns (the row version belongs to a stored record, so taking one constituent's would make the index depend on which side it was read through); and unnesting combined with a join.

Some of these are logical extension to the work in this PR, but has been deferred to be either be part of separate work or out of scope.

No planner support. This PR only adds support in the Relational. The query will fail to match the index owing to lack of support in the Cascades.

Implementation notes

RecordLayerJoinedSyntheticTableGenerator follows the same contract as its unnested counterpart — initIfNeeded / checkSupported / rewrite / generate — so MaterializedViewIndexGenerator only decides which kind applies. What it generates is the RecordLayerJoinedSyntheticTable from Add RecordLayerJoinedSyntheticTable and its metadata serialization #4737; this PR only adds the DDL path that builds one. The two kinds are exclusive; only a single-table definition goes on to need an unnested one.
Constituent aliases are derived from the definition (joined_0, joined_1, … in the order the tables were found).
QuantifierValues do not "resolve" a joined constituent's correlation.

Tests

JoinedSyntheticTableIndexTest (new): the four spellings agreeing; non-adjacent columns; three-way and self-joins; split point; residual predicate; an unnested and a joined table coexisting in one template; and one case per rejection above. Every positive case also round-trips its own output through RecordMetaData, which validates, so the metadata is shown to be legal rather than merely well-formed.

IndexTest — three existing tests updated, since joins are no longer rejected outright.

…4706) Preparatory refactor, split out of #4653. `MaterializedViewIndexGenerator` named `RecordLayerUnnestedSyntheticTableGenerator` in four places, so supporting a second kind of synthetic table meant branching on which-of-two at each of them. This introduces: - **`SyntheticTableGenerator`** — what a kind of synthetic table has to provide: `checkSupported`, `rewrite`, `getType`, `generate`. - **`SyntheticTableGeneratorFactory`** — the single place that decides which kind a definition needs. `MaterializedViewIndexGenerator` now holds no reference to any concrete generator. Unnesting is the only kind here, so this is **behaviour-preserving**. There are no test changes, by design: the existing `UnnestedSyntheticTableIndexTest`, `IndexTest`, `DdlStatementParsingTest` and `SchemaTemplateSerDeTests` passing unchanged is the acceptance criterion. #4653 (DDL support for indexes over `JoinedRecordType`s) is stacked on top of this and adds the second implementation.

An index whose columns come from more than one table is defined on a joined synthetic record type, with one inner-joined constituent per table and key expressions rooted at a constituent alias. The equalities relating the tables become the synthetic table's join conditions; any remaining filter is stored as the index predicate and evaluated against the joined record. Builds on the generator interface introduced in the parent commit, adding the second implementation.

robert-brunel · 2026-10-14T17:04:26Z

+        }
+
+        boolean hasSingleSource() {
+            return syntheticTableBuilder() == null || tables.size() == 1;


Was this meant to be inverted?

Suggested change

return syntheticTableBuilder() == null || tables.size() == 1;

return syntheticTableBuilder() != null || tables.size() == 1;

robert-brunel · 2026-10-14T17:05:31Z

-                "Unsupported query, expected to find exactly one type filter operator");
+                "Unsupported index definition, multiscan is not supported");
+        // throws unless there is exactly one source
+        Assert.notNullUnchecked(source.hasSingleSource() , ErrorCode.UNSUPPORTED_OPERATION,


Was this meant to be an assert-that?

Suggested change

Assert.notNullUnchecked(source.hasSingleSource() , ErrorCode.UNSUPPORTED_OPERATION,

Assert.thatUnchecked(source.hasSingleSource(), ErrorCode.UNSUPPORTED_OPERATION,

robert-brunel · 2026-10-14T17:07:57Z

+        @Nonnull
+        private final Map<CorrelationIdentifier, NonnullPair<String, RelationalExpression>> constituents = new HashMap<>();


The CorrelationIdentifier numbering is arbitrary. We probably rather want this map to be in a well-defined order, i.e. FROM-clause order?

robert-brunel · 2026-10-14T17:08:36Z

+                             final boolean hasOuterJoin) {
        this.valuesByQuantifier = valuesByQuantifier;
        this.explodes = explodes;
+        this.constituents = Map.copyOf(constituents);


Suggested change

this.constituents = Map.copyOf(constituents);

this.constituents = ImmutableMap.copyOf(constituents);

robert-brunel · 2026-10-14T17:12:24Z

+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2021-2025 Apple Inc. and the FoundationDB project authors


Suggested change

* Copyright 2021-2025 Apple Inc. and the FoundationDB project authors

* Copyright 2021-2026 Apple Inc. and the FoundationDB project authors

robert-brunel · 2026-10-14T17:12:54Z

+    }
+
+    @Nonnull
+    private AbstractMetadataOperationsFactory assertingFactory(@Nonnull final java.util.function.Consumer<SchemaTemplate> assertions) {


Nit: Import java.util.function.Consumer.

robert-brunel · 2026-10-14T17:13:40Z

-        return RecordLayerUnnestedSyntheticTableGenerator.initIfNeeded(spec, indexName, quantifierValues);
+        final var scanCount = spec.scanCount();
+        final var hasUnnesting = !quantifierValues.getExplodes().isEmpty();
+        if (scanCount != 1 && hasUnnesting) {


Was this also meant to be > 1 here?

Suggested change

if (scanCount != 1 && hasUnnesting) {

if (scanCount > 1 && hasUnnesting) {

robert-brunel · 2026-10-14T17:16:43Z

+        }
+        final var joinConditions = ImmutableList.<JoinCondition>builder();
+        final var residuals = ImmutableList.<QueryPredicate>builder();
+        final var childPredicates = predicate instanceof AndPredicate


Suggested change

final var childPredicates = predicate instanceof AndPredicate

final var childPredicates = predicate instanceof AndPredicate predicate

robert-brunel · 2026-10-14T17:18:40Z

+     * What an index is defined on: the stored tables its definition reads, as collected, or the synthetic record type a
+     * {@link SyntheticTableGenerator} rewrote it onto.
+     */
+    record Source(@Nonnull Map<RecordLayerTable, Integer> tables,


What are the integers for?

robert-brunel · 2026-10-14T17:19:50Z

+
+        Type.Record getRecordType() {
+            return syntheticTableBuilder == null ? Iterables.getOnlyElement(tables.keySet()).getType()
+                                                 : syntheticTableBuilder.build().getType();


Can the syntheticTableBuilder.build().getType() be made more efficient? We’re doing a build() just to get the type.

robert-brunel · 2026-10-14T17:20:56Z

        Assert.thatUnchecked(scanCount == 1, ErrorCode.UNSUPPORTED_OPERATION,
-                "Unsupported index definition, no iteration generator found");
-        // throws unless exactly one type filter was found
-        Assert.notNullUnchecked(table, ErrorCode.UNSUPPORTED_OPERATION,
-                "Unsupported query, expected to find exactly one type filter operator");
+                "Unsupported index definition, multiscan is not supported");


Can scanCount also be 0? Then the multiscan message would be confusing.

robert-brunel · 2026-10-14T17:22:01Z

+    @Nonnull
+    @Override
+    public IndexSpec rewrite(@Nonnull final IndexSpec spec) {
+        Assert.isNullUnchecked(spec.groupBy(), "group by on an index over a joined synthetic table");


Also ErrorCode.UNSUPPORTED_OPERATION here?

robert-brunel · 2026-10-14T17:22:51Z

+        Assert.thatUnchecked(projection.aggregate() == null, ErrorCode.UNSUPPORTED_OPERATION,
+                "Unsupported index definition, an aggregate cannot be defined on a joined synthetic table");


Also enforce projection.groupBy() == null?

g31pranjal added the enhancement New feature or request label Sep 22, 2026

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from 1af4681 to 8be6d28 Compare September 22, 2026 23:28

g31pranjal changed the base branch from main to apple/g31pranjal/sparse-index-synthetic-types September 23, 2026 15:00

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from 8be6d28 to 74e50b5 Compare September 23, 2026 15:00

Base automatically changed from apple/g31pranjal/sparse-index-synthetic-types to main September 24, 2026 13:27

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from 0253487 to 090f325 Compare September 24, 2026 13:31

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from eab11d4 to b52220b Compare October 2, 2026 10:49

g31pranjal mentioned this pull request Oct 2, 2026

Route index generation through a synthetic-table generator interface #4706

Merged

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from b52220b to 3e96455 Compare October 2, 2026 12:56

g31pranjal changed the base branch from main to apple/g31pranjal/synthetic-table-generator-interface October 2, 2026 12:56

g31pranjal force-pushed the apple/g31pranjal/synthetic-table-generator-interface branch from 32befa5 to daff945 Compare October 8, 2026 13:48

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from 3e96455 to c58b57c Compare October 8, 2026 13:48

g31pranjal mentioned this pull request Oct 13, 2026

Add RecordLayerJoinedSyntheticTable and its metadata serialization #4737

Open

g31pranjal changed the base branch from apple/g31pranjal/synthetic-table-generator-interface to apple/g31pranjal/joined-synthetic-table-metadata October 13, 2026 10:20

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from 7bc3d5f to 9c1bb12 Compare October 13, 2026 10:20

g31pranjal requested review from hatyo and robert-brunel October 13, 2026 11:51

g31pranjal marked this pull request as ready for review October 13, 2026 11:51

g31pranjal added 5 commits October 14, 2026 17:09

more refinements

21a7608

WIP: route on scanCount in SyntheticTableGeneratorFactory

0e07ef8

more test cases

d35dfa9

nit

ff6f15a

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-table-metadata branch from 273e837 to 6bb8b65 Compare October 14, 2026 16:27

g31pranjal force-pushed the apple/g31pranjal/joined-synthetic-tables branch from 12b6629 to ff6f15a Compare October 14, 2026 16:27

robert-brunel requested changes Oct 14, 2026

View reviewed changes

	return syntheticTableBuilder() == null \|\| tables.size() == 1;
	return syntheticTableBuilder() != null \|\| tables.size() == 1;

	Assert.notNullUnchecked(source.hasSingleSource() , ErrorCode.UNSUPPORTED_OPERATION,
	Assert.thatUnchecked(source.hasSingleSource(), ErrorCode.UNSUPPORTED_OPERATION,

		@Nonnull
		private final Map<CorrelationIdentifier, NonnullPair<String, RelationalExpression>> constituents = new HashMap<>();

	this.constituents = Map.copyOf(constituents);
	this.constituents = ImmutableMap.copyOf(constituents);

	* Copyright 2021-2025 Apple Inc. and the FoundationDB project authors
	* Copyright 2021-2026 Apple Inc. and the FoundationDB project authors

	if (scanCount != 1 && hasUnnesting) {
	if (scanCount > 1 && hasUnnesting) {

	final var childPredicates = predicate instanceof AndPredicate
	final var childPredicates = predicate instanceof AndPredicate predicate

		Assert.thatUnchecked(projection.aggregate() == null, ErrorCode.UNSUPPORTED_OPERATION,
		"Unsupported index definition, an aggregate cannot be defined on a joined synthetic table");

Conversation

g31pranjal commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is supported

Limitations

Implementation notes

Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-brunel Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

g31pranjal commented Sep 22, 2026 •

edited

Loading

robert-brunel Oct 14, 2026 •

edited

Loading