-
Notifications
You must be signed in to change notification settings - Fork 33
Clarify optimizer-controlled predicate ordering in FTS guidance #266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Vsevolod Kukol (sevoku)
wants to merge
2
commits into
AzureCosmosDB:main
Choose a base branch
from
sevoku:dev/sevoku/clarify-fts-predicate-order
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
90 changes: 32 additions & 58 deletions
90
skills/cosmosdb-best-practices/rules/query-order-filters.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,77 +1,51 @@ | ||
| --- | ||
| title: Order Filters by Selectivity | ||
| title: Let the Query Engine Order Filters | ||
|
sevoku marked this conversation as resolved.
|
||
| impact: MEDIUM | ||
| impactDescription: reduces intermediate result sets | ||
| impactDescription: avoids ineffective predicate-order tuning and incorrect execution assumptions | ||
| tags: query, filters, optimization, performance | ||
| --- | ||
|
|
||
| ## Order Filters by Selectivity | ||
| ## Let the Query Engine Order Filters | ||
|
|
||
| Place most selective filters first in WHERE clauses. The query engine processes filters left-to-right, so selective filters early reduce data scanned. | ||
| The textual order of equivalent predicates in a `WHERE` clause does not determine their execution order. Cosmos DB's query engine determines which predicates are more selective and how to execute the query. Moving a selective predicate earlier in the SQL text is not a performance optimization. | ||
|
|
||
| **Incorrect (least selective filter first):** | ||
| **Incorrect (assuming lower RU cost solely from reordering equivalent predicates):** | ||
|
|
||
| ```csharp | ||
| // Status has low selectivity (few unique values) | ||
| // Filters 1M items to 300K, then to 100 | ||
| var query = @" | ||
| SELECT * FROM c | ||
| WHERE c.status = 'active' -- 30% of items match | ||
| AND c.type = 'order' -- 10% of items match | ||
| AND c.customerId = @customerId"; -- 0.01% match (highly selective) | ||
|
|
||
| // Processes: 1M → 300K → 100K → 100 | ||
| // More intermediate processing than necessary | ||
| var originalQuery = @" | ||
| SELECT * FROM c | ||
| WHERE c.status = 'active' | ||
| AND c.type = 'order' | ||
| AND c.customerId = @customerId"; | ||
|
|
||
| var reorderedQuery = @" | ||
| SELECT * FROM c | ||
| WHERE c.customerId = @customerId | ||
| AND c.type = 'order' | ||
| AND c.status = 'active'"; | ||
| ``` | ||
|
|
||
| **Correct (most selective filter first):** | ||
| Both queries are valid and express the same filters. The mistake is claiming that `reorderedQuery` is cheaper merely because `customerId` appears first, or assigning intermediate row counts based on the clauses' textual positions. | ||
|
|
||
| ```csharp | ||
| // CustomerId is highly selective (unique per customer) | ||
| var query = @" | ||
| SELECT * FROM c | ||
| WHERE c.customerId = @customerId -- 0.01% match (filter first!) | ||
| AND c.type = 'order' -- Then narrow by type | ||
| AND c.status = 'active'"; -- Finally by status | ||
|
|
||
| // Processes: 1M → 1K → 100 → 100 | ||
| // Much less intermediate data | ||
| ``` | ||
| **Correct (use readable predicates and optimize actual index usage):** | ||
|
|
||
| ```csharp | ||
| // Selectivity guidelines (from most to least selective): | ||
| // 1. Unique identifiers: id, customerId, orderId (highest) | ||
| // 2. Foreign keys with many values: productId, userId | ||
| // 3. Timestamps (range queries): createdAt, modifiedAt | ||
| // 4. Categories with many values: categoryId, departmentId | ||
| // 5. Status fields: status, state (low selectivity) | ||
| // 6. Boolean flags: isActive, isDeleted (lowest - only 2 values) | ||
|
|
||
| // Example: Combining timestamp with status | ||
| var query = @" | ||
| SELECT * FROM c | ||
| var query = new QueryDefinition(@" | ||
| SELECT * FROM c | ||
| WHERE c.customerId = @customerId | ||
| AND c.orderDate >= @startDate | ||
| AND c.orderDate < @endDate | ||
| AND c.status = 'completed'"; | ||
|
|
||
| // Even better with composite index | ||
| AND c.type = 'order' | ||
| AND c.status = 'active'") | ||
| .WithParameter("@customerId", customerId); | ||
| ``` | ||
|
|
||
| ```csharp | ||
| // Use BETWEEN with high selectivity values | ||
| var query = @" | ||
| SELECT * FROM c | ||
| WHERE c.orderId >= @startId AND c.orderId <= @endId -- Very selective range | ||
| AND c.status = 'active'"; | ||
| The order above is for readability, not an execution hint. To tune performance, inspect index usage and measured request charges/query metrics rather than assuming that swapping the same predicates reduces work. | ||
|
|
||
| // For OR clauses, check if rewriting helps | ||
| // Less efficient: | ||
| var query1 = "SELECT * FROM c WHERE c.status = 'a' OR c.status = 'b' AND c.customerId = @id"; | ||
| // Better (explicit grouping): | ||
| var query2 = "SELECT * FROM c WHERE (c.status = 'a' OR c.status = 'b') AND c.customerId = @id"; | ||
| // Best (if possible, use IN): | ||
| var query3 = "SELECT * FROM c WHERE c.status IN ('a', 'b') AND c.customerId = @id"; | ||
| ``` | ||
| **Key points:** | ||
|
|
||
| - Selectivity depends on the data distribution, not just the property name or type. | ||
| - Add useful filters with appropriate index support; a partition-key equality filter can narrow the query's partition scope regardless of where it appears in the `WHERE` clause. | ||
| - Preserve Boolean grouping when rewriting queries. Adding parentheses around `OR` predicates can change which documents match; it is not merely a performance rewrite. | ||
|
|
||
| See also: [Avoid cross-partition queries](query-avoid-cross-partition.md), [avoid full scans](query-avoid-scans.md), [combine FTS with indexed filters](fts-hybrid-queries.md). | ||
|
|
||
| Reference: [Query optimization tips](https://learn.microsoft.com/azure/cosmos-db/nosql/performance-tips-query-sdk) | ||
| Reference: [Index usage and filter-clause ordering](https://learn.microsoft.com/azure/cosmos-db/index-overview#composite-indexes) | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.