diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b1c648c2..772cf0759 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ### Fixes +- Let Predictive Optimization manage models with `auto_liquid_cluster` instead of running synchronous `OPTIMIZE` after materialization ([#1658](https://github.com/databricks/dbt-databricks/pull/1658) resolves [#1655](https://github.com/databricks/dbt-databricks/issues/1655)) - Skip unnecessary Unity Catalog constraint metadata queries for incremental models without enforced contracts ([#1643](https://github.com/databricks/dbt-databricks/pull/1643) resolves [#1641](https://github.com/databricks/dbt-databricks/issues/1641)) - Recreate materialized views when query schema drifts, honoring `on_configuration_change` ([#1621](https://github.com/databricks/dbt-databricks/pull/1621) resolves [#1359](https://github.com/databricks/dbt-databricks/issues/1359)) - Warn when documented columns are missing from V1 and V2 table and incremental models, and honor `persist_docs.columns` for V2 column comments ([#1563](https://github.com/databricks/dbt-databricks/pull/1563) ports [dbt-adapters#1684](https://github.com/dbt-labs/dbt-adapters/pull/1684), resolving [dbt-adapters#1690](https://github.com/dbt-labs/dbt-adapters/issues/1690)). diff --git a/dbt/include/databricks/macros/relations/optimize.sql b/dbt/include/databricks/macros/relations/optimize.sql index 318c41217..054e3b7a5 100644 --- a/dbt/include/databricks/macros/relations/optimize.sql +++ b/dbt/include/databricks/macros/relations/optimize.sql @@ -7,7 +7,7 @@ {%- elif var('DATABRICKS_SKIP_OPTIMIZE', 'false')|lower != 'true' and var('databricks_skip_optimize', 'false')|lower != 'true' and adapter.resolve_file_format(config) == 'delta' -%} - {%- if (config.get('zorder', False) or config.get('liquid_clustered_by', False)) or config.get('auto_liquid_cluster', False) -%} + {%- if config.get('zorder', False) or config.get('liquid_clustered_by', False) -%} {%- call statement('run_optimize_stmt') -%} {{ get_optimize_sql(relation) }} {%- endcall -%} diff --git a/docs/flow/incremental_flow.md b/docs/flow/incremental_flow.md index 25ddf4518..f009b7aea 100644 --- a/docs/flow/incremental_flow.md +++ b/docs/flow/incremental_flow.md @@ -1,6 +1,6 @@ # Incremental Flow -_Last updated: 2026-08-20_ +_Last updated: 2026-08-31_ > Two diagrams follow: **Existing** is the default path, **New** is used when the > `use_materialization_v2` behavior flag is enabled. See [flow/README.md](README.md) for what the @@ -40,7 +40,7 @@ flowchart LR NEWCFG --> GRANTS[Apply grants] REPLACEDOCS --> GRANTS DOCS --> GRANTS - GRANTS --> OPT[Run optimize] + GRANTS --> OPT[Run optimize for zorder / liquid_clustered_by] OPT --> POST[Run post-hooks] POST --> STATIC[Restore static overwrite mode for non-full-refresh insert_overwrite] ``` @@ -78,7 +78,7 @@ flowchart LR CREATE --> GRANTS[Apply grants] SAFE --> GRANTS MERGE --> GRANTS - GRANTS --> OPT[Run optimize] + GRANTS --> OPT[Run optimize for zorder / liquid_clustered_by] OPT --> PYCLEAN{Python model?} PYCLEAN -- yes --> CLEAN[Drop intermediate relation] PYCLEAN -- no --> POST[Run post-hooks] diff --git a/docs/flow/snapshot_flow.md b/docs/flow/snapshot_flow.md index d66be58bd..fe3e549f4 100644 --- a/docs/flow/snapshot_flow.md +++ b/docs/flow/snapshot_flow.md @@ -1,6 +1,6 @@ # Snapshot Flow -_Last updated: 2026-08-10_ +_Last updated: 2026-08-31_ > Snapshots do **not** use the `use_materialization_v2` flag — there is a single path. Source: > `dbt/include/databricks/macros/materializations/snapshot.sql`. Strategy dispatch and the @@ -46,7 +46,7 @@ flowchart TD CLEAN -- yes --> POSTSNAP[post_snapshot cleanup] CLEAN -- no --> CONST POSTSNAP --> CONST[Persist constraints] - CONST --> OPT[Run optimize] + CONST --> OPT[Run optimize for zorder / liquid_clustered_by] OPT --> POSTOUT["Run post-hooks (outside transaction)"] ``` diff --git a/docs/flow/table_flow.md b/docs/flow/table_flow.md index f8599824e..57f453222 100644 --- a/docs/flow/table_flow.md +++ b/docs/flow/table_flow.md @@ -1,6 +1,6 @@ # Table Flow -_Last updated: 2026-08-10_ +_Last updated: 2026-08-31_ > Two diagrams follow: **V1** is the default path, **V2** is used when the `use_materialization_v2` > behavior flag is enabled. See [flow/README.md](README.md) for what the flag is and how the @@ -27,7 +27,7 @@ flowchart LR TAGS --> COLTAGS[Apply column tags] COLTAGS --> DOCS[Persist docs] DOCS --> CONSTRAINTS[Persist constraints] - CONSTRAINTS --> OPT[Run optimize] + CONSTRAINTS --> OPT[Run optimize for zorder / liquid_clustered_by] OPT --> POST[Run post-hooks] ``` @@ -53,7 +53,7 @@ flowchart LR CREATE --> GRANTS[Apply grants] SAFE --> GRANTS - GRANTS --> OPT[Run optimize] + GRANTS --> OPT[Run optimize for zorder / liquid_clustered_by] OPT --> PYCLEAN{Python model?} PYCLEAN -- yes --> CLEAN[Drop intermediate relation] PYCLEAN -- no --> POST[Run post-hooks] diff --git a/tests/unit/macros/relations/test_optimize_macros.py b/tests/unit/macros/relations/test_optimize_macros.py index efcf5032e..cb579ecb2 100644 --- a/tests/unit/macros/relations/test_optimize_macros.py +++ b/tests/unit/macros/relations/test_optimize_macros.py @@ -35,8 +35,16 @@ def test_macros_optimize_with_extraneous_info(self, config, var, template_bundle assert result == "run_optimize_stmt" + def test_macros_optimize_skips_auto_liquid_clustering(self, config, template_bundle): + config["auto_liquid_cluster"] = True + + result = self.render_bundle(template_bundle, "optimize") + + assert result == "" + @pytest.mark.parametrize("key_val", ["DATABRICKS_SKIP_OPTIMIZE", "databricks_skip_optimize"]) - def test_macros_optimize_with_skip(self, key_val, var, template_bundle): + def test_macros_optimize_with_skip(self, key_val, config, var, template_bundle): + config["liquid_clustered_by"] = ["foo"] var[key_val] = True r = self.render_bundle(template_bundle, "optimize")