perf: reuse columns from process_schema_changes in incremental materialization

[moomindani]

Summary

Closes #1411.

The incremental materialization currently discards the return value of
process_schema_changes, so each downstream strategy macro (merge,
append, delete+insert) re-issues another DESCRIBE TABLE EXTENDED
on the target relation even though check_for_schema_changes has just
DESCRIBEd it. This PR reuses those columns, eliminating one metadata
round-trip per incremental model, per run.

Changes

• dbt/include/databricks/macros/materializations/incremental/incremental.sql
  • Capture columns from process_schema_changes in both V1 and V2 paths.
  • When on_schema_change == 'ignore' (returns {}), fall back to a
    single adapter.get_columns_in_relation(existing_relation).
  • Thread the result through strategy_arg_dict['dest_columns']
    (previously hard-coded to none).
  • Extend get_build_sql with a dest_columns=none parameter so the
    V2 path can pass through.
• dbt/include/databricks/macros/materializations/incremental/strategies.sql
  • databricks__get_merge_sql: only DESCRIBE the target when
    dest_columns is none.
  • get_delete_insert_sql: honor arg_dict['dest_columns'] when set.
  • get_insert_into_sql: accept a dest_columns=none parameter and
    honor it; databricks__get_incremental_append_sql now passes
    arg_dict['dest_columns'] through.
• CHANGELOG.md: new entry under ## dbt-databricks next → Under the Hood.

Behavior

• When on_schema_change is 'fail', 'sync_all_columns', or
  'append_new_columns': process_schema_changes already DESCRIBEd
  both relations, so we reuse its result — one fewer DESCRIBE.
• When on_schema_change == 'ignore': we issue exactly one DESCRIBE on
  the existing relation, matching today's total count for that path.
• Existing public macro signatures are preserved. get_build_sql gains
  an optional keyword argument that defaults to none.

Test plan

Manually verified on a live Databricks SQL Warehouse with a project of
9 incremental stg models (on_schema_change: 'fail', 7 merge + 2
append strategies).

Target DESCRIBE TABLE EXTENDED … AS JSON count per incremental model:

Path	Before	After
V1 (use_materialization_v2: false)	2	1
V2 (use_materialization_v2: true)	2	1

Wall-clock impact on a full dbt run is within measurement noise at
this scale (9 small models, 16 threads); the saved round-trips get
absorbed by parallelism. The win here is fewer metadata round-trips
(lower warehouse load, less API traffic), not a dramatic wall-clock
speedup.

• Ruff lint clean on changed files.

[moomindani]

Hi @sd-db @tejassp-db — rebased onto latest main (2c3aa9fd), CHANGELOG conflict resolved, mergeable again.

I also pushed a follow-up fix (925d98bf) after re-checking the column reuse against every dest_columns consumer. Summary, in case it saves you the trace:

• process_schema_changes returns the source columns, so the reused list can omit a column the target still has under on_schema_change: append_new_columns.
• For merge that turns out to be harmless: dest_columns only reaches get_merge_update_columns, and the update/insert lists are built from a separate DESCRIBE of the source.
• For append it was not. insert_into_sql_impl branches on whether the dest and source column sets match, and reusing the source columns makes them match by construction. On DBR 12.2+ that yields insert into ... by name select *, which is correct — BY NAME fills unmatched target columns with their DEFAULT. On DBR < 12.2 there is no BY NAME, so the same branch degrades to a positional select * and drops the explicit column list that previously kept a target-only column aligned. The follow-up commit describes the target in that case instead.

Three macro tests pin the generated SQL for all three cases. hatch run unit: 1346 passed, 4 skipped; code-quality clean.

CI has not run on this PR either — the workflows are at action_required because it comes from a fork. Could one of you approve the run? /integration-test too if you would like the integration suite.

Open since April with review requested, so any feedback is welcome.

[moomindani]

One public reference I should have included with the previous comment: the column-matching failure the follow-up commit prevents is the same class as #1289 — a positional INSERT on a path where the columns need to be matched by name. That one was fixed for replace_where and microbatch in #1348, released in 1.11.7, by emitting INSERT ... BY NAME when the runtime supports it.

append differs in one respect: it can fall back to an explicit column list when BY NAME is unavailable, which replace_where cannot — there is no column list in INSERT ... TABLE <view>. So the guard keeps the explicit list on DBR < 12.2 instead of accepting a positional insert. Same intent as #1348, using the option append happens to have.

On scope: this PR only reuses columns the materialization has already fetched. It issues no new metadata call, and the number of calls is reduced or unchanged on every path — including on_schema_change: ignore, which still describes the existing relation exactly once and behaves as it does today.