chore: raise dbt-spark upper bound to <1.12.0 - #1673
Open
moomindani wants to merge 3 commits into
Open
Conversation
Picks up dbt-spark 1.11.0. Its three changes over 1.10.1 do not reach this adapter: `spark__persist_docs` and `list_relations_without_caching` are both overridden here, and the Thrift/SASL connection changes belong to a path this adapter does not use. The lowest-direct lock is unchanged. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
moomindani
requested review from
jprakash-db,
saishreeeee and
sd-db
as code owners
September 9, 2026 13:36
Co-authored-by: Isaac <no-reply@databricks.com>
4 tasks
sd-db
added a commit
that referenced
this pull request
Sep 10, 2026
### Description
`scripts/check_uv_lock_public_urls.py --fix` normalizes the `source = {
registry = ... }` entries but not the artifact `url` entries, and the
check it runs immediately afterwards requires both. On a lock produced
against a mirrored index that means `--fix` rewrites 114 registries,
leaves 1512 package URLs, and then reports failure — so the URLs have to
be substituted by hand before the pre-commit hook will pass.
This came up while raising four dependency bounds (#1670, #1671, #1672,
#1673): every one of them needed the same manual substitution after `uv
lock`.
`--fix` now rewrites the host of URLs shaped like the public artifact
tree. Only the host changes: the `/packages/` path identifies the
artifact and the hash recorded beside it is what guarantees integrity,
so the rewrite cannot silently point at different content. A URL without
a `/packages/` path is not a mirror of that tree, so rewriting its host
would be a guess — those are left for the check to report.
The `--fix` summary line now reports both counts, so `fix_uv_lock`
returns a `(registry_count, url_count)` pair.
### Testing
New `tests/unit/test_check_uv_lock_public_urls.py` (6 tests) covers the
check on a public and a mirrored lock, the fix on both, path and hash
preservation, and the non-mirror URL left alone. Three of them fail
against the current script.
End to end on this repo: regenerating `uv.lock` against a mirrored index
and running `--fix` alone now produces a file byte-identical to the
committed `uv.lock`, and `uv lock --check` agrees.
Unit suite 1349 passed, 4 skipped; `code-quality` clean.
### Checklist
- [x] I have run this code in development and it appears to resolve the
stated issue
- [x] This PR includes tests, or tests are not required/relevant for
this PR
- [x] I have updated the `CHANGELOG.md` and added information about my
change to the "dbt-databricks next" section.
- [ ] [Optional] I have run the `dbt-databricks-pr-ready` project skill
for this PR and addressed its merge-readiness feedback
---------
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Shubham Dhal <shubham.dhal@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The current ceiling
<1.11.0excludes dbt-spark 1.11.0, so the pinned version stayed at 1.10.1. This raises the bound and moves the lock to 1.11.0.requirements.lowest-direct.txtis unchanged.Since this adapter subclasses
SparkAdapter, I diffed 1.10.1 against 1.11.0. There are three changes and none reaches this adapter:spark__persist_docsnow filters columns throughvalidate_doc_columns— overridden here asdatabricks__persist_docs.SparkAdapter.list_relations_without_cachingbroadened its schema-not-found handling — overridden here.connections.pyswitched the SASL client import and passes an SSL context toTSSLSocket— the Thrift path, which this adapter does not use.Testing
Per bump: unit suite 1343 passed, 4 skipped;
code-qualityclean;uv lock --checkconsistent.Functional coverage was run against a serverless SQL warehouse with all four dependency bumps applied together:
tests/functional/adapter/incremental/andtests/functional/adapter/iceberg/gave 118 passed, 6 skipped, and 3 failures (TestAppendParquet, and the two column-tag tests) that reproduce identically onmainwith no bumps.Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.dbt-databricks-pr-readyproject skill for this PR and addressed its merge-readiness feedback