Conversation
`_BaseExecutor.__init__` snapshotted `connection._weaviate_version` and derived `uses_125_api` / `uses_127_api` from it, then built the `_QueryGRPC` request factory with those values. A connection reports version 0.0.0 until `connect()` reads `/v1/meta`, so a collection object made before the connection is open — `client.collections.use(...)` before `await client.connect()` with the async client — pinned the legacy paths for the lifetime of that object: search requests went out with `uses_125_api=False`, `generative.to_grpc()` was handed 0.0.0, and generated text was read from the deprecated metadata field instead of the generative one, so `generated` came back empty on a server that fills the new field. Read the version from the connection per call instead, which is what the other executors in the package already do, and derive the two feature flags inside `_QueryGRPC` from the version it is given so that they cannot disagree with it. The removed constructor arguments were computed from the same version by every caller, so the existing unit tests that build a `_QueryGRPC` directly only lose the redundant keyword arguments. Prior art: weaviate#2060 proposed the same lazy-read approach but stalled unmerged. Fixes weaviate#1831
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| SAST | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
|
To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge. |
Author
|
I agree to the CLA |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1831.
Credit where it is due: @2830500285 opened PR #2060 ("fix: refresh query version metadata") against this issue in June with the same core diagnosis and the same lazy-read approach. It never got a review because the CLA bot's request went unanswered, and it has since gone stale: it removes the
uses_125_api/uses_127_apiparameters from_QueryGRPC.__init__without updating the three test helpers that pass them (test/collection/test_bm25_operator.py,test_hybrid_diversity.py,test_target_vectors.py— all added after that PR was opened), so it no longer applies green. If you would rather revive #2060, this PR is redundant and I am happy to close it; I am posting it because the issue has been open a year and #2060 is structurally unmergeable.What this adds over #2060:
_QueryGRPCper call instead of mutating one long-lived instance through a_use_weaviate_version()setter, so the request builder stays immutable after construction and there is no cross-thread window where one caller's flags land in another caller's request._QueryGRPCtest helpers for the narrowed constructor, so the suite is green.__deserialize_list_value_prop_125ValueError) is no longer reachable — that fork was collapsed to an unconditional call, and the current protos have dropped the legacyListValue.valuesfield altogether. The surviving consequence is in__result_to_generative_object: withuses_127_apipinned False, generated text is read from the deprecated metadata field, soobj.generatedcomes backNoneon any supported server. Without that test this reads as "already fixed".client.collections.use(...)beforeawait client.connect(), then a realfetch_objects()over the mock gRPC server, asserting the capturedSearchRequestcarries both flags. On main the captured request isuses_123_api: trueand nothing else.Root cause is the one @tsmith023 identified in the thread:
_BaseExecutor.__init__snapshottedconnection._weaviate_version, which is 0.0.0 untilconnect()reads/v1/meta, and derived both feature flags plus the_QueryGRPCfrom it. Reading per call is whataliases/executor.py,backup/executor.py,export/executor.pyandtokenization/executor.pyalready do. Deriving the two flags from the version at the point of use removes the second source of truth that let them drift.Why make the stale read correct rather than delete the dead branch? Fair question, since
connect()already refuses anything below 1.27 (connect/v4.py:964sync,:1168async — both outside theskip_init_checksguard), so in a connected session both flags are necessarily True and__extract_generated_from_metadatais unreachable;uses_123_api=Truehas been hardcoded since 9a2de06 for exactly that reason. I went the lazy-read route because_weaviate_versionstill drives live 1.27/1.29/1.37-1.39 forks ingrpc/shared.py(:231, :251, :272-278, :404-433, :688-730, :758), so the version object has to stay threaded through_QueryGRPCregardless — and because the executor was reading it at the wrong time, which is the bug the issue reports. Happy to cut the dead branch instead, or as a follow-up, if you prefer the terminal fix.Not in this PR, on purpose (happy to follow up):
aggregations/base_executor.py:66,data/executor.py:87,93,tenants/executor.py:42,batch/base.py:302,batch/async_.py:87,batch/sync.py:73andcollection/sync.py:106take the same construction-time snapshot. The aggregate path in particular looks exposed the same way for pre-connect async collections — silently, since it falls back to the pre-1.29struct.packencoding rather than erroring — and I have not written a test for it. Also noticed but not touched:__deserialize_list_value_prop_125and__deserialize_non_ref_propboth pass"value"toWhichOneof, but that oneof is namedkindon bothListValueandValue, so the "unknown type" branch raisesValueErrorinstead of warning. TheListValueone is the literal exception in this issue's traceback; say the word and I will land it separately.Validation (offline; no Docker here, so
integration/andjourney_tests/were not run — worth watching those CI jobs):pytest test→ 497 passed, 1 skipped (490 + 1 on main)pytest mock_tests→ 66 passed (65 on main)weaviate/: 5 intest/collection/test_queries.py, 1 inmock_tests/test_collection.pyruff check/ruff format --diff/flake8over the changed files → cleanpyright 1.1.399inweaviate/(as CI runs it) → 0 errors