Skip to content

fix(query): resolve the server version at call time - #2167

Open
v0ropaev wants to merge 1 commit into
weaviate:mainfrom
v0ropaev:fix/query-executor-lazy-server-version
Open

v0ropaev wants to merge 1 commit into
weaviate:mainfrom
v0ropaev:fix/query-executor-lazy-server-version

Conversation

@v0ropaev

Copy link
Copy Markdown

Fixes #1831.

Credit where it is due: @2830500285 opened PR #2060 ("fix: refresh query version metadata") against this issue in June with the same core diagnosis and the same lazy-read approach. It never got a review because the CLA bot's request went unanswered, and it has since gone stale: it removes the uses_125_api / uses_127_api parameters from _QueryGRPC.__init__ without updating the three test helpers that pass them (test/collection/test_bm25_operator.py, test_hybrid_diversity.py, test_target_vectors.py — all added after that PR was opened), so it no longer applies green. If you would rather revive #2060, this PR is redundant and I am happy to close it; I am posting it because the issue has been open a year and #2060 is structurally unmergeable.

What this adds over #2060:

  • Rebuilds _QueryGRPC per call instead of mutating one long-lived instance through a _use_weaviate_version() setter, so the request builder stays immutable after construction and there is no cross-thread window where one caller's flags land in another caller's request.
  • Updates the three _QueryGRPC test helpers for the narrowed constructor, so the suite is green.
  • Adds the generative-extraction regression test. This matters because the symptom originally reported (the __deserialize_list_value_prop_125 ValueError) is no longer reachable — that fork was collapsed to an unconditional call, and the current protos have dropped the legacy ListValue.values field altogether. The surviving consequence is in __result_to_generative_object: with uses_127_api pinned False, generated text is read from the deprecated metadata field, so obj.generated comes back None on any supported server. Without that test this reads as "already fixed".
  • Adds @tsmith023's MRE as an offline mock test: client.collections.use(...) before await client.connect(), then a real fetch_objects() over the mock gRPC server, asserting the captured SearchRequest carries both flags. On main the captured request is uses_123_api: true and nothing else.

Root cause is the one @tsmith023 identified in the thread: _BaseExecutor.__init__ snapshotted connection._weaviate_version, which is 0.0.0 until connect() reads /v1/meta, and derived both feature flags plus the _QueryGRPC from it. Reading per call is what aliases/executor.py, backup/executor.py, export/executor.py and tokenization/executor.py already do. Deriving the two flags from the version at the point of use removes the second source of truth that let them drift.

Why make the stale read correct rather than delete the dead branch? Fair question, since connect() already refuses anything below 1.27 (connect/v4.py:964 sync, :1168 async — both outside the skip_init_checks guard), so in a connected session both flags are necessarily True and __extract_generated_from_metadata is unreachable; uses_123_api=True has been hardcoded since 9a2de06 for exactly that reason. I went the lazy-read route because _weaviate_version still drives live 1.27/1.29/1.37-1.39 forks in grpc/shared.py (:231, :251, :272-278, :404-433, :688-730, :758), so the version object has to stay threaded through _QueryGRPC regardless — and because the executor was reading it at the wrong time, which is the bug the issue reports. Happy to cut the dead branch instead, or as a follow-up, if you prefer the terminal fix.

Not in this PR, on purpose (happy to follow up): aggregations/base_executor.py:66, data/executor.py:87,93, tenants/executor.py:42, batch/base.py:302, batch/async_.py:87, batch/sync.py:73 and collection/sync.py:106 take the same construction-time snapshot. The aggregate path in particular looks exposed the same way for pre-connect async collections — silently, since it falls back to the pre-1.29 struct.pack encoding rather than erroring — and I have not written a test for it. Also noticed but not touched: __deserialize_list_value_prop_125 and __deserialize_non_ref_prop both pass "value" to WhichOneof, but that oneof is named kind on both ListValue and Value, so the "unknown type" branch raises ValueError instead of warning. The ListValue one is the literal exception in this issue's traceback; say the word and I will land it separately.

Validation (offline; no Docker here, so integration/ and journey_tests/ were not run — worth watching those CI jobs):

  • pytest test → 497 passed, 1 skipped (490 + 1 on main)
  • pytest mock_tests → 66 passed (65 on main)
  • new tests verified failing against unmodified weaviate/: 5 in test/collection/test_queries.py, 1 in mock_tests/test_collection.py
  • ruff check / ruff format --diff / flake8 over the changed files → clean
  • pyright 1.1.399 in weaviate/ (as CI runs it) → 0 errors

`_BaseExecutor.__init__` snapshotted `connection._weaviate_version` and
derived `uses_125_api` / `uses_127_api` from it, then built the `_QueryGRPC`
request factory with those values. A connection reports version 0.0.0 until
`connect()` reads `/v1/meta`, so a collection object made before the
connection is open — `client.collections.use(...)` before
`await client.connect()` with the async client — pinned the legacy paths for
the lifetime of that object: search requests went out with
`uses_125_api=False`, `generative.to_grpc()` was handed 0.0.0, and generated
text was read from the deprecated metadata field instead of the generative
one, so `generated` came back empty on a server that fills the new field.

Read the version from the connection per call instead, which is what the
other executors in the package already do, and derive the two feature flags
inside `_QueryGRPC` from the version it is given so that they cannot
disagree with it. The removed constructor arguments were computed from the
same version by every caller, so the existing unit tests that build a
`_QueryGRPC` directly only lose the redundant keyword arguments.

Prior art: weaviate#2060 proposed the same lazy-read approach but stalled unmerged.

Fixes weaviate#1831

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

@v0ropaev

Copy link
Copy Markdown
Author

I agree to the CLA

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] async client: query list value type mis-match

2 participants