Skip to content

fixtures: add the Qwen3-Embedding embed probe corpus so owned-cuda can certify - #11

Merged
ualtinok merged 1 commit into
cortexkit:masterfrom
Qiiks:feat/qwen3-embed-probe-fixture
Sep 16, 2026
Merged

ualtinok merged 1 commit into
cortexkit:masterfrom
Qiiks:feat/qwen3-embed-probe-fixture

Conversation

@Qiiks

@Qiiks Qiiks commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Owned-CUDA has no declared or inherited certification path: ensure_model_certified requires a measured probe row keyed on the machine-profile hash, and the only writer of those rows is probe.start. The embed probe resolves reference fixtures by (family, reference_model), but the shipped embed fixtures only cover minilm and gte-modernbert — so a Qwen3-Embedding model on the owned-CUDA lane dead-ends at:

"gate": "reference_fixture",
"blocking_reason": "reference_fixture_missing",
"reference": { "family": "qwen3-0.6b", "model": "<model_id>" }

with no supported way to certify. This PR adds the missing fixture set.

What's added

  • crates/synapse-module/src/fixtures/probe_corpus_qwen3_embedding_fp32.json — 64 items (same corpus texts as the shipped MiniLM fixture so cross-lane comparisons stay like-for-like), reference vectors for Qwen3-Embedding-0.6B, 1024 dims, pooling: "last", normalize: true.
  • probe_fixtures() loads the new set and computes its reference dims, matching the existing pattern.
  • probe_reference_key resolves the qwen3-embedding model-id family/reference names when the engine build flags carry no family (same fallback shape as minilm / gte-modernbert-base).
  • A registry test pinning family / reference_model / dims for all three embed sets.

Reference provenance (the part that matters)

The reference vectors were generated with candle-transformers 0.10.2 on CPU, f32 weights, last-token pooling + L2 normalization, from Qwen/Qwen3-Embedding-0.6B (safetensors revision 97b0c614be4d77ee51c0cef4e5f07c00f9eb65b3). The generation path shares no code with the CUDA kernels under test, which is the property the probe needs from a reference. The generation_command field records the invocation; the generator is a small candle-based tool (happy to land it under bench/lanes/ in a follow-up if you want it in-tree rather than as a documented command).

Tokenization follows the module's owned-Qwen3 policy: add_special_tokens=true, pop a trailing terminal, truncate to 2048, push the terminal — i.e. exactly what apply_owned_tokenizer_policy + tokenize_batch produce for this model, so the reference and the probe compare the same byte stream.

Verification

  • Live probe vs the owned-CUDA lane (RTX 4050, driver API 13030, CC 8.9): mean cosine 0.99999771, min cosine 0.99999539, worst-decile rank overlap 1.0000 — thresholds are 0.999 / 0.9. This is the first time an owned-CUDA Qwen3 embed model can reach the probe's quality gate at all.
  • cargo test -p synapse-module --lib — 423 passed, 0 failed (the new registry test included; one unrelated worker_host test flaked once on this box and passes on re-run).
  • cargo fmt --check and cargo clippy -p synapse-module --lib clean.

Scope

Fixture + matching only. I did not touch the certification flow itself; with this set present, a normal probe.start on an owned-CUDA qwen3-embedding model can produce a measured row and serve.

One adjacent gap I noticed while building this, filed separately if you want it: the CUDA capability floor is env-only (SYNAPSE_CUDA_DRIVER_API / SYNAPSE_CUDA_COMPUTE_CAPABILITY), and the driver value is a raw driver-API integer (13030 here), not the marketing version string — nothing probes it automatically, so a correct machine still refuses until the operator sets both. Happy to follow up with a probe that reads cuDriverGetVersion / cuDeviceGetAttribute at startup.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.


Summary by cubic

Adds the Qwen3-Embedding probe fixture corpus so owned-CUDA Qwen3 embed models can certify instead of dead-ending at reference_fixture_missing.

Changes

  • Adds a 64-item fixture with reference vectors from candle-transformers CPU f32 (last-token pool, L2), independent of the CUDA kernels under test.
  • Loads the fixture in probe_fixtures() and resolves qwen3-embedding model-ids to the new family/reference names, mirroring the existing minilm/gte-modernbert fallbacks.
  • Adds a registry test pinning family/reference/dims for all three embed fixture sets.

Verification

  • Live owned-CUDA probe scores 0.99999771 mean cosine (threshold 0.999) and worst-decile rank overlap 1.0.
  • cargo test -p synapse-module --lib passes (423 tests, 0 failures) and cargo fmt/clippy are clean.

Written for commit 6f7cdca. Summary will update on new commits.

Review in cubic

…n certify

Owned-CUDA has no declared or inherited certification path: a measured
probe row must exist before serving, and the probe resolves reference
fixtures by (family, reference_model). The shipped embed fixtures only
cover minilm and gte-modernbert, so a qwen3-0.6b/owned-cuda model
dead-ends at reference_fixture_missing with no supported way to
certify.

Add the missing set: 64 items, the same corpus texts as the shipped
fixtures, reference vectors generated on CPU with candle-transformers
0.10.2 f32 (last-token pool + L2), independent of the CUDA kernels
under test. Wire it into probe_fixtures() and resolve the
qwen3-embedding model-id family/reference names so the fixture matches
without an engine build flag, mirroring the minilm/modernbert fallbacks.

Verified locally: the fixture scores mean cosine 0.99999771 / worst
decile 1.0 against the owned-CUDA lane (thresholds 0.999 / 0.9), and
the new registry test pins family/reference/dims for all three sets.
Copilot AI lite review requested due to automatic review settings September 16, 2026 03:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Not reviewed (too large): crates/synapse-module/src/fixtures/probe_corpus_qwen3_embedding_fp32.json (~65,932 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic

@ualtinok
ualtinok merged commit 4e5ad1e into cortexkit:master Sep 16, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants