Skip to content

Vector index polish: chunked search_by_ids binds, single FTS pass, shared unpack_embedding #92

Description

@valiantone

Summary

Non-blocking polish items from the local review of the optional derived vector index (PR #85, issue #49). All are small; none affect correctness of the shipped contract.

Items

  1. Chunk search_by_ids SQLite bindsid IN (…) binds up to oversample + FTS hits parameters. Fine at the default oversample (1000) on modern SQLite (limit 32766), but oversample is user-configurable and older SQLite builds cap at 999 — chunk the id list or clamp oversample.
  2. Single FTS pass on the accelerated path_fetch_candidates runs db.fts_search to union text-match ids, then search_memories runs it again for BM25 scores. Pass the rows through instead of re-querying.
  3. Reuse embed.unpack_embeddingvector_index._unpack_blob reimplements it; sharing keeps the blob format (endianness) defined in one place.
  4. Cosmetics — docstring artifact in the /v1/vector-index/rebuild endpoint ("backing file. Returns 400"); _suppress_exception(Exception) around Chroma delete/clear is broad — consider logging swallowed errors at warn level.

Context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:storageStorage adapter abstraction + filesystem adaptersenhancementNew feature or requestpriority:p3Later optimization; implement only when benchmarks justify complexityv0.2.NEXTCommitted for a future 0.2.x release; not yet pinned to 0.2.4/0.2.5v0.2.xHotMem post-v0.2 follow-up work

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions