T3: Wire pathHints/languageHints into retrieval layer with metadata filtering

[MrDoe]

Summary

Wires \pathHints\ and \languageHints\ from \SearchOptions\ through to the retrieval layer, enabling metadata-based filtering across vector and keyword search.

Changes

• *\src/core/interfaces.ts* — Added \MetadataFilter\ interface (\pathPatterns, \languages), \searchWithFilter\ to \VectorStore, and optional \ilter\ param on \KeywordIndex.search\
• *\src/vectorstore/lancedb.ts* — \searchWithFilter\ pushes filter down via LanceDB .where()\ SQL clause using \�uildWhereClause\ (glob **\ → %, *\ → %, ?\ → _, SQL injection escaping)
• *\src/vectorstore/memory.ts* — \searchWithFilter\ with post-filter \matchesFilter/\globMatch\ helpers
• *\src/retriever/keyword-index.ts* — \search()\ accepts \ilter, applied before \slice\ via \matchesFilter/\globMatch\
• *\src/retriever/retriever.ts* — \RetrieveOptions.filter\ forwarded to both vector store and keyword index
• *\src/api.ts* — \search()\ forwards \pathHints/\languageHints\ as \ilter\ in \RetrieveOptions\
• 5 test files — Added \searchWithFilter\ to all mock \VectorStore\ stubs

Verification

• 
  pm run typecheck\ passes
• 
  pm test: 842/842 pass

Closes T3

[Copilot]

Pull request overview

This PR introduces metadata-based filtering (path/language) in the retrieval pipeline by threading hint inputs into both vector and keyword search, and adds backend support for applying those filters (pushdown in LanceDB; post-filtering in memory/keyword index). It also changes retrieval scoring behavior by switching LanceDB vector search to cosine distance and updating hybrid fusion to Reciprocal Rank Fusion (RRF), with accompanying config/test updates.

Changes:

• Add MetadataFilter and extend VectorStore/KeywordIndex APIs to support filtered search.
• Implement filtered search in LanceDB (SQL WHERE pushdown) and in-memory stores (glob-based post-filter).
• Update hybrid fusion scoring to RRF and adjust tests/config/example values accordingly.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/core/interfaces.ts	Adds MetadataFilter, VectorStore.searchWithFilter, and filterable KeywordIndex.search; extends explanation with rank fields.
src/vectorstore/lancedb.ts	Implements searchWithFilter, cosine distance search + normalization, and SQL WHERE clause construction for filters.
src/vectorstore/memory.ts	Adds searchWithFilter and applies metadata filtering with glob matching.
src/retriever/keyword-index.ts	Adds optional filter to search() and applies metadata filtering.
src/retriever/retriever.ts	Forwards RetrieveOptions.filter into vector/keyword search and switches hybrid fusion to RRF.
src/api.ts	Forwards pathHints/languageHints into retrieval via RetrieveOptions.filter.
src/core/manifest.ts	Bumps manifest schema constants (affects backward compatibility / rebuild behavior).
src/core/config.ts	Adjusts default retrieval minScore.
src/tests/vectorstore/lancedb.test.ts	Adds l2Normalize unit tests.
src/tests/retriever/retriever.test.ts	Updates expectations for RRF-scored results and explanations.
src/tests/plugin.test.ts	Updates VectorStore mocks to include searchWithFilter.
src/tests/opencode/create-read-tool.test.ts	Updates mock stores and expected displayed scores (reflecting new scoring scale).
src/tests/mcp/server.test.ts	Updates VectorStore mocks to include searchWithFilter.
src/tests/eval/token-comparison.test.ts	Updates mock store typing to include SearchResult and searchWithFilter.
src/tests/core/manifest.test.ts	Updates schema version expectations and version-compat test behavior.
src/tests/core/config.test.ts	Updates default minScore expectation.
opencode-rag.example.json	Updates example minScore.

Comments suppressed due to low confidence (1)

opencode-rag.example.json:52

• The example config sets retrieval.minScore to 0.35, but with the new RRF scoring the maximum combined score is ~0.016 (for rank-0 results). Using 0.35 will make searches return no results for most users who copy this example.

  "retrieval": {
    "topK": 20,
    "minScore": 0.35,
    "hybridSearch": {
      "enabled": true,
      "keywordWeight": 0.4
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Fixed in 7845a3b. search() now resolves each pathHint against cwd (falling back to process.cwd()) before passing it to filter.pathPatterns, leaving already-absolute patterns untouched via path.isAbsolute().

[Copilot]

Fixed in local commit b3502fb. Added a globPatternToRegex() helper that mirrors the globMatch() logic from the in-memory and keyword-index backends — ** is saved via a placeholder, special regex chars are escaped, then ** becomes .*, * becomes [^/]*, and ? becomes .. buildWhereClause() now uses regexp_matches(filePath, '...') instead of filePath LIKE '...', so src/*.ts correctly matches only direct children and won't cross into src/nested/file.ts.

The push is currently blocked by a branch protection rule on t1-cosine-l2. Here is the diff to apply:

+/**
+ * Convert a glob pattern to a regex string, matching the semantics of the
+ * in-memory and keyword-index backends: `*` matches within a single path
+ * segment (`[^/]*`), while `**` crosses path separators (`.*`).
+ */
+function globPatternToRegex(pattern: string): string {
+  const GLOBSTAR = "\x00GS\x00";
+  let re = pattern.replace(/\*\*/g, GLOBSTAR);
+  re = re.replace(/([.+^${}()|[\]\\])/g, "\\$1");
+  re = re.replace(new RegExp(GLOBSTAR, "g"), ".*");
+  re = re.replace(/\*/g, "[^/]*");
+  re = re.replace(/\?/g, ".");
+  return "^" + re + "$";
+}
+
 /**
  * Build a LanceDB SQL WHERE clause from a MetadataFilter.
  * Handles path glob patterns and language filters with proper escaping.
+ * Uses regexp_matches() for path patterns so that `*` is constrained to a
+ * single path segment while `**` can cross path separators, consistent with
+ * the in-memory and keyword-index backends.
  */
 function buildWhereClause(filter?: MetadataFilter): string | undefined {
   if (!filter) return undefined;
   const parts: string[] = [];
   if (filter.languages?.length) {
     const langs = filter.languages.map((l) => `'${l.replace(/'/g, "''")}'`).join(",");
     parts.push(`language IN (${langs})`);
   }
   if (filter.pathPatterns?.length) {
-    const likes = filter.pathPatterns.map((p) => {
-      const escaped = p.replace(/'/g, "''");
-      const like = escaped
-        .replace(/\*\*/g, "%")
-        .replace(/\*/g, "%")
-        .replace(/\?/g, "_");
-      return `filePath LIKE '${like}'`;
+    const regexps = filter.pathPatterns.map((p) => {
+      const regex = globPatternToRegex(p).replace(/'/g, "''");
+      return `regexp_matches(filePath, '${regex}')`;
     });
-    parts.push(`(${likes.join(" OR ")})`);
+    parts.push(`(${regexps.join(" OR ")})`);
   }
   return parts.length ? parts.join(" AND ") : undefined;
 }