Skip to content

SqliteTermToSemanticRefIndex discards relevance scores #321

Description

@KRRT7

Severity: High — silent search-quality divergence between storage backends
Area: storage/sqlite/semrefindex.py, storage/sqlite/schema.py

Summary

ITermToSemanticRefIndex.add_term accepts either a bare SemanticRefOrdinal or
a ScoredSemanticRefOrdinal. The memory backend round-trips the score; the
SQLite backend drops it on write and fabricates 1.0 on read. The same
conversation therefore produces different relevance rankings depending on which
provider backs it, with no warning.

Reproduction

await idx.add_term("cat", ScoredSemanticRefOrdinal(0, 0.25))
print(await idx.lookup_term("cat"))
memory  lookup_term('cat') -> [ScoredSemanticRefOrdinal(0, 0.25)]
sqlite  lookup_term('cat') -> [ScoredSemanticRefOrdinal(0, 1.0)]

Expected: both backends return 0.25.
Actual: SQLite returns 1.0.

Root cause

The table has no score column (storage/sqlite/schema.py:59-64):

CREATE TABLE IF NOT EXISTS SemanticRefIndex (
    term TEXT NOT NULL,             -- lowercased, not-unique/normalized
    semref_id INTEGER NOT NULL,
    ...
);

add_term (storage/sqlite/semrefindex.py:31-58) unwraps the ordinal and
throws the score away:

if isinstance(semantic_ref_ordinal, interfaces.ScoredSemanticRefOrdinal):
    semref_id = semantic_ref_ordinal.semantic_ref_ordinal   # score lost
else:
    semref_id = semantic_ref_ordinal

lookup_term then hardcodes the score (:111), as does serialize (:131):

results.append(ScoredSemanticRefOrdinal(semref_id, 1.0))

add_terms_batch (:60-85) has the same gap.

Impact

Scores flow into SemanticRefAccumulator.add_term_matches, are multiplied by
term weights, and drive get_top_n_scoring — so this changes result ordering,
not just reported numbers. It also means a memory-backed index serialized and
reloaded into SQLite loses information.

Suggested fix

Either add a score REAL NOT NULL DEFAULT 1.0 column and persist it (mirroring
what PropertyIndex already does), or — if scores are intentionally not
supported here — make add_term reject a ScoredSemanticRefOrdinal with a
non-default score rather than silently discarding it, and document the
limitation on the protocol.

Note there's a second divergence in the same call: memory's add_term appends
duplicates, while SQLite uses INSERT OR IGNORE, so repeated (term, semref)
pairs produce different result lengths too.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions