Severity: High — silent search-quality divergence between storage backends
Area: storage/sqlite/semrefindex.py, storage/sqlite/schema.py
Summary
ITermToSemanticRefIndex.add_term accepts either a bare SemanticRefOrdinal or
a ScoredSemanticRefOrdinal. The memory backend round-trips the score; the
SQLite backend drops it on write and fabricates 1.0 on read. The same
conversation therefore produces different relevance rankings depending on which
provider backs it, with no warning.
Reproduction
await idx.add_term("cat", ScoredSemanticRefOrdinal(0, 0.25))
print(await idx.lookup_term("cat"))
memory lookup_term('cat') -> [ScoredSemanticRefOrdinal(0, 0.25)]
sqlite lookup_term('cat') -> [ScoredSemanticRefOrdinal(0, 1.0)]
Expected: both backends return 0.25.
Actual: SQLite returns 1.0.
Root cause
The table has no score column (storage/sqlite/schema.py:59-64):
CREATE TABLE IF NOT EXISTS SemanticRefIndex (
term TEXT NOT NULL, -- lowercased, not-unique/normalized
semref_id INTEGER NOT NULL,
...
);
add_term (storage/sqlite/semrefindex.py:31-58) unwraps the ordinal and
throws the score away:
if isinstance(semantic_ref_ordinal, interfaces.ScoredSemanticRefOrdinal):
semref_id = semantic_ref_ordinal.semantic_ref_ordinal # score lost
else:
semref_id = semantic_ref_ordinal
lookup_term then hardcodes the score (:111), as does serialize (:131):
results.append(ScoredSemanticRefOrdinal(semref_id, 1.0))
add_terms_batch (:60-85) has the same gap.
Impact
Scores flow into SemanticRefAccumulator.add_term_matches, are multiplied by
term weights, and drive get_top_n_scoring — so this changes result ordering,
not just reported numbers. It also means a memory-backed index serialized and
reloaded into SQLite loses information.
Suggested fix
Either add a score REAL NOT NULL DEFAULT 1.0 column and persist it (mirroring
what PropertyIndex already does), or — if scores are intentionally not
supported here — make add_term reject a ScoredSemanticRefOrdinal with a
non-default score rather than silently discarding it, and document the
limitation on the protocol.
Note there's a second divergence in the same call: memory's add_term appends
duplicates, while SQLite uses INSERT OR IGNORE, so repeated (term, semref)
pairs produce different result lengths too.
Severity: High — silent search-quality divergence between storage backends
Area:
storage/sqlite/semrefindex.py,storage/sqlite/schema.pySummary
ITermToSemanticRefIndex.add_termaccepts either a bareSemanticRefOrdinalora
ScoredSemanticRefOrdinal. The memory backend round-trips the score; theSQLite backend drops it on write and fabricates
1.0on read. The sameconversation therefore produces different relevance rankings depending on which
provider backs it, with no warning.
Reproduction
Expected: both backends return
0.25.Actual: SQLite returns
1.0.Root cause
The table has no score column (
storage/sqlite/schema.py:59-64):add_term(storage/sqlite/semrefindex.py:31-58) unwraps the ordinal andthrows the score away:
lookup_termthen hardcodes the score (:111), as doesserialize(:131):add_terms_batch(:60-85) has the same gap.Impact
Scores flow into
SemanticRefAccumulator.add_term_matches, are multiplied byterm weights, and drive
get_top_n_scoring— so this changes result ordering,not just reported numbers. It also means a memory-backed index serialized and
reloaded into SQLite loses information.
Suggested fix
Either add a
score REAL NOT NULL DEFAULT 1.0column and persist it (mirroringwhat
PropertyIndexalready does), or — if scores are intentionally notsupported here — make
add_termreject aScoredSemanticRefOrdinalwith anon-default score rather than silently discarding it, and document the
limitation on the protocol.
Note there's a second divergence in the same call: memory's
add_termappendsduplicates, while SQLite uses
INSERT OR IGNORE, so repeated(term, semref)pairs produce different result lengths too.