Memory and SQLite term indexes normalize terms differently

[KRRT7]

Severity: High — the same query hits in one backend and misses in the other
Area: storage/memory/semrefindex.py, storage/sqlite/semrefindex.py

Summary

Both TermToSemanticRefIndex implementations funnel terms through a
_prepare_term hook before storing and before looking up, but the two hooks do
different things. Memory only lowercases; SQLite also strips, NFC-normalizes and
collapses whitespace. Terms that the SQLite index treats as equal are distinct
in memory, so lookups, get_terms() and size() all disagree.

Reproduction

await idx.add_term("Space  Needle ", 1)
print(sorted(await idx.get_terms()))
print(await idx.lookup_term("space needle"))

memory  get_terms()=['space  needle ']   lookup_term('space needle') -> []
sqlite  get_terms()=['space needle']     lookup_term('space needle') -> [ScoredSemanticRefOrdinal(1, 1.0)]

Further divergences:

property	input	memory	sqlite
lookup_term	add ('A', 0), probe 'A '	[]	[(0, 1.0)]
size()	add (' ', 0), ('\t', 0)	2	1
get_terms()	add (' ', 0)	[' ']	['']

Root cause

storage/memory/semrefindex.py:741-742:

def _prepare_term(self, term: str) -> str:
    return term.lower()

storage/sqlite/semrefindex.py:172-184:

def _prepare_term(self, term: str) -> str:
    term = term.strip()
    term = unicodedata.normalize("NFC", term)
    term = re.sub(r"\s+", " ", term)
    return term.lower()

Each backend is internally consistent — it normalizes the same way on write and
read — so neither is broken on its own. The defect is that normalization is a
property of the index contract, not of the storage engine, and it has been
implemented twice.

Impact

• Recall differs between deployments using different providers.
• A conversation indexed in memory and serialized into SQLite (or vice versa)
  will not match the terms it was built with, because the keys were normalized
  under different rules.
• Whitespace-only and empty terms are conflated in SQLite but not in memory,
  which also shifts size().

Suggested fix

Hoist normalization into one shared function (alongside make_property_term_text
in a neutral module, or on the protocol itself) and have both implementations
call it. The SQLite version looks like the intended behavior; promoting it to the
shared definition is probably the right direction, but that changes existing
memory-backed index keys, so it needs a deliberate call.

PropertyIndex._prepare_term has the same shape of duplication
(storage/memory/propindex.py:316 vs the inline .lower() at
storage/sqlite/propindex.py:55) and should move with it.