Prove retrieval quality, transfer invariance, and cold-hydration performance

[valiantone]

Priority and intent

P1 — execute after #67 and #69 establish the interchange contract and verified clone/hydrate path.

HotMem already combines the deterministic hotmem-hash-v1 character-trigram embedder, SQLite FTS5 BM25, and static importance. We do not yet have objective evidence showing where that retrieval stack succeeds, where it fails, or whether semantic embeddings, MMR, entity enrichment, or a second-stage reranker would provide the highest-value improvement.

Build a small, deterministic retrieval evaluation harness before changing production ranking. This ticket converts retrieval decisions from opinion into reproducible evidence while preserving HotMem's zero-dependency, local-first, portable-memory core.

This issue gates implementation work in #78 and #80. It does not implement either feature.

User outcome

A contributor can run one documented command and receive:

1. machine-readable retrieval metrics;
2. a concise human-readable report;
3. per-query failures that identify why relevant memory was missed;
4. an evidence-backed recommendation for the next retrieval change.

A junior contributor should be able to implement this ticket using only the contract below.

Current implementation to understand first

Read these files before changing anything:

• src/hotmem/embed.py — hotmem-hash-v1, dimension 64, deterministic character trigrams.
• src/hotmem/search.py — weighted fusion: cosine 0.6, normalized FTS5 BM25 0.2, importance 0.2.
• src/hotmem/db.py — cosine candidate retrieval and FTS5 search.
• tests/test_search.py and tests/test_embed.py — existing behavioral coverage.
• docs/snapshot-v2.md — embedding model/dimension portability contract.

Do not assume the hash vectors are transformer-style semantic embeddings. Do not call the current weighted fusion a separate reranker.

Required repository layout

Create:

benchmarks/
  retrieval/
    README.md
    corpus.jsonl
    queries.jsonl
    baseline.json
scripts/
  retrieval_eval.py
tests/
  test_retrieval_eval.py
docs/
  retrieval-quality.md

If the repository already has a more established benchmark location when implementation starts, use it consistently and explain the deviation in the PR.

Fixture contract

corpus.jsonl

Provide at least 50 synthetic memories. Every line must be valid UTF-8 JSON and contain:

{
  "memory_id": "mem-invoice-001",
  "identifier": "acme-finance",
  "fact": "Acme invoices require two-person approval above EUR 5,000.",
  "importance": 0.5,
  "tags": ["finance", "approval"]
}

Requirements:

• IDs are unique and stable.
• Content is synthetic and safe to publish.
• No API keys, personal information, customer data, or copied proprietary text.
• Include deliberate distractors, near-duplicates, conflicting revisions, and unrelated memories.
• Keep importance neutral at 0.5 except in cases explicitly testing importance.
• Fixture order must not determine the expected ranking.

queries.jsonl

Provide at least 40 evaluated queries, with at least five cases in each category:

1. exact_lexical — query shares important terms with the relevant memory.
2. semantic_paraphrase — meaning is equivalent but wording differs substantially.
3. identifier_or_entity_name — organization, project, product, or alias lookup; this tests existing text retrieval only, not entity extraction.
4. temporal_or_revision — newer and older facts coexist and expected relevance is explicitly graded.
5. near_duplicate_diversity — duplicates can consume multiple top-K positions.
6. negative_or_no_answer — corpus contains no relevant answer.
7. cross_project_isolation — identical names and conflicting facts from separate projects must remain correctly scoped and attributable.
8. snapshot_hydration_equivalence — retrieval expectations must remain stable after verified snapshot and clean-target hydration.

Every query line must contain:

{
  "query_id": "q-semantic-001",
  "category": "semantic_paraphrase",
  "query": "Who must sign off on a large Acme bill?",
  "relevance": {
    "mem-invoice-001": 3,
    "mem-invoice-002": 1
  },
  "notes": "Tests paraphrase without sharing approval terminology."
}

Relevance grades:

• 3: directly answers the query;
• 2: strongly relevant supporting memory;
• 1: partially relevant;
• omitted or 0: irrelevant.

Negative queries must use an empty relevance object.

Evaluator requirements

Implement scripts/retrieval_eval.py using the real production ingestion/database/search path. Do not reproduce the ranking algorithm in the evaluator.

The command must work from a clean development checkout:

uv run python scripts/retrieval_eval.py

Required options:

--corpus PATH       default: benchmarks/retrieval/corpus.jsonl
--queries PATH      default: benchmarks/retrieval/queries.jsonl
--output PATH       optional JSON result path
--report PATH       optional Markdown report path
--top-k INTEGER     default: 5
--repeat INTEGER    default: 1; use >1 only for latency sampling

Required behavior:

1. Create and use a temporary HotMem database; never write to a user's configured mount.
2. Hydrate all corpus records through the normal memory/database path.
3. Run every query through search_memories().
4. Validate malformed fixtures with a clear filename, line number, and reason.
5. Use a fixed clock or exclude clock-dependent fields from comparisons.
6. Produce stable logical results across repeated runs.
7. Exit non-zero for malformed input or evaluator errors.
8. Do not make network calls.
9. Do not download models.
10. Do not add a required production dependency.

Metrics

Calculate and report:

• Recall@1 and Recall@5 using memories graded 2 or 3 as relevant.
• Mean Reciprocal Rank at 5 (MRR@5).
• nDCG@5 using the 0–3 relevance grades.
• Negative-query false-positive rate at 5.
• Duplicate-slot rate at 5 for near_duplicate_diversity cases. The query fixture may declare duplicate groups if needed.
• Query latency p50 and p95 in milliseconds. Clearly label latency as local diagnostic data, not a cross-machine performance guarantee.
• Snapshot throughput, verification throughput, hydration records/second, package size, and cold-start-to-first-query time for the committed fixture.
• Logical top-K equivalence before and after snapshot -> verify -> clean hydrate, with any score/order drift reported per query.

Report every quality metric:

• overall;
• separately for each query category;
• with integer numerator/denominator where applicable.

Do not hide a category with zero cases; treat that as invalid fixture input.

Outputs

Machine-readable JSON

The JSON result must include:

• schema version;
• UTC generation timestamp;
• HotMem version or git commit when available;
• embedding model and dimension;
• scoring weights;
• corpus/query counts;
• requested top-K and repeat count;
• aggregate metrics;
• per-category metrics;
• ordered result IDs, relevance grades, and score for every query;
• failed or missed query IDs.

Do not serialize temporary paths or other machine-specific values into the committed baseline.

Human-readable Markdown

The report must show:

• the active retrieval configuration;
• a compact metrics table;
• the five worst-performing queries with expected and returned IDs;
• category-level observations;
• a conclusion using the decision rules below.

Decision rules

The generated recommendation must be deterministic:

1. If snapshot/hydration logical equivalence is not 100%, recommend fixing clone/index compatibility before any retrieval feature.
2. Otherwise, if semantic_paraphrase Recall@5 is at least 15 percentage points below exact_lexical Recall@5, recommend portable semantic embeddings (Portable derived-index contract and optional semantic embedder #78) next.
3. Otherwise, if duplicate-slot rate at 5 is greater than 20%, recommend optional deterministic reranking (Evidence-gated optional reranking hook with deterministic local fallback #80) next.
4. Otherwise, recommend retaining the current stack and expanding fixtures before adding retrieval complexity.
5. Never recommend entity extraction from this benchmark alone. Entity enrichment requires a separate product and schema decision.
6. Never recommend a hosted or learned reranker until a first-stage retrieval benchmark demonstrates a candidate-ranking problem that fusion or MMR cannot address.

These are prioritization rules, not permanent release thresholds. Document that distinction.

Baseline handling

Commit benchmarks/retrieval/baseline.json generated from the unchanged production retrieval stack.

The baseline is an observability artifact, not permission to encode weak results forever:

• CI tests must validate its schema, fixture counts, deterministic result ordering, and metric calculations.
• CI must fail if an implementation PR changes ordered result IDs or metrics without intentionally regenerating the baseline.
• A changed baseline must be called out in the PR description with category-level before/after values and an explanation.
• Do not add arbitrary quality thresholds that the existing implementation cannot meet.

Normalize timestamps, timing, temporary paths, and commit-specific data before comparing a run with the committed baseline.

Automated tests

Add tests covering at minimum:

• JSONL parsing and useful line-numbered validation errors;
• unique IDs and valid relevance references;
• all eight required categories and minimum fixture counts;
• exact hand-calculated Recall, MRR, nDCG, false-positive, and duplicate-slot examples;
• stable result ordering for the committed fixtures;
• output JSON schema/version;
• baseline comparison excluding nondeterministic metadata;
• no writes outside a temporary directory;
• evaluator subprocess exits 0 for valid fixtures and non-zero for invalid fixtures.

Existing tests/test_search.py, API, MCP, snapshot, and hydration tests must remain unchanged in behavior and pass.

Documentation

Add docs/retrieval-quality.md explaining in plain language:

• what HotMem currently uses: hash vectors + FTS5 BM25 + importance;
• why hash vectors are deterministic and portable but not learned semantic embeddings;
• why weighted score fusion is not a second-stage reranker;
• how to run and interpret the benchmark;
• what the benchmark does not prove;
• why entities and rerankers remain optional future extensions rather than core requirements.

Link this page from the existing appropriate docs navigation without reorganizing unrelated documentation.

Compatibility and architectural constraints

• Default runtime remains local-first and network-free.
• The hotmem core dependency list does not grow.
• Public HTTP, Python, TypeScript, CLI, and MCP response shapes do not change.
• Snapshot/hydration schema and embedding compatibility rules do not change.
• No model or benchmark result is fetched at runtime.
• Production scoring weights and ranking behavior remain unchanged in this ticket.
• Benchmark code must call production code; production code must not import benchmark code.

Explicitly out of scope

• Implementing sentence-transformers, OpenAI, Cohere, Voyage, or any other embedding provider.
• Anthropic embeddings; Anthropic does not provide an embeddings API and must not be listed as one.
• Entity extraction, entity linking, knowledge graphs, mental models, or relationship inference.
• Cross-encoder, LLM, or hosted reranking.
• MMR implementation.
• Changing hybrid scoring weights.
• Vector databases, ANN indexes, or replacing SQLite.
• New production API parameters.
• Benchmarking Mem0, OpenViking, Hindsight, or remote services.
• Marketing claims based on a small synthetic benchmark.

Definition of done

• Required fixture files and minimum case counts exist.
• One documented command produces valid JSON and Markdown outputs.
• All required metrics are correct and tested.
• Baseline comes from the unchanged production stack.
• Per-query failures are inspectable.
• Deterministic decision rule identifies the next retrieval investment.
• No production dependency, API, schema, or ranking behavior changes.
• uv run ruff check . passes.
• uv run pytest passes.
• uv run mkdocs build --strict passes.
• PR description contains the generated metrics table and recommendation.

Suggested implementation order for a junior contributor

1. Read the current implementation and run the full test suite before editing.
2. Add the corpus and query fixtures; write fixture-validation tests first.
3. Implement and unit-test the pure metric functions using tiny hand-calculated examples.
4. Implement temporary-database ingestion and real search_memories() execution.
5. Add JSON output, then Markdown rendering.
6. Generate and normalize the baseline.
7. Add the baseline regression test.
8. Add the documentation and navigation entry.
9. Run lint, tests, and strict docs build.
10. In the PR, paste before/after status, metrics, known limitations, and the deterministic recommendation.

Related issues

• Define HotMem interchange and company-brain clone package v1 #67 and Ship verified portable company-brain dump and hydrate workflow #69 — establish the interchange and verified clone/hydrate path before this benchmark.
• Portable derived-index contract and optional semantic embedder #78 — portable derived-index and semantic embedding boundary; do not implement until this benchmark identifies the need.
• 1a: FTS5 full-text search #2 — FTS5/BM25; already represented in the current production stack.
• Evidence-gated optional reranking hook with deterministic local fallback #80 — optional reranking hook; do not implement until duplicate-slot or candidate-ordering evidence justifies it.
• Define a lossless, namespaced annotation envelope for portable company brains #79 — portable annotations; intentionally independent from retrieval quality and entity extraction.
• Compatibility golden tests for file-native evolution #54 — compatibility golden tests; preserve those contracts.