Portable derived-index contract and optional semantic embedder

[valiantone]

Priority

P2 — implement only after #77 establishes the baseline and demonstrates that semantic retrieval is the next highest-value improvement.

Goal

Introduce a small, production-grade embedding boundary that lets HotMem use a compatible semantic embedder without compromising its zero-configuration default, canonical JSONL portability, or clean hydration behavior.

The differentiator is not the number of embedding vendors supported. The differentiator is that a company brain remains valid and recoverable when it moves between machines, providers, models, and offline environments.

Required outcome

• hotmem-hash-v1 remains the deterministic, dependency-free default.
• Core HotMem defines a minimal Embedder protocol and immutable embedding descriptor.
• One optional local semantic adapter proves the interface end to end.
• Canonical memories remain readable and hydratable without the source embedding implementation.
• Compatible stored vectors are reused; incompatible vectors are rebuilt explicitly and observably.
• Retrieval never compares vectors from incompatible embedding spaces.

Implementation contract

1. Minimal protocol

Define an interface equivalent to:

class Embedder(Protocol):
    @property
    def descriptor(self) -> EmbeddingDescriptor: ...

    def embed(self, text: str) -> list[float]: ...

EmbeddingDescriptor must contain enough information to decide compatibility:

• implementation/provider identifier;
• model identifier;
• model revision when available;
• vector dimension;
• normalization (none or l2 initially);
• similarity metric (cosine initially);
• preprocessing/version identifier.

Compatibility must be determined by one tested function. Do not scatter string comparisons through snapshot, hydration, server, MCP, and import code.

2. Implementations

• Preserve the current algorithm as HashEmbedder with the exact hotmem-hash-v1 output.
• Add exactly one optional local semantic reference adapter behind an optional dependency extra.
• Do not add the model runtime to core dependencies.
• Do not download a model during import, server startup, tests, snapshot, or hydration.
• Tests for the optional adapter must use a fake/injected model unless an explicitly marked integration test is invoked.
• Do not add hosted provider SDKs in this ticket. Future hosted adapters must implement the same protocol without changing core.

3. Dependency injection

Pass the selected embedder explicitly through the memory write, search, hydration/reindex, HTTP server, CLI, MCP, playground, bundle, and snapshot-reader paths that currently import global embedding constants or embed_text().

Keep compatibility wrappers for the existing public functions and constants where required so current clients and tests do not break.

Avoid a global mutable singleton. Two independent HotMem runtimes in one process must be able to use different embedder instances safely.

4. Canonical record versus derived index

Treat fact text, identity, provenance, and annotations as canonical. Treat embeddings as derived acceleration data.

Snapshot/interchange records must preserve the embedding descriptor or a stable descriptor fingerprint when a vector is included. A valid record without an embedding must remain valid.

On hydration, report:

• embedding_reused;
• embedding_rebuilt;
• embedding_missing;
• embedding_failed.

Rules:

1. Reuse only when the stored descriptor is compatible with the active embedder.
2. Rebuild from canonical searchable text when incompatible and an active embedder is available.
3. Never silently relabel a stored vector as belonging to another model.
4. Never compare a query vector against a stored vector from an incompatible space.
5. Never make a company-brain package unusable merely because its original provider is unavailable.
6. Fail clearly when reindexing is required but canonical searchable text is unavailable.

5. Configuration

• Default remains hash with no new flags required.
• Add one explicit CLI/server configuration path for selecting the optional local adapter.
• Status/health output must identify the active descriptor without exposing secrets or machine-specific cache paths.
• Invalid configuration must fail before serving requests with an actionable error.

Benchmark gate

Use the committed #77 fixtures and report before/after values for:

• exact lexical Recall@5;
• semantic paraphrase Recall@5;
• MRR@5 and nDCG@5;
• query p50/p95 latency;
• cold startup time;
• snapshot/hydration retrieval equivalence.

The PR must not claim improvement unless the checked-in benchmark demonstrates it. A semantic adapter may not regress exact lexical Recall@5 by more than 5 percentage points without an explicit maintainer decision.

Tests

• Hash vectors are byte-compatible with the current implementation.
• Descriptor equality and incompatibility cases are exhaustive.
• Two runtimes with different embedders do not leak configuration.
• Compatible snapshot vector is reused without calling the embedder.
• Incompatible vector is rebuilt and reported.
• Missing optional package produces an actionable install message.
• No network or model download occurs in the default test suite.
• Mixed incompatible spaces are never scored together.
• JSONL, JSONL.GZ, Snapshot v2, API, MCP, and existing golden tests remain compatible.
• Snapshot -> clean hydrate preserves logical memories and retrieval expectations.

Performance and quality constraints

• Hash default startup and retrieval must not regress materially; report measured before/after values.
• The core package gains no required dependency.
• No provider-specific branching outside the adapter/configuration boundary.
• No secret values in logs, traces, manifests, or snapshots.
• Existing databases open without destructive migration.
• Existing snapshots remain readable.

Explicitly out of scope

• Multiple hosted provider implementations.
• Entity extraction or knowledge graphs.
• Reranking, MMR, RRF, or scoring-weight changes.
• Vector databases or mandatory ANN indexes.
• Automatic background model downloads.
• Automatic migration of every stored embedding at process startup.
• Declaring mixed embeddings safe merely because dimensions happen to match.

Definition of done

• Prove retrieval quality, transfer invariance, and cold-hydration performance #77 evidence justifies semantic embeddings.
• Minimal embedder protocol and descriptor are documented and tested.
• hotmem-hash-v1 remains exact and default.
• One optional local semantic adapter works end to end.
• Compatibility/rebuild behavior is deterministic and observable.
• Snapshot/hydration remains valid without the optional adapter.
• Core required dependencies remain unchanged.
• Full tests, lint, type checks if configured, and strict docs build pass.
• PR includes benchmark and startup/hydration before/after tables.

Dependencies

• Define HotMem interchange and company-brain clone package v1 #67 interchange contract.
• Ship verified portable company-brain dump and hydrate workflow #69 verified clone/hydrate workflow.
• Prove retrieval quality, transfer invariance, and cold-hydration performance #77 retrieval and portability benchmark.

Historical context

#1 was closed as not planned and moved to EMOS. This issue supersedes only the HotMem-relevant portability boundary; it intentionally does not restore the old provider-heavy scope.