Define a lossless, namespaced annotation envelope for portable company brains

[valiantone]

Priority

P2 — implement after the interchange contract and verified clone workflow are stable. It may run after or alongside the embedding-boundary work, but must not delay P0/P1 delivery.

Goal

Define a lean, forward-compatible annotation envelope so a HotMem company brain can transport entity aliases, relationships, and other externally produced enrichment without HotMem becoming an entity-extraction or knowledge-graph engine.

HotMem's job is preservation, validation, provenance, deterministic merge, and lossless movement.

Required outcome

• Optional annotations survive JSONL, JSONL.GZ, Snapshot v2, clone, repeated hydration, and synchronization unchanged.
• Unknown future annotation namespaces are preserved losslessly.
• Every annotation can identify its producer, schema/version, timestamp, confidence, and source memory/evidence when supplied.
• HotMem performs no mandatory LLM call and adds no production dependency.
• Existing records and clients remain valid without annotations.

Format contract

Use one reserved, versioned envelope rather than new database columns for every enrichment type. Prefer the existing JSON metadata storage unless implementation evidence shows that lossless round-trip cannot be guaranteed there.

The logical shape must support:

{
  "annotations": {
    "schema_version": 1,
    "namespaces": {
      "org.hotmem.entities": {
        "items": [
          {
            "id": "entity:acme",
            "type": "organization",
            "name": "Acme GmbH",
            "aliases": ["Acme"],
            "confidence": 0.98,
            "evidence": ["memory-id"]
          }
        ]
      }
    },
    "producers": {
      "external-enricher": {
        "version": "1.0",
        "created_at": "..."
      }
    }
  }
}

The final field placement and exact names may differ, but the following are mandatory:

• namespaced ownership;
• schema version;
• stable item identity where an annotation type supports it;
• producer/extraction provenance;
• optional confidence;
• optional evidence references;
• preservation of unknown keys.

Validation rules

• Validate envelope structure, JSON types, namespace syntax, version, finite confidence in [0, 1], and local evidence references when present.
• Distinguish malformed from unknown: malformed known structure fails clearly; unknown namespaces/fields are preserved.
• Set explicit size and nesting limits to prevent pathological metadata from exhausting memory. Limits must be documented and return actionable errors.
• Never interpret an extracted entity or relationship as verified truth.
• Never fetch evidence URIs during validation.
• Never execute or import producer-supplied code.

Deterministic merge and replay

Specify and test:

1. byte/logical canonicalization rules;
2. duplicate annotation identity handling;
3. same-ID/same-content replay as a no-op;
4. same-ID/different-content as an explicit conflict;
5. ordering rules for deterministic snapshot output;
6. preservation of source and transfer provenance;
7. behavior when the receiving runtime does not understand a namespace.

No silent last-write-wins behavior.

Query behavior

This ticket may provide exact filtering by namespace, annotation ID, entity name, or alias only if it can reuse SQLite/FTS5 without new required dependencies.

Do not add semantic entity resolution, graph traversal, automatic alias inference, or ranking-weight changes. Preservation and portability are the acceptance boundary.

Import/export interoperability

Provide synthetic mapping fixtures showing how an external enrichment source could map:

• entity and alias annotations;
• typed relationships;
• producer/model/version metadata;
• confidence and evidence references.

Fixtures must not depend on Mem0, Hindsight, or any remote service at test time and must not claim full vendor-format compatibility.

Tests

• Legacy records without annotations remain unchanged.
• Known annotations validate and round-trip.
• Unknown namespace and fields survive byte-equivalent logical round-trip.
• Malformed confidence, excessive depth/size, and invalid references fail safely.
• Duplicate replay is idempotent.
• Conflicting same-ID annotations are reported and neither version is silently lost.
• Snapshot -> verify -> hydrate preserves the envelope.
• JSONL.GZ behaves identically to JSONL.
• One-way delta replay preserves annotations once Design incremental HotMem company-brain synchronization #73 exists.
• Deterministic snapshot identity is not affected by map insertion order.
• Existing API, MCP, TypeScript, Python, snapshot, and golden compatibility tests pass.

Performance and lean-core constraints

• No required dependency.
• No LLM or network call.
• No graph database.
• No table per annotation type.
• Validation and serialization are linear in annotation payload size.
• Records without annotations incur negligible overhead; include a microbenchmark.
• Annotation size is visible in diagnostics and package manifests where appropriate.

Explicitly out of scope

• Entity extraction, NER, entity linking, embeddings, or enrichment models.
• Knowledge-graph storage or traversal.
• Mental-model generation.
• Claim verification.
• Hosted enrichment services.
• Automatic conflict resolution.
• Mandatory schema understanding by receiving runtimes.

Definition of done

• Versioned namespaced envelope is documented.
• Unknown annotations are preserved losslessly.
• Validation limits and conflict behavior are tested.
• Clone/hydrate and repeat replay are deterministic.
• External mapping fixtures exist without vendor runtime dependencies.
• No core dependency or network behavior is added.
• Full tests, lint, and strict docs build pass.

Dependencies

• Define HotMem interchange and company-brain clone package v1 #67 interchange contract.
• Ship verified portable company-brain dump and hydrate workflow #69 verified clone/hydrate workflow.
• Design incremental HotMem company-brain synchronization #73 for synchronization-specific acceptance criteria only.

[valiantone]

Additive enterprise annotation boundary

The envelope already correctly avoids turning HotMem into a graph engine. For enterprise use, make source namespace or workspace, source authority, valid time, recorded time, and access or classification propagation explicit when those attributes are supplied. An extracted entity or relationship remains an attributed candidate, not verified shared truth.

The exact field contract is a schema decision that needs accountable approval before implementation. Preserve lossless unknown namespaces and do not make HotMem interpret access policy or traverse relationships.

This is the applicable ADR-003 guardrail: https://github.com/KnowGuard-AI/knowguard-inc-atlas/blob/main/wiki/research/emos-organization-brain-and-bounded-loop-engineering.md