From 0a808da840611f8bb1d8163d0b0c539d3ac21cd2 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Sun, 30 Aug 2026 15:58:26 -0400 Subject: [PATCH 1/6] docs(specs): competitive positioning refresh and developer-docs architecture Ground the positioning in the current agent-memory and schema-governance landscape: registries (Confluent/Glue), migration tooling (Liquibase/ Flyway/Hibernate), agent-memory systems (Zep, Mem0, Letta, Cognee), data contracts, and graph schema governance (Neo4j GRAPH TYPE, SHACL). Claims split shipped vs in-delivery vs planned. New developer-docs-architecture spec sets the docs IA: concept-anchored, 15-minute quickstart with real coordinates, opt-in feature pages, README migration plan, and the per-PR docs contract. --- specs/competitive-positioning.md | 534 +++++++++++++++++++-------- specs/developer-docs-architecture.md | 311 ++++++++++++++++ 2 files changed, 696 insertions(+), 149 deletions(-) create mode 100644 specs/developer-docs-architecture.md diff --git a/specs/competitive-positioning.md b/specs/competitive-positioning.md index a71c78c3..8001f8fc 100644 --- a/specs/competitive-positioning.md +++ b/specs/competitive-positioning.md @@ -1,198 +1,434 @@ -# DICE Competitive Positioning +# DICE competitive positioning + +Where DICE sits against the agent-memory field (Zep/Graphiti, Mem0, Letta, Cognee, Hindsight, +LangMem, and the three hyperscaler memory services), and against the enterprise data-governance +tooling whose vocabulary we should be borrowing rather than inventing. + +Two halves. The first is the claim we lead with and the evidence behind it. The second is the +per-competitor detail — kept because it's still the fastest way to answer "how do we compare on X". + +Surveyed mid-2026. Competitor surfaces move fast and several of them publish roadmap blogs as if +they were release notes, so re-verify anything here before it goes into a bakeoff or a deck. + +## Delivery status: read this before quoting anything below + +The governance story is **direction, not shipped surface**. `dice-metamodel` — `MetamodelVersion`, +`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `MetamodelVersionStore` — is an +unmerged PR train. Detection and quarantine are designed and not built. Everywhere below, a +metamodel or drift capability is marked **(in delivery)** or **(planned)**, and those markers are +load-bearing: this doc is for positioning conversations, and a present-tense claim about an unmerged +module is the exact thing that collapses in an eval. Say "this is where we're going and here's the +design" — that's still a differentiated answer, because nobody else is going there at all. + +What *is* shipped and can be claimed flatly: the proposition model with confidence and decay, +multi-strategy entity resolution, source provenance (`ProvenanceEntry`), collector decision traces +(`CollectorTraceStore`), the conflict-detection SPI, and the pluggable store family. + +## The claim + +**DICE is the agent-memory substrate with schema governance an enterprise data team will +recognise.** + +Every serious competitor has solved *extraction* and most have solved *entity resolution*. None of +them has solved *governance of the schema that extraction runs against* — and none of them has it on +a public roadmap. Across Zep/Graphiti, Mem0, Letta, Cognee and Hindsight, not one ships, and DICE is +building: + +- governed metamodel versioning — a schema identity you can write down, store, and compare later + **(in delivery)**; +- provenance from a stored claim back to the source text it came from **(shipped)**, and a + correlatable trace of the extraction run that produced it **(planned — see below)**; +- drift quarantine as a first-class state, distinct from delete **(planned)**; +- conflict resolution behind an explicit, pluggable SPI rather than an opaque LLM call **(shipped)**, + with versioned and audited policies **(planned)**. + +That is a category gap, not a feature gap. The systems that treat memory as a datastore have +skipped the layer that every enterprise data platform grew in its second year. Our advantage is that +we're building it deliberately; it is not yet an advantage we can demo end to end. + +## What the competition actually has + +Differentiation that lies dies in the first eval. Stated straight, and generously: + +| System | What it genuinely has | +|---|---| +| **Zep / Graphiti** | Bi-temporal edges (`valid_from`/`valid_until` plus ingestion time), cross-session entity dedup, Leiden community clustering with summaries, custom entity types via Pydantic, five reranking strategies, SOC 2 on the managed offering. The strongest temporal model in the field. | +| **Mem0** | Very fast single-pass ingestion (v3, April 2026, dropped the UPDATE/DELETE phases to cut latency), hybrid semantic + BM25 + entity-match retrieval, graph memory across Neo4j/Memgraph/Neptune/Kuzu, full SQLite change history, mentions counting. | +| **Letta (MemGPT)** | Agent-managed context with archival memory in Postgres/pgvector; the agent itself decides what to promote and how to resolve conflicts, via tool calls. Honest about being unstructured. | +| **Cognee** | The closest thing to declared schema in the field: LLM extraction into RDF triples with Pydantic validation, auto-generated ontology, and a custom-schema override. Shape validation, not versioning. | +| **Hindsight** | Structured facts into a knowledge graph with real entity resolution ("Alice" vs "my coworker Alice") and quality that compounds across sessions. | +| **LangMem** | Excellent extraction prompts — confidence-qualified, surprise-prioritised, SNR-shaped — plus gradient-style prompt optimisation and dilated-window retrieval. | +| **Google / AWS / Microsoft** | Fully managed, zero-infrastructure, IAM-scoped, framework-integrated. Flat fact strings underneath. | + +Two of these are ahead of us on things we care about. Zep's bi-temporal model is better than our +system timestamps. Cognee's Pydantic validation is a real declared-shape check at write time. Say so. + +## Where the governance line falls + +| Capability | Zep | Mem0 | Letta | Cognee | Hindsight | DICE | +|---|---|---|---|---|---|---| +| Schema versioning | — | — | — | Pydantic override, unversioned | — | Content-hashed `MetamodelVersion` (in delivery) | +| Type hierarchy / ontology | Implicit (clustering) | — | — | RDF/RDFS, auto-generated | — | Declared `DataDictionary`, per-type governed (in delivery) | +| Source provenance | Bi-temporal timestamps only | — | — | — | — | `ProvenanceEntry` edges, append-only (shipped) | +| Extraction-run trace | — | — | — | — | — | Planned (no run identifier today — issue #67) | +| Collector decision audit | — | Full SQLite change history | — | — | — | `CollectorTraceStore` (shipped) | +| Contradiction handling | Temporal windows, manual | ADD-only (sidesteps it) | Manual, agent-driven | Unclear | Unclear | `ConflictDetector` SPI, both claims retained (shipped) | +| Versioned / audited conflict policy | — | — | — | — | — | Planned | +| Drift detection (declared vs observed) | — | — | — | — | — | Planned (tier 2) | +| Drift quarantine | — | — | — | — | — | Planned (tier 3) | + +Four honest caveats. Zep and Cognee may have unpublished lineage APIs; this table is built from +public docs. Mem0's change history is genuinely better than anything we have for "what changed and +who changed it" — it is only weaker on *what schema it was valid under*. Our audit story is two +separate mechanisms, not one chain: `ProvenanceEntry` links a proposition to the source chunks it +came from, and `CollectorTraceStore` records why a collapse or merge decision was made. Neither +carries an extraction-run identifier, so there is no end-to-end "this claim came from that run under +that schema" trace today — that's planned work (issue #67), and the metamodel stamp is the piece +that would complete it. And the whole tier ladder — set out in `docs/design/metamodel-versioning.md`, +which lands with the `dice-metamodel` module — is designed, with only tier 1 in delivery. + +Claim the architecture, not the checkbox. + +## Identity semantics: content-addressed, not sequential + +Confluent Schema Registry and AWS Glue both assign **sequential** version identity — Confluent a +global immutable integer schema ID plus a per-subject version number, Glue a per-schema version +number plus an opaque version ID. Registration is a stateful event: the same schema posted twice +gets one ID and one version because the registry remembers, not because the identity is derived from +the content. + +DICE derives identity from the content. That design is in delivery, not merged, so the whole of this +section is a statement of direction. `MetamodelVersion.contentHash` is a SHA-256 over the structural +fields — sorted type names, per-type labels and properties, sorted relationship descriptors — with +every token length-prefixed so `["a;b"]` can't collide with `["a", "b"]`. Three consequences worth +stating in any comparison: + +- **Registration is idempotent by construction.** `saveVersion` upserts on `(schemaName, + contentHash)`; a re-save lands on identical content by definition. No sequence counter to get out + of step across environments. +- **Schema is data.** A dev and a prod schema of identical shape hash identically, so they compare + directly. The registries can't do that — same schema, different subject, different version number. +- **Stamps are portable.** A proposition carries `dice.metamodel.version` as metadata, so the schema + a claim was extracted under travels with the claim rather than living in a registry it must be + looked up in. + +The second divergence is **granularity**. Both registries configure compatibility **per subject**, +flat: one mode for the whole schema. DICE governs **per type**, via `GovernedTypeSelector`. That's +not a stylistic difference — it's what lets one domain be closed-world where it matters (`Person`, +`Company`) and open-world everywhere else, so an exploratory type LLM extraction just invented +doesn't churn the version history. + +**Adopt their vocabulary where it fits.** BACKWARD / FORWARD / FULL, and the transitive variants, +are well-understood terms for what a compatibility check means, and our detect tier should use them +rather than coining synonyms. Glue's *checkpoint* — a selectable reference point to compare against, +rather than "previous version" or "all versions" — is more expressive than transitive-yes/no and is +the right model if governed types ever get release windows. + +**Name the divergence.** Both registries **hard-reject** an incompatible registration: HTTP 409, no +warn mode, no soft landing. DICE does not, and won't. Extraction is LLM-driven; a type nobody +declared is often a real finding. Rejecting at write time is the one decision that can't be undone +later. + +## Opt-in and drift: observe, never auto-mutate + +The migration and ORM world has already run this experiment, and the developer verdict is +unambiguous. + +**Loved:** + +- **`liquibase diff` / `diff-changelog`** — compares declared (reference) against observed (target) + and *generates* the changesets that would close the gap. It reports; a human decides whether to + apply or mark as run. +- **`flyway validate`** — checksums applied migrations against local ones and fails loudly on + mismatch. Explicitly bound to a build phase, not fired automatically on every deploy. +- **Hibernate `hbm2ddl.auto=validate`** — check only, fail on mismatch, change nothing. And + `hbm2ddl.auto=none` as the shipped default: no schema management unless you asked for it. +- **`@Version` per entity** — versioning is an annotation you put on the entities you chose. Nothing + is versioned by inference. + +**Hated:** `hbm2ddl.auto=update`. Uncontrolled schema evolution at startup, silently creating +columns and leaving zombies behind when a `@Column(name=...)` changes. A production anti-pattern in +every guide that mentions it. + +DICE's stance follows directly, and it's a one-line elevator pitch: **liquibase diff for knowledge +graphs.** Observe and quarantine, never auto-mutate. That's the design; here is what it commits us +to, tier by tier: + +- Nothing is versioned unless an application supplies a `DeclaredSchemaSource` (in delivery). No + declared schema, no versioning — the Hibernate `none` default, not the `update` one. +- Governance is per governed type via `GovernedTypeSelector` (in delivery), the `@Version` model + rather than a global switch. +- The detect tier (planned) will report the declared-vs-observed gap. It will not close it. Closing + it stays a human act, the way Liquibase generates changesets rather than applying them. +- The quarantine tier (planned) marks affected propositions stale. Stale is reversible; delete is + not. + +The failure mode to watch, straight from the same brief: **opt-in fatigue**. All three tools require +explicit configuration, teams ship with defaults, and later regret having no drift visibility. Our +answer isn't to turn governance on by default — it's to make the on-switch a one-bean decision and to +lead every governance doc page with the gate. + +## Escalation vocabulary: say what the data team says + +OpenMetadata, DataHub, Great Expectations, Soda and the Open Data Contract Standard have converged on +one vocabulary. Use it verbatim. Every term we coin instead is a term a buyer has to translate. + +| Term | Their meaning | Ours | Status | +|---|---|---|---| +| **Drift** | Structural or statistical divergence over time | Declared metamodel vs what the graph actually holds | Planned | +| **Contract** | Declared structure + quality + ownership | `DeclaredSchema` — the governed types and their shape | In delivery | +| **Assertion** | A comparison check fired on change | The declared-vs-observed comparison in the detect tier | Planned | +| **Policy** | Enforcement rule, configurable strictness | `SchemaAdherence` (STRICT/DEFAULT/RELAXED) shipped; `ConflictDetector` SPI shipped; versioned policy planned | Mixed | +| **Incident** | Raised on assertion failure, routed to an owner | `DriftReport` — the record a detection produces | Planned | +| **Quarantine** | Invalid records split off; the valid stream continues | Affected propositions marked stale, not deleted | Planned | + +The escalation ladder they all implement is **observe → alert → block**. DICE's three tiers are +designed to map onto it directly: -Based on deep analysis of Zep/Graphiti, Mem0, LangChain/LangMem, Google Vertex AI Memory Bank, AWS Bedrock AgentCore Memory, Microsoft Foundry Agent Service, and Neo4j Agent Memory. +| DICE tier | Their stage | What happens | Status | +|---|---|---|---| +| Stamp and observe | Observe | Schema gets an identity; propositions carry the stamp. No opinions. | In delivery | +| Detect and report | Alert | Compare declaration against declaration, and declaration against live graph. Emit a `DriftReport`. | Planned | +| Quarantine | Block (soft) | Mark affected propositions stale. The valid set stays queryable; the suspect set is excluded and recoverable. | Planned | + +The diagram below is the target flow, not current behaviour. + +```mermaid +flowchart LR + W["extraction writes"] --> Q{"governed type?"} + Q -->|"no (open-world)"| ACC["accepted, unstamped"] + Q -->|yes| ST["stamped with contentHash"] + ST --> D{"declared vs observed"} + D -->|conforms| VALID["valid set — queryable"] + D -->|"diverges"| DR["DriftReport"] + DR --> QU["quarantined — marked stale,
excluded from reads, reversible"] + QU -.->|"human review"| VALID +``` + +Note the shape of the split: the pipeline **continues**. Quarantine is a valid/invalid partition, not +a halt — exactly the hybrid default the Spark/Soda world settled on. Fail-fast is what you escalate +to when quality drops below a threshold, not what you do on the first bad record. + +Two decisions this vocabulary forces on us, both still open: + +- **Is quarantine a metadata flag or physical routing?** Enterprise tools do both and neither is + canonical. DICE's stale-marking is a flag; whether reads exclude by default is the real question. +- **Do we route incidents?** Their incident model assumes metadata ownership drives notification. + DICE has events; it has no owner model. `DriftReport` without a routing story is a log line. + +## Graph-native convergence: Neo4j GRAPH TYPE + +Neo4j's GRAPH TYPE (preview, 2025) declares a schema once in Cypher DDL and enforces it at write +time, with SET/ADD/ALTER/DROP lifecycle commands and an Open variant that requires declared fields +while tolerating extras. TypeDB does the same thing more strictly: nothing can be written that +doesn't conform, and `redefine` checks existing instances before a migration commits. + +**Read this as validation, not threat.** The store vendors are independently concluding that a +property graph needs a declared schema coupled to enforcement. That's the direction we're already +walking. The differentiation is *where the governance sits*. + +DICE governs **above the store** — that's the architectural bet, with tier 1 in delivery and the +rest designed: + +- **Per context, not per database.** Governance scopes to a `contextId`, so one deployment can hold + a governed tenant and an exploratory one. GRAPH TYPE is a property of the graph. +- **Open-world by default.** GRAPH TYPE and TypeDB reject the undeclared. DICE tolerates it — an + exploratory type is a finding, and per-type governance is what makes that safe rather than sloppy. + Today's `SchemaAdherence` already takes this side of the trade at extraction time. +- **Quarantine rather than reject** (planned). Write-time enforcement means a non-conforming + extraction is lost. Ours would be stored, flagged, and reviewable. +- **Survives a backend swap.** DICE's schema lives in `SchemaRegistry`/`DataDictionary`, not in the + database. In-memory, Neo4j, or something later — the governance story is the same one. Adopting + GRAPH TYPE would rewrite it. + +The honest cost of governing above the store: our schema is application-enforced, so it *can* drift +if some entry point skips validation. Neo4j's cannot. That's a real trade, and the detect tier is +planned precisely because we took the flexible side of it — which also means that until that tier +lands, the drift it's meant to catch is undetected rather than prevented. + +**SHACL validation reports are the prior art for `DriftReport`.** SHACL is the only standardised +violation-report format in this space: a conformance flag at the root, one result node per violation, +each carrying the focus node, the failed constraint, severity, and a human-readable message. Copy the +shape. Don't copy the serialisation — SHACL reports are RDF graphs, which would make every consumer +of a `DriftReport` parse RDF. PG-Schema and ProGS are the property-graph analogues and are worth +watching, but neither has a canonical report format to borrow. + +## What we copy, what we reject, what only we do + +| | Copy | Reject | Only us | +|---|---|---|---| +| **Schema registries** (Confluent, Glue) | BACKWARD/FORWARD/FULL vocabulary; Glue's checkpoint idea; idempotent registration | Sequential version IDs; flat per-subject compatibility; hard-reject on incompatible registration | Content-addressed identity; per-type governed selection; the stamp travelling with the data | +| **Migration & ORM** (Liquibase, Flyway, Hibernate) | Declared-vs-observed diff that reports rather than applies; validate-mode; per-entity `@Version` opt-in | `hbm2ddl.auto=update` auto-mutation; anything that changes a schema at startup | Drift detection over LLM-extracted knowledge, where the "observed" side is a graph nobody wrote by hand | +| **Data governance** (DataHub, Soda, ODCS) | drift / contract / assertion / policy / incident / quarantine, verbatim; observe→alert→block; valid/invalid split | Blocking at the producer boundary as a default; contract YAML as the primary authoring surface | Governance over probabilistic extraction — confidence-weighted claims, not rows | +| **Graph schema** (Neo4j GRAPH TYPE, TypeDB, SHACL) | SHACL's validation-report structure; GRAPH TYPE's Open variant semantics | Write-time rejection; RDF as a wire format; store-coupled schema declaration | Governance that works per context, tolerates open-world types, and survives a backend swap | +| **Agent memory** (Zep, Mem0, Cognee, LangMem) | Zep's bi-temporal model (GAP-4B); Mem0's integer re-indexing and change history; LangMem's extraction prompts; Cognee's declared-shape validation | ADD-only ingestion; hard delete on contradiction; opaque LLM consolidation | Source provenance and collector traces (shipped); governed metamodel versioning (in delivery); drift quarantine and versioned conflict policy (planned) — nobody else has any of the four, shipped or roadmapped | -## DICE Strengths +## Per-competitor detail -| Capability | DICE Implementation | Competitors | -|---|---|---| -| Batch classification | N propositions in 1 LLM call via `classifyBatch()` | Zep: sequential per-edge. Mem0: sequential per-fact. LangMem: sequential tool calls. | -| Auto-merge fast path | Embedding score >= 0.95 skips LLM entirely | Zep: text-identical fast path only (no similarity threshold). Others: none. | -| Outcome-dependent decay | Contradicted +0.15, merged *0.7, reinforced *0.85 | Zep: binary temporal invalidation. Mem0: hard delete. LangMem: none. | -| Canonical text dedup | Cheap string match before vector search | Zep: similar for identical edge text but only after vector search. | -| 5-way classification | IDENTICAL/SIMILAR/CONTRADICTORY/UNRELATED/GENERALIZES with edge-case guidance + few-shot | Mem0: only ADD/UPDATE/DELETE/NONE. LangMem: insert/update/delete tool calls. | -| JVM native | Only JVM-based memory system | All competitors are Python (Zep also Go). | -| Confidence + decay model | Per-proposition confidence with exponential time decay from GUM paper | Zep: no decay. Mem0: no confidence model. LangMem: asks for p(x) in prompts but no decay math. | -| Extraction quality | SNR-maximizing, confidence-qualified with hedging detection, role-aware (USER/AGENT/ALL), schema-bound | Comparable to LangMem prompts; better than Zep/Mem0. | -| Reinforcement frequency | `reinforceCount` tracks merge/reinforce frequency, queryable via `PropositionQuery` | Mem0: `mentions` counter on graph nodes. Zep/LangMem: none. Google/AWS/Microsoft: none. | -| Entity resolution | Fuzzy name, vector, exact name, partial name, agentic candidate searchers; LLM disambiguation | Zep: Neo4j entity nodes. Mem0: graph nodes. Google/AWS/Microsoft: none — flat fact strings. | -| Abstraction hierarchy | Multi-level propositions (level 0 = raw, level 1+ = synthesized) with source tracking | None of the competitors support multi-level abstraction. | -| Contradiction retention | Both propositions retained with reduced confidence | Zep: temporal invalidation. Mem0/Google/AWS/Microsoft: hard delete. LangMem: none. | -| Portability | Embeddable JVM library, no cloud dependency | Google: GCP only. AWS: AWS only. Microsoft: Azure only. Zep: Neo4j required. Mem0: self-hosted or cloud. LangMem: Python/LangGraph. | - -## vs Zep/Graphiti +### vs Zep/Graphiti | Dimension | DICE | Zep | Edge | |---|---|---|---| | Ingestion speed | Batch classify + auto-merge + canonical dedup | Sequential only ("must be awaited") | **DICE** | | Classification nuance | 5-way with edge cases + few-shot | Duplicate vs contradicted (binary) | **DICE** | | Confidence model | Exponential decay + outcome-dependent adjustment + reinforceCount | No decay, no confidence scoring | **DICE** | -| Extraction quality | SNR, confidence-qualified, role-aware, schema-bound | Custom ontology via Pydantic, entity validation | Tie | +| Extraction quality | SNR, confidence-qualified, role-aware, schema-bound | Custom entity types via Pydantic, entity validation | Tie | | Temporal model | System timestamps only (created/revised) | Bi-temporal (valid_at/invalid_at/expired_at) | **Zep** | | Graph structure | Propositions + entity mentions, no graph DB required | Full knowledge graph in Neo4j with community detection | **Zep** | | Retrieval | Vector similarity + canonical match | Cosine + BM25 + BFS + 5 rerankers | **Zep** | +| Schema governance | Content-hashed metamodel versions, per-type opt-in (in delivery) | None — entity types are code, not versioned artifacts | **DICE**, on direction | | Infrastructure weight | Embeddable, JVM-native, no external deps | Requires Neo4j + embedding service + LLM | **DICE** | -**Their moat**: Bi-temporal fact model, custom ontology via Pydantic, 5 reranking strategies, community subgraph summaries, Neo4j-backed graph traversal. +**Their moat**: bi-temporal fact model, custom entity types via Pydantic, five reranking strategies, +community subgraph summaries, Neo4j-backed traversal. -**Their weakness**: Sequential-only ingestion ("episodes must be added sequentially and awaited"), Python/Go only, heavy infrastructure (Neo4j required), no batch classification. +**Their weakness**: sequential-only ingestion, Python/Go only, heavy infrastructure, no schema +versioning. Their audit story is SOC 2 on the managed service — an org-level control, not a +data-lineage API. -**Attack angle**: DICE is embeddable — no Neo4j dependency. Batch pipeline is faster for high-throughput ingestion. Position as "memory for JVM agents" vs their "memory infrastructure platform." Bi-temporal model (GAP-4B) closes their biggest technical advantage. +**Attack angle**: embeddable, no Neo4j requirement, faster at high-throughput ingestion. Governance +is the clean differentiator; the temporal gap (GAP-4B) is the thing to close before claiming parity. -## vs Mem0 +### vs Mem0 | Dimension | DICE | Mem0 | Edge | |---|---|---|---| -| Classification | 5-way taxonomy with edge-case guidance | 4-op (ADD/UPDATE/DELETE/NONE) | **DICE** | -| Dedup pipeline | Canonical + auto-merge + batch LLM | Sequential per-fact, top-5 candidates | **DICE** | -| Frequency signal | `reinforceCount` on propositions, queryable | `mentions` counter on graph nodes/edges | Tie | -| Confidence model | Decay + outcome adjustment + qualification at extraction | None — no confidence scores | **DICE** | +| Classification | 5-way taxonomy with edge-case guidance | v3 is ADD-only; supersession and contradiction aren't expressible | **DICE** | +| Dedup pipeline | Canonical + auto-merge + batch LLM | Entity linking at retrieval time, not at write | **DICE** | +| Ingestion latency | Batch LLM call per chunk | Single-pass, deliberately minimal | **Mem0** | +| Confidence model | Decay + outcome adjustment + qualification at extraction | None | **DICE** | | ID safety | Integer re-indexing prevents hallucination | Integer re-indexing prevents hallucination | Tie | -| Role-aware extraction | `ExtractionPerspective` enum (ALL/USER/AGENT) | Separate user vs agent prompts with penalty framing | Tie | -| Graph memory | No graph DB | Neo4j/Memgraph/Neptune/Kuzu | **Mem0** | -| Audit trail | Grounding chain + reinforceCount | Full SQLite history (old/new/event/actor) | **Mem0** | +| Graph memory | Entity mentions + Neo4j projection | Neo4j/Memgraph/Neptune/Kuzu | **Mem0** | +| Audit trail | Source provenance + collector decision traces, no run-level trace | Full SQLite history (old/new/event/actor) | **Mem0** | +| Schema governance | Governed metamodel versions (in delivery) | None | **DICE**, on direction | -**Their moat**: Graph memory with Neo4j/Memgraph/Neptune/Kuzu, vision support, procedural memory for agent traces, mentions counting. +**Their moat**: graph memory across four stores, vision, procedural memory for agent traces, a real +change history, and the lowest ingestion latency in the field. -**Their weakness**: Coarse 4-operation model (ADD/UPDATE/DELETE/NONE) — no SIMILAR/GENERALIZES distinction. Sequential per-fact processing. Graph memory is a separate bolted-on pipeline. +**Their weakness**: v3 traded expressiveness for speed. ADD-only means contradictions accumulate +unresolved; there is no supersession semantics to appeal to. -**Attack angle**: DICE's 5-way classification preserves nuance that Mem0's 4-op model loses. Unified revision pipeline vs their split vector+graph paths. ID hallucination prevention (GAP-6) adopts their best defensive technique. +**Attack angle**: their change history is genuinely good and ours should be at least as queryable. +Where we win is that their history records *what changed*, not *what schema it was valid under*. -## vs LangChain/LangMem +### vs Cognee -| Dimension | DICE | LangMem | Edge | +| Dimension | DICE | Cognee | Edge | |---|---|---|---| -| Extraction prompts | SNR, confidence-qualified, role-aware, few-shot | Confidence-qualified, surprise-prioritized, SNR | Tie | -| Dedup/classification | Structured 5-way pipeline with fast paths | LLM tool calls (insert/update/delete), no structured classification | **DICE** | -| Batch processing | N propositions in 1 LLM call | Sequential tool calls | **DICE** | -| Prompt optimization | Not applicable | Gradient-based prompt evolution | **LangMem** | -| Retrieval | Vector similarity | Dilated windows + LLM-generated queries | **LangMem** | -| Graph memory | Entity mentions on propositions | Commented-out prototype | **DICE** | -| Background processing | Synchronous pipeline | Debounced async reflection | **LangMem** | -| Ecosystem | JVM/Spring native | Python/LangGraph locked | Depends on stack | +| Declared shape | `DataDictionary` + `SchemaAdherence` (STRICT/DEFAULT/RELAXED), shipped | Pydantic models, optionally overriding an auto-generated ontology | Tie | +| Schema versioning | Content-hashed, stored, comparable (in delivery) | None — validation is per-run | **DICE**, on direction | +| Ontology | Declared types and relationships | RDF/RDFS triples, auto-generated from the corpus | **Cognee** | +| Validation failure handling | Adherence policy; non-conforming extraction retained and flaggable | Pydantic rejects the shape; no error feedback loop | **DICE** | +| Provenance | `ProvenanceEntry` edges to source chunks | Not documented | **DICE** | -**Their moat**: Excellent extraction prompts, prompt optimization via gradient analogy, debounced background reflection, dilated windows retrieval. +**Their moat**: the only competitor with a real declared-schema check at write time, and +auto-generated ontology lowers the setup cost a lot. -**Their weakness**: Graph memory is literally commented-out code. No structured dedup pipeline — relies on LLM tool calls for consolidation. Tightly coupled to LangGraph ecosystem. +**Their weakness**: shape validation without versioning. Change the Pydantic model and nothing tells +you what previously-stored data no longer conforms — which is the exact problem the detect tier +exists to solve. -**Attack angle**: LangMem has great prompts but weak infrastructure. DICE's explicit classification taxonomy + fast paths + batch processing is more reliable at scale. We've adopted their best prompt ideas and combined them with our superior pipeline mechanics. +**Attack angle**: closest competitor conceptually, and the one worth watching. Pydantic validation is +a good idea we should match at the extraction boundary. -## vs Google Vertex AI Memory Bank +### vs Letta (MemGPT) and Hindsight -| Dimension | DICE | Google Memory Bank | Edge | -|---|---|---|---| -| Data model | Rich `Proposition` with entity mentions, confidence, decay, grounding, reinforceCount | Flat `fact` string with no structured sub-components | **DICE** | -| Confidence/decay | Exponential decay from GUM paper, outcome-dependent adjustment | None — facts are binary (exist or deleted) | **DICE** | -| Entity resolution | Multi-strategy entity resolution with LLM disambiguation | None — no entity model | **DICE** | -| Classification nuance | 5-way (IDENTICAL/SIMILAR/CONTRADICTORY/UNRELATED/GENERALIZES) | 3-outcome (CREATED/UPDATED/DELETED), opaque LLM consolidation | **DICE** | -| Abstraction | Multi-level hierarchy with source tracking | None — all facts at same level | **DICE** | -| Contradiction handling | Both retained with reduced confidence | Contradicting memory deleted — history lost | **DICE** | -| Dedup transparency | Configurable thresholds + LLM classification | Opaque LLM-based consolidation, no control over merge logic | **DICE** | -| Graph projection | Entity mentions map to Neo4j relationships, Prolog facts | None | **DICE** | -| Managed service | No — self-hosted | Fully managed, zero infrastructure | **Google** | -| Multimodal | Text only | Images, video, audio extraction | **Google** | -| Topic-based extraction | Schema hints + extraction guidance | Managed + custom topics with per-topic filtering | **Google** | -| TTL | Threshold-based retirement via decay | Granular TTL (create, generate-created, generate-updated) | **Google** | -| Retrieval | Vector similarity + canonical match + entity-based queries | Euclidean distance similarity + regex/metadata filtering | **DICE** | +Letta declines to have a schema at all: core memory managed by the LLM, archival memory as +pgvector passages, conflicts resolved by the agent through tool calls. Consolidation is agent-driven, +not governed. Nothing to compare on governance; the honest comparison is philosophical — Letta trusts +the agent, DICE trusts the substrate. -**Their moat**: Fully managed GCP service with zero infrastructure overhead, multimodal extraction (images/video/audio), topic-based extraction with managed topics, built-in TTL, IAM-scoped access control, ADK integration with PreloadMemoryTool. +Hindsight structures facts into a graph with entity resolution good enough to link "Alice" to "my +coworker Alice", and improves across sessions. Smaller product surface than the incumbents, no +documented temporal invalidation or contradiction framework. Worth re-checking; the entity-resolution +quality is the part that could matter. -**Their weakness**: Flat fact model with no entity resolution, no confidence or decay, no abstraction hierarchy, no reinforcement counting, opaque consolidation logic, `CreateMemory` bypasses dedup entirely, top_k defaults to 3, GCP vendor lock-in. +### vs LangChain/LangMem -**Attack angle**: Memory Bank solves "I don't want to build memory infrastructure" but trades away all the expressiveness that makes memory useful at scale. DICE's proposition model carries structured metadata (confidence, decay, entity mentions, grounding, reinforceCount) enabling richer consolidation, retrieval, and reasoning. For teams that need more than flat fact storage — entity-centric queries, confidence-weighted retrieval, graph projection — DICE is fundamentally more capable. +| Dimension | DICE | LangMem | Edge | +|---|---|---|---| +| Extraction prompts | SNR, confidence-qualified, role-aware, few-shot | Confidence-qualified, surprise-prioritised, SNR | Tie | +| Dedup/classification | Structured 5-way pipeline with fast paths | LLM tool calls (insert/update/delete) | **DICE** | +| Batch processing | N propositions in 1 LLM call | Sequential tool calls | **DICE** | +| Prompt optimisation | Not applicable | Gradient-based prompt evolution | **LangMem** | +| Retrieval | Vector similarity | Dilated windows + LLM-generated queries | **LangMem** | +| Graph memory | Entity mentions on propositions | Commented-out prototype | **DICE** | +| Background processing | Synchronous pipeline | Debounced async reflection | **LangMem** | -## vs AWS Bedrock AgentCore Memory +**Attack angle**: great prompts, weak infrastructure. We've taken the prompt ideas; the pipeline +mechanics are ours. -| Dimension | DICE | AWS AgentCore Memory | Edge | -|---|---|---|---| -| Data model | Rich `Proposition` with entity mentions, confidence, decay, grounding, reinforceCount | Flat `fact` string (`{"fact": "..."}`) with rigid output schema | **DICE** | -| Confidence/decay | Exponential decay + outcome-dependent adjustment | None — no confidence field on memory records | **DICE** | -| Entity resolution | Multi-strategy resolution with LLM disambiguation | None — facts are flat text strings, no entity linking | **DICE** | -| Classification nuance | 5-way taxonomy with edge-case guidance | 3-op consolidation (AddMemory/UpdateMemory/SkipMemory) | **DICE** | -| Contradiction handling | Both retained with reduced confidence | AddMemory creates new entry (contradictions not explicitly detected) | **DICE** | -| Provenance | Grounding chain links propositions to source chunks | None — no link back to source text | **DICE** | -| Dedup transparency | Configurable thresholds + deterministic fast paths + LLM classification | LLM-based only — non-deterministic, expensive per consolidation | **DICE** | -| Knowledge types | Explicit `KnowledgeType` enum (SEMANTIC/EPISODIC/PROCEDURAL/WORKING) with classifier | Implicit via strategy choice (semantic/preference/summary/episodic) | Tie | -| Managed service | No — self-hosted | Fully managed with control plane + data plane | **AWS** | -| Strategy tiers | N/A | Built-in, built-in with overrides, self-managed — three customization levels | **AWS** | -| Episodic memory | Propositions with EPISODIC knowledge type | Automatic episode detection with cross-episode reflection | **AWS** | -| Session management | Application-managed | Built-in session/actor model with branching | **AWS** | -| Framework integrations | JVM/Spring native | LangChain, LangGraph, AutoGen, Strands out of the box | **AWS** | -| Retrieval | Vector similarity + canonical match + entity-based + composable PropositionQuery | Cosine similarity only on flat text with metadata filters | **DICE** | -| Sync extraction | Synchronous pipeline | Async-only with `time.sleep(60)` in examples | **DICE** | - -**Their moat**: Fully managed AWS service with three strategy tiers (built-in → override → self-managed), episodic strategy with automatic episode detection and cross-episode reflection, session/actor model with branching, CDK/IaC support, framework integrations (LangChain/AutoGen/Strands). - -**Their weakness**: Flat `{"fact": "..."}` model with no entity resolution. No confidence/decay — all memories equally weighted. No provenance tracking. LLM-dependent deduplication is expensive and non-deterministic. Built-in schema is rigid and non-editable. Async-only extraction means stored information isn't immediately available. Deep AWS vendor lock-in (IAM, S3, SNS, KMS, Lambda). - -**Attack angle**: AgentCore Memory is a reasonable managed service for simple preference/fact storage in chatbots. But for systems that need structured knowledge with entity resolution, confidence-weighted decaying memory, provenance tracking, deterministic deduplication, and multi-strategy retrieval, DICE's proposition model is fundamentally more expressive. Both Google and AWS validate that agent memory is a critical capability — DICE provides it without vendor lock-in and with far richer knowledge representation. - -## vs Microsoft Foundry Agent Service - -| Dimension | DICE | Microsoft Foundry | Edge | -|---|---|---|---| -| Data model | Rich `Proposition` with entity mentions, confidence, decay, grounding, reinforceCount | Flat memory "items" — no structured sub-components | **DICE** | -| Memory types | Unified proposition model with `KnowledgeType` classifier (SEMANTIC/EPISODIC/PROCEDURAL/WORKING) | Two types only: user profile (static preferences) and chat summary (distilled conversation) | **DICE** | -| Confidence/decay | Exponential decay + outcome-dependent adjustment | None — memories are binary (exist or removed) | **DICE** | -| Entity resolution | Multi-strategy resolution with LLM disambiguation | None — no entity model | **DICE** | -| Classification nuance | 5-way (IDENTICAL/SIMILAR/CONTRADICTORY/UNRELATED/GENERALIZES) | Opaque LLM-based consolidation, no user-visible classification taxonomy | **DICE** | -| Contradiction handling | Both retained with reduced confidence | "Conflicting facts are resolved" — old value discarded | **DICE** | -| Abstraction | Multi-level hierarchy with source tracking | None — all memories at same level | **DICE** | -| Graph projection | Entity mentions map to Neo4j relationships, Prolog facts | None | **DICE** | -| Dedup transparency | Configurable thresholds + deterministic fast paths + LLM classification | Opaque LLM-based consolidation, "behavior can vary by memory type and may change during preview" | **DICE** | -| Retrieval | Vector similarity + canonical match + entity-based + composable PropositionQuery | Memory search (details opaque), scope-based filtering | **DICE** | -| Managed service | No — self-hosted | Fully managed Azure service, zero infrastructure | **Microsoft** | -| Integration model | JVM/Spring native, embeddable library | Memory search tool auto-attached to prompt agents, or low-level Memory Store APIs | **Microsoft** | -| Scale limits | Application-determined | 100 scopes/store, 10K memories/scope, 1K req/min | **DICE** | +### vs the managed services (Google, AWS, Microsoft) -**Their moat**: Fully managed Azure service with zero infrastructure overhead, two-method access (agent tool for simple use, APIs for advanced), integration with Azure AI Content Safety for prompt injection detection, built-in scope-based multi-tenancy. +All three converge on the same shape: a flat fact string, opaque LLM consolidation, hard delete on +contradiction, no entity model, no confidence, no provenance, and full vendor lock-in — traded +against zero infrastructure and native IAM. -**Their weakness**: Simplest memory model of any competitor — only two memory types (user profile + chat summary), no entity resolution, no confidence or decay, no abstraction hierarchy, no reinforcement counting, opaque consolidation logic that "may change during preview," hard 10K memory limit per scope, Azure vendor lock-in. +| Dimension | DICE | Google Memory Bank | AWS AgentCore | Microsoft Foundry | +|---|---|---|---|---| +| Data model | Structured `Proposition` | Flat `fact` string | Flat `{"fact": "..."}` | Flat memory "items" | +| Memory types | `KnowledgeType` classifier | Managed + custom topics | Strategy-scoped | Profile + chat summary only | +| Confidence/decay | Decay + outcome adjustment | None | None | None | +| Entity resolution | Multi-strategy + LLM disambiguation | None | None | None | +| Contradiction | Both retained, reduced confidence | Old deleted | New entry, no detection | Old value discarded | +| Provenance | `ProvenanceEntry` to source chunks | None | None | None | +| Governance | Metamodel versioning (in delivery), drift (planned) | None | None | None | +| Managed | Self-hosted | Fully managed | Fully managed | Fully managed | +| Scale limits | Application-determined | Not published | Not published | 100 scopes, 10K memories/scope | -**Attack angle**: Foundry's memory is the thinnest of the managed offerings — even simpler than Google Memory Bank or AWS AgentCore. It's adequate for remembering user preferences across chatbot sessions, but lacks the structured knowledge representation needed for serious agent memory. DICE's proposition model, entity resolution, confidence decay, and composable queries operate in a fundamentally different capability tier. Microsoft validates the market need but their implementation is a minimal viable feature, not a memory system. +**Attack angle**: all three validate the category and none of them is a competitor for a team that +cares what happens to knowledge over time. The interesting question these raise isn't technical, it's +whether DICE eventually wants a managed offering — the embeddable library model is a strength today. -## vs Neo4j Agent Memory +### vs Neo4j Agent Memory | Dimension | DICE | Neo4j Agent Memory | Edge | |---|---|---|---| -| Classification nuance | 5-way with edge cases + few-shot | No explicit classification taxonomy — entity resolution handles dedup | **DICE** | -| Batch processing | N propositions in 1 LLM call | Sequential extraction through cascade stages | **DICE** | -| Confidence/decay | Exponential decay + outcome-dependent adjustment + reinforceCount | No decay model — entities are binary (exist or merged) | **DICE** | -| Contradiction handling | Both retained with reduced confidence | Entities merged or left distinct — no contradiction retention | **DICE** | -| Abstraction hierarchy | Multi-level propositions with source tracking | Flat — all entities/facts at same level | **DICE** | -| Extraction pipeline | Single LLM call, SNR-maximizing | spaCy → GLiNER → LLM cascade with 5 merge strategies | **Neo4j** | -| Graph structure | Propositions + entity mentions, Neo4j as projection | Full native Neo4j knowledge graph with POLE+O ontology | **Neo4j** | -| Temporal model | System timestamps only | Facts with valid_from/valid_until + geospatial queries | **Neo4j** | -| Reasoning traces | Not applicable | Trace → Step → ToolCall hierarchy with aggregated tool stats | **Neo4j** | -| Retrieval | Vector similarity + canonical match + entity-based + composable query | Hybrid vector + graph traversal (up to 3 hops) | **Neo4j** | -| Framework integrations | JVM/Spring native | 9 frameworks (LangChain, LlamaIndex, CrewAI, etc.) + MCP server | **Neo4j** | -| Infrastructure weight | Embeddable JVM library, no external deps | Requires Neo4j 5.11+ plus spaCy/GLiNER models | **DICE** | -| Observability | Application-managed | OpenTelemetry + Opik built-in | **Neo4j** | - -**Their moat**: Native Neo4j graph with multi-hop traversal and POLE+O ontology, multi-stage extraction cascade (spaCy → GLiNER → LLM) for cost/quality tradeoffs, reasoning memory with tool call statistics, 9 framework adapters + MCP server, geospatial and temporal fact queries, entity enrichment via Wikipedia/Diffbot. - -**Their weakness**: Neo4j 5.11+ hard dependency (significant infrastructure commitment). No classification taxonomy — entities merge or stay distinct with no SIMILAR/CONTRADICTORY/GENERALIZES nuance. No confidence decay — everything equally weighted forever. No abstraction hierarchy. No batch classification. No contradiction retention. Python only. Experimental Labs project with no SLAs. - -**Attack angle**: Neo4j Agent Memory is the most architecturally sophisticated competitor — both systems take knowledge representation seriously. But the competition is proposition-centric (DICE) vs entity-centric (Neo4j). DICE manages the **lifecycle** of knowledge claims — how they evolve, conflict, reinforce, and decay. Neo4j builds a **static graph** of entities and relationships. DICE's 5-way classification, confidence decay, and abstraction hierarchy address the harder problem of knowledge evolution. And DICE is embeddable with no infrastructure requirements — it already projects to Neo4j when graph structure is needed, without requiring it. - -## Key Remaining Gaps +| Classification nuance | 5-way with edge cases + few-shot | No taxonomy — entity resolution handles dedup | **DICE** | +| Batch processing | N propositions in 1 LLM call | Sequential cascade stages | **DICE** | +| Confidence/decay | Decay + outcome adjustment + reinforceCount | None | **DICE** | +| Contradiction | Both retained with reduced confidence | Merged or left distinct | **DICE** | +| Extraction pipeline | Single LLM call, SNR-maximising | spaCy → GLiNER → LLM cascade | **Neo4j** | +| Graph structure | Propositions + mentions, Neo4j as projection | Native graph with POLE+O ontology | **Neo4j** | +| Temporal model | System timestamps only | valid_from/valid_until + geospatial | **Neo4j** | +| Retrieval | Vector + canonical + entity + composable query | Hybrid vector + up to 3-hop traversal | **Neo4j** | +| Schema enforcement | Application-layer, per context; quarantine planned | Heading toward store-level GRAPH TYPE enforcement | Different bets | +| Infrastructure weight | Embeddable, no external deps | Neo4j 5.11+ plus spaCy/GLiNER models | **DICE** | + +**Attack angle**: the most architecturally serious competitor, and the one whose direction most +validates ours. The competition is proposition-centric versus entity-centric: DICE manages the +*lifecycle* of claims, they build a graph of entities. Their store-level schema bet (GRAPH TYPE) and +our substrate-level bet are the interesting divergence — theirs is stronger inside Neo4j, ours is the +only one that survives leaving it. + +## Key remaining gaps | Gap | Blocks us against | Impact | |---|---|---| | ~~ID hallucination prevention (GAP-6)~~ | ~~Mem0~~ | ~~DONE — integer re-indexing~~ | -| Surprise-prioritized retention (GAP-2) | LangMem | Novel facts don't get durable treatment | +| Detect tier (declared vs observed) | The whole governance claim | Tier 1 is still in delivery; the claim needs tier 2 | +| Quarantine tier | Nobody, yet — but it's the headline | Claim stays architectural until it lands | +| Bi-temporal model (GAP-4B) | Zep, Neo4j | No point-in-time queries or temporal contradiction resolution | | Temporal anchoring (GAP-4A) | Zep | Relative dates stored as literal text | -| Bi-temporal model (GAP-4B) | Zep | No point-in-time queries or temporal contradiction resolution | - -## Not Worth Chasing - -- **Zep's 5-reranker retrieval**: Deep feature tied to Neo4j graph traversal. Better to invest in bi-temporal model. -- **LangMem's prompt optimization**: Gradient-based prompt evolution is interesting but orthogonal to memory quality. DICE's pipeline mechanics matter more. -- **Mem0's graph memory**: DICE already has entity mentions + Neo4j projection via the graph projector. Adding a separate graph memory pipeline would duplicate effort. -- **Google's multimodal extraction**: Interesting for image/video-heavy use cases but orthogonal to memory quality. Can be added later if needed — the proposition model is format-agnostic. -- **AWS's episodic reflection**: Cross-episode insight generation is valuable but DICE's abstraction pipeline already synthesizes higher-level insights from proposition groups. Different mechanism, similar outcome. -- **Microsoft's user profile/chat summary model**: The simplest memory system of any competitor — just two flat memory types. Even less capable than Google or AWS. -- **Neo4j Agent Memory's POLE+O ontology**: Domain-specific entity subtypes (Person → Suspect/Witness/Victim) are useful for law enforcement/intelligence domains but add complexity for general-purpose memory. DICE's proposition model is domain-agnostic by design. -- **Neo4j Agent Memory's reasoning traces**: Capturing tool call statistics and decision workflows is interesting but orthogonal to memory quality. Could be added as a projection type if needed. -- **Neo4j Agent Memory's multi-stage extraction cascade**: Their spaCy → GLiNER → LLM pipeline is cost-effective but adds operational complexity (model downloads, dependency management). DICE's single LLM call is simpler; the cost savings don't justify the complexity for proposition extraction. -- **Managed service hosting**: Google, AWS, Microsoft, and Neo4j all validate that agent memory is a product category. DICE's value is in the richness of its knowledge model, not in being a managed service. The embeddable library model is a strength, not a gap. +| Surprise-prioritised retention (GAP-2) | LangMem | Novel facts don't get durable treatment | +| Incident routing for `DriftReport` | DataHub, Soda | A drift report nobody is notified about is a log line | + +## Not worth chasing + +- **Zep's 5-reranker retrieval** — deep feature tied to Neo4j traversal. Bi-temporal is the better + investment. +- **LangMem's prompt optimisation** — interesting, orthogonal to memory quality. +- **Mem0's separate graph pipeline** — we already have entity mentions plus Neo4j projection. +- **Google's multimodal extraction** — the proposition model is format-agnostic; add later if a use + case demands it. +- **AWS's episodic reflection** — our abstraction pipeline already synthesises across propositions. +- **Neo4j's POLE+O ontology** — domain-specific subtypes; the proposition model is domain-agnostic by + design. +- **Neo4j's spaCy → GLiNER → LLM cascade** — cost-effective, operationally heavy (model downloads, + dependency management). Not worth it for proposition extraction. +- **Contract YAML as the authoring surface** — ODCS is a good spec, but DICE's declared schema is + already a JVM type an application owns. A second YAML dialect would be two sources of truth. +- **Write-time rejection of undeclared types** — the one governance move that destroys information + irreversibly. Quarantine is strictly better for an LLM-driven substrate. +- **Managed hosting** — every hyperscaler validates the category; none of them makes the embeddable + model less valuable. diff --git a/specs/developer-docs-architecture.md b/specs/developer-docs-architecture.md new file mode 100644 index 00000000..6b8cdaa1 --- /dev/null +++ b/specs/developer-docs-architecture.md @@ -0,0 +1,311 @@ +# Developer docs architecture + +The target information architecture for DICE's developer documentation — what pages exist, in what +order, and who owns keeping them true. This is a plan, not a description: most of what's below +doesn't exist yet. + +DICE ships to Maven Central as a set of JVM modules with Spring Boot autoconfiguration. The docs +that fit that shape are the ones Confluent Schema Registry, Flyway, Zep and Spring Boot itself +write: **concept-anchored, workflow-layered**. None of them uses strict Diátaxis. All of them open +with a runnable quickstart, then a small set of mental models, then task-shaped how-tos, then dense +reference — with optional features flagged at the top of their own page and a compatibility matrix +that answers "will this work with my stack" without reading anything else. + +## What we already have, and what's missing + +`docs/design/` is the *why*: rationale notes aimed at someone changing DICE. They're good and they +stay as they are. `specs/` is strategy and planning, also internal. + +The *how* exists, but in one place and the wrong shape: **`README.md` is 2,621 lines**. It already +holds Spring pipeline setup, the proposition pipeline walkthrough, mention filtering, entity +extraction and resolution, `ContextId` and `PropositionQuery`, graph and Prolog projection, agent +memory, the Oracle, the REST API, Spring Boot integration, graph-backed storage config, API-key +security and installation. That's most of a docs site pretending to be a landing page, and nobody +reads to line 2,000. + +So this isn't a greenfield IA. It's mostly a **migration**: + +| Directory | Audience | Question it answers | +|---|---|---| +| `docs/design/` | DICE contributors | Why is it built this way? | +| `docs/` (new tree below) | DICE consumers | How do I use it? | +| `specs/` | Us | What are we building and why does it matter commercially? | +| `README.md` | Anyone landing on the repo | What is this, should I care, where do I start? | + +Design docs and developer docs cross-link but never duplicate. A concept page says what a +proposition is and how to make one; the design note says why confidence decays the way it does. + +### Migrating the README + +Target: a README under ~250 lines. Line numbers are as of this branch; they'll move, so re-check +before acting on the table. + +| README section (line) | Fate | +|---|---| +| What is DICE, benefits table, architecture overview (24–116) | **Keep**, trimmed. This is the landing page's actual job. | +| Real-world example: Impromptu (117–176) | **Keep**, cut to a paragraph plus a link. Proof it's used, not a tutorial. | +| Pipeline setup, conversation analysis (127–176) | **Move** → `quickstart/` | +| Proposition pipeline, content dedup, mention filtering (177–489) | **Move** → `concepts/propositions.md`, `how-to/extract-from-documents.md`, `how-to/mention-filtering.md` | +| Entity extraction, entity resolution, resolution service (490–1225) | **Move** → `concepts/entity-resolution.md` + `how-to/tune-entity-resolution.md`. The single biggest block; split it. | +| Source analysis context, `ContextId`, `PropositionQuery` (1226–1419) | **Move** → `concepts/context-and-schema.md`, `how-to/query-propositions.md` | +| Relations, projector architecture, graph and Prolog projection (1420–1647) | **Move** → `concepts/storage-and-projections.md`, `how-to/project-to-graph.md`, `features/prolog-inference.md` | +| Agent memory, memory projection, memory maintenance (1648–1979) | **Move** → `how-to/agent-memory.md`, `concepts/knowledge-hygiene.md` | +| Proposition operations, Oracle (1980–2090) | **Move** → `how-to/query-propositions.md`, `how-to/oracle.md` | +| Package structure (2091–2205) | **Move** → `reference/` | +| REST API and endpoints (2206–2334) | **Move** → `features/web-api.md` (it's opt-in and gated by an API key — it belongs behind a gate page) | +| Spring Boot integration, graph-backed storage, API-key security (2335–2572) | **Move** → `how-to/choose-a-backend.md`, `reference/configuration-properties.md`, `features/web-api.md` | +| Installation (2573–2585) | **Keep** as coordinates only; the working version lives in the quickstart. | +| Technology stack, references, license (2586–2621) | **Keep**. | + +Two rules for the migration, or it will half-happen and leave two sources of truth. Content **moves** +— no section is copied into `docs/` and left in the README. And every removed section leaves a +one-line link where it was, so an existing bookmark still lands somewhere useful. Do it as one PR per +destination page, not one giant PR nobody can review. + +## The IA + +```mermaid +flowchart TD + R["docs/README.md
what DICE is, when to use it"] + QS["quickstart/
15 minutes to first extraction"] + C["concepts/
the spine — read in order"] + H["how-to/
by use case"] + F["features/
opt-in, gate first"] + REF["reference/
config, API, properties"] + P["production/
deploy, tune, observe"] + S["support/
compatibility matrix, FAQ"] + + R --> QS + QS --> C + C --> H + H --> F + H --> P + F --> REF + P --> REF + R --> S + + C --- C1["propositions
entity resolution
storage and projections
context and schema
knowledge hygiene"] + H --- H1["extract from documents
query what you stored
project to a graph
generate a report
wire it into an agent"] + F --- F1["metamodel versioning
governance and drift
Prolog inference
web API
decay and pruning"] +``` + +Five concepts, in that order, because each one needs the one before it. Everything else is entered +from any direction. + +## The concept spine + +The whole substrate is one chain, and cherry-picking it produces confusion: + +1. **Propositions** — natural-language claims are the system of record. Confidence, importance, + decay, lifecycle states. Everything else is a projection of these. +2. **Entity resolution** — how mentions in text get matched to entities that already exist, or minted + as new ones. Resolution outcomes, the escalating resolver chain, cross-chunk dedup. +3. **Storage and projections** — the `PropositionStore` SPI family, in-memory versus the durable + Neo4j backend, and the materialised views (vector, graph, Prolog, memory, oracle). +4. **Context and schema** — `contextId` scoping, the `DataDictionary`, `SchemaAdherence`, and what + "extraction against a schema" actually constrains. +5. **Knowledge hygiene** — admission gates, reclamation, consolidation. Why they're three + interventions at three moments rather than one. + +Every concept page ends with **Try it now**: five to ten lines that exercise the thing just +explained against the quickstart's setup, and the output to expect. A concept page with no runnable +code is a design note in the wrong directory. + +## The 15-minute quickstart + +One page, one path, no branches. Copy, paste, run, see output. + +### Prerequisite: there is no starter artifact + +DICE ships `dice`, `dice-ingestion`, `dice-storage`, `dice-storage-autoconfigure` and `dice-report`. +There is **no** `dice-spring-boot-starter`. A one-coordinate quickstart therefore has a build +prerequisite, and it has to be named rather than assumed: + +- **Either** create a `dice-spring-boot-starter` aggregator module that depends on + `dice-storage-autoconfigure` (which pulls `dice-storage` and `dice`) plus `dice-report`, publish it + to Maven Central, and let the quickstart use one coordinate. This is the better outcome and it's a + small module — but it's a build task that has to land *before* the quickstart page can be written + as one line. +- **Or** write the quickstart against the real coordinates today: `dice-storage-autoconfigure` and + `dice-report`, both explicit, with a comment saying what each buys. Honest, two lines instead of + one, and it doesn't block the docs on a release. + +Pick one before writing the page. What is not acceptable is documenting a starter that doesn't exist. + +### The page + +1. **The dependencies.** Whichever of the two above we picked, with the version from the BOM. +2. **Configuration.** The minimum that actually works: an LLM provider (inherited from + embabel-agent), an **embedding model** — `InMemoryPropositionRepository` does vector search, and + without an embedding model configured the vector-query step in step 6 returns nothing — and the + default in-memory store, which means no Neo4j and no Docker for the first run. If we'd rather keep + the config to one key, drop vector retrieval from the minimal path and query by entity and + `ContextId` only, then introduce embeddings in the retrieval how-to. Either is fine; silently + showing a vector query that can't work is not. +3. **What autoconfiguration gave you.** The beans that now exist and what each is for. Not a list of + `@Configuration` classes — a list of things you can `@Autowired`. +4. **Extract.** Feed one paragraph of text through the pipeline. Print the propositions with their + confidence. +5. **Persist.** `PropositionPipeline` returns **unsaved** results — that's deliberate, so the caller + owns the transaction, and it's the step every naive quickstart forgets. Call + `persist(propositionRepository, namedEntityDataRepository)` on the `PersistablePropositions` + result, and say in one sentence why the pipeline doesn't do it for you. A reader who skips this + queries an empty store and concludes DICE is broken. +6. **Query.** Retrieve what was just stored, by entity and `ContextId`, and by vector similarity if + step 2 configured an embedding model. +7. **Report.** Produce one human-readable artifact so the run ends in something visible. +8. **Where to go next**, split by intent — see the audience note below. + +Rules. The 15 minutes is a promise: no Neo4j, no Docker, no `create` of anything external. The code +is real and compiles; a test in `dice-integration-tests` runs the quickstart's exact snippets so +that promise can't rot silently — and that test is what would have caught the missing persist step. +Anything that needs a decision — backend choice, resolver chain, schema — takes the default and +links out. + +## How-to guides, by use case + +Task-shaped, titled by what the reader wants, and each one states its prerequisites at the top +rather than assuming a reading order: + +- Extract knowledge from a document set (chunking, dedup before extraction, batch, and persisting + the unsaved results the pipeline hands back) +- Tune entity resolution (resolver chain, thresholds, when to spend an LLM call) +- Choose and configure a storage backend (in-memory → Neo4j, what changes) +- Query propositions (composable `PropositionQuery`, retrieval modes, trust filtering) +- Project to a Neo4j graph (typed nodes, lineage back to evidence, re-run idempotence) +- Query with Prolog (facts, rules, transitive reasoning) +- Generate a report +- Use DICE as agent memory in an embabel-agent application +- Handle conflicts and contradictions (the conflict policy SPI) +- Process a stream incrementally (windowing, dedup across windows) + +Each names what it costs — LLM calls, latency, infrastructure. A how-to that hides the cost is how +you get an angry issue three weeks later. + +## Feature pages lead with the gate + +Every optional feature gets its own page and every one of those pages opens with the gate, not the +pitch. This matters most for versioning and governance, where the whole design is that **nothing +happens unless you asked for it** — no `DeclaredSchemaSource`, no versioning; no governed types, no +drift detection. A page that opens by explaining metamodel versioning teaches the reader that DICE +versions their schema. It does not. + +Template, in order: + +- **Availability** — which DICE version, which modules, what it costs to add. +- **The gate** — the exact bean, property, or annotation that switches it on. First screen, always. + "Off unless you define a `DeclaredSchemaSource` bean" is the first thing on the versioning page. +- **When to use it** — and, explicitly, when not to. +- **Impact** — latency, memory, extra infrastructure, extra LLM calls. +- **How to enable** — full working configuration. +- **Example** — real code with real output. + +Opt-in surfaces that need this treatment: metamodel versioning, drift detection and quarantine, +Prolog inference, graph projection, the web API, decay and stale-pruning, the multi-signal collector, +concurrent extraction. + +## Compatibility matrix + +Its own page, linked from the README, answering "will this work with my stack" before anyone reads +prose. Four axes, and the embabel-agent one is a **range**, not a point — DICE tracks a moving +platform and the supported window is the single most-asked question a library like this gets. + +| Axis | What we state | +|---|---| +| embabel-agent | Supported version range per DICE release, with the tested point release called out | +| Spring Boot | Minimum and maximum tested, per DICE release | +| JDK | Baseline is 21 (inherited from `embabel-build-parent`); newer JDKs listed as tested or not | +| Neo4j | Required version floor for the durable backend, via Drivine; explicitly "not required" for in-memory | + +Plus a feature-availability column, so a reader can see at a glance that governance arrives after +1.0 rather than discovering it from a 404. Exact values get pinned at the 1.0 release; the page ships +with the axes and the honest "tested against" values we have. + +## Production guide + +The page a team reads before going live, not a "deployment" afterthought: + +- Backend choice and the migration from in-memory to Neo4j +- Schema declarations and index creation on startup +- Decay tick configuration, and why `prune-stale` defaults to false +- Concurrency: what's safe to run in parallel and where entity-resolution ordering matters +- LLM cost and latency budgets per pipeline stage +- Observability: the events DICE emits, what to alert on +- Backup, and what "system of record" obliges you to keep +- Governance rollout: stamp first, detect later, quarantine last + +## Audience split + +Two audiences arrive with different vocabularies and different first questions: + +- **Data engineers** want to know where the data lands, what the schema is, how drift is handled, and + what Cypher they can run. They read storage, schema, governance, projections. +- **LLM/agent engineers** want extraction quality, entity resolution, memory retrieval, and how it + plugs into an agent. They read propositions, resolution, retrieval, agent memory. + +**One spine, multiple doors.** The concept order doesn't fork — propositions still come before +resolution before storage for everybody. What forks is the entry point: the quickstart's "where to +go next" offers a data-engineer path and an agent-engineer path through the same pages in different +orders, and the README says which is which. Two parallel doc sets would drift within a quarter. + +A third audience (researchers wanting Prolog and reasoning) is real but small; served by the Prolog +feature page, not by a third path. + +## Risks + +**Reading-order enforcement.** The concept chain only makes sense in order, and readers arrive from +search engines in the middle of it. Nothing in a static docs site forces sequence. Mitigations, all +cheap: number the concept pages, put a one-line "assumes you've read X" at the top of each, and end +each with an explicit next link. Accept that some readers land mid-chain and make each page survive +it — a page that only works as chapter 4 is a page most readers will bounce off. + +**Extraction variability.** The docs assume the extractor returns clean, schema-adhering +propositions. It won't always. Mention filtering and `SchemaAdherence` need to appear early and be +weighted heavily, not buried in a how-to nobody reaches. + +**Backend coupling surfaces late.** Prolog needs a store; graph projection needs decisions made +before extraction runs. The quickstart deliberately defers all of this, so the "choose your backend" +how-to has to be prominent and has to say plainly that changing it later means rework. + +**Duplication across modules, and with the README.** Spring configuration examples exist in +`dice-storage-autoconfigure`, `dice-ingestion` and elsewhere, and a 2,621-line README holds a version +of most of them. One canonical example set in `docs/`; module READMEs and the root README link to it +and never restate it. The migration table above only works if content moves rather than being copied +— two sources of truth is worse than one badly-placed one. + +**Doc rot.** Handled by the contract below, and by making the quickstart executable. + +## The per-PR docs contract + +**Every feature PR ships its design-doc delta and its developer-doc page in the same PR. Missing +docs are a review-blocking finding, on the same footing as a missing test.** + +What that means concretely: + +- New or changed behaviour visible to a consumer → the developer-doc page is updated in that PR. +- New or changed *rationale* → the `docs/design/` note is updated in that PR. +- A new opt-in feature → a feature page that leads with its gate, in that PR. +- A new configuration property → the reference page, in that PR. +- A version-support change → the compatibility matrix, in that PR. + +The reasoning is the same one that makes tests non-negotiable. Docs written later are written by +someone who has forgotten the edge cases, or by nobody. And in a library whose central promise is +that governance is opt-in, an undocumented opt-in switch is indistinguishable from a feature that +doesn't exist. + +Reviewers ask one question: *if I only had this PR's docs, could I use this feature?* If not, the +finding is blocking. "Docs to follow" is not an accepted resolution — the follow-up either doesn't +happen or lands after someone has already been confused by the gap. + +Small carve-outs, stated so the rule stays credible: internal refactors with no consumer-visible +change need no doc delta, and neither do test-only or build changes. + +## Open questions + +- Does DICE ever ship a CLI (schema validation, drift check)? Flyway's docs work partly because the + CLI gives every concept an executable form. If a CLI happens, it needs its own quickstart. +- Rendered site or Markdown in the repo? The IA works either way; the decision affects cross-linking + and whether the compatibility matrix can be generated. +- Do the `docs/design/` notes stay contributor-facing, or do the best of them get promoted into + concept pages? Current answer: they stay, because their audience is genuinely different — but + concept pages will lift explanations from them, and that's fine. From 7a9c8612bef04ae1beda9ae43aa843433a7640eb Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Sun, 30 Aug 2026 20:22:01 -0400 Subject: [PATCH 2/6] docs(specs): voice pass on positioning and docs-architecture Rewrite both specs in the repo's documentation voice. Delivery status now reports the open PR train (#83-#88) per PR and corrects entries that were marked planned after the implementing PRs opened. --- specs/competitive-positioning.md | 624 ++++++++++++--------------- specs/developer-docs-architecture.md | 237 +++++----- 2 files changed, 389 insertions(+), 472 deletions(-) diff --git a/specs/competitive-positioning.md b/specs/competitive-positioning.md index 8001f8fc..39b11406 100644 --- a/specs/competitive-positioning.md +++ b/specs/competitive-positioning.md @@ -1,197 +1,166 @@ # DICE competitive positioning Where DICE sits against the agent-memory field (Zep/Graphiti, Mem0, Letta, Cognee, Hindsight, -LangMem, and the three hyperscaler memory services), and against the enterprise data-governance -tooling whose vocabulary we should be borrowing rather than inventing. +LangMem, Neo4j Agent Memory, and the three hyperscaler memory services), and which vocabulary to +borrow from enterprise data-governance tooling. -Two halves. The first is the claim we lead with and the evidence behind it. The second is the -per-competitor detail — kept because it's still the fastest way to answer "how do we compare on X". +DICE differs from that field in one place: governance of the schema extraction runs against. No +surveyed competitor versions its extraction schema, compares a declaration against what a live +graph holds, or quarantines divergent records, and none has that work on a public roadmap. -Surveyed mid-2026. Competitor surfaces move fast and several of them publish roadmap blogs as if -they were release notes, so re-verify anything here before it goes into a bakeoff or a deck. +Surveyed mid-2026. Several competitors publish roadmap posts in the register of release notes, so +re-verify anything here before it goes into a bakeoff or a deck. -## Delivery status: read this before quoting anything below +## Delivery status -The governance story is **direction, not shipped surface**. `dice-metamodel` — `MetamodelVersion`, -`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `MetamodelVersionStore` — is an -unmerged PR train. Detection and quarantine are designed and not built. Everywhere below, a -metamodel or drift capability is marked **(in delivery)** or **(planned)**, and those markers are -load-bearing: this doc is for positioning conversations, and a present-tense claim about an unmerged -module is the exact thing that collapses in an eval. Say "this is where we're going and here's the -design" — that's still a differentiated answer, because nobody else is going there at all. +On main: the proposition model with confidence and decay, multi-strategy entity resolution, source +provenance (`ProvenanceEntry`), collector decision traces (`CollectorTraceStore`), the +`ConflictDetector` SPI, `SchemaAdherence` (STRICT/DEFAULT/RELAXED), declared types via +`SchemaRegistry`/`DataDictionary`, integer re-indexing of proposition IDs across LLM calls (GAP-6), +and the pluggable store family. -What *is* shipped and can be claimed flatly: the proposition model with confidence and decay, -multi-strategy entity resolution, source provenance (`ProvenanceEntry`), collector decision traces -(`CollectorTraceStore`), the conflict-detection SPI, and the pluggable store family. +In the open PR train, unmerged as of this writing: -## The claim - -**DICE is the agent-memory substrate with schema governance an enterprise data team will -recognise.** - -Every serious competitor has solved *extraction* and most have solved *entity resolution*. None of -them has solved *governance of the schema that extraction runs against* — and none of them has it on -a public roadmap. Across Zep/Graphiti, Mem0, Letta, Cognee and Hindsight, not one ships, and DICE is -building: - -- governed metamodel versioning — a schema identity you can write down, store, and compare later - **(in delivery)**; -- provenance from a stored claim back to the source text it came from **(shipped)**, and a - correlatable trace of the extraction run that produced it **(planned — see below)**; -- drift quarantine as a first-class state, distinct from delete **(planned)**; -- conflict resolution behind an explicit, pluggable SPI rather than an opaque LLM call **(shipped)**, - with versioned and audited policies **(planned)**. +| PR | Contents | +|---|---| +| #83 | `dice-metamodel` core: `MetamodelVersion` (content-addressed stamp), `GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `MetamodelVersionStore`, and `docs/design/metamodel-versioning.md` with the tier ladder | +| #84 | `DrivineMetamodelVersionStore`: Neo4j persistence, MERGE on natural key, per-schema monotonic sequence for write order | +| #85 | `ObservedSchema`/`ObservedSchemaSource`, `MetamodelDiff`/`MetamodelChange`, declared-vs-declared and declared-vs-observed differs | +| #86 | `DriftReport`/`DriftReportStore`, `DriftCheckRunner` (`dryRun` defaults true), quarantine contracts and `MentionTypeDriftQuarantinePolicy` | +| #87 | `DrivineDriftReportStore` and `DrivineObservedSchemaSource`, plus the end-to-end governance loop against a live Neo4j | +| #88 | `MetamodelAutoConfiguration`, activated by the presence of a `DeclaredSchemaSource` bean; `embabel.dice.metamodel.enabled` kill switch and `drift.mode` = `off` \| `observe` (default) \| `quarantine` | -That is a category gap, not a feature gap. The systems that treat memory as a datastore have -skipped the layer that every enterprise data platform grew in its second year. Our advantage is that -we're building it deliberately; it is not yet an advantage we can demo end to end. +Not built: an extraction-run identifier linking a stored claim to the run that produced it (issue +#67), versioned and audited conflict policies, incident routing for `DriftReport`, a bi-temporal +fact model, and temporal anchoring of relative dates. -## What the competition actually has +The rest of this document describes the design across main and that train. This section is the +reference for where a given capability currently lives. -Differentiation that lies dies in the first eval. Stated straight, and generously: +## What each competitor has -| System | What it genuinely has | +| System | Capabilities | |---|---| -| **Zep / Graphiti** | Bi-temporal edges (`valid_from`/`valid_until` plus ingestion time), cross-session entity dedup, Leiden community clustering with summaries, custom entity types via Pydantic, five reranking strategies, SOC 2 on the managed offering. The strongest temporal model in the field. | -| **Mem0** | Very fast single-pass ingestion (v3, April 2026, dropped the UPDATE/DELETE phases to cut latency), hybrid semantic + BM25 + entity-match retrieval, graph memory across Neo4j/Memgraph/Neptune/Kuzu, full SQLite change history, mentions counting. | -| **Letta (MemGPT)** | Agent-managed context with archival memory in Postgres/pgvector; the agent itself decides what to promote and how to resolve conflicts, via tool calls. Honest about being unstructured. | -| **Cognee** | The closest thing to declared schema in the field: LLM extraction into RDF triples with Pydantic validation, auto-generated ontology, and a custom-schema override. Shape validation, not versioning. | -| **Hindsight** | Structured facts into a knowledge graph with real entity resolution ("Alice" vs "my coworker Alice") and quality that compounds across sessions. | -| **LangMem** | Excellent extraction prompts — confidence-qualified, surprise-prioritised, SNR-shaped — plus gradient-style prompt optimisation and dilated-window retrieval. | +| **Zep / Graphiti** | Bi-temporal edges (`valid_from`/`valid_until` plus ingestion time), cross-session entity dedup, Leiden community clustering with summaries, custom entity types via Pydantic, five reranking strategies, SOC 2 on the managed offering. | +| **Mem0** | Single-pass ingestion (v3, April 2026, dropped the UPDATE/DELETE phases to cut latency), hybrid semantic + BM25 + entity-match retrieval, graph memory across Neo4j/Memgraph/Neptune/Kuzu, full SQLite change history, mentions counting. | +| **Letta (MemGPT)** | Agent-managed context with archival memory in Postgres/pgvector. The agent decides what to promote and how to resolve conflicts, via tool calls. No declared structure. | +| **Cognee** | LLM extraction into RDF triples with Pydantic validation, auto-generated ontology, custom-schema override. Validates shape per run; no stored version to compare a later run against. | +| **Hindsight** | Structured facts into a knowledge graph with entity resolution that links "Alice" to "my coworker Alice", and quality that compounds across sessions. | +| **LangMem** | Confidence-qualified, surprise-prioritised, SNR-shaped extraction prompts, gradient-style prompt optimisation, dilated-window retrieval. | | **Google / AWS / Microsoft** | Fully managed, zero-infrastructure, IAM-scoped, framework-integrated. Flat fact strings underneath. | -Two of these are ahead of us on things we care about. Zep's bi-temporal model is better than our -system timestamps. Cognee's Pydantic validation is a real declared-shape check at write time. Say so. +Two of these exceed DICE's equivalents: Zep's bi-temporal model against DICE's system timestamps, +and Cognee's Pydantic validation, which is a declared-shape check at write time. -## Where the governance line falls +## Governance capabilities | Capability | Zep | Mem0 | Letta | Cognee | Hindsight | DICE | |---|---|---|---|---|---|---| -| Schema versioning | — | — | — | Pydantic override, unversioned | — | Content-hashed `MetamodelVersion` (in delivery) | -| Type hierarchy / ontology | Implicit (clustering) | — | — | RDF/RDFS, auto-generated | — | Declared `DataDictionary`, per-type governed (in delivery) | -| Source provenance | Bi-temporal timestamps only | — | — | — | — | `ProvenanceEntry` edges, append-only (shipped) | -| Extraction-run trace | — | — | — | — | — | Planned (no run identifier today — issue #67) | -| Collector decision audit | — | Full SQLite change history | — | — | — | `CollectorTraceStore` (shipped) | -| Contradiction handling | Temporal windows, manual | ADD-only (sidesteps it) | Manual, agent-driven | Unclear | Unclear | `ConflictDetector` SPI, both claims retained (shipped) | -| Versioned / audited conflict policy | — | — | — | — | — | Planned | -| Drift detection (declared vs observed) | — | — | — | — | — | Planned (tier 2) | -| Drift quarantine | — | — | — | — | — | Planned (tier 3) | - -Four honest caveats. Zep and Cognee may have unpublished lineage APIs; this table is built from -public docs. Mem0's change history is genuinely better than anything we have for "what changed and -who changed it" — it is only weaker on *what schema it was valid under*. Our audit story is two -separate mechanisms, not one chain: `ProvenanceEntry` links a proposition to the source chunks it +| Schema versioning | — | — | — | Pydantic override, unversioned | — | Content-hashed `MetamodelVersion` (#83) | +| Type hierarchy / ontology | Implicit (clustering) | — | — | RDF/RDFS, auto-generated | — | Declared `DataDictionary`, per-type governed (main + #83) | +| Source provenance | Bi-temporal timestamps only | — | — | — | — | `ProvenanceEntry` edges, append-only (main) | +| Extraction-run trace | — | — | — | — | — | None (issue #67) | +| Collector decision audit | — | Full SQLite change history | — | — | — | `CollectorTraceStore` (main) | +| Contradiction handling | Temporal windows, manual | ADD-only | Manual, agent-driven | Unclear | Unclear | `ConflictDetector` SPI, both claims retained (main) | +| Versioned / audited conflict policy | — | — | — | — | — | None | +| Drift detection (declared vs observed) | — | — | — | — | — | `DeclaredObservedDiffer`, `DriftCheckRunner` (#85, #86) | +| Drift quarantine | — | — | — | — | — | `DriftQuarantinePolicy`, STALE marking (#86) | + +Caveats on that table. It is built from public documentation, so Zep and Cognee may have +unpublished lineage APIs. Mem0's change history records old value, new value, event and actor, +which is more than DICE records, and omits the schema the change was valid under. DICE's audit +trail is two separate mechanisms: `ProvenanceEntry` links a proposition to the source chunks it came from, and `CollectorTraceStore` records why a collapse or merge decision was made. Neither -carries an extraction-run identifier, so there is no end-to-end "this claim came from that run under -that schema" trace today — that's planned work (issue #67), and the metamodel stamp is the piece -that would complete it. And the whole tier ladder — set out in `docs/design/metamodel-versioning.md`, -which lands with the `dice-metamodel` module — is designed, with only tier 1 in delivery. +carries an extraction-run identifier, so there is no end-to-end "this claim came from that run +under that schema" trace (issue #67); the metamodel stamp is the piece that would complete it. -Claim the architecture, not the checkbox. +## Metamodel identity: content-addressed -## Identity semantics: content-addressed, not sequential +Confluent Schema Registry and AWS Glue assign sequential version identity. Confluent uses a global +immutable integer schema ID plus a per-subject version number, Glue a per-schema version number +plus an opaque version ID. Registration is a stateful event: the same schema posted twice gets one +ID and one version because the registry remembers it. -Confluent Schema Registry and AWS Glue both assign **sequential** version identity — Confluent a -global immutable integer schema ID plus a per-subject version number, Glue a per-schema version -number plus an opaque version ID. Registration is a stateful event: the same schema posted twice -gets one ID and one version because the registry remembers, not because the identity is derived from -the content. - -DICE derives identity from the content. That design is in delivery, not merged, so the whole of this -section is a statement of direction. `MetamodelVersion.contentHash` is a SHA-256 over the structural -fields — sorted type names, per-type labels and properties, sorted relationship descriptors — with -every token length-prefixed so `["a;b"]` can't collide with `["a", "b"]`. Three consequences worth -stating in any comparison: +DICE derives identity from content. `MetamodelVersion.contentHash` is a SHA-256 over the structural +fields (sorted type names, per-type labels and property signatures, sorted relationship +descriptors) with every token length-prefixed, so `["a;b"]` cannot collide with `["a", "b"]`. Three +consequences: - **Registration is idempotent by construction.** `saveVersion` upserts on `(schemaName, - contentHash)`; a re-save lands on identical content by definition. No sequence counter to get out - of step across environments. -- **Schema is data.** A dev and a prod schema of identical shape hash identically, so they compare - directly. The registries can't do that — same schema, different subject, different version number. -- **Stamps are portable.** A proposition carries `dice.metamodel.version` as metadata, so the schema - a claim was extracted under travels with the claim rather than living in a registry it must be - looked up in. - -The second divergence is **granularity**. Both registries configure compatibility **per subject**, -flat: one mode for the whole schema. DICE governs **per type**, via `GovernedTypeSelector`. That's -not a stylistic difference — it's what lets one domain be closed-world where it matters (`Person`, -`Company`) and open-world everywhere else, so an exploratory type LLM extraction just invented -doesn't churn the version history. - -**Adopt their vocabulary where it fits.** BACKWARD / FORWARD / FULL, and the transitive variants, -are well-understood terms for what a compatibility check means, and our detect tier should use them -rather than coining synonyms. Glue's *checkpoint* — a selectable reference point to compare against, -rather than "previous version" or "all versions" — is more expressive than transitive-yes/no and is -the right model if governed types ever get release windows. - -**Name the divergence.** Both registries **hard-reject** an incompatible registration: HTTP 409, no -warn mode, no soft landing. DICE does not, and won't. Extraction is LLM-driven; a type nobody -declared is often a real finding. Rejecting at write time is the one decision that can't be undone -later. - -## Opt-in and drift: observe, never auto-mutate - -The migration and ORM world has already run this experiment, and the developer verdict is -unambiguous. - -**Loved:** - -- **`liquibase diff` / `diff-changelog`** — compares declared (reference) against observed (target) - and *generates* the changesets that would close the gap. It reports; a human decides whether to - apply or mark as run. -- **`flyway validate`** — checksums applied migrations against local ones and fails loudly on - mismatch. Explicitly bound to a build phase, not fired automatically on every deploy. -- **Hibernate `hbm2ddl.auto=validate`** — check only, fail on mismatch, change nothing. And - `hbm2ddl.auto=none` as the shipped default: no schema management unless you asked for it. -- **`@Version` per entity** — versioning is an annotation you put on the entities you chose. Nothing - is versioned by inference. - -**Hated:** `hbm2ddl.auto=update`. Uncontrolled schema evolution at startup, silently creating -columns and leaving zombies behind when a `@Column(name=...)` changes. A production anti-pattern in -every guide that mentions it. - -DICE's stance follows directly, and it's a one-line elevator pitch: **liquibase diff for knowledge -graphs.** Observe and quarantine, never auto-mutate. That's the design; here is what it commits us -to, tier by tier: - -- Nothing is versioned unless an application supplies a `DeclaredSchemaSource` (in delivery). No - declared schema, no versioning — the Hibernate `none` default, not the `update` one. -- Governance is per governed type via `GovernedTypeSelector` (in delivery), the `@Version` model - rather than a global switch. -- The detect tier (planned) will report the declared-vs-observed gap. It will not close it. Closing - it stays a human act, the way Liquibase generates changesets rather than applying them. -- The quarantine tier (planned) marks affected propositions stale. Stale is reversible; delete is - not. - -The failure mode to watch, straight from the same brief: **opt-in fatigue**. All three tools require -explicit configuration, teams ship with defaults, and later regret having no drift visibility. Our -answer isn't to turn governance on by default — it's to make the on-switch a one-bean decision and to -lead every governance doc page with the gate. - -## Escalation vocabulary: say what the data team says - -OpenMetadata, DataHub, Great Expectations, Soda and the Open Data Contract Standard have converged on -one vocabulary. Use it verbatim. Every term we coin instead is a term a buyer has to translate. - -| Term | Their meaning | Ours | Status | -|---|---|---|---| -| **Drift** | Structural or statistical divergence over time | Declared metamodel vs what the graph actually holds | Planned | -| **Contract** | Declared structure + quality + ownership | `DeclaredSchema` — the governed types and their shape | In delivery | -| **Assertion** | A comparison check fired on change | The declared-vs-observed comparison in the detect tier | Planned | -| **Policy** | Enforcement rule, configurable strictness | `SchemaAdherence` (STRICT/DEFAULT/RELAXED) shipped; `ConflictDetector` SPI shipped; versioned policy planned | Mixed | -| **Incident** | Raised on assertion failure, routed to an owner | `DriftReport` — the record a detection produces | Planned | -| **Quarantine** | Invalid records split off; the valid stream continues | Affected propositions marked stale, not deleted | Planned | - -The escalation ladder they all implement is **observe → alert → block**. DICE's three tiers are -designed to map onto it directly: + contentHash)`, and a re-save of identical content lands on the same row. There is no sequence + counter to get out of step across environments. +- **Schema is data.** A dev and a prod schema of identical shape hash identically and compare + directly. In the registries the same schema under a different subject gets a different version + number. +- **Stamps are portable.** A proposition carries `dice.metamodel.version` as metadata, so the + schema a claim was extracted under travels with the claim and needs no registry lookup. + +The second divergence is granularity. Both registries configure compatibility per subject: one mode +for the whole schema. DICE governs per type via `GovernedTypeSelector`, which lets one domain be +closed-world where it matters (`Person`, `Company`) and open-world elsewhere, so an exploratory type +the LLM invented leaves the version history unchanged. + +DICE adopts the registries' compatibility vocabulary: BACKWARD, FORWARD, FULL and the transitive +variants are well-understood terms for what a compatibility check means. Glue's *checkpoint*, a +selectable reference point to compare against, is more expressive than transitive yes/no and is the +right model if governed types get release windows. + +Both registries hard-reject an incompatible registration with HTTP 409 and no warn mode. DICE +accepts the write and records the divergence. Extraction is LLM-driven, a type nobody declared is +often a real finding, and a write-time rejection discards it irrecoverably. + +## Opt-in and drift: prior art from migration tooling + +Patterns developers adopt: + +- **`liquibase diff` / `diff-changelog`** compares declared (reference) against observed (target) + and generates the changesets that would close the gap. A human decides whether to apply them or + mark them as run. +- **`flyway validate`** checksums applied migrations against local ones and fails loudly on + mismatch. It is bound to a build phase. +- **Hibernate `hbm2ddl.auto=validate`** checks only: fail on mismatch, change nothing. + `hbm2ddl.auto=none` is the shipped default, so no schema management happens unasked. +- **`@Version` per entity** makes versioning an annotation on the entities you chose. Nothing is + versioned by inference. + +The pattern developers warn against is `hbm2ddl.auto=update`: uncontrolled schema evolution at +startup, silently creating columns and leaving zombies behind when a `@Column(name=...)` changes. + +DICE follows the first set: + +- Versioning activates on an application-supplied `DeclaredSchemaSource` bean. With no declared + schema there is no versioning. +- Governance is per governed type via `GovernedTypeSelector`. +- The drift check reports the declared-vs-observed gap and leaves closing it to a human act, the + way Liquibase generates changesets and applies none of them. +- Quarantine marks affected propositions STALE and leaves them stored and recoverable. `drift.mode` + defaults to `observe`. + +The failure mode all three tools share is opt-in fatigue: they require explicit configuration, +teams ship with defaults, and later have no drift visibility. DICE's mitigation is a one-bean +switch and an activation condition stated in the first paragraph of every governance doc page. + +## Escalation vocabulary + +OpenMetadata, DataHub, Great Expectations, Soda and the Open Data Contract Standard have converged +on one vocabulary. DICE uses these terms with their existing meanings. + +| Term | Their meaning | DICE | +|---|---|---| +| **Drift** | Structural or statistical divergence over time | Declared metamodel versus what the graph holds | +| **Contract** | Declared structure + quality + ownership | `DeclaredSchema`: the governed types and their shape | +| **Assertion** | A comparison check fired on change | The declared-vs-observed comparison in the detect tier | +| **Policy** | Enforcement rule, configurable strictness | `SchemaAdherence` (STRICT/DEFAULT/RELAXED) and the `ConflictDetector` SPI | +| **Incident** | Raised on assertion failure, routed to an owner | `DriftReport`: the record a detection produces | +| **Quarantine** | Invalid records split off, the valid stream continues | Affected propositions marked STALE | -| DICE tier | Their stage | What happens | Status | -|---|---|---|---| -| Stamp and observe | Observe | Schema gets an identity; propositions carry the stamp. No opinions. | In delivery | -| Detect and report | Alert | Compare declaration against declaration, and declaration against live graph. Emit a `DriftReport`. | Planned | -| Quarantine | Block (soft) | Mark affected propositions stale. The valid set stays queryable; the suspect set is excluded and recoverable. | Planned | +Their escalation ladder is observe → alert → block. DICE's three tiers map onto it: -The diagram below is the target flow, not current behaviour. +| DICE tier | Their stage | What happens | +|---|---|---| +| Stamp and observe | Observe | Schema gets an identity; propositions carry the stamp | +| Detect and report | Alert | Compare declaration against declaration, and declaration against live graph; emit a `DriftReport` | +| Quarantine | Block (soft) | Mark affected propositions STALE. The valid set stays queryable; the suspect set is excluded and recoverable | ```mermaid flowchart LR @@ -205,164 +174,140 @@ flowchart LR QU -.->|"human review"| VALID ``` -Note the shape of the split: the pipeline **continues**. Quarantine is a valid/invalid partition, not -a halt — exactly the hybrid default the Spark/Soda world settled on. Fail-fast is what you escalate -to when quality drops below a threshold, not what you do on the first bad record. +Quarantine partitions the write stream into a valid and a suspect set, and the pipeline continues, +which is the hybrid default the Spark and Soda world settled on. Fail-fast is the escalation for +when quality drops below a threshold. -Two decisions this vocabulary forces on us, both still open: +Two decisions this vocabulary forces, both open: -- **Is quarantine a metadata flag or physical routing?** Enterprise tools do both and neither is - canonical. DICE's stale-marking is a flag; whether reads exclude by default is the real question. +- **Is quarantine a metadata flag or physical routing?** Enterprise tools do both. DICE's + STALE-marking is a flag; whether reads exclude it by default is the open question. - **Do we route incidents?** Their incident model assumes metadata ownership drives notification. - DICE has events; it has no owner model. `DriftReport` without a routing story is a log line. + DICE emits events and has no owner model, so a `DriftReport` has no routing destination today. ## Graph-native convergence: Neo4j GRAPH TYPE Neo4j's GRAPH TYPE (preview, 2025) declares a schema once in Cypher DDL and enforces it at write time, with SET/ADD/ALTER/DROP lifecycle commands and an Open variant that requires declared fields -while tolerating extras. TypeDB does the same thing more strictly: nothing can be written that -doesn't conform, and `redefine` checks existing instances before a migration commits. - -**Read this as validation, not threat.** The store vendors are independently concluding that a -property graph needs a declared schema coupled to enforcement. That's the direction we're already -walking. The differentiation is *where the governance sits*. - -DICE governs **above the store** — that's the architectural bet, with tier 1 in delivery and the -rest designed: - -- **Per context, not per database.** Governance scopes to a `contextId`, so one deployment can hold - a governed tenant and an exploratory one. GRAPH TYPE is a property of the graph. -- **Open-world by default.** GRAPH TYPE and TypeDB reject the undeclared. DICE tolerates it — an - exploratory type is a finding, and per-type governance is what makes that safe rather than sloppy. - Today's `SchemaAdherence` already takes this side of the trade at extraction time. -- **Quarantine rather than reject** (planned). Write-time enforcement means a non-conforming - extraction is lost. Ours would be stored, flagged, and reviewable. -- **Survives a backend swap.** DICE's schema lives in `SchemaRegistry`/`DataDictionary`, not in the - database. In-memory, Neo4j, or something later — the governance story is the same one. Adopting - GRAPH TYPE would rewrite it. - -The honest cost of governing above the store: our schema is application-enforced, so it *can* drift -if some entry point skips validation. Neo4j's cannot. That's a real trade, and the detect tier is -planned precisely because we took the flexible side of it — which also means that until that tier -lands, the drift it's meant to catch is undetected rather than prevented. - -**SHACL validation reports are the prior art for `DriftReport`.** SHACL is the only standardised -violation-report format in this space: a conformance flag at the root, one result node per violation, -each carrying the focus node, the failed constraint, severity, and a human-readable message. Copy the -shape. Don't copy the serialisation — SHACL reports are RDF graphs, which would make every consumer -of a `DriftReport` parse RDF. PG-Schema and ProGS are the property-graph analogues and are worth -watching, but neither has a canonical report format to borrow. - -## What we copy, what we reject, what only we do - -| | Copy | Reject | Only us | +while tolerating extras. TypeDB is stricter: nothing that fails to conform can be written, and +`redefine` checks existing instances before a migration commits. Store vendors are independently +concluding that a property graph needs a declared schema coupled to enforcement. The divergence is +where the governance sits. + +DICE governs above the store: + +- **Per context.** Governance scopes to a `contextId`, so one deployment can hold a governed tenant + and an exploratory one. GRAPH TYPE is a property of the graph. +- **Open-world by default.** GRAPH TYPE and TypeDB reject undeclared types. DICE stores them as + findings, and per-type governance is what keeps that safe. `SchemaAdherence` already takes this + side of the trade at extraction time. +- **Quarantine on divergence.** Write-time enforcement loses a non-conforming extraction. DICE + stores it, flags it, and leaves it reviewable. +- **Backend-independent.** DICE's schema lives in `SchemaRegistry`/`DataDictionary`, outside the + database, so the same governance applies across in-memory, Neo4j, or a later backend. Adopting + GRAPH TYPE would move it into the store. + +The cost of governing above the store is that the schema is application-enforced, so it can drift +if some entry point skips validation. Neo4j's is enforced by the database. + +SHACL validation reports are the prior art for `DriftReport`, and the only standardised +violation-report format in this space: a conformance flag at the root, one result node per +violation, each carrying the focus node, the failed constraint, severity, and a human-readable +message. `DriftReport` follows that structure. SHACL serialises as RDF graphs, which +`DriftReport` does not adopt, so its consumers need no RDF parser. PG-Schema and ProGS are the +property-graph analogues and are worth watching; neither has a canonical report format to borrow. + +## Adopted, rejected, unique + +| | Adopt | Reject | Unique to DICE | |---|---|---|---| -| **Schema registries** (Confluent, Glue) | BACKWARD/FORWARD/FULL vocabulary; Glue's checkpoint idea; idempotent registration | Sequential version IDs; flat per-subject compatibility; hard-reject on incompatible registration | Content-addressed identity; per-type governed selection; the stamp travelling with the data | -| **Migration & ORM** (Liquibase, Flyway, Hibernate) | Declared-vs-observed diff that reports rather than applies; validate-mode; per-entity `@Version` opt-in | `hbm2ddl.auto=update` auto-mutation; anything that changes a schema at startup | Drift detection over LLM-extracted knowledge, where the "observed" side is a graph nobody wrote by hand | -| **Data governance** (DataHub, Soda, ODCS) | drift / contract / assertion / policy / incident / quarantine, verbatim; observe→alert→block; valid/invalid split | Blocking at the producer boundary as a default; contract YAML as the primary authoring surface | Governance over probabilistic extraction — confidence-weighted claims, not rows | -| **Graph schema** (Neo4j GRAPH TYPE, TypeDB, SHACL) | SHACL's validation-report structure; GRAPH TYPE's Open variant semantics | Write-time rejection; RDF as a wire format; store-coupled schema declaration | Governance that works per context, tolerates open-world types, and survives a backend swap | -| **Agent memory** (Zep, Mem0, Cognee, LangMem) | Zep's bi-temporal model (GAP-4B); Mem0's integer re-indexing and change history; LangMem's extraction prompts; Cognee's declared-shape validation | ADD-only ingestion; hard delete on contradiction; opaque LLM consolidation | Source provenance and collector traces (shipped); governed metamodel versioning (in delivery); drift quarantine and versioned conflict policy (planned) — nobody else has any of the four, shipped or roadmapped | +| **Schema registries** (Confluent, Glue) | BACKWARD/FORWARD/FULL vocabulary; Glue's checkpoint; idempotent registration | Sequential version IDs; flat per-subject compatibility; hard-reject on incompatible registration | Content-addressed identity; per-type governed selection; the stamp travelling with the data | +| **Migration & ORM** (Liquibase, Flyway, Hibernate) | Declared-vs-observed diff that reports; validate-mode; per-entity `@Version` opt-in | `hbm2ddl.auto=update` auto-mutation; anything that changes a schema at startup | Drift detection over LLM-extracted knowledge, where the observed side is a graph nobody wrote by hand | +| **Data governance** (DataHub, Soda, ODCS) | drift / contract / assertion / policy / incident / quarantine, verbatim; observe→alert→block; valid/invalid split | Blocking at the producer boundary by default; contract YAML as the primary authoring surface | Governance over probabilistic extraction: confidence-weighted claims | +| **Graph schema** (Neo4j GRAPH TYPE, TypeDB, SHACL) | SHACL's validation-report structure; GRAPH TYPE's Open variant semantics | Write-time rejection; RDF as a wire format; store-coupled schema declaration | Governance that is per context, tolerates open-world types, and is backend-independent | +| **Agent memory** (Zep, Mem0, Cognee, LangMem) | Zep's bi-temporal model (GAP-4B); Mem0's integer re-indexing and change history; LangMem's extraction prompts; Cognee's declared-shape validation | ADD-only ingestion; hard delete on contradiction; opaque LLM consolidation | Source provenance, collector traces, governed metamodel versioning, drift quarantine, versioned conflict policy | ## Per-competitor detail ### vs Zep/Graphiti -| Dimension | DICE | Zep | Edge | -|---|---|---|---| -| Ingestion speed | Batch classify + auto-merge + canonical dedup | Sequential only ("must be awaited") | **DICE** | -| Classification nuance | 5-way with edge cases + few-shot | Duplicate vs contradicted (binary) | **DICE** | -| Confidence model | Exponential decay + outcome-dependent adjustment + reinforceCount | No decay, no confidence scoring | **DICE** | -| Extraction quality | SNR, confidence-qualified, role-aware, schema-bound | Custom entity types via Pydantic, entity validation | Tie | -| Temporal model | System timestamps only (created/revised) | Bi-temporal (valid_at/invalid_at/expired_at) | **Zep** | -| Graph structure | Propositions + entity mentions, no graph DB required | Full knowledge graph in Neo4j with community detection | **Zep** | -| Retrieval | Vector similarity + canonical match | Cosine + BM25 + BFS + 5 rerankers | **Zep** | -| Schema governance | Content-hashed metamodel versions, per-type opt-in (in delivery) | None — entity types are code, not versioned artifacts | **DICE**, on direction | -| Infrastructure weight | Embeddable, JVM-native, no external deps | Requires Neo4j + embedding service + LLM | **DICE** | - -**Their moat**: bi-temporal fact model, custom entity types via Pydantic, five reranking strategies, -community subgraph summaries, Neo4j-backed traversal. - -**Their weakness**: sequential-only ingestion, Python/Go only, heavy infrastructure, no schema -versioning. Their audit story is SOC 2 on the managed service — an org-level control, not a -data-lineage API. - -**Attack angle**: embeddable, no Neo4j requirement, faster at high-throughput ingestion. Governance -is the clean differentiator; the temporal gap (GAP-4B) is the thing to close before claiming parity. +| Dimension | DICE | Zep | +|---|---|---| +| Ingestion | Batch classify + auto-merge + canonical dedup | Sequential only ("must be awaited") | +| Classification | 5-way with edge cases + few-shot | Duplicate vs contradicted (binary) | +| Confidence model | Exponential decay + outcome-dependent adjustment + reinforceCount | No decay, no confidence scoring | +| Extraction quality | SNR, confidence-qualified, role-aware, schema-bound | Custom entity types via Pydantic, entity validation | +| Temporal model | System timestamps only (created/revised) | Bi-temporal (valid_at/invalid_at/expired_at) | +| Graph structure | Propositions + entity mentions, no graph DB required | Full knowledge graph in Neo4j with community detection | +| Retrieval | Vector similarity + canonical match | Cosine + BM25 + BFS + 5 rerankers | +| Schema governance | Content-hashed metamodel versions, per-type opt-in | Entity types are code, unversioned | +| Infrastructure | Embeddable, JVM-native, no external deps | Neo4j + embedding service + LLM | + +Zep is Python and Go only. Its audit story is SOC 2 on the managed service, an org-level control +with no data-lineage API. The temporal difference is GAP-4B. ### vs Mem0 -| Dimension | DICE | Mem0 | Edge | -|---|---|---|---| -| Classification | 5-way taxonomy with edge-case guidance | v3 is ADD-only; supersession and contradiction aren't expressible | **DICE** | -| Dedup pipeline | Canonical + auto-merge + batch LLM | Entity linking at retrieval time, not at write | **DICE** | -| Ingestion latency | Batch LLM call per chunk | Single-pass, deliberately minimal | **Mem0** | -| Confidence model | Decay + outcome adjustment + qualification at extraction | None | **DICE** | -| ID safety | Integer re-indexing prevents hallucination | Integer re-indexing prevents hallucination | Tie | -| Graph memory | Entity mentions + Neo4j projection | Neo4j/Memgraph/Neptune/Kuzu | **Mem0** | -| Audit trail | Source provenance + collector decision traces, no run-level trace | Full SQLite history (old/new/event/actor) | **Mem0** | -| Schema governance | Governed metamodel versions (in delivery) | None | **DICE**, on direction | - -**Their moat**: graph memory across four stores, vision, procedural memory for agent traces, a real -change history, and the lowest ingestion latency in the field. - -**Their weakness**: v3 traded expressiveness for speed. ADD-only means contradictions accumulate -unresolved; there is no supersession semantics to appeal to. - -**Attack angle**: their change history is genuinely good and ours should be at least as queryable. -Where we win is that their history records *what changed*, not *what schema it was valid under*. +| Dimension | DICE | Mem0 | +|---|---|---| +| Classification | 5-way taxonomy with edge-case guidance | v3 is ADD-only; supersession and contradiction are inexpressible | +| Dedup pipeline | Canonical + auto-merge + batch LLM | Entity linking at retrieval time | +| Ingestion latency | Batch LLM call per chunk | Single-pass, deliberately minimal | +| Confidence model | Decay + outcome adjustment + qualification at extraction | None | +| ID safety | Integer re-indexing prevents hallucination | Integer re-indexing prevents hallucination | +| Graph memory | Entity mentions + Neo4j projection | Neo4j/Memgraph/Neptune/Kuzu | +| Audit trail | Source provenance + collector decision traces, no run-level trace | Full SQLite history (old/new/event/actor) | +| Schema governance | Governed metamodel versions | None | + +Mem0 also has vision and procedural memory for agent traces, and the lowest ingestion latency in +the field. Its v3 traded expressiveness for speed: ADD-only means contradictions accumulate +unresolved with no supersession semantics. Its change history records what changed and omits the +schema the change was valid under. ### vs Cognee -| Dimension | DICE | Cognee | Edge | -|---|---|---|---| -| Declared shape | `DataDictionary` + `SchemaAdherence` (STRICT/DEFAULT/RELAXED), shipped | Pydantic models, optionally overriding an auto-generated ontology | Tie | -| Schema versioning | Content-hashed, stored, comparable (in delivery) | None — validation is per-run | **DICE**, on direction | -| Ontology | Declared types and relationships | RDF/RDFS triples, auto-generated from the corpus | **Cognee** | -| Validation failure handling | Adherence policy; non-conforming extraction retained and flaggable | Pydantic rejects the shape; no error feedback loop | **DICE** | -| Provenance | `ProvenanceEntry` edges to source chunks | Not documented | **DICE** | - -**Their moat**: the only competitor with a real declared-schema check at write time, and -auto-generated ontology lowers the setup cost a lot. - -**Their weakness**: shape validation without versioning. Change the Pydantic model and nothing tells -you what previously-stored data no longer conforms — which is the exact problem the detect tier -exists to solve. +| Dimension | DICE | Cognee | +|---|---|---| +| Declared shape | `DataDictionary` + `SchemaAdherence` (STRICT/DEFAULT/RELAXED) | Pydantic models, optionally overriding an auto-generated ontology | +| Schema versioning | Content-hashed, stored, comparable | Validation is per-run | +| Ontology | Declared types and relationships | RDF/RDFS triples, auto-generated from the corpus | +| Validation failure handling | Adherence policy; non-conforming extraction retained and flaggable | Pydantic rejects the shape; no error feedback loop | +| Provenance | `ProvenanceEntry` edges to source chunks | Not documented | -**Attack angle**: closest competitor conceptually, and the one worth watching. Pydantic validation is -a good idea we should match at the extraction boundary. +Cognee is the only surveyed competitor with a declared-schema check at write time, and its +auto-generated ontology lowers setup cost. Changing the Pydantic model tells you nothing about +which previously-stored data no longer conforms, which is what the detect tier addresses. ### vs Letta (MemGPT) and Hindsight -Letta declines to have a schema at all: core memory managed by the LLM, archival memory as -pgvector passages, conflicts resolved by the agent through tool calls. Consolidation is agent-driven, -not governed. Nothing to compare on governance; the honest comparison is philosophical — Letta trusts -the agent, DICE trusts the substrate. +Letta has no schema: core memory managed by the LLM, archival memory as pgvector passages, +conflicts resolved by the agent through tool calls, consolidation agent-driven. There is no +governance surface to compare against. Hindsight structures facts into a graph with entity resolution good enough to link "Alice" to "my -coworker Alice", and improves across sessions. Smaller product surface than the incumbents, no -documented temporal invalidation or contradiction framework. Worth re-checking; the entity-resolution -quality is the part that could matter. +coworker Alice", and improves across sessions. Smaller product surface than the incumbents, with no +documented temporal invalidation or contradiction framework. The entity-resolution quality is the +part worth re-checking. ### vs LangChain/LangMem -| Dimension | DICE | LangMem | Edge | -|---|---|---|---| -| Extraction prompts | SNR, confidence-qualified, role-aware, few-shot | Confidence-qualified, surprise-prioritised, SNR | Tie | -| Dedup/classification | Structured 5-way pipeline with fast paths | LLM tool calls (insert/update/delete) | **DICE** | -| Batch processing | N propositions in 1 LLM call | Sequential tool calls | **DICE** | -| Prompt optimisation | Not applicable | Gradient-based prompt evolution | **LangMem** | -| Retrieval | Vector similarity | Dilated windows + LLM-generated queries | **LangMem** | -| Graph memory | Entity mentions on propositions | Commented-out prototype | **DICE** | -| Background processing | Synchronous pipeline | Debounced async reflection | **LangMem** | +| Dimension | DICE | LangMem | +|---|---|---| +| Extraction prompts | SNR, confidence-qualified, role-aware, few-shot | Confidence-qualified, surprise-prioritised, SNR | +| Dedup/classification | Structured 5-way pipeline with fast paths | LLM tool calls (insert/update/delete) | +| Batch processing | N propositions in 1 LLM call | Sequential tool calls | +| Prompt optimisation | None | Gradient-based prompt evolution | +| Retrieval | Vector similarity | Dilated windows + LLM-generated queries | +| Graph memory | Entity mentions on propositions | Commented-out prototype | +| Background processing | Synchronous pipeline | Debounced async reflection | -**Attack angle**: great prompts, weak infrastructure. We've taken the prompt ideas; the pipeline -mechanics are ours. +DICE has taken several of LangMem's prompt ideas. ### vs the managed services (Google, AWS, Microsoft) -All three converge on the same shape: a flat fact string, opaque LLM consolidation, hard delete on -contradiction, no entity model, no confidence, no provenance, and full vendor lock-in — traded -against zero infrastructure and native IAM. +All three share a shape: a flat fact string, opaque LLM consolidation, hard delete on +contradiction, no entity model, no confidence, no provenance, vendor lock-in, zero infrastructure, +native IAM. | Dimension | DICE | Google Memory Bank | AWS AgentCore | Microsoft Foundry | |---|---|---|---|---| @@ -372,63 +317,54 @@ against zero infrastructure and native IAM. | Entity resolution | Multi-strategy + LLM disambiguation | None | None | None | | Contradiction | Both retained, reduced confidence | Old deleted | New entry, no detection | Old value discarded | | Provenance | `ProvenanceEntry` to source chunks | None | None | None | -| Governance | Metamodel versioning (in delivery), drift (planned) | None | None | None | -| Managed | Self-hosted | Fully managed | Fully managed | Fully managed | +| Governance | Metamodel versioning, drift checking | None | None | None | +| Hosting | Self-hosted | Fully managed | Fully managed | Fully managed | | Scale limits | Application-determined | Not published | Not published | 100 scopes, 10K memories/scope | -**Attack angle**: all three validate the category and none of them is a competitor for a team that -cares what happens to knowledge over time. The interesting question these raise isn't technical, it's -whether DICE eventually wants a managed offering — the embeddable library model is a strength today. +The open question these raise is whether DICE eventually wants a managed offering. ### vs Neo4j Agent Memory -| Dimension | DICE | Neo4j Agent Memory | Edge | -|---|---|---|---| -| Classification nuance | 5-way with edge cases + few-shot | No taxonomy — entity resolution handles dedup | **DICE** | -| Batch processing | N propositions in 1 LLM call | Sequential cascade stages | **DICE** | -| Confidence/decay | Decay + outcome adjustment + reinforceCount | None | **DICE** | -| Contradiction | Both retained with reduced confidence | Merged or left distinct | **DICE** | -| Extraction pipeline | Single LLM call, SNR-maximising | spaCy → GLiNER → LLM cascade | **Neo4j** | -| Graph structure | Propositions + mentions, Neo4j as projection | Native graph with POLE+O ontology | **Neo4j** | -| Temporal model | System timestamps only | valid_from/valid_until + geospatial | **Neo4j** | -| Retrieval | Vector + canonical + entity + composable query | Hybrid vector + up to 3-hop traversal | **Neo4j** | -| Schema enforcement | Application-layer, per context; quarantine planned | Heading toward store-level GRAPH TYPE enforcement | Different bets | -| Infrastructure weight | Embeddable, no external deps | Neo4j 5.11+ plus spaCy/GLiNER models | **DICE** | - -**Attack angle**: the most architecturally serious competitor, and the one whose direction most -validates ours. The competition is proposition-centric versus entity-centric: DICE manages the -*lifecycle* of claims, they build a graph of entities. Their store-level schema bet (GRAPH TYPE) and -our substrate-level bet are the interesting divergence — theirs is stronger inside Neo4j, ours is the -only one that survives leaving it. - -## Key remaining gaps - -| Gap | Blocks us against | Impact | +| Dimension | DICE | Neo4j Agent Memory | |---|---|---| -| ~~ID hallucination prevention (GAP-6)~~ | ~~Mem0~~ | ~~DONE — integer re-indexing~~ | -| Detect tier (declared vs observed) | The whole governance claim | Tier 1 is still in delivery; the claim needs tier 2 | -| Quarantine tier | Nobody, yet — but it's the headline | Claim stays architectural until it lands | -| Bi-temporal model (GAP-4B) | Zep, Neo4j | No point-in-time queries or temporal contradiction resolution | -| Temporal anchoring (GAP-4A) | Zep | Relative dates stored as literal text | -| Surprise-prioritised retention (GAP-2) | LangMem | Novel facts don't get durable treatment | -| Incident routing for `DriftReport` | DataHub, Soda | A drift report nobody is notified about is a log line | - -## Not worth chasing - -- **Zep's 5-reranker retrieval** — deep feature tied to Neo4j traversal. Bi-temporal is the better +| Classification | 5-way with edge cases + few-shot | No taxonomy; entity resolution handles dedup | +| Batch processing | N propositions in 1 LLM call | Sequential cascade stages | +| Confidence/decay | Decay + outcome adjustment + reinforceCount | None | +| Contradiction | Both retained with reduced confidence | Merged or left distinct | +| Extraction pipeline | Single LLM call, SNR-maximising | spaCy → GLiNER → LLM cascade | +| Graph structure | Propositions + mentions, Neo4j as projection | Native graph with POLE+O ontology | +| Temporal model | System timestamps only | valid_from/valid_until + geospatial | +| Retrieval | Vector + canonical + entity + composable query | Hybrid vector + up to 3-hop traversal | +| Schema enforcement | Application-layer, per context, with quarantine | Store-level GRAPH TYPE enforcement | +| Infrastructure | Embeddable, no external deps | Neo4j 5.11+ plus spaCy/GLiNER models | + +The division is proposition-centric against entity-centric: DICE manages the lifecycle of claims, +Neo4j builds a graph of entities. Their store-level schema bet is enforced by the database and +scoped to it; DICE's substrate-level bet applies across backends. + +## Remaining gaps + +- No extraction-run identifier, so a stored claim cannot be traced to the run and schema that + produced it (issue #67). +- No bi-temporal model (GAP-4B): system timestamps only, so no point-in-time query and no temporal + contradiction resolution. +- No temporal anchoring (GAP-4A): relative dates are stored as literal text. +- No surprise-prioritised retention (GAP-2): novel facts get no durability preference. +- No incident routing for `DriftReport`. DICE emits events and has no owner model. +- Conflict policy has no versioning or audit trail. + +## Out of scope + +- **Zep's 5-reranker retrieval.** Deep feature tied to Neo4j traversal. Bi-temporal is the better investment. -- **LangMem's prompt optimisation** — interesting, orthogonal to memory quality. -- **Mem0's separate graph pipeline** — we already have entity mentions plus Neo4j projection. -- **Google's multimodal extraction** — the proposition model is format-agnostic; add later if a use - case demands it. -- **AWS's episodic reflection** — our abstraction pipeline already synthesises across propositions. -- **Neo4j's POLE+O ontology** — domain-specific subtypes; the proposition model is domain-agnostic by - design. -- **Neo4j's spaCy → GLiNER → LLM cascade** — cost-effective, operationally heavy (model downloads, - dependency management). Not worth it for proposition extraction. -- **Contract YAML as the authoring surface** — ODCS is a good spec, but DICE's declared schema is - already a JVM type an application owns. A second YAML dialect would be two sources of truth. -- **Write-time rejection of undeclared types** — the one governance move that destroys information - irreversibly. Quarantine is strictly better for an LLM-driven substrate. -- **Managed hosting** — every hyperscaler validates the category; none of them makes the embeddable - model less valuable. +- **LangMem's prompt optimisation.** Orthogonal to memory quality. +- **Mem0's separate graph pipeline.** DICE has entity mentions plus Neo4j projection. +- **Google's multimodal extraction.** The proposition model is format-agnostic; add on demand. +- **AWS's episodic reflection.** The abstraction pipeline already synthesises across propositions. +- **Neo4j's POLE+O ontology.** Domain-specific subtypes; the proposition model is domain-agnostic. +- **Neo4j's spaCy → GLiNER → LLM cascade.** Cost-effective and operationally heavy (model + downloads, dependency management). +- **Contract YAML as the authoring surface.** DICE's declared schema is a JVM type an application + owns; a YAML dialect would be a second source of truth. +- **Write-time rejection of undeclared types.** It discards an extraction irrecoverably. +- **Managed hosting.** The embeddable library is the shipped distribution model. diff --git a/specs/developer-docs-architecture.md b/specs/developer-docs-architecture.md index 6b8cdaa1..54d330fb 100644 --- a/specs/developer-docs-architecture.md +++ b/specs/developer-docs-architecture.md @@ -1,29 +1,26 @@ # Developer docs architecture -The target information architecture for DICE's developer documentation — what pages exist, in what -order, and who owns keeping them true. This is a plan, not a description: most of what's below -doesn't exist yet. +The target information architecture for DICE's developer documentation: what pages exist, in what +order, and who owns keeping them true. Most of what's below doesn't exist yet. DICE ships to Maven Central as a set of JVM modules with Spring Boot autoconfiguration. The docs that fit that shape are the ones Confluent Schema Registry, Flyway, Zep and Spring Boot itself -write: **concept-anchored, workflow-layered**. None of them uses strict Diátaxis. All of them open -with a runnable quickstart, then a small set of mental models, then task-shaped how-tos, then dense -reference — with optional features flagged at the top of their own page and a compatibility matrix -that answers "will this work with my stack" without reading anything else. +write: concept-anchored and workflow-layered, none of them strict Diátaxis. All of them open with a +runnable quickstart, then a small set of mental models, then task-shaped how-tos, then dense +reference, with optional features flagged at the top of their own page and a compatibility matrix +that answers "will this work with my stack" on its own. ## What we already have, and what's missing -`docs/design/` is the *why*: rationale notes aimed at someone changing DICE. They're good and they -stay as they are. `specs/` is strategy and planning, also internal. +`docs/design/` is the *why*: rationale notes aimed at someone changing DICE. They stay as they are. +`specs/` is strategy and planning, also internal. -The *how* exists, but in one place and the wrong shape: **`README.md` is 2,621 lines**. It already -holds Spring pipeline setup, the proposition pipeline walkthrough, mention filtering, entity -extraction and resolution, `ContextId` and `PropositionQuery`, graph and Prolog projection, agent -memory, the Oracle, the REST API, Spring Boot integration, graph-backed storage config, API-key -security and installation. That's most of a docs site pretending to be a landing page, and nobody -reads to line 2,000. +The *how* is all in `README.md`, which is 2,621 lines. It holds Spring pipeline setup, the +proposition pipeline walkthrough, mention filtering, entity extraction and resolution, `ContextId` +and `PropositionQuery`, graph and Prolog projection, agent memory, the Oracle, the REST API, Spring +Boot integration, graph-backed storage config, API-key security and installation. -So this isn't a greenfield IA. It's mostly a **migration**: +So the work is mostly a migration: | Directory | Audience | Question it answers | |---|---|---| @@ -32,35 +29,34 @@ So this isn't a greenfield IA. It's mostly a **migration**: | `specs/` | Us | What are we building and why does it matter commercially? | | `README.md` | Anyone landing on the repo | What is this, should I care, where do I start? | -Design docs and developer docs cross-link but never duplicate. A concept page says what a +Design docs and developer docs cross-link without duplicating. A concept page says what a proposition is and how to make one; the design note says why confidence decays the way it does. ### Migrating the README -Target: a README under ~250 lines. Line numbers are as of this branch; they'll move, so re-check +Target: a README under ~250 lines. Line numbers are as of this branch and will move, so re-check before acting on the table. | README section (line) | Fate | |---|---| -| What is DICE, benefits table, architecture overview (24–116) | **Keep**, trimmed. This is the landing page's actual job. | -| Real-world example: Impromptu (117–176) | **Keep**, cut to a paragraph plus a link. Proof it's used, not a tutorial. | +| What is DICE, benefits table, architecture overview (24–116) | **Keep**, trimmed. This is the landing page's job. | +| Real-world example: Impromptu (117–176) | **Keep**, cut to a paragraph plus a link. | | Pipeline setup, conversation analysis (127–176) | **Move** → `quickstart/` | | Proposition pipeline, content dedup, mention filtering (177–489) | **Move** → `concepts/propositions.md`, `how-to/extract-from-documents.md`, `how-to/mention-filtering.md` | -| Entity extraction, entity resolution, resolution service (490–1225) | **Move** → `concepts/entity-resolution.md` + `how-to/tune-entity-resolution.md`. The single biggest block; split it. | +| Entity extraction, entity resolution, resolution service (490–1225) | **Move** → `concepts/entity-resolution.md` + `how-to/tune-entity-resolution.md`. The largest block; split it. | | Source analysis context, `ContextId`, `PropositionQuery` (1226–1419) | **Move** → `concepts/context-and-schema.md`, `how-to/query-propositions.md` | | Relations, projector architecture, graph and Prolog projection (1420–1647) | **Move** → `concepts/storage-and-projections.md`, `how-to/project-to-graph.md`, `features/prolog-inference.md` | | Agent memory, memory projection, memory maintenance (1648–1979) | **Move** → `how-to/agent-memory.md`, `concepts/knowledge-hygiene.md` | | Proposition operations, Oracle (1980–2090) | **Move** → `how-to/query-propositions.md`, `how-to/oracle.md` | | Package structure (2091–2205) | **Move** → `reference/` | -| REST API and endpoints (2206–2334) | **Move** → `features/web-api.md` (it's opt-in and gated by an API key — it belongs behind a gate page) | +| REST API and endpoints (2206–2334) | **Move** → `features/web-api.md`; it is opt-in and gated by an API key. | | Spring Boot integration, graph-backed storage, API-key security (2335–2572) | **Move** → `how-to/choose-a-backend.md`, `reference/configuration-properties.md`, `features/web-api.md` | | Installation (2573–2585) | **Keep** as coordinates only; the working version lives in the quickstart. | | Technology stack, references, license (2586–2621) | **Keep**. | -Two rules for the migration, or it will half-happen and leave two sources of truth. Content **moves** -— no section is copied into `docs/` and left in the README. And every removed section leaves a -one-line link where it was, so an existing bookmark still lands somewhere useful. Do it as one PR per -destination page, not one giant PR nobody can review. +Two rules for the migration. Content moves: each section is deleted from the README as it lands in +`docs/`, so there is one copy. And every removed section leaves a one-line link where it was, so an +existing bookmark still lands somewhere useful. Do it as one PR per destination page. ## The IA @@ -70,7 +66,7 @@ flowchart TD QS["quickstart/
15 minutes to first extraction"] C["concepts/
the spine — read in order"] H["how-to/
by use case"] - F["features/
opt-in, gate first"] + F["features/
opt-in, activation condition first"] REF["reference/
config, API, properties"] P["production/
deploy, tune, observe"] S["support/
compatibility matrix, FAQ"] @@ -89,27 +85,24 @@ flowchart TD F --- F1["metamodel versioning
governance and drift
Prolog inference
web API
decay and pruning"] ``` -Five concepts, in that order, because each one needs the one before it. Everything else is entered +Five concepts, in that order, because each needs the one before it. Every other section is entered from any direction. ## The concept spine -The whole substrate is one chain, and cherry-picking it produces confusion: - 1. **Propositions** — natural-language claims are the system of record. Confidence, importance, decay, lifecycle states. Everything else is a projection of these. -2. **Entity resolution** — how mentions in text get matched to entities that already exist, or minted - as new ones. Resolution outcomes, the escalating resolver chain, cross-chunk dedup. -3. **Storage and projections** — the `PropositionStore` SPI family, in-memory versus the durable - Neo4j backend, and the materialised views (vector, graph, Prolog, memory, oracle). +2. **Entity resolution** — how mentions in text get matched to entities that already exist, or + minted as new ones. Resolution outcomes, the escalating resolver chain, cross-chunk dedup. +3. **Storage and projections** — the `PropositionStore` SPI family, in-memory and the durable Neo4j + backend, and the materialised views (vector, graph, Prolog, memory, oracle). 4. **Context and schema** — `contextId` scoping, the `DataDictionary`, `SchemaAdherence`, and what - "extraction against a schema" actually constrains. -5. **Knowledge hygiene** — admission gates, reclamation, consolidation. Why they're three - interventions at three moments rather than one. + "extraction against a schema" constrains. +5. **Knowledge hygiene** — admission gates, reclamation, consolidation, and why they are three + interventions at three moments. -Every concept page ends with **Try it now**: five to ten lines that exercise the thing just -explained against the quickstart's setup, and the output to expect. A concept page with no runnable -code is a design note in the wrong directory. +Every concept page ends with **Try it now**: five to ten lines that exercise what was just +explained against the quickstart's setup, and the output to expect. ## The 15-minute quickstart @@ -117,55 +110,50 @@ One page, one path, no branches. Copy, paste, run, see output. ### Prerequisite: there is no starter artifact -DICE ships `dice`, `dice-ingestion`, `dice-storage`, `dice-storage-autoconfigure` and `dice-report`. -There is **no** `dice-spring-boot-starter`. A one-coordinate quickstart therefore has a build -prerequisite, and it has to be named rather than assumed: +DICE ships `dice`, `dice-ingestion`, `dice-storage`, `dice-storage-autoconfigure` and +`dice-report`. There is no `dice-spring-boot-starter`, so a one-coordinate quickstart has a build +prerequisite: -- **Either** create a `dice-spring-boot-starter` aggregator module that depends on - `dice-storage-autoconfigure` (which pulls `dice-storage` and `dice`) plus `dice-report`, publish it - to Maven Central, and let the quickstart use one coordinate. This is the better outcome and it's a - small module — but it's a build task that has to land *before* the quickstart page can be written - as one line. -- **Or** write the quickstart against the real coordinates today: `dice-storage-autoconfigure` and - `dice-report`, both explicit, with a comment saying what each buys. Honest, two lines instead of - one, and it doesn't block the docs on a release. +- **Either** create a `dice-spring-boot-starter` aggregator module depending on + `dice-storage-autoconfigure` (which pulls `dice-storage` and `dice`) plus `dice-report`, publish + it to Maven Central, and let the quickstart use one coordinate. Small module, and it has to land + before the quickstart page can be written as one line. +- **Or** write the quickstart against today's coordinates, `dice-storage-autoconfigure` and + `dice-report`, both explicit, with a comment saying what each buys. Two lines, and it unblocks + the docs from a release. -Pick one before writing the page. What is not acceptable is documenting a starter that doesn't exist. +Pick one before writing the page. The page uses coordinates that are published. ### The page 1. **The dependencies.** Whichever of the two above we picked, with the version from the BOM. -2. **Configuration.** The minimum that actually works: an LLM provider (inherited from - embabel-agent), an **embedding model** — `InMemoryPropositionRepository` does vector search, and - without an embedding model configured the vector-query step in step 6 returns nothing — and the - default in-memory store, which means no Neo4j and no Docker for the first run. If we'd rather keep - the config to one key, drop vector retrieval from the minimal path and query by entity and - `ContextId` only, then introduce embeddings in the retrieval how-to. Either is fine; silently - showing a vector query that can't work is not. -3. **What autoconfiguration gave you.** The beans that now exist and what each is for. Not a list of - `@Configuration` classes — a list of things you can `@Autowired`. +2. **Configuration.** The minimum that works: an LLM provider (inherited from embabel-agent), an + embedding model, and the default in-memory store, which means no Neo4j and no Docker for the + first run. `InMemoryPropositionRepository` does vector search, so without a configured embedding + model the vector query in step 6 returns nothing. To keep the config to one key, drop vector + retrieval from the minimal path, query by entity and `ContextId` only, and introduce embeddings + in the retrieval how-to. Either way the page's queries match the configuration it gave. +3. **What autoconfiguration gave you.** The beans that now exist and what each is for: a list of + things you can `@Autowired`. 4. **Extract.** Feed one paragraph of text through the pipeline. Print the propositions with their confidence. -5. **Persist.** `PropositionPipeline` returns **unsaved** results — that's deliberate, so the caller - owns the transaction, and it's the step every naive quickstart forgets. Call - `persist(propositionRepository, namedEntityDataRepository)` on the `PersistablePropositions` - result, and say in one sentence why the pipeline doesn't do it for you. A reader who skips this - queries an empty store and concludes DICE is broken. +5. **Persist.** `PropositionPipeline` returns unsaved results, so the caller owns the transaction. + Call `persist(propositionRepository, namedEntityDataRepository)` on the `PersistablePropositions` + result, and say in one sentence why the pipeline leaves it to the caller. Skipping this step + leaves an empty store. 6. **Query.** Retrieve what was just stored, by entity and `ContextId`, and by vector similarity if step 2 configured an embedding model. 7. **Report.** Produce one human-readable artifact so the run ends in something visible. -8. **Where to go next**, split by intent — see the audience note below. +8. **Where to go next**, split by intent; see the audience note below. -Rules. The 15 minutes is a promise: no Neo4j, no Docker, no `create` of anything external. The code -is real and compiles; a test in `dice-integration-tests` runs the quickstart's exact snippets so -that promise can't rot silently — and that test is what would have caught the missing persist step. -Anything that needs a decision — backend choice, resolver chain, schema — takes the default and -links out. +Constraints. The 15 minutes holds: no Neo4j, no Docker, no `create` of anything external. The code +is real and compiles, and a test in `dice-integration-tests` runs the quickstart's exact snippets, +which is what catches a missing persist step. Anything that needs a decision (backend choice, +resolver chain, schema) takes the default and links out. ## How-to guides, by use case -Task-shaped, titled by what the reader wants, and each one states its prerequisites at the top -rather than assuming a reading order: +Task-shaped, titled by what the reader wants, each stating its prerequisites at the top: - Extract knowledge from a document set (chunking, dedup before extraction, batch, and persisting the unsaved results the pipeline hands back) @@ -179,51 +167,49 @@ rather than assuming a reading order: - Handle conflicts and contradictions (the conflict policy SPI) - Process a stream incrementally (windowing, dedup across windows) -Each names what it costs — LLM calls, latency, infrastructure. A how-to that hides the cost is how -you get an angry issue three weeks later. +Each states what it costs in LLM calls, latency, and infrastructure. -## Feature pages lead with the gate +## Feature pages -Every optional feature gets its own page and every one of those pages opens with the gate, not the -pitch. This matters most for versioning and governance, where the whole design is that **nothing -happens unless you asked for it** — no `DeclaredSchemaSource`, no versioning; no governed types, no -drift detection. A page that opens by explaining metamodel versioning teaches the reader that DICE -versions their schema. It does not. +Every optional feature gets its own page, and each page states its activation condition in the +first paragraph. This matters most for versioning and governance, where nothing happens until an +application asks for it: no `DeclaredSchemaSource`, no versioning; no governed types, no drift +detection. DICE versions a schema only when an application declares one. Template, in order: - **Availability** — which DICE version, which modules, what it costs to add. -- **The gate** — the exact bean, property, or annotation that switches it on. First screen, always. - "Off unless you define a `DeclaredSchemaSource` bean" is the first thing on the versioning page. -- **When to use it** — and, explicitly, when not to. +- **Activation condition** — the exact bean, property, or annotation that switches it on, on the + first screen. "Off unless you define a `DeclaredSchemaSource` bean" opens the versioning page. +- **When to use it**, and when to leave it off. - **Impact** — latency, memory, extra infrastructure, extra LLM calls. - **How to enable** — full working configuration. - **Example** — real code with real output. Opt-in surfaces that need this treatment: metamodel versioning, drift detection and quarantine, -Prolog inference, graph projection, the web API, decay and stale-pruning, the multi-signal collector, -concurrent extraction. +Prolog inference, graph projection, the web API, decay and stale-pruning, the multi-signal +collector, concurrent extraction. ## Compatibility matrix -Its own page, linked from the README, answering "will this work with my stack" before anyone reads -prose. Four axes, and the embabel-agent one is a **range**, not a point — DICE tracks a moving -platform and the supported window is the single most-asked question a library like this gets. +Its own page, linked from the README, answering "will this work with my stack" before any prose. +Four axes. The embabel-agent one is stated as a supported range, since DICE tracks a moving +platform. | Axis | What we state | |---|---| | embabel-agent | Supported version range per DICE release, with the tested point release called out | | Spring Boot | Minimum and maximum tested, per DICE release | -| JDK | Baseline is 21 (inherited from `embabel-build-parent`); newer JDKs listed as tested or not | -| Neo4j | Required version floor for the durable backend, via Drivine; explicitly "not required" for in-memory | +| JDK | Baseline is 21 (inherited from `embabel-build-parent`); newer JDKs listed as tested or untested | +| Neo4j | Required version floor for the durable backend, via Drivine; "not required" for in-memory | -Plus a feature-availability column, so a reader can see at a glance that governance arrives after -1.0 rather than discovering it from a 404. Exact values get pinned at the 1.0 release; the page ships -with the axes and the honest "tested against" values we have. +Plus a feature-availability column, so a reader can see that governance arrives after 1.0. Exact +values get pinned at the 1.0 release; the page ships with the axes and the tested-against values we +have. ## Production guide -The page a team reads before going live, not a "deployment" afterthought: +The page a team reads before going live: - Backend choice and the migration from in-memory to Neo4j - Schema declarations and index creation on startup @@ -238,40 +224,38 @@ The page a team reads before going live, not a "deployment" afterthought: Two audiences arrive with different vocabularies and different first questions: -- **Data engineers** want to know where the data lands, what the schema is, how drift is handled, and - what Cypher they can run. They read storage, schema, governance, projections. +- **Data engineers** want to know where the data lands, what the schema is, how drift is handled, + and what Cypher they can run. They read storage, schema, governance, projections. - **LLM/agent engineers** want extraction quality, entity resolution, memory retrieval, and how it plugs into an agent. They read propositions, resolution, retrieval, agent memory. -**One spine, multiple doors.** The concept order doesn't fork — propositions still come before -resolution before storage for everybody. What forks is the entry point: the quickstart's "where to -go next" offers a data-engineer path and an agent-engineer path through the same pages in different -orders, and the README says which is which. Two parallel doc sets would drift within a quarter. +The concept order is the same for both: propositions, then resolution, then storage. What differs +is the entry point. The quickstart's "where to go next" offers a data-engineer path and an +agent-engineer path through the same pages in different orders, and the README says which is which. +Two parallel doc sets drift. -A third audience (researchers wanting Prolog and reasoning) is real but small; served by the Prolog -feature page, not by a third path. +A third audience, researchers wanting Prolog and reasoning, is real and small, and is served by the +Prolog feature page. ## Risks -**Reading-order enforcement.** The concept chain only makes sense in order, and readers arrive from +**Reading-order enforcement.** The concept chain makes sense in order, and readers arrive from search engines in the middle of it. Nothing in a static docs site forces sequence. Mitigations, all cheap: number the concept pages, put a one-line "assumes you've read X" at the top of each, and end -each with an explicit next link. Accept that some readers land mid-chain and make each page survive -it — a page that only works as chapter 4 is a page most readers will bounce off. +each with an explicit next link. Each page has to stand on its own for a reader who lands mid-chain. **Extraction variability.** The docs assume the extractor returns clean, schema-adhering -propositions. It won't always. Mention filtering and `SchemaAdherence` need to appear early and be -weighted heavily, not buried in a how-to nobody reaches. +propositions. It won't always. Mention filtering and `SchemaAdherence` appear early and are weighted +heavily. **Backend coupling surfaces late.** Prolog needs a store; graph projection needs decisions made -before extraction runs. The quickstart deliberately defers all of this, so the "choose your backend" -how-to has to be prominent and has to say plainly that changing it later means rework. +before extraction runs. The quickstart defers all of this, so the "choose your backend" how-to has +to be prominent and has to say plainly that changing it later means rework. **Duplication across modules, and with the README.** Spring configuration examples exist in -`dice-storage-autoconfigure`, `dice-ingestion` and elsewhere, and a 2,621-line README holds a version -of most of them. One canonical example set in `docs/`; module READMEs and the root README link to it -and never restate it. The migration table above only works if content moves rather than being copied -— two sources of truth is worse than one badly-placed one. +`dice-storage-autoconfigure`, `dice-ingestion` and elsewhere, and the 2,621-line README holds a +version of most of them. One canonical example set in `docs/`; module READMEs and the root README +link to it. The migration table above works only if content moves. **Doc rot.** Handled by the contract below, and by making the quickstart executable. @@ -280,25 +264,22 @@ and never restate it. The migration table above only works if content moves rath **Every feature PR ships its design-doc delta and its developer-doc page in the same PR. Missing docs are a review-blocking finding, on the same footing as a missing test.** -What that means concretely: +Concretely: - New or changed behaviour visible to a consumer → the developer-doc page is updated in that PR. - New or changed *rationale* → the `docs/design/` note is updated in that PR. -- A new opt-in feature → a feature page that leads with its gate, in that PR. +- A new opt-in feature → a feature page stating its activation condition, in that PR. - A new configuration property → the reference page, in that PR. - A version-support change → the compatibility matrix, in that PR. -The reasoning is the same one that makes tests non-negotiable. Docs written later are written by -someone who has forgotten the edge cases, or by nobody. And in a library whose central promise is -that governance is opt-in, an undocumented opt-in switch is indistinguishable from a feature that -doesn't exist. +Docs written later are written by someone who has forgotten the edge cases. Governance is opt-in, +so an undocumented switch cannot be turned on by the consumer it was built for. -Reviewers ask one question: *if I only had this PR's docs, could I use this feature?* If not, the -finding is blocking. "Docs to follow" is not an accepted resolution — the follow-up either doesn't -happen or lands after someone has already been confused by the gap. +Reviewers ask one question: *if I only had this PR's docs, could I use this feature?* If the answer +is no, the finding is blocking. "Docs to follow" does not resolve it. -Small carve-outs, stated so the rule stays credible: internal refactors with no consumer-visible -change need no doc delta, and neither do test-only or build changes. +Carve-outs: internal refactors with no consumer-visible change need no doc delta, and neither do +test-only or build changes. ## Open questions @@ -307,5 +288,5 @@ change need no doc delta, and neither do test-only or build changes. - Rendered site or Markdown in the repo? The IA works either way; the decision affects cross-linking and whether the compatibility matrix can be generated. - Do the `docs/design/` notes stay contributor-facing, or do the best of them get promoted into - concept pages? Current answer: they stay, because their audience is genuinely different — but - concept pages will lift explanations from them, and that's fine. + concept pages? Current answer: they stay, because their audience is different. Concept pages will + lift explanations from them. From 2c95a07bce544c817bf352ce3d4fc4307d1dd16f Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Sun, 30 Aug 2026 21:36:39 -0400 Subject: [PATCH 3/6] docs(specs): research-grounded gaps, modules, and orthogonal-research sections Rank remaining gaps by prevalence across the four surveyed memory services, each with its citation. Move convergent mechanisms to their own section framed as parallel arrival at the same solutions. Add candidate-modules (governed working memory per ArcMem, background consolidation, procedural memory) and orthogonal-research sections (GUM lineage, TMS/ATMS, ACT-R activation, provenance semirings, with the three strongest borrowing opportunities). Correct the write-time rejection claim: recoverable by re-extraction in principle, at LLM cost and without reliable reproduction. --- specs/competitive-positioning.md | 196 ++++++++++++++++++++++++++++--- 1 file changed, 178 insertions(+), 18 deletions(-) diff --git a/specs/competitive-positioning.md b/specs/competitive-positioning.md index 39b11406..5d711788 100644 --- a/specs/competitive-positioning.md +++ b/specs/competitive-positioning.md @@ -106,8 +106,10 @@ selectable reference point to compare against, is more expressive than transitiv right model if governed types get release windows. Both registries hard-reject an incompatible registration with HTTP 409 and no warn mode. DICE -accepts the write and records the divergence. Extraction is LLM-driven, a type nobody declared is -often a real finding, and a write-time rejection discards it irrecoverably. +accepts the write and records the divergence. Extraction is LLM-driven, and a type nobody declared +is often a real finding. A write-time rejection discards it with no review path; re-extraction +costs another LLM pass and does not reliably reproduce the same proposition. Quarantine keeps the +proposition out of reads and keeps it reviewable. ## Opt-in and drift: prior art from migration tooling @@ -225,7 +227,7 @@ property-graph analogues and are worth watching; neither has a canonical report | **Migration & ORM** (Liquibase, Flyway, Hibernate) | Declared-vs-observed diff that reports; validate-mode; per-entity `@Version` opt-in | `hbm2ddl.auto=update` auto-mutation; anything that changes a schema at startup | Drift detection over LLM-extracted knowledge, where the observed side is a graph nobody wrote by hand | | **Data governance** (DataHub, Soda, ODCS) | drift / contract / assertion / policy / incident / quarantine, verbatim; observe→alert→block; valid/invalid split | Blocking at the producer boundary by default; contract YAML as the primary authoring surface | Governance over probabilistic extraction: confidence-weighted claims | | **Graph schema** (Neo4j GRAPH TYPE, TypeDB, SHACL) | SHACL's validation-report structure; GRAPH TYPE's Open variant semantics | Write-time rejection; RDF as a wire format; store-coupled schema declaration | Governance that is per context, tolerates open-world types, and is backend-independent | -| **Agent memory** (Zep, Mem0, Cognee, LangMem) | Zep's bi-temporal model (GAP-4B); Mem0's integer re-indexing and change history; LangMem's extraction prompts; Cognee's declared-shape validation | ADD-only ingestion; hard delete on contradiction; opaque LLM consolidation | Source provenance, collector traces, governed metamodel versioning, drift quarantine, versioned conflict policy | +| **Agent memory** (Zep, Mem0, Cognee, LangMem) | Zep's bi-temporal model (GAP-4B), ahead of DICE's system timestamps; Mem0's change-history record of old value / new value / event / actor; Cognee's declared-shape validation | ADD-only ingestion; hard delete on contradiction; opaque LLM consolidation | Source provenance, collector traces, governed metamodel versioning, drift quarantine, versioned conflict policy | ## Per-competitor detail @@ -289,20 +291,6 @@ coworker Alice", and improves across sessions. Smaller product surface than the documented temporal invalidation or contradiction framework. The entity-resolution quality is the part worth re-checking. -### vs LangChain/LangMem - -| Dimension | DICE | LangMem | -|---|---|---| -| Extraction prompts | SNR, confidence-qualified, role-aware, few-shot | Confidence-qualified, surprise-prioritised, SNR | -| Dedup/classification | Structured 5-way pipeline with fast paths | LLM tool calls (insert/update/delete) | -| Batch processing | N propositions in 1 LLM call | Sequential tool calls | -| Prompt optimisation | None | Gradient-based prompt evolution | -| Retrieval | Vector similarity | Dilated windows + LLM-generated queries | -| Graph memory | Entity mentions on propositions | Commented-out prototype | -| Background processing | Synchronous pipeline | Debounced async reflection | - -DICE has taken several of LangMem's prompt ideas. - ### vs the managed services (Google, AWS, Microsoft) All three share a shape: a flat fact string, opaque LLM consolidation, hard delete on @@ -342,8 +330,100 @@ The division is proposition-centric against entity-centric: DICE manages the lif Neo4j builds a graph of entities. Their store-level schema bet is enforced by the database and scoped to it; DICE's substrate-level bet applies across backends. +## Convergent mechanisms + +Places where DICE and a surveyed system reached the same solution to the same problem. They are +tracked here so the gaps section stays about capabilities DICE lacks. + +- **LLM extraction into a structured store.** All four surveyed memory services take a conversation + turn, run an LLM extraction, and persist the result into a vector, graph or relational store + (brief 15, from https://rywalker.com/research/langmem, + https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview + 2025-07-08, https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html, + https://neo4j.com/labs/agent-memory/). DICE's pipeline has the same shape with the proposition as + the stored unit. +- **Decay and consolidation ahead of hard deletion.** DICE decays effective confidence over time + and retires softly to STALE (`DecaySweepPass`, dual-threshold hysteresis 0.1/0.25). AWS and + Google merge or supersede through consolidation, Neo4j tracks temporal fact validity, LangMem + consolidates to suppress duplication. No surveyed system documents a hard TTL delete. +- **Integer re-indexing of proposition IDs across LLM calls.** DICE (GAP-6, on main) and Mem0 both + renumber IDs per call so the model cannot invent one. +- **Confidence-qualified, SNR-shaped extraction prompts.** DICE and LangMem settled on the same + prompt controls independently (https://rywalker.com/research/langmem). +- **A declared schema coupled to enforcement.** Neo4j GRAPH TYPE, TypeDB and DICE's metamodel work + are three independent arrivals at the same conclusion; the divergence is where the governance + sits, covered above. + +Where a converged mechanism is implemented better elsewhere: Zep's bi-temporal edges against DICE's +system timestamps, Mem0's change history (old value, new value, event, actor) against DICE's +collector traces, and Cognee's Pydantic check as a write-time declared-shape validation. Each is +cited in the section that covers that competitor. + ## Remaining gaps +The focus-first items, ranked by prevalence across the four surveyed memory services (LangMem, +Vertex AI Memory Bank, Bedrock AgentCore Memory, Neo4j Agent Memory) — a capability all four ship +outranks one vendor's experiment. + +**1. Operational tooling and memory inspection.** All four ship an inspection surface; DICE has +none. Bedrock AgentCore emits CloudWatch metrics for latency, invocations, errors and memory +creation count, with spans over CreateEvent, GetEvent, ListEvents, DeleteEvent and +RetrieveMemoryRecords, plus extraction and consolidation logs +(https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html). +Vertex AI Memory Bank lists stored memories in the Cloud Console (Memory Bank UI announcement, +Google Developer Forums; brief 15 records the citation without a URL). Neo4j's NAMS dashboard shows +health, entity count and queue lag with interactive graph visualisation and Cypher queryability +(https://medium.com/neo4j/a-tour-of-the-neo4j-agent-memory-service-nams-0f2d535a4fdb, 2026-06; +brief 15 lists GA status as unconfirmed). LangSmith traces execution paths and state transitions +with cost/latency/error dashboards and human-in-the-loop inspection +(langchain.com/resources/llm-observability-tools, cited by brief 15). In DICE, "why was this +proposition formed" needs custom logging over `CollectorTraceStore`. + +**2. Background memory formation and consolidation pipelines.** All four run formation off the +request path. AWS runs extraction then consolidation as background processes with start/completion +and success counts in the logs (observability-memory-metrics.html). Google's extraction is +asynchronous, ingested via `add_memory()` or an end-of-conversation callback +(https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, +2025-07-08). LangMem offers hot-path tools or background managers +(https://rywalker.com/research/langmem). Neo4j runs background enrichment (Wikipedia, Diffbot) and +a multi-stage extraction cascade (https://neo4j.com/labs/agent-memory/). DICE has an async +`@EventListener` on conversation analysis and a non-blocking `PropositionIncrementalAnalyzer`, with +consolidation available as passes an application invokes. The managed pipeline that schedules them +is missing. + +**3. TTL and eviction controls.** Decay itself is convergent (above); the gap is a retention policy +an operator can set, where DICE's decay constant and sweep thresholds are code-level configuration +(README:103-108). No surveyed system publishes a TTL API either: AWS and Google document no +expiration policy, Neo4j relies on temporal validity, LangMem on consolidation. Brief 15 lists the +AWS TTL knob and the Google retention policy as unverified, so this item rests on weaker evidence +than the two above it. + +**4. Procedural memory.** Two of the four ship it. LangMem has a `procedural` memory type where +agents update their own prompt rules from feedback (https://rywalker.com/research/langmem). Neo4j +stores tool usage and reasoning traces with similarity search over trace lineage +(https://neo4j.com/labs/agent-memory/). AWS and Google extract facts only. DICE projects +propositions into Prolog as a view layer and has no rule-formation path. + +**5. Namespacing and scoping APIs.** Two of the four expose one. AWS uses hierarchical namespaces +for fine-grained access control plus protocol session headers (`Mcp-Session-Id`, +`X-Amzn-Bedrock-AgentCore-Runtime-Session-Id`) +(https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html). Neo4j chains +`(Message)` nodes by `[:NEXT]` per session with entity and fact nodes shared across sessions +(https://neo4j.com/labs/agent-memory/). Google scopes by identity and LangMem by embedding space, +neither with a namespace API. DICE's `ContextId` is a query filter over shared storage +(README:1241-1270), so it isolates reads and not the storage layer. + +**6. Cross-session profile modeling.** All four aggregate across sessions, none publishes a profile +schema. AWS extracts user preferences, facts and session summaries across sessions; Google +personalises long-term memories per user +(https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, +2025-07-08); Neo4j persists preference and fact entities in a shared graph +(https://neo4j.com/labs/agent-memory/); LangMem keeps preferences in semantic memory with optional +typed Pydantic profiles (https://rywalker.com/research/langmem). DICE's `ContextId` scopes +knowledge and defines no profile structure or cross-session aggregation. + +Gaps DICE already tracks: + - No extraction-run identifier, so a stored claim cannot be traced to the run and schema that produced it (issue #67). - No bi-temporal model (GAP-4B): system timestamps only, so no point-in-time query and no temporal @@ -353,6 +433,85 @@ scoped to it; DICE's substrate-level bet applies across backends. - No incident routing for `DriftReport`. DICE emits events and has no owner model. - Conflict policy has no versioning or audit trail. +## Candidate modules + +The capabilities the managed services hold over DICE sit above the proposition substrate and +consume it, so each is a module or add-on rather than a change to the substrate. + +**Governed working memory.** ArcMem is an activation-ranked working-memory system built on DICE +APIs, and is the existing proof that this layer works over the substrate. It keeps a bounded active +set (20 units plus a token budget) in a protected prompt region, ranks units by an activation score blending recency, +reinforcement and decay, protects units by authority level (PROVISIONAL → UNRELIABLE → RELIABLE → +CANON) from automatic demotion, reinforces on access by anchoring decay at `max(contentRevised, +lastAccessed)`, and evicts into durable storage so evicted material stays queryable. Its gate order +is confidence → deduplication → conflict → trust → promotion → budget enforcement, and changing the +order changes the semantics. It consumes DICE's storage, extraction, decay and revision as +primitives (brief 17). A DICE-native module needs five SPIs ArcMem hand-rolls today: + +- activation ranking as a pluggable policy — ArcMem uses hand-tuned parameters and a fixed rank + clamp of [100, 900]; +- authority and trust levels with the promotion gate; +- pinning and protection for the resident set; +- per-turn lifecycle events for reinforce, evict and reactivate; +- batched or concurrent extraction — `PropositionPipeline.process()` is serial and consumers batch + by hand. + +Which of these belong in DICE and which in the consuming application is unsettled (brief 17, +Risks), and entity resolution has to stay serial where shared identity is involved. + +**Session memory management.** In DICE today: `MemoryProjector` classifies propositions by +knowledge type for prompt injection +(`dice/src/main/kotlin/com/embabel/dice/projection/memory/MemoryProjector.kt:46`), and the `Memory` +tool runs hybrid vector plus keyword retrieval over context-scoped propositions +(`dice/src/main/kotlin/com/embabel/dice/agent/Memory.kt:112`). A module would add session-scoped +active-context assembly and summarisation over those two. + +**Background consolidation.** In DICE today: consolidation passes including +`ContradictionResolutionPass` and `DecaySweepPass`, plus async event listeners on analysis. A module +would add the scheduler, the extraction-then-consolidation staging AWS and Google run, and the +lifecycle logging that makes a run inspectable. + +**Procedural memory.** In DICE today: Prolog projection over propositions as a view layer. A module +would add the formation path — turning agent feedback into stored rules that later runs read back. + +## Orthogonal research + +DICE's proposition extraction sits in the lineage of general user modeling: "Creating General User +Models from Computer Use" (Shaikh, Sapkota, Rizvi, Horvitz, Park, Yang, Bernstein; arXiv:2505.10831, +UIST 2025, ACM DOI 10.1145/3746059.3747722), cited in the README for the confidence-weighted +proposition and the exponential decay formula. That is user-modeling research rather than agent +memory. Scanning adjacent non-LLM fields for mechanisms is standing practice here, and these map +onto problems DICE already has: + +| Field | Mechanism | DICE analog | Citation | +|---|---|---|---| +| Truth maintenance (TMS/ATMS) | Justifications record why a belief holds; retracting a premise un-derives its dependents; ATMS labels a node with the assumption sets that support it | `ContradictionResolutionPass` retires the weaker of a contradictory pair to CONTRADICTED by comparing `effectiveConfidence()`, with no dependency record | Doyle, "A Truth Maintenance System," *Artificial Intelligence* 12(3), 1979; de Kleer, "An Assumption-Based TMS," *Artificial Intelligence* 28, 1986 | +| AGM belief revision | Revision and contraction obey minimal-change postulates over a selection function | Confidence adjustments (contradicted +0.15, merged ×0.7, reinforced ×0.85) are hand-tuned constants | Alchourrón, Gärdenfors, Makinson, "On the Logic of Theory Change," *J. Symbolic Logic* 50, 1985 | +| Provenance semirings | Provenance of a derived fact is a semiring expression over source tokens, composed under the query's + and × operators | `ProvenanceEntry` is a flat per-proposition list with no algebra for merge or abstraction | Green, Karvounarakis, Tannen, "Provenance Semirings," PODS 2007, DOI 10.1145/1265530.1265535 | +| Bitemporal databases | Valid time and transaction time as orthogonal axes, with defined as-of and point-in-time queries | `TemporalMetadata` carries `observedAt`/`validFrom`/`validTo`/`invalidatedAt` and no transaction-time axis (GAP-4B) | Snodgrass, *Developing Time-Oriented Database Applications in SQL*, Morgan Kaufmann, 1999 | +| Record linkage | Fellegi-Sunter match decisions from per-field m/u agreement probabilities against a likelihood-ratio threshold | Entity resolution uses fuzzy, vector, exact, partial and agentic searchers with LLM disambiguation | Fellegi, Sunter, "A Theory for Record Linkage," *JASA* 64, 1969, DOI 10.1080/01621459.1969.10501049 | +| ACT-R declarative memory | Base-level activation `B_i = ln(Σ_j t_j^-d)` folds recency and frequency into one retrieval score | `effectiveConfidence()` decays on recency alone; `reinforceCount` sits outside the decay maths — the same two signals a working-memory module needs to rank on | Anderson & Schooler, "Reflections of the Environment in Memory," *Psychological Science* 2, 1991 | +| Argumentation frameworks | Arguments plus an attack relation; admissible, preferred and grounded semantics decide which sets survive collectively | `ContradictionResolutionPass` is pairwise strongest-wins, with pinned propositions branched out into a review event | Dung, "On the Acceptability of Arguments...," *Artificial Intelligence* 77(2), 1995 | + +The three strongest borrowing opportunities, per brief 16: + +- **ATMS justification tracking, to make CONTRADICTED reversible.** The status flip today is driven + by a confidence comparison at classification time and records no reason for the loss. A + justification set per proposition lets retracting the evidence un-derive the dependent status, + without a fresh classification pass. +- **ACT-R base-level activation, to unify `reinforceCount` and decay.** Both signals exist and never + combine. The activation equation is a closed form for the ranking score a working-memory module + needs and for decay that counts frequency of use. +- **Semiring-formalised provenance composition.** Treating auto-merge as the union-like operator and + abstraction synthesis (which requires all its sources) as the product-like one gives one queryable + answer to "what composed this fact" across both, in place of per-pass list splicing. + +AGM and Fellegi-Sunter are weaker fits: AGM's postulates are about logical theories rather than +graded beliefs and are useful as a checklist, and LLM disambiguation already covers what +Fellegi-Sunter scoring would buy. The first two opportunities touch `Proposition.kt` and +`ContradictionResolutionPass.kt`, which consolidation passes, storage projection and retrieval +ranking all depend on. + ## Out of scope - **Zep's 5-reranker retrieval.** Deep feature tied to Neo4j traversal. Bi-temporal is the better @@ -366,5 +525,6 @@ scoped to it; DICE's substrate-level bet applies across backends. downloads, dependency management). - **Contract YAML as the authoring surface.** DICE's declared schema is a JVM type an application owns; a YAML dialect would be a second source of truth. -- **Write-time rejection of undeclared types.** It discards an extraction irrecoverably. +- **Write-time rejection of undeclared types.** It discards an extraction with no review path, and + re-extraction costs another LLM pass without reliably reproducing the same proposition. - **Managed hosting.** The embeddable library is the shipped distribution model. From 6987f887b9da8e5a0b3667808197e468708ab0ae Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Sun, 30 Aug 2026 21:55:25 -0400 Subject: [PATCH 4/6] docs(specs): recenter positioning on the proposition substrate Open with what DICE is; compare at the substrate level per system. Schema governance becomes one capability section corrected by a per-system fact-check: declaration is common, Graphiti validates against Pydantic models, Cognee flags non-conforming nodes, Neo4j GRAPH TYPE hard-rejects at write time as a general database feature. The claims that survive are narrower: no surveyed system versions its extraction schema or compares a declaration against the live store. Remove all delivery status and PR references. --- specs/competitive-positioning.md | 661 ++++++++++++------------------- 1 file changed, 259 insertions(+), 402 deletions(-) diff --git a/specs/competitive-positioning.md b/specs/competitive-positioning.md index 5d711788..cc90366b 100644 --- a/specs/competitive-positioning.md +++ b/specs/competitive-positioning.md @@ -1,168 +1,194 @@ # DICE competitive positioning -Where DICE sits against the agent-memory field (Zep/Graphiti, Mem0, Letta, Cognee, Hindsight, -LangMem, Neo4j Agent Memory, and the three hyperscaler memory services), and which vocabulary to -borrow from enterprise data-governance tooling. - -DICE differs from that field in one place: governance of the schema extraction runs against. No -surveyed competitor versions its extraction schema, compares a declaration against what a live -graph holds, or quarantines divergent records, and none has that work on a public roadmap. - -Surveyed mid-2026. Several competitors publish roadmap posts in the register of release notes, so -re-verify anything here before it goes into a bakeoff or a deck. - -## Delivery status - -On main: the proposition model with confidence and decay, multi-strategy entity resolution, source -provenance (`ProvenanceEntry`), collector decision traces (`CollectorTraceStore`), the -`ConflictDetector` SPI, `SchemaAdherence` (STRICT/DEFAULT/RELAXED), declared types via -`SchemaRegistry`/`DataDictionary`, integer re-indexing of proposition IDs across LLM calls (GAP-6), -and the pluggable store family. - -In the open PR train, unmerged as of this writing: - -| PR | Contents | -|---|---| -| #83 | `dice-metamodel` core: `MetamodelVersion` (content-addressed stamp), `GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `MetamodelVersionStore`, and `docs/design/metamodel-versioning.md` with the tier ladder | -| #84 | `DrivineMetamodelVersionStore`: Neo4j persistence, MERGE on natural key, per-schema monotonic sequence for write order | -| #85 | `ObservedSchema`/`ObservedSchemaSource`, `MetamodelDiff`/`MetamodelChange`, declared-vs-declared and declared-vs-observed differs | -| #86 | `DriftReport`/`DriftReportStore`, `DriftCheckRunner` (`dryRun` defaults true), quarantine contracts and `MentionTypeDriftQuarantinePolicy` | -| #87 | `DrivineDriftReportStore` and `DrivineObservedSchemaSource`, plus the end-to-end governance loop against a live Neo4j | -| #88 | `MetamodelAutoConfiguration`, activated by the presence of a `DeclaredSchemaSource` bean; `embabel.dice.metamodel.enabled` kill switch and `drift.mode` = `off` \| `observe` (default) \| `quarantine` | - -Not built: an extraction-run identifier linking a stored claim to the run that produced it (issue -#67), versioned and audited conflict policies, incident routing for `DriftReport`, a bi-temporal -fact model, and temporal anchoring of relative dates. - -The rest of this document describes the design across main and that train. This section is the -reference for where a given capability currently lives. - -## What each competitor has - -| System | Capabilities | -|---|---| -| **Zep / Graphiti** | Bi-temporal edges (`valid_from`/`valid_until` plus ingestion time), cross-session entity dedup, Leiden community clustering with summaries, custom entity types via Pydantic, five reranking strategies, SOC 2 on the managed offering. | -| **Mem0** | Single-pass ingestion (v3, April 2026, dropped the UPDATE/DELETE phases to cut latency), hybrid semantic + BM25 + entity-match retrieval, graph memory across Neo4j/Memgraph/Neptune/Kuzu, full SQLite change history, mentions counting. | -| **Letta (MemGPT)** | Agent-managed context with archival memory in Postgres/pgvector. The agent decides what to promote and how to resolve conflicts, via tool calls. No declared structure. | -| **Cognee** | LLM extraction into RDF triples with Pydantic validation, auto-generated ontology, custom-schema override. Validates shape per run; no stored version to compare a later run against. | -| **Hindsight** | Structured facts into a knowledge graph with entity resolution that links "Alice" to "my coworker Alice", and quality that compounds across sessions. | -| **LangMem** | Confidence-qualified, surprise-prioritised, SNR-shaped extraction prompts, gradient-style prompt optimisation, dilated-window retrieval. | -| **Google / AWS / Microsoft** | Fully managed, zero-infrastructure, IAM-scoped, framework-integrated. Flat fact strings underneath. | - -Two of these exceed DICE's equivalents: Zep's bi-temporal model against DICE's system timestamps, -and Cognee's Pydantic validation, which is a declared-shape check at write time. - -## Governance capabilities - -| Capability | Zep | Mem0 | Letta | Cognee | Hindsight | DICE | -|---|---|---|---|---|---|---| -| Schema versioning | — | — | — | Pydantic override, unversioned | — | Content-hashed `MetamodelVersion` (#83) | -| Type hierarchy / ontology | Implicit (clustering) | — | — | RDF/RDFS, auto-generated | — | Declared `DataDictionary`, per-type governed (main + #83) | -| Source provenance | Bi-temporal timestamps only | — | — | — | — | `ProvenanceEntry` edges, append-only (main) | -| Extraction-run trace | — | — | — | — | — | None (issue #67) | -| Collector decision audit | — | Full SQLite change history | — | — | — | `CollectorTraceStore` (main) | -| Contradiction handling | Temporal windows, manual | ADD-only | Manual, agent-driven | Unclear | Unclear | `ConflictDetector` SPI, both claims retained (main) | -| Versioned / audited conflict policy | — | — | — | — | — | None | -| Drift detection (declared vs observed) | — | — | — | — | — | `DeclaredObservedDiffer`, `DriftCheckRunner` (#85, #86) | -| Drift quarantine | — | — | — | — | — | `DriftQuarantinePolicy`, STALE marking (#86) | - -Caveats on that table. It is built from public documentation, so Zep and Cognee may have -unpublished lineage APIs. Mem0's change history records old value, new value, event and actor, -which is more than DICE records, and omits the schema the change was valid under. DICE's audit -trail is two separate mechanisms: `ProvenanceEntry` links a proposition to the source chunks it -came from, and `CollectorTraceStore` records why a collapse or merge decision was made. Neither -carries an extraction-run identifier, so there is no end-to-end "this claim came from that run -under that schema" trace (issue #67); the metamodel stamp is the piece that would complete it. - -## Metamodel identity: content-addressed - -Confluent Schema Registry and AWS Glue assign sequential version identity. Confluent uses a global -immutable integer schema ID plus a per-subject version number, Glue a per-schema version number -plus an opaque version ID. Registration is a stateful event: the same schema posted twice gets one -ID and one version because the registry remembers it. - -DICE derives identity from content. `MetamodelVersion.contentHash` is a SHA-256 over the structural -fields (sorted type names, per-type labels and property signatures, sorted relationship -descriptors) with every token length-prefixed, so `["a;b"]` cannot collide with `["a", "b"]`. Three -consequences: - -- **Registration is idempotent by construction.** `saveVersion` upserts on `(schemaName, - contentHash)`, and a re-save of identical content lands on the same row. There is no sequence - counter to get out of step across environments. -- **Schema is data.** A dev and a prod schema of identical shape hash identically and compare - directly. In the registries the same schema under a different subject gets a different version - number. -- **Stamps are portable.** A proposition carries `dice.metamodel.version` as metadata, so the - schema a claim was extracted under travels with the claim and needs no registry lookup. - -The second divergence is granularity. Both registries configure compatibility per subject: one mode -for the whole schema. DICE governs per type via `GovernedTypeSelector`, which lets one domain be -closed-world where it matters (`Person`, `Company`) and open-world elsewhere, so an exploratory type -the LLM invented leaves the version history unchanged. - -DICE adopts the registries' compatibility vocabulary: BACKWARD, FORWARD, FULL and the transitive -variants are well-understood terms for what a compatibility check means. Glue's *checkpoint*, a -selectable reference point to compare against, is more expressive than transitive yes/no and is the -right model if governed types get release windows. - -Both registries hard-reject an incompatible registration with HTTP 409 and no warn mode. DICE -accepts the write and records the divergence. Extraction is LLM-driven, and a type nobody declared -is often a real finding. A write-time rejection discards it with no review path; re-extraction -costs another LLM pass and does not reliably reproduce the same proposition. Quarantine keeps the -proposition out of reads and keeps it reviewable. - -## Opt-in and drift: prior art from migration tooling - -Patterns developers adopt: - -- **`liquibase diff` / `diff-changelog`** compares declared (reference) against observed (target) - and generates the changesets that would close the gap. A human decides whether to apply them or - mark them as run. -- **`flyway validate`** checksums applied migrations against local ones and fails loudly on - mismatch. It is bound to a build phase. -- **Hibernate `hbm2ddl.auto=validate`** checks only: fail on mismatch, change nothing. - `hbm2ddl.auto=none` is the shipped default, so no schema management happens unasked. -- **`@Version` per entity** makes versioning an annotation on the entities you chose. Nothing is - versioned by inference. - -The pattern developers warn against is `hbm2ddl.auto=update`: uncontrolled schema evolution at -startup, silently creating columns and leaving zombies behind when a `@Column(name=...)` changes. - -DICE follows the first set: - -- Versioning activates on an application-supplied `DeclaredSchemaSource` bean. With no declared - schema there is no versioning. -- Governance is per governed type via `GovernedTypeSelector`. -- The drift check reports the declared-vs-observed gap and leaves closing it to a human act, the - way Liquibase generates changesets and applies none of them. -- Quarantine marks affected propositions STALE and leaves them stored and recoverable. `drift.mode` - defaults to `observe`. - -The failure mode all three tools share is opt-in fatigue: they require explicit configuration, -teams ship with defaults, and later have no drift visibility. DICE's mitigation is a one-bean -switch and an activation condition stated in the first paragraph of every governance doc page. - -## Escalation vocabulary - -OpenMetadata, DataHub, Great Expectations, Soda and the Open Data Contract Standard have converged -on one vocabulary. DICE uses these terms with their existing meanings. - -| Term | Their meaning | DICE | -|---|---|---| -| **Drift** | Structural or statistical divergence over time | Declared metamodel versus what the graph holds | -| **Contract** | Declared structure + quality + ownership | `DeclaredSchema`: the governed types and their shape | -| **Assertion** | A comparison check fired on change | The declared-vs-observed comparison in the detect tier | -| **Policy** | Enforcement rule, configurable strictness | `SchemaAdherence` (STRICT/DEFAULT/RELAXED) and the `ConflictDetector` SPI | -| **Incident** | Raised on assertion failure, routed to an owner | `DriftReport`: the record a detection produces | -| **Quarantine** | Invalid records split off, the valid stream continues | Affected propositions marked STALE | +What DICE is, and where that puts it against the agent-memory field: Zep/Graphiti, Mem0, Letta, +Cognee, Hindsight, LangMem, Neo4j's agent-memory offerings, and the three hyperscaler memory +services. The last sections cover vocabulary and mechanisms DICE takes from enterprise +data-governance and database tooling. + +Surveyed mid-2026. Several of these systems publish roadmap posts in the register of release notes, +so re-verify anything here before it goes into a bakeoff or a deck. + +## What DICE is + +DICE is a proposition-first knowledge substrate. Heterogeneous artifacts — conversation, documents, +feeds, files — become durable propositions: confidence-weighted natural language statements with +typed entity mentions. Task-specific projections derive from them: a Neo4j graph, a Prolog fact +base, vector embeddings, agent working memory, reports. The proposition is the system of record. + +A proposition is an epistemic object. It carries: -Their escalation ladder is observe → alert → block. DICE's three tiers map onto it: +- **confidence** and **importance**, each a `ZeroToOne`; +- **decay** — effective confidence falls exponentially with age, anchored at + `max(contentRevised, lastAccessed)`, so an unused claim loses standing without being deleted + (`Proposition.kt:358-403`); +- **a status lifecycle** — ACTIVE, SUPERSEDED, CONTRADICTED, PROMOTED, STALE, with revival from + STALE on reinforcement (`Proposition.kt:31-54`); +- **grounding** — `DerivedFrom` edges to the source chunks the claim was extracted from; +- **provenance entries** — typed source locator, chunk, offsets and content hash, append-only + (`ProvenanceEntry.kt:18-52`); +- **reinforceCount** — how often the claim has been re-observed. -| DICE tier | Their stage | What happens | +The confidence-weighted proposition and the exponential decay formula come from general user +modeling: "Creating General User Models from Computer Use" (Shaikh, Sapkota, Rizvi, Horvitz, Park, +Yang, Bernstein; arXiv:2505.10831, UIST 2025, ACM DOI 10.1145/3746059.3747722), cited in the README +beside the decay maths. That is user-modeling research rather than agent memory. + +## The substrate compared + +Each system's unit of knowledge, and what that unit carries. + +| System | Unit of knowledge | What the unit carries | |---|---|---| -| Stamp and observe | Observe | Schema gets an identity; propositions carry the stamp | -| Detect and report | Alert | Compare declaration against declaration, and declaration against live graph; emit a `DriftReport` | -| Quarantine | Block (soft) | Mark affected propositions STALE. The valid set stays queryable; the suspect set is excluded and recoverable | +| **Zep / Graphiti** | Episodes ingested into a temporal knowledge graph of entity nodes and edges | Bi-temporal edges: `valid_from`/`valid_until` for world time plus ingestion time; entity and edge attributes typed by Pydantic models; cross-session entity dedup | +| **Mem0** | Extracted facts — short natural language strings | Full SQLite change history (old value, new value, event, actor), mention counts. No confidence | +| **Letta (MemGPT)** | Memory blocks in core context (label, value, limit, description) and archival passages in Postgres/pgvector | Free text. The agent decides promotion and conflict resolution through tool calls | +| **Cognee** | LLM-extracted nodes and RDF triples | An `ontology_valid` flag from matching against a declared OWL/RDF ontology. No confidence or decay | +| **Neo4j agent-memory (labs SDK, NAMS)** | Entity and fact nodes, plus `(Message)` nodes chained by `[:NEXT]` per session | POLE+O typing, `valid_from`/`valid_until`, geospatial attributes | +| **LangMem** | Semantic, episodic and procedural memory items | Optional typed Pydantic profiles; consolidation state | +| **Hindsight** | Structured facts in a knowledge graph | Entity resolution that links "Alice" to "my coworker Alice" | +| **Google / AWS / Microsoft** | Extracted memories as flat fact strings or items | A strategy or topic label and an IAM scope | +| **DICE** | Proposition | Confidence, importance, decay, status lifecycle, grounding, provenance entries, reinforce count | + +Three of these carry something DICE's proposition does not. + +- **Zep's bi-temporal model** separates when a fact was true in the world from when the system + learned it. DICE stores system timestamps only, so it cannot express that split. +- **Cognee's ontology grounding** resolves extracted entity and type names against a declared + OWL/RDF ontology, with fuzzy matching, before any graph node is built + (docs.cognee.ai/core-concepts/ontologies). DICE declares types in a `DataDictionary` and applies + them at extraction, without resolution against a stored ontology. +- **Mem0's change history** records old value, new value, event and actor per change. DICE's + `CollectorTraceStore` records why a collapse or merge decision was made and `ProvenanceEntry` + records where a claim came from; neither carries an extraction-run identifier, so there is no + end-to-end trace from a stored claim back to the run that produced it. + +## Projections + +DICE projects one proposition set into several representations: a Neo4j property graph and a Prolog +fact base (`dice-storage`), vector embeddings, agent working memory (`MemoryProjector`), rationale +and structured reports (`dice-report`), and lineage. Each derives from the propositions and can be +rebuilt from them; adding a representation means adding a projector, and the propositions are +unchanged. + +Where the surveyed systems put their knowledge: + +- **Zep / Graphiti** — a Neo4j property graph, with Leiden community clustering and summaries over + it; retrieval is cosine plus BM25 plus BFS with five reranking strategies. +- **Mem0** — a vector store, plus a separate graph pipeline over Neo4j, Memgraph, Neptune or Kuzu + fed by its own extraction path. +- **Letta** — pgvector passages, retrieved by the agent through tool calls. +- **Cognee** — an RDF-grounded graph plus embeddings; search returns nodes. +- **Neo4j agent-memory** — a native property graph; retrieval is hybrid vector plus up to 3-hop + traversal. +- **LangMem** — embedding space, with dilated-window retrieval. +- **Google / AWS / Microsoft** — similarity search over stored memories, with no exposed + intermediate model. + +For each of those, the store is the knowledge model, so a new representation means a new extraction +path or a migration. Mem0's separate graph pipeline is the field's closest analogue to a second +projection, and it re-extracts rather than deriving from the facts already stored. + +## Knowledge hygiene + +DICE treats hygiene as three interventions at three moments — what is let in, what is reclaimed, +what is consolidated between sessions (`docs/design/knowledge-hygiene.md`). + +- **Admission.** Gates run at extraction: confidence qualification, deduplication against canonical + and stored propositions, conflict classification, trust scoring. The reason to gate at admission + rather than clean up afterwards is cost and clarity. +- **Reclamation.** `DecaySweepPass` retires softly to STALE and never hard-deletes, with + dual-threshold hysteresis — stale below 0.1 effective confidence, recovery edge at 0.25. Revival + is never sweep-driven; it happens on reinforcement. +- **Consolidation.** Between-session passes: `ContradictionResolutionPass` retires the weaker of a + contradictory pair to CONTRADICTED, auto-merge collapses duplicates, abstraction synthesises + higher-level propositions from groups. + +Background consolidation is convergent. Bedrock AgentCore runs extraction then consolidation as +background processes, with start and completion marks and success counts in the logs +(https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html). +Vertex AI Memory Bank extracts asynchronously, ingested via `add_memory()` or an end-of-conversation +callback +(https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, +2025-07-08). LangMem offers hot-path tools or background managers +(https://rywalker.com/research/langmem). Neo4j runs background enrichment and a multi-stage +extraction cascade (https://neo4j.com/labs/agent-memory/). No surveyed system documents a hard TTL +delete: decay, supersession and consolidation are the industry's answer as well as DICE's. + +Two hygiene mechanisms are absent from the surveyed field: a lifecycle status on the stored unit, +and a retirement path that keeps the retired unit. Vertex AI Memory Bank deletes the superseded +memory on contradiction; Microsoft Foundry discards the old value; Bedrock AgentCore writes a new +entry with no contradiction detection; Mem0 v3 (April 2026) is ADD-only, so supersession and +contradiction are inexpressible in the model. Cognee keeps its non-conforming nodes, but they are +"stored, embedded, and returned by search exactly like grounded ones" +(docs.cognee.ai/core-concepts/ontologies), so the flag has no effect on reads. + +## Schema governance + +Declaring a schema for extraction is common in this field, and what happens to the declaration +varies. + +- **Graphiti** takes Pydantic entity and edge models and a `set_ontology` API with a + `strict_ontology` flag, and validates: "Each entity is validated against the appropriate Pydantic + model" before graph construction + (https://help.getzep.com/graphiti/core-concepts/custom-entity-and-edge-types). +- **Cognee** takes an OWL/RDF ontology file and checks extracted entities and types against it + before any graph nodes are built, then keeps what does not match, tagged `ontology_valid=False`: + "Nothing is rejected or discarded" (https://docs.cognee.ai/core-concepts/ontologies). +- **Neo4j's labs agent-memory SDK and NAMS** take a `DomainSchema` of entity types and descriptions. + It steers GLiNER extraction; no validation, versioning or rejection is documented + (https://neo4j.com/labs/agent-memory/faq/). +- **Bedrock AgentCore** gives full output-schema control only in the self-managed strategy; + built-in and overridden strategies "do not let you change the final output schema" + (https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory-custom-strategy.html). +- **Vertex AI Memory Bank's "custom topics"** and **Mem0's "custom categories"** are a label plus + free-text instructions inserted into the extraction prompt, with few-shot examples recommended — + prompt text rather than enforced types + (https://cloud.google.com/agent-builder/agent-engine/memory-bank/generate-memories; + https://docs.mem0.ai/open-source/features/custom-fact-extraction-prompt). +- **Letta** has no typed extraction schema: memory blocks are free-text segments. + +The strongest enforcement in reach is a general-purpose database feature. Neo4j **GRAPH TYPE**, a +Cypher 25 preview in Neo4j 2026.02 across Enterprise Edition, Infinigraph Edition and all Aura +tiers, declares nodes, labels and relationship connections with `SET`/`ADD`/`ALTER`/`DROP`, and +hard-rejects a non-conforming write at write time +(https://neo4j.com/blog/developer/graph-type-schema-enforcement-made-easy-preview/). GRAPH TYPE is +not part of Neo4j's agent-memory product and is not wired into it; `DomainSchema` there remains +unenforced extraction guidance. `SHOW CURRENT GRAPH TYPE` returns the declaration, with no +comparison against what the graph holds. + +Two capabilities are absent from every system surveyed. Neither gives an extraction schema a version +identity or history — Graphiti's docs describe evolution by adding attributes with no version +numbers, and GRAPH TYPE's `ADD`/`ALTER`/`DROP` are an evolution mechanism with no discrete version +record. And none compares a declaration against what a live store actually holds. No surveyed vendor +has published roadmap intent for either. + +DICE's design puts both above the store: + +- **Content-addressed version identity.** SHA-256 over the structural fields — sorted type names, + per-type labels and property signatures, sorted relationship descriptors — with every token + length-prefixed, so `["a;b"]` cannot collide with `["a", "b"]`. Registration is idempotent by + construction, two environments holding the same shape hash identically, and a proposition carries + the stamp as metadata, so the schema a claim was extracted under travels with the claim. + Confluent Schema Registry and AWS Glue assign sequential identity per subject instead, which makes + registration a stateful event and cross-environment comparison a lookup. +- **Per-type governance.** A governed-type selector decides which types are closed-world, so one + domain can be governed (`Person`, `Company`) while an exploratory type the LLM invented leaves the + version history unchanged. Both registries configure compatibility per subject: one mode for the + whole schema. +- **Drift detection.** Declaration against declaration, and declaration against the live graph, + producing a drift report. +- **Quarantine as the disposition.** Affected propositions are marked STALE — excluded from reads, + stored, recoverable. + +The version stamp, drift detection and quarantine are design-stage; what runs in the extraction path +today is the declared type dictionary (`SchemaRegistry`/`DataDictionary`) and the `SchemaAdherence` +policy (STRICT/DEFAULT/RELAXED). + +DICE chose quarantine as the disposition for extracted data. GRAPH TYPE shows write-time rejection +is a real design, and a workable one for writes an application authors. DICE's writes come from an +LLM, and a type nobody declared is often a real finding: rejection discards it with no review path, +and re-extraction costs another LLM pass without reliably reproducing the same proposition. +Quarantine keeps the proposition out of reads and keeps it reviewable. ```mermaid flowchart LR @@ -171,193 +197,43 @@ flowchart LR Q -->|yes| ST["stamped with contentHash"] ST --> D{"declared vs observed"} D -->|conforms| VALID["valid set — queryable"] - D -->|"diverges"| DR["DriftReport"] + D -->|"diverges"| DR["drift report"] DR --> QU["quarantined — marked stale,
excluded from reads, reversible"] QU -.->|"human review"| VALID ``` -Quarantine partitions the write stream into a valid and a suspect set, and the pipeline continues, -which is the hybrid default the Spark and Soda world settled on. Fail-fast is the escalation for -when quality drops below a threshold. - -Two decisions this vocabulary forces, both open: - -- **Is quarantine a metadata flag or physical routing?** Enterprise tools do both. DICE's - STALE-marking is a flag; whether reads exclude it by default is the open question. -- **Do we route incidents?** Their incident model assumes metadata ownership drives notification. - DICE emits events and has no owner model, so a `DriftReport` has no routing destination today. - -## Graph-native convergence: Neo4j GRAPH TYPE - -Neo4j's GRAPH TYPE (preview, 2025) declares a schema once in Cypher DDL and enforces it at write -time, with SET/ADD/ALTER/DROP lifecycle commands and an Open variant that requires declared fields -while tolerating extras. TypeDB is stricter: nothing that fails to conform can be written, and -`redefine` checks existing instances before a migration commits. Store vendors are independently -concluding that a property graph needs a declared schema coupled to enforcement. The divergence is -where the governance sits. - -DICE governs above the store: - -- **Per context.** Governance scopes to a `contextId`, so one deployment can hold a governed tenant - and an exploratory one. GRAPH TYPE is a property of the graph. -- **Open-world by default.** GRAPH TYPE and TypeDB reject undeclared types. DICE stores them as - findings, and per-type governance is what keeps that safe. `SchemaAdherence` already takes this - side of the trade at extraction time. -- **Quarantine on divergence.** Write-time enforcement loses a non-conforming extraction. DICE - stores it, flags it, and leaves it reviewable. -- **Backend-independent.** DICE's schema lives in `SchemaRegistry`/`DataDictionary`, outside the - database, so the same governance applies across in-memory, Neo4j, or a later backend. Adopting - GRAPH TYPE would move it into the store. - -The cost of governing above the store is that the schema is application-enforced, so it can drift -if some entry point skips validation. Neo4j's is enforced by the database. - -SHACL validation reports are the prior art for `DriftReport`, and the only standardised -violation-report format in this space: a conformance flag at the root, one result node per -violation, each carrying the focus node, the failed constraint, severity, and a human-readable -message. `DriftReport` follows that structure. SHACL serialises as RDF graphs, which -`DriftReport` does not adopt, so its consumers need no RDF parser. PG-Schema and ProGS are the -property-graph analogues and are worth watching; neither has a canonical report format to borrow. - -## Adopted, rejected, unique - -| | Adopt | Reject | Unique to DICE | -|---|---|---|---| -| **Schema registries** (Confluent, Glue) | BACKWARD/FORWARD/FULL vocabulary; Glue's checkpoint; idempotent registration | Sequential version IDs; flat per-subject compatibility; hard-reject on incompatible registration | Content-addressed identity; per-type governed selection; the stamp travelling with the data | -| **Migration & ORM** (Liquibase, Flyway, Hibernate) | Declared-vs-observed diff that reports; validate-mode; per-entity `@Version` opt-in | `hbm2ddl.auto=update` auto-mutation; anything that changes a schema at startup | Drift detection over LLM-extracted knowledge, where the observed side is a graph nobody wrote by hand | -| **Data governance** (DataHub, Soda, ODCS) | drift / contract / assertion / policy / incident / quarantine, verbatim; observe→alert→block; valid/invalid split | Blocking at the producer boundary by default; contract YAML as the primary authoring surface | Governance over probabilistic extraction: confidence-weighted claims | -| **Graph schema** (Neo4j GRAPH TYPE, TypeDB, SHACL) | SHACL's validation-report structure; GRAPH TYPE's Open variant semantics | Write-time rejection; RDF as a wire format; store-coupled schema declaration | Governance that is per context, tolerates open-world types, and is backend-independent | -| **Agent memory** (Zep, Mem0, Cognee, LangMem) | Zep's bi-temporal model (GAP-4B), ahead of DICE's system timestamps; Mem0's change-history record of old value / new value / event / actor; Cognee's declared-shape validation | ADD-only ingestion; hard delete on contradiction; opaque LLM consolidation | Source provenance, collector traces, governed metamodel versioning, drift quarantine, versioned conflict policy | - -## Per-competitor detail - -### vs Zep/Graphiti - -| Dimension | DICE | Zep | -|---|---|---| -| Ingestion | Batch classify + auto-merge + canonical dedup | Sequential only ("must be awaited") | -| Classification | 5-way with edge cases + few-shot | Duplicate vs contradicted (binary) | -| Confidence model | Exponential decay + outcome-dependent adjustment + reinforceCount | No decay, no confidence scoring | -| Extraction quality | SNR, confidence-qualified, role-aware, schema-bound | Custom entity types via Pydantic, entity validation | -| Temporal model | System timestamps only (created/revised) | Bi-temporal (valid_at/invalid_at/expired_at) | -| Graph structure | Propositions + entity mentions, no graph DB required | Full knowledge graph in Neo4j with community detection | -| Retrieval | Vector similarity + canonical match | Cosine + BM25 + BFS + 5 rerankers | -| Schema governance | Content-hashed metamodel versions, per-type opt-in | Entity types are code, unversioned | -| Infrastructure | Embeddable, JVM-native, no external deps | Neo4j + embedding service + LLM | - -Zep is Python and Go only. Its audit story is SOC 2 on the managed service, an org-level control -with no data-lineage API. The temporal difference is GAP-4B. - -### vs Mem0 - -| Dimension | DICE | Mem0 | -|---|---|---| -| Classification | 5-way taxonomy with edge-case guidance | v3 is ADD-only; supersession and contradiction are inexpressible | -| Dedup pipeline | Canonical + auto-merge + batch LLM | Entity linking at retrieval time | -| Ingestion latency | Batch LLM call per chunk | Single-pass, deliberately minimal | -| Confidence model | Decay + outcome adjustment + qualification at extraction | None | -| ID safety | Integer re-indexing prevents hallucination | Integer re-indexing prevents hallucination | -| Graph memory | Entity mentions + Neo4j projection | Neo4j/Memgraph/Neptune/Kuzu | -| Audit trail | Source provenance + collector decision traces, no run-level trace | Full SQLite history (old/new/event/actor) | -| Schema governance | Governed metamodel versions | None | - -Mem0 also has vision and procedural memory for agent traces, and the lowest ingestion latency in -the field. Its v3 traded expressiveness for speed: ADD-only means contradictions accumulate -unresolved with no supersession semantics. Its change history records what changed and omits the -schema the change was valid under. - -### vs Cognee - -| Dimension | DICE | Cognee | -|---|---|---| -| Declared shape | `DataDictionary` + `SchemaAdherence` (STRICT/DEFAULT/RELAXED) | Pydantic models, optionally overriding an auto-generated ontology | -| Schema versioning | Content-hashed, stored, comparable | Validation is per-run | -| Ontology | Declared types and relationships | RDF/RDFS triples, auto-generated from the corpus | -| Validation failure handling | Adherence policy; non-conforming extraction retained and flaggable | Pydantic rejects the shape; no error feedback loop | -| Provenance | `ProvenanceEntry` edges to source chunks | Not documented | - -Cognee is the only surveyed competitor with a declared-schema check at write time, and its -auto-generated ontology lowers setup cost. Changing the Pydantic model tells you nothing about -which previously-stored data no longer conforms, which is what the detect tier addresses. - -### vs Letta (MemGPT) and Hindsight - -Letta has no schema: core memory managed by the LLM, archival memory as pgvector passages, -conflicts resolved by the agent through tool calls, consolidation agent-driven. There is no -governance surface to compare against. - -Hindsight structures facts into a graph with entity resolution good enough to link "Alice" to "my -coworker Alice", and improves across sessions. Smaller product surface than the incumbents, with no -documented temporal invalidation or contradiction framework. The entity-resolution quality is the -part worth re-checking. - -### vs the managed services (Google, AWS, Microsoft) - -All three share a shape: a flat fact string, opaque LLM consolidation, hard delete on -contradiction, no entity model, no confidence, no provenance, vendor lock-in, zero infrastructure, -native IAM. - -| Dimension | DICE | Google Memory Bank | AWS AgentCore | Microsoft Foundry | -|---|---|---|---|---| -| Data model | Structured `Proposition` | Flat `fact` string | Flat `{"fact": "..."}` | Flat memory "items" | -| Memory types | `KnowledgeType` classifier | Managed + custom topics | Strategy-scoped | Profile + chat summary only | -| Confidence/decay | Decay + outcome adjustment | None | None | None | -| Entity resolution | Multi-strategy + LLM disambiguation | None | None | None | -| Contradiction | Both retained, reduced confidence | Old deleted | New entry, no detection | Old value discarded | -| Provenance | `ProvenanceEntry` to source chunks | None | None | None | -| Governance | Metamodel versioning, drift checking | None | None | None | -| Hosting | Self-hosted | Fully managed | Fully managed | Fully managed | -| Scale limits | Application-determined | Not published | Not published | 100 scopes, 10K memories/scope | - -The open question these raise is whether DICE eventually wants a managed offering. - -### vs Neo4j Agent Memory - -| Dimension | DICE | Neo4j Agent Memory | -|---|---|---| -| Classification | 5-way with edge cases + few-shot | No taxonomy; entity resolution handles dedup | -| Batch processing | N propositions in 1 LLM call | Sequential cascade stages | -| Confidence/decay | Decay + outcome adjustment + reinforceCount | None | -| Contradiction | Both retained with reduced confidence | Merged or left distinct | -| Extraction pipeline | Single LLM call, SNR-maximising | spaCy → GLiNER → LLM cascade | -| Graph structure | Propositions + mentions, Neo4j as projection | Native graph with POLE+O ontology | -| Temporal model | System timestamps only | valid_from/valid_until + geospatial | -| Retrieval | Vector + canonical + entity + composable query | Hybrid vector + up to 3-hop traversal | -| Schema enforcement | Application-layer, per context, with quarantine | Store-level GRAPH TYPE enforcement | -| Infrastructure | Embeddable, no external deps | Neo4j 5.11+ plus spaCy/GLiNER models | - -The division is proposition-centric against entity-centric: DICE manages the lifecycle of claims, -Neo4j builds a graph of entities. Their store-level schema bet is enforced by the database and -scoped to it; DICE's substrate-level bet applies across backends. +Three sets of prior art shape the surface. Drift, contract, assertion, policy, incident and +quarantine are used with the meanings OpenMetadata, DataHub, Great Expectations, Soda and the Open +Data Contract Standard converged on, and their observe → alert → block ladder maps onto stamp-and- +observe, detect-and-report, quarantine. Drift reports follow SHACL's validation-report structure — a +conformance flag at the root, one result per violation carrying focus node, failed constraint, +severity and message — without SHACL's RDF serialisation, so consumers need no RDF parser. The +opt-in model comes from migration tooling: versioning activates on an application-supplied declared +schema, the drift check reports and leaves closing the gap to a human act the way `liquibase diff` +generates changesets and applies none, and nothing mutates a schema at startup the way +`hbm2ddl.auto=update` does. Those tools share one failure mode, opt-in fatigue: teams ship with +defaults and later have no drift visibility. + +Two questions the design leaves open: whether reads exclude quarantined propositions by default, and +where a drift report goes — DICE emits events and has no owner model, so there is no routing +destination. ## Convergent mechanisms -Places where DICE and a surveyed system reached the same solution to the same problem. They are -tracked here so the gaps section stays about capabilities DICE lacks. +Places where DICE and a surveyed system reached the same solution to the same problem, beyond the +decay-and-consolidation convergence covered above. - **LLM extraction into a structured store.** All four surveyed memory services take a conversation turn, run an LLM extraction, and persist the result into a vector, graph or relational store - (brief 15, from https://rywalker.com/research/langmem, - https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview - 2025-07-08, https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html, + (https://rywalker.com/research/langmem; + https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, + 2025-07-08; https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html; https://neo4j.com/labs/agent-memory/). DICE's pipeline has the same shape with the proposition as the stored unit. -- **Decay and consolidation ahead of hard deletion.** DICE decays effective confidence over time - and retires softly to STALE (`DecaySweepPass`, dual-threshold hysteresis 0.1/0.25). AWS and - Google merge or supersede through consolidation, Neo4j tracks temporal fact validity, LangMem - consolidates to suppress duplication. No surveyed system documents a hard TTL delete. -- **Integer re-indexing of proposition IDs across LLM calls.** DICE (GAP-6, on main) and Mem0 both - renumber IDs per call so the model cannot invent one. -- **Confidence-qualified, SNR-shaped extraction prompts.** DICE and LangMem settled on the same +- **Integer re-indexing of proposition IDs across LLM calls.** DICE and Mem0 both renumber IDs per + call so the model cannot invent one. +- **Confidence-qualified, SNR-shaped extraction prompts.** DICE and LangMem arrived at the same prompt controls independently (https://rywalker.com/research/langmem). -- **A declared schema coupled to enforcement.** Neo4j GRAPH TYPE, TypeDB and DICE's metamodel work - are three independent arrivals at the same conclusion; the divergence is where the governance - sits, covered above. - -Where a converged mechanism is implemented better elsewhere: Zep's bi-temporal edges against DICE's -system timestamps, Mem0's change history (old value, new value, event, actor) against DICE's -collector traces, and Cognee's Pydantic check as a write-time declared-shape validation. Each is -cited in the section that covers that competitor. ## Remaining gaps @@ -373,30 +249,20 @@ RetrieveMemoryRecords, plus extraction and consolidation logs Vertex AI Memory Bank lists stored memories in the Cloud Console (Memory Bank UI announcement, Google Developer Forums; brief 15 records the citation without a URL). Neo4j's NAMS dashboard shows health, entity count and queue lag with interactive graph visualisation and Cypher queryability -(https://medium.com/neo4j/a-tour-of-the-neo4j-agent-memory-service-nams-0f2d535a4fdb, 2026-06; -brief 15 lists GA status as unconfirmed). LangSmith traces execution paths and state transitions -with cost/latency/error dashboards and human-in-the-loop inspection -(langchain.com/resources/llm-observability-tools, cited by brief 15). In DICE, "why was this -proposition formed" needs custom logging over `CollectorTraceStore`. - -**2. Background memory formation and consolidation pipelines.** All four run formation off the -request path. AWS runs extraction then consolidation as background processes with start/completion -and success counts in the logs (observability-memory-metrics.html). Google's extraction is -asynchronous, ingested via `add_memory()` or an end-of-conversation callback -(https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, -2025-07-08). LangMem offers hot-path tools or background managers -(https://rywalker.com/research/langmem). Neo4j runs background enrichment (Wikipedia, Diffbot) and -a multi-stage extraction cascade (https://neo4j.com/labs/agent-memory/). DICE has an async -`@EventListener` on conversation analysis and a non-blocking `PropositionIncrementalAnalyzer`, with -consolidation available as passes an application invokes. The managed pipeline that schedules them -is missing. - -**3. TTL and eviction controls.** Decay itself is convergent (above); the gap is a retention policy -an operator can set, where DICE's decay constant and sweep thresholds are code-level configuration -(README:103-108). No surveyed system publishes a TTL API either: AWS and Google document no -expiration policy, Neo4j relies on temporal validity, LangMem on consolidation. Brief 15 lists the -AWS TTL knob and the Google retention policy as unverified, so this item rests on weaker evidence -than the two above it. +(https://medium.com/neo4j/a-tour-of-the-neo4j-agent-memory-service-nams-0f2d535a4fdb, 2026-06; GA +status unconfirmed). LangSmith traces execution paths and state transitions with cost, latency and +error dashboards and human-in-the-loop inspection (langchain.com/resources/llm-observability-tools). +In DICE, "why was this proposition formed" needs custom logging over `CollectorTraceStore`. + +**2. Background memory formation pipelines.** All four run formation off the request path, cited +under knowledge hygiene above. DICE has an async `@EventListener` on conversation analysis and a +non-blocking `PropositionIncrementalAnalyzer`, with consolidation available as passes an application +invokes. The managed pipeline that schedules them is missing. + +**3. TTL and eviction controls.** The gap is a retention policy an operator can set; DICE's decay +constant and sweep thresholds are code-level configuration (README:103-108). No surveyed system +publishes a TTL API either, so this item rests on weaker evidence than the two above it (brief 15 +lists the AWS TTL knob and Google's retention policy as unverified). **4. Procedural memory.** Two of the four ship it. LangMem has a `procedural` memory type where agents update their own prompt rules from feedback (https://rywalker.com/research/langmem). Neo4j @@ -419,34 +285,30 @@ personalises long-term memories per user (https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, 2025-07-08); Neo4j persists preference and fact entities in a shared graph (https://neo4j.com/labs/agent-memory/); LangMem keeps preferences in semantic memory with optional -typed Pydantic profiles (https://rywalker.com/research/langmem). DICE's `ContextId` scopes -knowledge and defines no profile structure or cross-session aggregation. - -Gaps DICE already tracks: +typed Pydantic profiles (https://rywalker.com/research/langmem). DICE's `ContextId` scopes knowledge +and defines no profile structure or cross-session aggregation. -- No extraction-run identifier, so a stored claim cannot be traced to the run and schema that - produced it (issue #67). -- No bi-temporal model (GAP-4B): system timestamps only, so no point-in-time query and no temporal - contradiction resolution. -- No temporal anchoring (GAP-4A): relative dates are stored as literal text. -- No surprise-prioritised retention (GAP-2): novel facts get no durability preference. -- No incident routing for `DriftReport`. DICE emits events and has no owner model. -- Conflict policy has no versioning or audit trail. +Also missing, and not held by any surveyed system either: an extraction-run identifier linking a +stored claim to the run and schema that produced it; a bi-temporal fact model, so no point-in-time +query and no temporal contradiction resolution; temporal anchoring, so relative dates are stored as +literal text; surprise-prioritised retention, so novel facts get no durability preference; and +versioning or an audit trail on conflict policy. ## Candidate modules -The capabilities the managed services hold over DICE sit above the proposition substrate and -consume it, so each is a module or add-on rather than a change to the substrate. +The capabilities the managed services hold over DICE sit above the proposition substrate and consume +it, so each is a module or add-on rather than a change to the substrate. **Governed working memory.** ArcMem is an activation-ranked working-memory system built on DICE -APIs, and is the existing proof that this layer works over the substrate. It keeps a bounded active -set (20 units plus a token budget) in a protected prompt region, ranks units by an activation score blending recency, -reinforcement and decay, protects units by authority level (PROVISIONAL → UNRELIABLE → RELIABLE → -CANON) from automatic demotion, reinforces on access by anchoring decay at `max(contentRevised, -lastAccessed)`, and evicts into durable storage so evicted material stays queryable. Its gate order -is confidence → deduplication → conflict → trust → promotion → budget enforcement, and changing the -order changes the semantics. It consumes DICE's storage, extraction, decay and revision as -primitives (brief 17). A DICE-native module needs five SPIs ArcMem hand-rolls today: +APIs, and the existing proof that this layer works over the substrate. It keeps a bounded active set +(20 units plus a token budget) in a protected prompt region, ranks units by an activation score +blending recency, reinforcement and decay, protects units by authority level (PROVISIONAL → +UNRELIABLE → RELIABLE → CANON) from automatic demotion, reinforces on access by anchoring decay at +`max(contentRevised, lastAccessed)`, and evicts into durable storage so evicted material stays +queryable. Its gate order is confidence → deduplication → conflict → trust → promotion → budget +enforcement, and changing the order changes the semantics. It consumes DICE's storage, extraction, +decay and revision as primitives (brief 17). A DICE-native module needs five SPIs ArcMem hand-rolls +today: - activation ranking as a pluggable policy — ArcMem uses hand-tuned parameters and a fixed rank clamp of [100, 900]; @@ -456,11 +318,11 @@ primitives (brief 17). A DICE-native module needs five SPIs ArcMem hand-rolls to - batched or concurrent extraction — `PropositionPipeline.process()` is serial and consumers batch by hand. -Which of these belong in DICE and which in the consuming application is unsettled (brief 17, -Risks), and entity resolution has to stay serial where shared identity is involved. +Which of these belong in DICE and which in the consuming application is unsettled (brief 17, Risks), +and entity resolution has to stay serial where shared identity is involved. -**Session memory management.** In DICE today: `MemoryProjector` classifies propositions by -knowledge type for prompt injection +**Session memory management.** In DICE today: `MemoryProjector` classifies propositions by knowledge +type for prompt injection (`dice/src/main/kotlin/com/embabel/dice/projection/memory/MemoryProjector.kt:46`), and the `Memory` tool runs hybrid vector plus keyword retrieval over context-scoped propositions (`dice/src/main/kotlin/com/embabel/dice/agent/Memory.kt:112`). A module would add session-scoped @@ -476,19 +338,15 @@ would add the formation path — turning agent feedback into stored rules that l ## Orthogonal research -DICE's proposition extraction sits in the lineage of general user modeling: "Creating General User -Models from Computer Use" (Shaikh, Sapkota, Rizvi, Horvitz, Park, Yang, Bernstein; arXiv:2505.10831, -UIST 2025, ACM DOI 10.1145/3746059.3747722), cited in the README for the confidence-weighted -proposition and the exponential decay formula. That is user-modeling research rather than agent -memory. Scanning adjacent non-LLM fields for mechanisms is standing practice here, and these map -onto problems DICE already has: +Scanning adjacent non-LLM fields for mechanisms is standing practice here, on the same footing as +the GUM lineage above. These map onto problems DICE already has. | Field | Mechanism | DICE analog | Citation | |---|---|---|---| | Truth maintenance (TMS/ATMS) | Justifications record why a belief holds; retracting a premise un-derives its dependents; ATMS labels a node with the assumption sets that support it | `ContradictionResolutionPass` retires the weaker of a contradictory pair to CONTRADICTED by comparing `effectiveConfidence()`, with no dependency record | Doyle, "A Truth Maintenance System," *Artificial Intelligence* 12(3), 1979; de Kleer, "An Assumption-Based TMS," *Artificial Intelligence* 28, 1986 | | AGM belief revision | Revision and contraction obey minimal-change postulates over a selection function | Confidence adjustments (contradicted +0.15, merged ×0.7, reinforced ×0.85) are hand-tuned constants | Alchourrón, Gärdenfors, Makinson, "On the Logic of Theory Change," *J. Symbolic Logic* 50, 1985 | | Provenance semirings | Provenance of a derived fact is a semiring expression over source tokens, composed under the query's + and × operators | `ProvenanceEntry` is a flat per-proposition list with no algebra for merge or abstraction | Green, Karvounarakis, Tannen, "Provenance Semirings," PODS 2007, DOI 10.1145/1265530.1265535 | -| Bitemporal databases | Valid time and transaction time as orthogonal axes, with defined as-of and point-in-time queries | `TemporalMetadata` carries `observedAt`/`validFrom`/`validTo`/`invalidatedAt` and no transaction-time axis (GAP-4B) | Snodgrass, *Developing Time-Oriented Database Applications in SQL*, Morgan Kaufmann, 1999 | +| Bitemporal databases | Valid time and transaction time as orthogonal axes, with defined as-of and point-in-time queries | `TemporalMetadata` carries `observedAt`/`validFrom`/`validTo`/`invalidatedAt` and no transaction-time axis | Snodgrass, *Developing Time-Oriented Database Applications in SQL*, Morgan Kaufmann, 1999 | | Record linkage | Fellegi-Sunter match decisions from per-field m/u agreement probabilities against a likelihood-ratio threshold | Entity resolution uses fuzzy, vector, exact, partial and agentic searchers with LLM disambiguation | Fellegi, Sunter, "A Theory for Record Linkage," *JASA* 64, 1969, DOI 10.1080/01621459.1969.10501049 | | ACT-R declarative memory | Base-level activation `B_i = ln(Σ_j t_j^-d)` folds recency and frequency into one retrieval score | `effectiveConfidence()` decays on recency alone; `reinforceCount` sits outside the decay maths — the same two signals a working-memory module needs to rank on | Anderson & Schooler, "Reflections of the Environment in Memory," *Psychological Science* 2, 1991 | | Argumentation frameworks | Arguments plus an attack relation; admissible, preferred and grounded semantics decide which sets survive collectively | `ContradictionResolutionPass` is pairwise strongest-wins, with pinned propositions branched out into a review event | Dung, "On the Acceptability of Arguments...," *Artificial Intelligence* 77(2), 1995 | @@ -506,25 +364,24 @@ The three strongest borrowing opportunities, per brief 16: abstraction synthesis (which requires all its sources) as the product-like one gives one queryable answer to "what composed this fact" across both, in place of per-pass list splicing. -AGM and Fellegi-Sunter are weaker fits: AGM's postulates are about logical theories rather than -graded beliefs and are useful as a checklist, and LLM disambiguation already covers what -Fellegi-Sunter scoring would buy. The first two opportunities touch `Proposition.kt` and +AGM and Fellegi-Sunter are weaker fits: AGM's postulates cover logical theories rather than graded +beliefs and are useful as a checklist, and LLM disambiguation already covers what Fellegi-Sunter +scoring would buy. The first two opportunities touch `Proposition.kt` and `ContradictionResolutionPass.kt`, which consolidation passes, storage projection and retrieval ranking all depend on. ## Out of scope -- **Zep's 5-reranker retrieval.** Deep feature tied to Neo4j traversal. Bi-temporal is the better - investment. +- **Zep's five-reranker retrieval.** Deep feature tied to Neo4j traversal; the bi-temporal model is + the better investment from that system. - **LangMem's prompt optimisation.** Orthogonal to memory quality. -- **Mem0's separate graph pipeline.** DICE has entity mentions plus Neo4j projection. +- **Mem0's separate graph pipeline.** DICE has entity mentions plus a Neo4j projection over the same + propositions. - **Google's multimodal extraction.** The proposition model is format-agnostic; add on demand. - **AWS's episodic reflection.** The abstraction pipeline already synthesises across propositions. - **Neo4j's POLE+O ontology.** Domain-specific subtypes; the proposition model is domain-agnostic. -- **Neo4j's spaCy → GLiNER → LLM cascade.** Cost-effective and operationally heavy (model - downloads, dependency management). +- **Neo4j's spaCy → GLiNER → LLM cascade.** Cost-effective and operationally heavy: model downloads + and dependency management. - **Contract YAML as the authoring surface.** DICE's declared schema is a JVM type an application owns; a YAML dialect would be a second source of truth. -- **Write-time rejection of undeclared types.** It discards an extraction with no review path, and - re-extraction costs another LLM pass without reliably reproducing the same proposition. -- **Managed hosting.** The embeddable library is the shipped distribution model. +- **Managed hosting.** The embeddable library is the distribution model. From 43b3bc88b02fc3b12939c08791fbc6099e478745 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Wed, 2 Sep 2026 16:33:14 -0400 Subject: [PATCH 5/6] Rewrite the two specs in the house voice competitive-positioning.md derives from what main states, drops the material that only exists on unmerged branches, and corrects four claims that were wrong against the code. developer-docs-architecture.md becomes readme-to-docs-tree.md: the docs/ layout, which README section lands on which page, the concept and feature page shapes, the quickstart, the compatibility matrix, and the docs rule for feature PRs. --- specs/competitive-positioning.md | 500 ++++++++++----------------- specs/developer-docs-architecture.md | 292 ---------------- specs/readme-to-docs-tree.md | 140 ++++++++ 3 files changed, 332 insertions(+), 600 deletions(-) delete mode 100644 specs/developer-docs-architecture.md create mode 100644 specs/readme-to-docs-tree.md diff --git a/specs/competitive-positioning.md b/specs/competitive-positioning.md index cc90366b..fdc57d8b 100644 --- a/specs/competitive-positioning.md +++ b/specs/competitive-positioning.md @@ -1,235 +1,161 @@ # DICE competitive positioning -What DICE is, and where that puts it against the agent-memory field: Zep/Graphiti, Mem0, Letta, -Cognee, Hindsight, LangMem, Neo4j's agent-memory offerings, and the three hyperscaler memory -services. The last sections cover vocabulary and mechanisms DICE takes from enterprise -data-governance and database tooling. - -Surveyed mid-2026. Several of these systems publish roadmap posts in the register of release notes, -so re-verify anything here before it goes into a bakeoff or a deck. +Survey date: mid-2026. Zep/Graphiti, Mem0, Letta (MemGPT), Cognee, Hindsight, LangMem, Neo4j agent +memory (labs SDK and NAMS), Google Vertex AI Memory Bank, AWS Bedrock AgentCore Memory, Microsoft +Foundry. Several of them publish roadmap intent and shipped behaviour in the same posts, so +re-verify a claim before reusing it. ## What DICE is -DICE is a proposition-first knowledge substrate. Heterogeneous artifacts — conversation, documents, -feeds, files — become durable propositions: confidence-weighted natural language statements with -typed entity mentions. Task-specific projections derive from them: a Neo4j graph, a Prolog fact -base, vector embeddings, agent working memory, reports. The proposition is the system of record. +DICE is Domain-Integrated Context Engineering. It uses a domain model to structure the context an +LLM reads, and applies the same structure to what the LLM produces (README.md:24-36). -A proposition is an epistemic object. It carries: +Natural language propositions are the system of record. Everything else derives from them: a Neo4j +graph, a Prolog fact base, vector embeddings, agent working memory, and reports (README.md:48-92, +`docs/design/architecture.md`). -- **confidence** and **importance**, each a `ZeroToOne`; -- **decay** — effective confidence falls exponentially with age, anchored at - `max(contentRevised, lastAccessed)`, so an unused claim loses standing without being deleted - (`Proposition.kt:358-403`); -- **a status lifecycle** — ACTIVE, SUPERSEDED, CONTRADICTED, PROMOTED, STALE, with revival from - STALE on reinforcement (`Proposition.kt:31-54`); -- **grounding** — `DerivedFrom` edges to the source chunks the claim was extracted from; -- **provenance entries** — typed source locator, chunk, offsets and content hash, append-only - (`ProvenanceEntry.kt:18-52`); -- **reinforceCount** — how often the claim has been re-observed. +A proposition carries: -The confidence-weighted proposition and the exponential decay formula come from general user -modeling: "Creating General User Models from Computer Use" (Shaikh, Sapkota, Rizvi, Horvitz, Park, -Yang, Bernstein; arXiv:2505.10831, UIST 2025, ACM DOI 10.1145/3746059.3747722), cited in the README -beside the decay maths. That is user-modeling research rather than agent memory. +- `confidence` and `importance`, each a `ZeroToOne` + (`dice/src/main/kotlin/com/embabel/dice/proposition/Proposition.kt:101-103`). +- `decay`. Effective confidence falls exponentially with age, anchored at the later of + `contentRevised` and `lastAccessed`, so using a claim refreshes it (`Proposition.kt:358-403`). +- A status: ACTIVE, SUPERSEDED, CONTRADICTED, PROMOTED, STALE. Reinforcement lifts a STALE + proposition back to ACTIVE (`Proposition.kt:31-54`). +- `grounding`. References to the source chunks the claim was extracted from. +- `provenanceEntries`. Typed source locator, chunk id, character offsets, content hash + (`dice/src/main/kotlin/com/embabel/dice/provenance/ProvenanceEntry.kt:18-52`). +- `reinforceCount`. How often the claim has been re-observed (`Proposition.kt:118`). -## The substrate compared +The confidence-weighted proposition and the exponential decay formula come from "Creating General +User Models from Computer Use" (Shaikh, Sapkota, Rizvi, Horvitz, Park, Yang, Bernstein; +arXiv:2505.10831, UIST 2025, ACM DOI 10.1145/3746059.3747722), cited in the README beside the decay +maths. GUM is user-modelling research. -Each system's unit of knowledge, and what that unit carries. +## Unit of knowledge | System | Unit of knowledge | What the unit carries | |---|---|---| | **Zep / Graphiti** | Episodes ingested into a temporal knowledge graph of entity nodes and edges | Bi-temporal edges: `valid_from`/`valid_until` for world time plus ingestion time; entity and edge attributes typed by Pydantic models; cross-session entity dedup | -| **Mem0** | Extracted facts — short natural language strings | Full SQLite change history (old value, new value, event, actor), mention counts. No confidence | +| **Mem0** | Extracted facts, short natural language strings | Full SQLite change history (old value, new value, event, actor), mention counts. No confidence | | **Letta (MemGPT)** | Memory blocks in core context (label, value, limit, description) and archival passages in Postgres/pgvector | Free text. The agent decides promotion and conflict resolution through tool calls | | **Cognee** | LLM-extracted nodes and RDF triples | An `ontology_valid` flag from matching against a declared OWL/RDF ontology. No confidence or decay | | **Neo4j agent-memory (labs SDK, NAMS)** | Entity and fact nodes, plus `(Message)` nodes chained by `[:NEXT]` per session | POLE+O typing, `valid_from`/`valid_until`, geospatial attributes | | **LangMem** | Semantic, episodic and procedural memory items | Optional typed Pydantic profiles; consolidation state | | **Hindsight** | Structured facts in a knowledge graph | Entity resolution that links "Alice" to "my coworker Alice" | | **Google / AWS / Microsoft** | Extracted memories as flat fact strings or items | A strategy or topic label and an IAM scope | -| **DICE** | Proposition | Confidence, importance, decay, status lifecycle, grounding, provenance entries, reinforce count | +| **DICE** | Proposition | Confidence, importance, decay, status, grounding, provenance entries, reinforce count | -Three of these carry something DICE's proposition does not. +Three of those units carry a field DICE's proposition lacks. -- **Zep's bi-temporal model** separates when a fact was true in the world from when the system - learned it. DICE stores system timestamps only, so it cannot express that split. -- **Cognee's ontology grounding** resolves extracted entity and type names against a declared +- **Bi-temporal edges (Zep / Graphiti).** World time is stored separately from ingestion time. + DICE's `TemporalMetadata` carries `observedAt`, `validFrom`, `validTo` and `invalidatedAt`, with + no transaction-time axis, so a query for what the store believed at a past instant has no basis. +- **Ontology grounding (Cognee).** Extracted entity and type names resolve against a declared OWL/RDF ontology, with fuzzy matching, before any graph node is built - (docs.cognee.ai/core-concepts/ontologies). DICE declares types in a `DataDictionary` and applies - them at extraction, without resolution against a stored ontology. -- **Mem0's change history** records old value, new value, event and actor per change. DICE's - `CollectorTraceStore` records why a collapse or merge decision was made and `ProvenanceEntry` - records where a claim came from; neither carries an extraction-run identifier, so there is no - end-to-end trace from a stored claim back to the run that produced it. + (https://docs.cognee.ai/core-concepts/ontologies). DICE declares types in a `DataDictionary` and + applies them at extraction, with no resolution against a stored ontology. +- **Change history (Mem0).** Old value, new value, event and actor per change. DICE records why a + collapse or merge decision was made + (`dice/src/main/kotlin/com/embabel/dice/spi/CollectorSignals.kt:115`) and where a claim came from + (`ProvenanceEntry`). Neither carries an extraction-run identifier, so a stored claim has no link + back to the run that produced it. ## Projections -DICE projects one proposition set into several representations: a Neo4j property graph and a Prolog -fact base (`dice-storage`), vector embeddings, agent working memory (`MemoryProjector`), rationale -and structured reports (`dice-report`), and lineage. Each derives from the propositions and can be -rebuilt from them; adding a representation means adding a projector, and the propositions are -unchanged. - -Where the surveyed systems put their knowledge: - -- **Zep / Graphiti** — a Neo4j property graph, with Leiden community clustering and summaries over - it; retrieval is cosine plus BM25 plus BFS with five reranking strategies. -- **Mem0** — a vector store, plus a separate graph pipeline over Neo4j, Memgraph, Neptune or Kuzu - fed by its own extraction path. -- **Letta** — pgvector passages, retrieved by the agent through tool calls. -- **Cognee** — an RDF-grounded graph plus embeddings; search returns nodes. -- **Neo4j agent-memory** — a native property graph; retrieval is hybrid vector plus up to 3-hop - traversal. -- **LangMem** — embedding space, with dilated-window retrieval. -- **Google / AWS / Microsoft** — similarity search over stored memories, with no exposed - intermediate model. - -For each of those, the store is the knowledge model, so a new representation means a new extraction -path or a migration. Mem0's separate graph pipeline is the field's closest analogue to a second -projection, and it re-extracts rather than deriving from the facts already stored. +Every DICE projection derives from the one proposition set and can be rebuilt from it, so adding a +representation means adding a projector: `dice-storage` for the Neo4j graph, the Prolog fact base +and lineage, `dice/src/main/kotlin/com/embabel/dice/projection/memory/MemoryProjector.kt:46` for +agent working memory, `dice-report` for rationale and structured reports. + +| System | Where knowledge lives | Retrieval | +|---|---|---| +| **Zep / Graphiti** | Neo4j property graph, with Leiden community clustering and summaries over it | Cosine plus BM25 plus BFS, with five reranking strategies | +| **Mem0** | A vector store, plus a separate graph pipeline over Neo4j, Memgraph, Neptune or Kuzu | Vector search; the graph pipeline runs its own extraction path | +| **Letta** | pgvector passages | Agent tool calls | +| **Cognee** | RDF-grounded graph plus embeddings | Search returns nodes | +| **Neo4j agent memory** | Native property graph | Hybrid vector plus up to 3-hop traversal | +| **LangMem** | Embedding space | Dilated-window retrieval | +| **Google / AWS / Microsoft** | Stored memories, with no exposed intermediate model | Similarity search | +| **DICE** | Propositions, projected into Neo4j, Prolog, vectors, memory and reports | `RetrievalRouter` over vector, entity, graph walk, temporal and hybrid modes | + +In every surveyed system the store is the knowledge model, so a second representation costs a second +extraction path or a migration. Mem0's graph pipeline is such a second representation, fed by its +own extraction over the source material. ## Knowledge hygiene -DICE treats hygiene as three interventions at three moments — what is let in, what is reclaimed, -what is consolidated between sessions (`docs/design/knowledge-hygiene.md`). +DICE splits hygiene into three interventions at three moments (`docs/design/knowledge-hygiene.md`). - **Admission.** Gates run at extraction: confidence qualification, deduplication against canonical - and stored propositions, conflict classification, trust scoring. The reason to gate at admission - rather than clean up afterwards is cost and clarity. + and stored propositions, conflict classification, trust scoring. Gating costs less than removing + junk later, and a low-confidence fact is easiest to judge while its extraction context is close. - **Reclamation.** `DecaySweepPass` retires softly to STALE and never hard-deletes, with - dual-threshold hysteresis — stale below 0.1 effective confidence, recovery edge at 0.25. Revival - is never sweep-driven; it happens on reinforcement. -- **Consolidation.** Between-session passes: `ContradictionResolutionPass` retires the weaker of a + dual-threshold hysteresis: stale below 0.1 effective confidence, recovery edge at 0.25 + (`dice/src/main/kotlin/com/embabel/dice/operations/consolidation/DecaySweepPass.kt:63-64`). + Revival happens only on reinforcement. +- **Consolidation.** Between-session passes. `ContradictionResolutionPass` retires the weaker of a contradictory pair to CONTRADICTED, auto-merge collapses duplicates, abstraction synthesises higher-level propositions from groups. -Background consolidation is convergent. Bedrock AgentCore runs extraction then consolidation as -background processes, with start and completion marks and success counts in the logs -(https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html). -Vertex AI Memory Bank extracts asynchronously, ingested via `add_memory()` or an end-of-conversation -callback -(https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, -2025-07-08). LangMem offers hot-path tools or background managers -(https://rywalker.com/research/langmem). Neo4j runs background enrichment and a multi-stage -extraction cascade (https://neo4j.com/labs/agent-memory/). No surveyed system documents a hard TTL -delete: decay, supersession and consolidation are the industry's answer as well as DICE's. - -Two hygiene mechanisms are absent from the surveyed field: a lifecycle status on the stored unit, -and a retirement path that keeps the retired unit. Vertex AI Memory Bank deletes the superseded -memory on contradiction; Microsoft Foundry discards the old value; Bedrock AgentCore writes a new -entry with no contradiction detection; Mem0 v3 (April 2026) is ADD-only, so supersession and -contradiction are inexpressible in the model. Cognee keeps its non-conforming nodes, but they are -"stored, embedded, and returned by search exactly like grounded ones" -(docs.cognee.ai/core-concepts/ontologies), so the flag has no effect on reads. +The four managed services also form memory off the request path. + +| Service | Background formation | +|---|---| +| Bedrock AgentCore | Extraction then consolidation as background processes, with start and completion marks and success counts in the logs ([metrics](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html)) | +| Vertex AI Memory Bank | Asynchronous extraction, ingested via `add_memory()` or an end-of-conversation callback ([announcement](https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview), 2025-07-08) | +| LangMem | Hot-path tools or background managers ([writeup](https://rywalker.com/research/langmem)) | +| Neo4j | Background enrichment and a multi-stage extraction cascade ([labs](https://neo4j.com/labs/agent-memory/)) | + +No surveyed system documents a hard TTL delete. Decay, supersession and consolidation are the common +answer. + +Two DICE hygiene mechanisms are absent from every other system surveyed: a lifecycle status on the +stored unit, and a retirement path that keeps the retired unit. + +- Vertex AI Memory Bank deletes the superseded memory on contradiction. +- Microsoft Foundry discards the old value. +- Bedrock AgentCore writes a new entry, with no contradiction detection. +- Mem0 v3 (April 2026) is ADD-only, so supersession and contradiction are inexpressible in the model. +- Cognee keeps its non-conforming nodes. They are "stored, embedded, and returned by search exactly + like grounded ones" (https://docs.cognee.ai/core-concepts/ontologies), so the flag has no effect + on reads. ## Schema governance -Declaring a schema for extraction is common in this field, and what happens to the declaration -varies. - -- **Graphiti** takes Pydantic entity and edge models and a `set_ontology` API with a - `strict_ontology` flag, and validates: "Each entity is validated against the appropriate Pydantic - model" before graph construction - (https://help.getzep.com/graphiti/core-concepts/custom-entity-and-edge-types). -- **Cognee** takes an OWL/RDF ontology file and checks extracted entities and types against it - before any graph nodes are built, then keeps what does not match, tagged `ontology_valid=False`: - "Nothing is rejected or discarded" (https://docs.cognee.ai/core-concepts/ontologies). -- **Neo4j's labs agent-memory SDK and NAMS** take a `DomainSchema` of entity types and descriptions. - It steers GLiNER extraction; no validation, versioning or rejection is documented - (https://neo4j.com/labs/agent-memory/faq/). -- **Bedrock AgentCore** gives full output-schema control only in the self-managed strategy; - built-in and overridden strategies "do not let you change the final output schema" - (https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory-custom-strategy.html). -- **Vertex AI Memory Bank's "custom topics"** and **Mem0's "custom categories"** are a label plus - free-text instructions inserted into the extraction prompt, with few-shot examples recommended — - prompt text rather than enforced types - (https://cloud.google.com/agent-builder/agent-engine/memory-bank/generate-memories; - https://docs.mem0.ai/open-source/features/custom-fact-extraction-prompt). -- **Letta** has no typed extraction schema: memory blocks are free-text segments. - -The strongest enforcement in reach is a general-purpose database feature. Neo4j **GRAPH TYPE**, a -Cypher 25 preview in Neo4j 2026.02 across Enterprise Edition, Infinigraph Edition and all Aura -tiers, declares nodes, labels and relationship connections with `SET`/`ADD`/`ALTER`/`DROP`, and -hard-rejects a non-conforming write at write time -(https://neo4j.com/blog/developer/graph-type-schema-enforcement-made-easy-preview/). GRAPH TYPE is -not part of Neo4j's agent-memory product and is not wired into it; `DomainSchema` there remains -unenforced extraction guidance. `SHOW CURRENT GRAPH TYPE` returns the declaration, with no -comparison against what the graph holds. - -Two capabilities are absent from every system surveyed. Neither gives an extraction schema a version -identity or history — Graphiti's docs describe evolution by adding attributes with no version -numbers, and GRAPH TYPE's `ADD`/`ALTER`/`DROP` are an evolution mechanism with no discrete version -record. And none compares a declaration against what a live store actually holds. No surveyed vendor -has published roadmap intent for either. - -DICE's design puts both above the store: - -- **Content-addressed version identity.** SHA-256 over the structural fields — sorted type names, - per-type labels and property signatures, sorted relationship descriptors — with every token - length-prefixed, so `["a;b"]` cannot collide with `["a", "b"]`. Registration is idempotent by - construction, two environments holding the same shape hash identically, and a proposition carries - the stamp as metadata, so the schema a claim was extracted under travels with the claim. - Confluent Schema Registry and AWS Glue assign sequential identity per subject instead, which makes - registration a stateful event and cross-environment comparison a lookup. -- **Per-type governance.** A governed-type selector decides which types are closed-world, so one - domain can be governed (`Person`, `Company`) while an exploratory type the LLM invented leaves the - version history unchanged. Both registries configure compatibility per subject: one mode for the - whole schema. -- **Drift detection.** Declaration against declaration, and declaration against the live graph, - producing a drift report. -- **Quarantine as the disposition.** Affected propositions are marked STALE — excluded from reads, - stored, recoverable. - -The version stamp, drift detection and quarantine are design-stage; what runs in the extraction path -today is the declared type dictionary (`SchemaRegistry`/`DataDictionary`) and the `SchemaAdherence` -policy (STRICT/DEFAULT/RELAXED). - -DICE chose quarantine as the disposition for extracted data. GRAPH TYPE shows write-time rejection -is a real design, and a workable one for writes an application authors. DICE's writes come from an -LLM, and a type nobody declared is often a real finding: rejection discards it with no review path, -and re-extraction costs another LLM pass without reliably reproducing the same proposition. -Quarantine keeps the proposition out of reads and keeps it reviewable. - -```mermaid -flowchart LR - W["extraction writes"] --> Q{"governed type?"} - Q -->|"no (open-world)"| ACC["accepted, unstamped"] - Q -->|yes| ST["stamped with contentHash"] - ST --> D{"declared vs observed"} - D -->|conforms| VALID["valid set — queryable"] - D -->|"diverges"| DR["drift report"] - DR --> QU["quarantined — marked stale,
excluded from reads, reversible"] - QU -.->|"human review"| VALID -``` - -Three sets of prior art shape the surface. Drift, contract, assertion, policy, incident and -quarantine are used with the meanings OpenMetadata, DataHub, Great Expectations, Soda and the Open -Data Contract Standard converged on, and their observe → alert → block ladder maps onto stamp-and- -observe, detect-and-report, quarantine. Drift reports follow SHACL's validation-report structure — a -conformance flag at the root, one result per violation carrying focus node, failed constraint, -severity and message — without SHACL's RDF serialisation, so consumers need no RDF parser. The -opt-in model comes from migration tooling: versioning activates on an application-supplied declared -schema, the drift check reports and leaves closing the gap to a human act the way `liquibase diff` -generates changesets and applies none, and nothing mutates a schema at startup the way -`hbm2ddl.auto=update` does. Those tools share one failure mode, opt-in fatigue: teams ship with -defaults and later have no drift visibility. - -Two questions the design leaves open: whether reads exclude quarantined propositions by default, and -where a drift report goes — DICE emits events and has no owner model, so there is no routing -destination. - -## Convergent mechanisms - -Places where DICE and a surveyed system reached the same solution to the same problem, beyond the -decay-and-consolidation convergence covered above. +Declaring a schema for extraction is common. What happens to the declaration varies. + +| System | Declaration | Enforcement | +|---|---|---| +| **Graphiti** | Pydantic entity and edge models; `set_ontology` with a `strict_ontology` flag | "Each entity is validated against the appropriate Pydantic model" before graph construction ([docs](https://help.getzep.com/graphiti/core-concepts/custom-entity-and-edge-types)) | +| **Cognee** | An OWL/RDF ontology file | Checked before any graph node is built. Content that fails to match is kept, tagged `ontology_valid=False`: "Nothing is rejected or discarded" ([docs](https://docs.cognee.ai/core-concepts/ontologies)) | +| **Neo4j labs SDK / NAMS** | `DomainSchema` of entity types and descriptions | Steers GLiNER extraction. No validation, versioning or rejection documented ([faq](https://neo4j.com/labs/agent-memory/faq/)) | +| **Bedrock AgentCore** | Output schema, self-managed strategy only | Built-in and overridden strategies "do not let you change the final output schema" ([docs](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory-custom-strategy.html)) | +| **Vertex Memory Bank / Mem0** | "Custom topics" and "custom categories": a label plus free-text prompt instructions, few-shot examples recommended | Prompt text, no type enforcement ([google](https://cloud.google.com/agent-builder/agent-engine/memory-bank/generate-memories); [mem0](https://docs.mem0.ai/open-source/features/custom-fact-extraction-prompt)) | +| **Letta** | None. Memory blocks are free-text segments | None | +| **Neo4j GRAPH TYPE** (general database feature, Cypher 25 preview in 2026.02, Enterprise, Infinigraph and all Aura tiers) | Nodes, labels and relationship connections via `SET`/`ADD`/`ALTER`/`DROP` | Hard-rejects a non-conforming write at write time ([blog](https://neo4j.com/blog/developer/graph-type-schema-enforcement-made-easy-preview/)) | +| **DICE** | A `DataDictionary` from embabel-agent (`com.embabel.agent.core.DataDictionary`), held by name in `SchemaRegistry` (`dice/src/main/kotlin/com/embabel/dice/common/SchemaRegistry.kt`) | `SchemaAdherence` over `entities` and `predicates` flags, with STRICT, DEFAULT and RELAXED presets (`dice/src/main/kotlin/com/embabel/dice/common/SchemaAdherence.kt:26-49`) | + +GRAPH TYPE is the strongest enforcement found, and it sits outside Neo4j's agent-memory product, +where `DomainSchema` remains unenforced extraction guidance. `SHOW CURRENT GRAPH TYPE` returns the +declaration, with no comparison against what the graph holds. + +Two capabilities are absent from every system surveyed, DICE included. No system gives an extraction +schema a version identity or history: Graphiti evolves a schema by adding attributes with no version +numbers, GRAPH TYPE's `ADD`/`ALTER`/`DROP` leave no discrete version record, and neither a DICE +declaration nor a stored proposition carries a version stamp. No system compares a declaration +against what a live store holds. No surveyed vendor has published roadmap intent for either. + +## Shared mechanisms + +Places where DICE and a surveyed system reached the same solution independently, beyond the decay +and consolidation overlap above. - **LLM extraction into a structured store.** All four surveyed memory services take a conversation turn, run an LLM extraction, and persist the result into a vector, graph or relational store - (https://rywalker.com/research/langmem; - https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, - 2025-07-08; https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html; - https://neo4j.com/labs/agent-memory/). DICE's pipeline has the same shape with the proposition as - the stored unit. + (cited under knowledge hygiene above, plus + https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html). DICE's pipeline has the + same shape, with the proposition as the stored unit. - **Integer re-indexing of proposition IDs across LLM calls.** DICE and Mem0 both renumber IDs per call so the model cannot invent one. - **Confidence-qualified, SNR-shaped extraction prompts.** DICE and LangMem arrived at the same @@ -237,151 +163,109 @@ decay-and-consolidation convergence covered above. ## Remaining gaps -The focus-first items, ranked by prevalence across the four surveyed memory services (LangMem, -Vertex AI Memory Bank, Bedrock AgentCore Memory, Neo4j Agent Memory) — a capability all four ship -outranks one vendor's experiment. - -**1. Operational tooling and memory inspection.** All four ship an inspection surface; DICE has -none. Bedrock AgentCore emits CloudWatch metrics for latency, invocations, errors and memory -creation count, with spans over CreateEvent, GetEvent, ListEvents, DeleteEvent and -RetrieveMemoryRecords, plus extraction and consolidation logs -(https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html). -Vertex AI Memory Bank lists stored memories in the Cloud Console (Memory Bank UI announcement, -Google Developer Forums; brief 15 records the citation without a URL). Neo4j's NAMS dashboard shows -health, entity count and queue lag with interactive graph visualisation and Cypher queryability -(https://medium.com/neo4j/a-tour-of-the-neo4j-agent-memory-service-nams-0f2d535a4fdb, 2026-06; GA -status unconfirmed). LangSmith traces execution paths and state transitions with cost, latency and -error dashboards and human-in-the-loop inspection (langchain.com/resources/llm-observability-tools). -In DICE, "why was this proposition formed" needs custom logging over `CollectorTraceStore`. - -**2. Background memory formation pipelines.** All four run formation off the request path, cited -under knowledge hygiene above. DICE has an async `@EventListener` on conversation analysis and a -non-blocking `PropositionIncrementalAnalyzer`, with consolidation available as passes an application -invokes. The managed pipeline that schedules them is missing. - -**3. TTL and eviction controls.** The gap is a retention policy an operator can set; DICE's decay -constant and sweep thresholds are code-level configuration (README:103-108). No surveyed system -publishes a TTL API either, so this item rests on weaker evidence than the two above it (brief 15 -lists the AWS TTL knob and Google's retention policy as unverified). - -**4. Procedural memory.** Two of the four ship it. LangMem has a `procedural` memory type where -agents update their own prompt rules from feedback (https://rywalker.com/research/langmem). Neo4j -stores tool usage and reasoning traces with similarity search over trace lineage -(https://neo4j.com/labs/agent-memory/). AWS and Google extract facts only. DICE projects -propositions into Prolog as a view layer and has no rule-formation path. - -**5. Namespacing and scoping APIs.** Two of the four expose one. AWS uses hierarchical namespaces -for fine-grained access control plus protocol session headers (`Mcp-Session-Id`, -`X-Amzn-Bedrock-AgentCore-Runtime-Session-Id`) -(https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html). Neo4j chains -`(Message)` nodes by `[:NEXT]` per session with entity and fact nodes shared across sessions -(https://neo4j.com/labs/agent-memory/). Google scopes by identity and LangMem by embedding space, -neither with a namespace API. DICE's `ContextId` is a query filter over shared storage -(README:1241-1270), so it isolates reads and not the storage layer. - -**6. Cross-session profile modeling.** All four aggregate across sessions, none publishes a profile -schema. AWS extracts user preferences, facts and session summaries across sessions; Google -personalises long-term memories per user -(https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview, -2025-07-08); Neo4j persists preference and fact entities in a shared graph -(https://neo4j.com/labs/agent-memory/); LangMem keeps preferences in semantic memory with optional -typed Pydantic profiles (https://rywalker.com/research/langmem). DICE's `ContextId` scopes knowledge -and defines no profile structure or cross-session aggregation. - -Also missing, and not held by any surveyed system either: an extraction-run identifier linking a -stored claim to the run and schema that produced it; a bi-temporal fact model, so no point-in-time -query and no temporal contradiction resolution; temporal anchoring, so relative dates are stored as -literal text; surprise-prioritised retention, so novel facts get no durability preference; and -versioning or an audit trail on conflict policy. +Capabilities DICE lacks, against the four managed memory services: LangMem, Vertex AI Memory Bank, +Bedrock AgentCore Memory, Neo4j Agent Memory. The count column says how many of the four ship it. + +| Gap | Count | Evidence | DICE today | +|---|---|---|---| +| Operational tooling and memory inspection | 4 | Bedrock emits CloudWatch metrics for latency, invocations, errors and memory creation count, with spans over CreateEvent, GetEvent, ListEvents, DeleteEvent and RetrieveMemoryRecords, plus extraction and consolidation logs ([metrics](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html)). Vertex lists stored memories in the Cloud Console (Memory Bank UI announcement, Google Developer Forums, no stable URL recorded). The NAMS dashboard shows health, entity count and queue lag with graph visualisation and Cypher queryability ([tour](https://medium.com/neo4j/a-tour-of-the-neo4j-agent-memory-service-nams-0f2d535a4fdb), 2026-06, GA unconfirmed). LangSmith traces execution paths and state transitions with cost, latency and error dashboards (langchain.com/resources/llm-observability-tools) | "Why was this proposition formed" needs custom logging over `CollectorTraceStore` | +| Background memory formation pipelines | 4 | Formation runs off the request path, cited under knowledge hygiene above | Async `@EventListener` on conversation analysis, a non-blocking `PropositionIncrementalAnalyzer` (`dice/src/main/kotlin/com/embabel/dice/incremental/proposition/PropositionIncrementalAnalyzer.kt`), and consolidation passes an application invokes. The scheduler that runs them is missing | +| TTL and eviction controls | 0 publish a TTL API | The AWS TTL setting and Google's retention policy are unverified | Decay rate multiplier defaults to 2.0 (`Proposition.kt:365`); sweep thresholds default to 0.1 and 0.25 (`DecaySweepPass.kt:63-64`). All are constructor parameters, so an operator has no retention policy to set | +| Procedural memory | 2 | LangMem's `procedural` type lets agents update their own prompt rules from feedback ([writeup](https://rywalker.com/research/langmem)). Neo4j stores tool usage and reasoning traces with similarity search over trace lineage ([labs](https://neo4j.com/labs/agent-memory/)). AWS and Google extract facts only | Prolog projection over propositions as a view layer, with no rule-formation path | +| Namespacing and scoping APIs | 2 | AWS uses hierarchical namespaces for fine-grained access control plus session headers `Mcp-Session-Id` and `X-Amzn-Bedrock-AgentCore-Runtime-Session-Id` ([memory](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html)). Neo4j chains `(Message)` nodes by `[:NEXT]` per session, with entity and fact nodes shared across sessions ([labs](https://neo4j.com/labs/agent-memory/)). Google scopes by identity and LangMem by embedding space, neither with a namespace API | `ContextId` is a query filter over shared storage (README.md:1238-1256). It scopes reads. The storage layer stays shared | +| Cross-session profile modelling | 4 aggregate, 0 publish a profile schema | AWS extracts user preferences, facts and session summaries across sessions. Google personalises long-term memories per user ([announcement](https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview), 2025-07-08). Neo4j persists preference and fact entities in a shared graph ([labs](https://neo4j.com/labs/agent-memory/)). LangMem keeps preferences in semantic memory with optional typed Pydantic profiles ([writeup](https://rywalker.com/research/langmem)) | `ContextId` scopes knowledge and defines no profile structure or cross-session aggregation | + +Absent from DICE and from every surveyed system: an extraction-run identifier linking a stored claim +to the run and schema that produced it; temporal anchoring, so relative dates are stored as literal +text; surprise-prioritised retention, so novel facts get no durability preference; versioning or an +audit trail on conflict policy. + +Absent from DICE and present in Zep/Graphiti: a transaction-time axis, so DICE has no point-in-time +query over past belief and no temporal contradiction resolution. ## Candidate modules -The capabilities the managed services hold over DICE sit above the proposition substrate and consume -it, so each is a module or add-on rather than a change to the substrate. - -**Governed working memory.** ArcMem is an activation-ranked working-memory system built on DICE -APIs, and the existing proof that this layer works over the substrate. It keeps a bounded active set -(20 units plus a token budget) in a protected prompt region, ranks units by an activation score -blending recency, reinforcement and decay, protects units by authority level (PROVISIONAL → -UNRELIABLE → RELIABLE → CANON) from automatic demotion, reinforces on access by anchoring decay at -`max(contentRevised, lastAccessed)`, and evicts into durable storage so evicted material stays -queryable. Its gate order is confidence → deduplication → conflict → trust → promotion → budget -enforcement, and changing the order changes the semantics. It consumes DICE's storage, extraction, -decay and revision as primitives (brief 17). A DICE-native module needs five SPIs ArcMem hand-rolls -today: - -- activation ranking as a pluggable policy — ArcMem uses hand-tuned parameters and a fixed rank - clamp of [100, 900]; -- authority and trust levels with the promotion gate; -- pinning and protection for the resident set; -- per-turn lifecycle events for reinforce, evict and reactivate; -- batched or concurrent extraction — `PropositionPipeline.process()` is serial and consumers batch - by hand. +Each gap above consumes propositions and adds no field to the proposition model, so each can ship as +a separate module over the DICE state the table records. + +**Working memory.** A bounded active set held in a protected prompt region, ranked by an activation +score blending recency, reinforcement and decay, evicted into durable storage so evicted material +stays queryable. DICE supplies storage, extraction, decay, revision and pinning +(`dice/src/main/kotlin/com/embabel/dice/proposition/PropositionStore.kt:234-241`) as primitives. The +module adds four SPIs: -Which of these belong in DICE and which in the consuming application is unsettled (brief 17, Risks), -and entity resolution has to stay serial where shared identity is involved. +- activation ranking as a pluggable policy; +- authority and trust levels with a promotion gate; +- per-turn lifecycle events for reinforce, evict and reactivate; +- a token and unit budget over the resident set. -**Session memory management.** In DICE today: `MemoryProjector` classifies propositions by knowledge -type for prompt injection -(`dice/src/main/kotlin/com/embabel/dice/projection/memory/MemoryProjector.kt:46`), and the `Memory` -tool runs hybrid vector plus keyword retrieval over context-scoped propositions -(`dice/src/main/kotlin/com/embabel/dice/agent/Memory.kt:112`). A module would add session-scoped -active-context assembly and summarisation over those two. +**Session memory management.** Session-scoped active-context assembly and summarisation over +`MemoryProjector`, which classifies propositions by knowledge type for prompt injection +(`MemoryProjector.kt:46`), and the `Memory` tool, which runs hybrid vector plus keyword retrieval +over context-scoped propositions (`dice/src/main/kotlin/com/embabel/dice/agent/Memory.kt:112`). -**Background consolidation.** In DICE today: consolidation passes including -`ContradictionResolutionPass` and `DecaySweepPass`, plus async event listeners on analysis. A module -would add the scheduler, the extraction-then-consolidation staging AWS and Google run, and the -lifecycle logging that makes a run inspectable. +**Background consolidation.** A scheduler over the existing passes, the +extraction-then-consolidation staging AWS and Google run, and lifecycle logging that makes a run +inspectable. -**Procedural memory.** In DICE today: Prolog projection over propositions as a view layer. A module -would add the formation path — turning agent feedback into stored rules that later runs read back. +**Procedural memory.** A formation path over the existing Prolog projection, turning agent feedback +into stored rules that later runs read back. ## Orthogonal research -Scanning adjacent non-LLM fields for mechanisms is standing practice here, on the same footing as -the GUM lineage above. These map onto problems DICE already has. +Mechanisms from non-LLM fields that map onto problems DICE has. | Field | Mechanism | DICE analog | Citation | |---|---|---|---| -| Truth maintenance (TMS/ATMS) | Justifications record why a belief holds; retracting a premise un-derives its dependents; ATMS labels a node with the assumption sets that support it | `ContradictionResolutionPass` retires the weaker of a contradictory pair to CONTRADICTED by comparing `effectiveConfidence()`, with no dependency record | Doyle, "A Truth Maintenance System," *Artificial Intelligence* 12(3), 1979; de Kleer, "An Assumption-Based TMS," *Artificial Intelligence* 28, 1986 | -| AGM belief revision | Revision and contraction obey minimal-change postulates over a selection function | Confidence adjustments (contradicted +0.15, merged ×0.7, reinforced ×0.85) are hand-tuned constants | Alchourrón, Gärdenfors, Makinson, "On the Logic of Theory Change," *J. Symbolic Logic* 50, 1985 | +| Truth maintenance (TMS/ATMS) | Justifications record why a belief holds; retracting a premise un-derives its dependents; ATMS labels a node with the assumption sets that support it | `ContradictionResolutionPass` retires the weaker of a contradictory pair to CONTRADICTED by comparing `effectiveConfidence()`, with no dependency record (`dice/src/main/kotlin/com/embabel/dice/operations/consolidation/ContradictionResolutionPass.kt:84-87`) | Doyle, "A Truth Maintenance System," *Artificial Intelligence* 12(3), 1979; de Kleer, "An Assumption-Based TMS," *Artificial Intelligence* 28, 1986 | +| AGM belief revision | Revision and contraction obey minimal-change postulates over a selection function | Decay adjustments (contradiction +0.15, merge ×0.7, reinforcement ×0.85) are hand-tuned constants (`dice/src/main/kotlin/com/embabel/dice/proposition/revision/LlmPropositionReviser.kt:528,670,694`) | Alchourrón, Gärdenfors, Makinson, "On the Logic of Theory Change," *J. Symbolic Logic* 50, 1985 | | Provenance semirings | Provenance of a derived fact is a semiring expression over source tokens, composed under the query's + and × operators | `ProvenanceEntry` is a flat per-proposition list with no algebra for merge or abstraction | Green, Karvounarakis, Tannen, "Provenance Semirings," PODS 2007, DOI 10.1145/1265530.1265535 | | Bitemporal databases | Valid time and transaction time as orthogonal axes, with defined as-of and point-in-time queries | `TemporalMetadata` carries `observedAt`/`validFrom`/`validTo`/`invalidatedAt` and no transaction-time axis | Snodgrass, *Developing Time-Oriented Database Applications in SQL*, Morgan Kaufmann, 1999 | | Record linkage | Fellegi-Sunter match decisions from per-field m/u agreement probabilities against a likelihood-ratio threshold | Entity resolution uses fuzzy, vector, exact, partial and agentic searchers with LLM disambiguation | Fellegi, Sunter, "A Theory for Record Linkage," *JASA* 64, 1969, DOI 10.1080/01621459.1969.10501049 | -| ACT-R declarative memory | Base-level activation `B_i = ln(Σ_j t_j^-d)` folds recency and frequency into one retrieval score | `effectiveConfidence()` decays on recency alone; `reinforceCount` sits outside the decay maths — the same two signals a working-memory module needs to rank on | Anderson & Schooler, "Reflections of the Environment in Memory," *Psychological Science* 2, 1991 | +| ACT-R declarative memory | Base-level activation `B_i = ln(Σ_j t_j^-d)` folds recency and frequency into one retrieval score | `effectiveConfidence()` decays on recency alone. `reinforceCount` sits outside the decay maths, and a working-memory module needs both signals to rank on | Anderson & Schooler, "Reflections of the Environment in Memory," *Psychological Science* 2, 1991 | | Argumentation frameworks | Arguments plus an attack relation; admissible, preferred and grounded semantics decide which sets survive collectively | `ContradictionResolutionPass` is pairwise strongest-wins, with pinned propositions branched out into a review event | Dung, "On the Acceptability of Arguments...," *Artificial Intelligence* 77(2), 1995 | -The three strongest borrowing opportunities, per brief 16: +Three borrowing opportunities: -- **ATMS justification tracking, to make CONTRADICTED reversible.** The status flip today is driven - by a confidence comparison at classification time and records no reason for the loss. A - justification set per proposition lets retracting the evidence un-derive the dependent status, - without a fresh classification pass. +- **ATMS justification tracking, to make CONTRADICTED reversible.** The status flip follows a + confidence comparison at classification time and records no reason for the loss. A justification + set per proposition lets retracting the evidence un-derive the dependent status. - **ACT-R base-level activation, to unify `reinforceCount` and decay.** Both signals exist and never combine. The activation equation is a closed form for the ranking score a working-memory module - needs and for decay that counts frequency of use. -- **Semiring-formalised provenance composition.** Treating auto-merge as the union-like operator and - abstraction synthesis (which requires all its sources) as the product-like one gives one queryable - answer to "what composed this fact" across both, in place of per-pass list splicing. - -AGM and Fellegi-Sunter are weaker fits: AGM's postulates cover logical theories rather than graded -beliefs and are useful as a checklist, and LLM disambiguation already covers what Fellegi-Sunter -scoring would buy. The first two opportunities touch `Proposition.kt` and -`ContradictionResolutionPass.kt`, which consolidation passes, storage projection and retrieval -ranking all depend on. + needs, and for decay that counts frequency of use. +- **Semiring-formalised provenance composition.** Treat auto-merge as the union-like operator and + abstraction synthesis (which requires all its sources) as the product-like one. That gives one + queryable answer to "what composed this fact" across both passes. + +AGM's postulates cover logical theories and DICE's beliefs are graded, so AGM serves as a checklist. +LLM disambiguation already covers what Fellegi-Sunter scoring would add. The first two opportunities +touch `Proposition.kt` and `ContradictionResolutionPass.kt`, which consolidation, projection and +retrieval ranking all depend on. + +## Open questions + +- **Which working-memory capabilities belong in DICE and which in the consuming application.** + Recommendation: activation ranking and per-turn lifecycle events in DICE, prompt-region and budget + policy in the application, since only the application knows its token budget. +- **Whether to close the schema version-identity gap.** Recommendation: treat it as a candidate + module. No surveyed system offers it and no surveyed vendor has published roadmap intent, so + nothing external forces the timing. +- **Whether entity resolution can run concurrently in a working-memory module.** Recommendation: keep + it serial where shared identity is involved, matching the pipeline's serial resolution stage + (`docs/design/architecture.md`). ## Out of scope -- **Zep's five-reranker retrieval.** Deep feature tied to Neo4j traversal; the bi-temporal model is +- **Zep's five-reranker retrieval.** Deep feature tied to Neo4j traversal. The bi-temporal model is the better investment from that system. - **LangMem's prompt optimisation.** Orthogonal to memory quality. - **Mem0's separate graph pipeline.** DICE has entity mentions plus a Neo4j projection over the same propositions. -- **Google's multimodal extraction.** The proposition model is format-agnostic; add on demand. +- **Google's multimodal extraction.** The proposition model is format-agnostic. Add on demand. - **AWS's episodic reflection.** The abstraction pipeline already synthesises across propositions. -- **Neo4j's POLE+O ontology.** Domain-specific subtypes; the proposition model is domain-agnostic. -- **Neo4j's spaCy → GLiNER → LLM cascade.** Cost-effective and operationally heavy: model downloads - and dependency management. +- **Neo4j's POLE+O ontology.** Domain-specific subtypes. The proposition model is domain-agnostic. +- **Neo4j's spaCy to GLiNER to LLM cascade.** Cost-effective and operationally heavy: model + downloads and dependency management. - **Contract YAML as the authoring surface.** DICE's declared schema is a JVM type an application - owns; a YAML dialect would be a second source of truth. + owns. A YAML dialect would be a second source of truth. - **Managed hosting.** The embeddable library is the distribution model. diff --git a/specs/developer-docs-architecture.md b/specs/developer-docs-architecture.md deleted file mode 100644 index 54d330fb..00000000 --- a/specs/developer-docs-architecture.md +++ /dev/null @@ -1,292 +0,0 @@ -# Developer docs architecture - -The target information architecture for DICE's developer documentation: what pages exist, in what -order, and who owns keeping them true. Most of what's below doesn't exist yet. - -DICE ships to Maven Central as a set of JVM modules with Spring Boot autoconfiguration. The docs -that fit that shape are the ones Confluent Schema Registry, Flyway, Zep and Spring Boot itself -write: concept-anchored and workflow-layered, none of them strict Diátaxis. All of them open with a -runnable quickstart, then a small set of mental models, then task-shaped how-tos, then dense -reference, with optional features flagged at the top of their own page and a compatibility matrix -that answers "will this work with my stack" on its own. - -## What we already have, and what's missing - -`docs/design/` is the *why*: rationale notes aimed at someone changing DICE. They stay as they are. -`specs/` is strategy and planning, also internal. - -The *how* is all in `README.md`, which is 2,621 lines. It holds Spring pipeline setup, the -proposition pipeline walkthrough, mention filtering, entity extraction and resolution, `ContextId` -and `PropositionQuery`, graph and Prolog projection, agent memory, the Oracle, the REST API, Spring -Boot integration, graph-backed storage config, API-key security and installation. - -So the work is mostly a migration: - -| Directory | Audience | Question it answers | -|---|---|---| -| `docs/design/` | DICE contributors | Why is it built this way? | -| `docs/` (new tree below) | DICE consumers | How do I use it? | -| `specs/` | Us | What are we building and why does it matter commercially? | -| `README.md` | Anyone landing on the repo | What is this, should I care, where do I start? | - -Design docs and developer docs cross-link without duplicating. A concept page says what a -proposition is and how to make one; the design note says why confidence decays the way it does. - -### Migrating the README - -Target: a README under ~250 lines. Line numbers are as of this branch and will move, so re-check -before acting on the table. - -| README section (line) | Fate | -|---|---| -| What is DICE, benefits table, architecture overview (24–116) | **Keep**, trimmed. This is the landing page's job. | -| Real-world example: Impromptu (117–176) | **Keep**, cut to a paragraph plus a link. | -| Pipeline setup, conversation analysis (127–176) | **Move** → `quickstart/` | -| Proposition pipeline, content dedup, mention filtering (177–489) | **Move** → `concepts/propositions.md`, `how-to/extract-from-documents.md`, `how-to/mention-filtering.md` | -| Entity extraction, entity resolution, resolution service (490–1225) | **Move** → `concepts/entity-resolution.md` + `how-to/tune-entity-resolution.md`. The largest block; split it. | -| Source analysis context, `ContextId`, `PropositionQuery` (1226–1419) | **Move** → `concepts/context-and-schema.md`, `how-to/query-propositions.md` | -| Relations, projector architecture, graph and Prolog projection (1420–1647) | **Move** → `concepts/storage-and-projections.md`, `how-to/project-to-graph.md`, `features/prolog-inference.md` | -| Agent memory, memory projection, memory maintenance (1648–1979) | **Move** → `how-to/agent-memory.md`, `concepts/knowledge-hygiene.md` | -| Proposition operations, Oracle (1980–2090) | **Move** → `how-to/query-propositions.md`, `how-to/oracle.md` | -| Package structure (2091–2205) | **Move** → `reference/` | -| REST API and endpoints (2206–2334) | **Move** → `features/web-api.md`; it is opt-in and gated by an API key. | -| Spring Boot integration, graph-backed storage, API-key security (2335–2572) | **Move** → `how-to/choose-a-backend.md`, `reference/configuration-properties.md`, `features/web-api.md` | -| Installation (2573–2585) | **Keep** as coordinates only; the working version lives in the quickstart. | -| Technology stack, references, license (2586–2621) | **Keep**. | - -Two rules for the migration. Content moves: each section is deleted from the README as it lands in -`docs/`, so there is one copy. And every removed section leaves a one-line link where it was, so an -existing bookmark still lands somewhere useful. Do it as one PR per destination page. - -## The IA - -```mermaid -flowchart TD - R["docs/README.md
what DICE is, when to use it"] - QS["quickstart/
15 minutes to first extraction"] - C["concepts/
the spine — read in order"] - H["how-to/
by use case"] - F["features/
opt-in, activation condition first"] - REF["reference/
config, API, properties"] - P["production/
deploy, tune, observe"] - S["support/
compatibility matrix, FAQ"] - - R --> QS - QS --> C - C --> H - H --> F - H --> P - F --> REF - P --> REF - R --> S - - C --- C1["propositions
entity resolution
storage and projections
context and schema
knowledge hygiene"] - H --- H1["extract from documents
query what you stored
project to a graph
generate a report
wire it into an agent"] - F --- F1["metamodel versioning
governance and drift
Prolog inference
web API
decay and pruning"] -``` - -Five concepts, in that order, because each needs the one before it. Every other section is entered -from any direction. - -## The concept spine - -1. **Propositions** — natural-language claims are the system of record. Confidence, importance, - decay, lifecycle states. Everything else is a projection of these. -2. **Entity resolution** — how mentions in text get matched to entities that already exist, or - minted as new ones. Resolution outcomes, the escalating resolver chain, cross-chunk dedup. -3. **Storage and projections** — the `PropositionStore` SPI family, in-memory and the durable Neo4j - backend, and the materialised views (vector, graph, Prolog, memory, oracle). -4. **Context and schema** — `contextId` scoping, the `DataDictionary`, `SchemaAdherence`, and what - "extraction against a schema" constrains. -5. **Knowledge hygiene** — admission gates, reclamation, consolidation, and why they are three - interventions at three moments. - -Every concept page ends with **Try it now**: five to ten lines that exercise what was just -explained against the quickstart's setup, and the output to expect. - -## The 15-minute quickstart - -One page, one path, no branches. Copy, paste, run, see output. - -### Prerequisite: there is no starter artifact - -DICE ships `dice`, `dice-ingestion`, `dice-storage`, `dice-storage-autoconfigure` and -`dice-report`. There is no `dice-spring-boot-starter`, so a one-coordinate quickstart has a build -prerequisite: - -- **Either** create a `dice-spring-boot-starter` aggregator module depending on - `dice-storage-autoconfigure` (which pulls `dice-storage` and `dice`) plus `dice-report`, publish - it to Maven Central, and let the quickstart use one coordinate. Small module, and it has to land - before the quickstart page can be written as one line. -- **Or** write the quickstart against today's coordinates, `dice-storage-autoconfigure` and - `dice-report`, both explicit, with a comment saying what each buys. Two lines, and it unblocks - the docs from a release. - -Pick one before writing the page. The page uses coordinates that are published. - -### The page - -1. **The dependencies.** Whichever of the two above we picked, with the version from the BOM. -2. **Configuration.** The minimum that works: an LLM provider (inherited from embabel-agent), an - embedding model, and the default in-memory store, which means no Neo4j and no Docker for the - first run. `InMemoryPropositionRepository` does vector search, so without a configured embedding - model the vector query in step 6 returns nothing. To keep the config to one key, drop vector - retrieval from the minimal path, query by entity and `ContextId` only, and introduce embeddings - in the retrieval how-to. Either way the page's queries match the configuration it gave. -3. **What autoconfiguration gave you.** The beans that now exist and what each is for: a list of - things you can `@Autowired`. -4. **Extract.** Feed one paragraph of text through the pipeline. Print the propositions with their - confidence. -5. **Persist.** `PropositionPipeline` returns unsaved results, so the caller owns the transaction. - Call `persist(propositionRepository, namedEntityDataRepository)` on the `PersistablePropositions` - result, and say in one sentence why the pipeline leaves it to the caller. Skipping this step - leaves an empty store. -6. **Query.** Retrieve what was just stored, by entity and `ContextId`, and by vector similarity if - step 2 configured an embedding model. -7. **Report.** Produce one human-readable artifact so the run ends in something visible. -8. **Where to go next**, split by intent; see the audience note below. - -Constraints. The 15 minutes holds: no Neo4j, no Docker, no `create` of anything external. The code -is real and compiles, and a test in `dice-integration-tests` runs the quickstart's exact snippets, -which is what catches a missing persist step. Anything that needs a decision (backend choice, -resolver chain, schema) takes the default and links out. - -## How-to guides, by use case - -Task-shaped, titled by what the reader wants, each stating its prerequisites at the top: - -- Extract knowledge from a document set (chunking, dedup before extraction, batch, and persisting - the unsaved results the pipeline hands back) -- Tune entity resolution (resolver chain, thresholds, when to spend an LLM call) -- Choose and configure a storage backend (in-memory → Neo4j, what changes) -- Query propositions (composable `PropositionQuery`, retrieval modes, trust filtering) -- Project to a Neo4j graph (typed nodes, lineage back to evidence, re-run idempotence) -- Query with Prolog (facts, rules, transitive reasoning) -- Generate a report -- Use DICE as agent memory in an embabel-agent application -- Handle conflicts and contradictions (the conflict policy SPI) -- Process a stream incrementally (windowing, dedup across windows) - -Each states what it costs in LLM calls, latency, and infrastructure. - -## Feature pages - -Every optional feature gets its own page, and each page states its activation condition in the -first paragraph. This matters most for versioning and governance, where nothing happens until an -application asks for it: no `DeclaredSchemaSource`, no versioning; no governed types, no drift -detection. DICE versions a schema only when an application declares one. - -Template, in order: - -- **Availability** — which DICE version, which modules, what it costs to add. -- **Activation condition** — the exact bean, property, or annotation that switches it on, on the - first screen. "Off unless you define a `DeclaredSchemaSource` bean" opens the versioning page. -- **When to use it**, and when to leave it off. -- **Impact** — latency, memory, extra infrastructure, extra LLM calls. -- **How to enable** — full working configuration. -- **Example** — real code with real output. - -Opt-in surfaces that need this treatment: metamodel versioning, drift detection and quarantine, -Prolog inference, graph projection, the web API, decay and stale-pruning, the multi-signal -collector, concurrent extraction. - -## Compatibility matrix - -Its own page, linked from the README, answering "will this work with my stack" before any prose. -Four axes. The embabel-agent one is stated as a supported range, since DICE tracks a moving -platform. - -| Axis | What we state | -|---|---| -| embabel-agent | Supported version range per DICE release, with the tested point release called out | -| Spring Boot | Minimum and maximum tested, per DICE release | -| JDK | Baseline is 21 (inherited from `embabel-build-parent`); newer JDKs listed as tested or untested | -| Neo4j | Required version floor for the durable backend, via Drivine; "not required" for in-memory | - -Plus a feature-availability column, so a reader can see that governance arrives after 1.0. Exact -values get pinned at the 1.0 release; the page ships with the axes and the tested-against values we -have. - -## Production guide - -The page a team reads before going live: - -- Backend choice and the migration from in-memory to Neo4j -- Schema declarations and index creation on startup -- Decay tick configuration, and why `prune-stale` defaults to false -- Concurrency: what's safe to run in parallel and where entity-resolution ordering matters -- LLM cost and latency budgets per pipeline stage -- Observability: the events DICE emits, what to alert on -- Backup, and what "system of record" obliges you to keep -- Governance rollout: stamp first, detect later, quarantine last - -## Audience split - -Two audiences arrive with different vocabularies and different first questions: - -- **Data engineers** want to know where the data lands, what the schema is, how drift is handled, - and what Cypher they can run. They read storage, schema, governance, projections. -- **LLM/agent engineers** want extraction quality, entity resolution, memory retrieval, and how it - plugs into an agent. They read propositions, resolution, retrieval, agent memory. - -The concept order is the same for both: propositions, then resolution, then storage. What differs -is the entry point. The quickstart's "where to go next" offers a data-engineer path and an -agent-engineer path through the same pages in different orders, and the README says which is which. -Two parallel doc sets drift. - -A third audience, researchers wanting Prolog and reasoning, is real and small, and is served by the -Prolog feature page. - -## Risks - -**Reading-order enforcement.** The concept chain makes sense in order, and readers arrive from -search engines in the middle of it. Nothing in a static docs site forces sequence. Mitigations, all -cheap: number the concept pages, put a one-line "assumes you've read X" at the top of each, and end -each with an explicit next link. Each page has to stand on its own for a reader who lands mid-chain. - -**Extraction variability.** The docs assume the extractor returns clean, schema-adhering -propositions. It won't always. Mention filtering and `SchemaAdherence` appear early and are weighted -heavily. - -**Backend coupling surfaces late.** Prolog needs a store; graph projection needs decisions made -before extraction runs. The quickstart defers all of this, so the "choose your backend" how-to has -to be prominent and has to say plainly that changing it later means rework. - -**Duplication across modules, and with the README.** Spring configuration examples exist in -`dice-storage-autoconfigure`, `dice-ingestion` and elsewhere, and the 2,621-line README holds a -version of most of them. One canonical example set in `docs/`; module READMEs and the root README -link to it. The migration table above works only if content moves. - -**Doc rot.** Handled by the contract below, and by making the quickstart executable. - -## The per-PR docs contract - -**Every feature PR ships its design-doc delta and its developer-doc page in the same PR. Missing -docs are a review-blocking finding, on the same footing as a missing test.** - -Concretely: - -- New or changed behaviour visible to a consumer → the developer-doc page is updated in that PR. -- New or changed *rationale* → the `docs/design/` note is updated in that PR. -- A new opt-in feature → a feature page stating its activation condition, in that PR. -- A new configuration property → the reference page, in that PR. -- A version-support change → the compatibility matrix, in that PR. - -Docs written later are written by someone who has forgotten the edge cases. Governance is opt-in, -so an undocumented switch cannot be turned on by the consumer it was built for. - -Reviewers ask one question: *if I only had this PR's docs, could I use this feature?* If the answer -is no, the finding is blocking. "Docs to follow" does not resolve it. - -Carve-outs: internal refactors with no consumer-visible change need no doc delta, and neither do -test-only or build changes. - -## Open questions - -- Does DICE ever ship a CLI (schema validation, drift check)? Flyway's docs work partly because the - CLI gives every concept an executable form. If a CLI happens, it needs its own quickstart. -- Rendered site or Markdown in the repo? The IA works either way; the decision affects cross-linking - and whether the compatibility matrix can be generated. -- Do the `docs/design/` notes stay contributor-facing, or do the best of them get promoted into - concept pages? Current answer: they stay, because their audience is different. Concept pages will - lift explanations from them. diff --git a/specs/readme-to-docs-tree.md b/specs/readme-to-docs-tree.md new file mode 100644 index 00000000..531791fa --- /dev/null +++ b/specs/readme-to-docs-tree.md @@ -0,0 +1,140 @@ +# Split the README into a docs tree + +Decides the `docs/` layout, which README section lands on which page, the shape of the quickstart +and feature pages, and the rule that keeps pages current. Writing the pages is out of scope. + +## Target tree + +| Directory | What goes there | +|---|---| +| `docs/README.md` | What DICE is, when to use it, links into the tree. | +| `docs/quickstart/` | One page, one path: dependencies through first report in 15 minutes. | +| `docs/concepts/` | Five pages, read in order, propositions through knowledge hygiene. | +| `docs/how-to/` | Task-shaped pages titled by what the reader wants, prerequisites at the top. | +| `docs/features/` | One page per opt-in surface, activation condition first. | +| `docs/reference/` | Configuration properties, package structure, REST endpoints. | +| `docs/production/` | Backend migration, decay settings, concurrency, LLM cost, observability, backup. | +| `docs/support/` | Compatibility matrix and FAQ. | + +| Location | Audience | Question it answers | +|---|---|---| +| `docs/design/` (19 notes and an index, unchanged) | DICE contributors | Why is it built this way? | +| `docs/` | DICE consumers | How do I use it? | +| `specs/` | Us | What are we building, and why does it matter commercially? | +| `README.md` | Anyone landing on the repo | What is this, and where do I start? | + +## README migration + +`README.md` is 2621 lines, target around 250. Line numbers move, so re-check a row before acting. + +| README section (lines) | Fate | +|---|---| +| What is DICE, benefits table, architecture overview, design notes (24-116) | Keep, trimmed. This is the landing page's job. | +| Real-world example: Impromptu (117-126) | Keep, cut to a paragraph plus a link. | +| Pipeline setup, conversation analysis (127-174) | Move to `docs/quickstart/`. | +| Key features, proposition pipeline, content dedup, mention filtering (175-489) | Move to `docs/concepts/propositions.md`, `docs/how-to/extract-from-documents.md`, `docs/how-to/mention-filtering.md`. | +| Entity extraction, entity resolution, resolution service (490-1225) | Move to `docs/concepts/entity-resolution.md` and `docs/how-to/tune-entity-resolution.md`. Largest block, split it. | +| Source analysis context, `ContextId`, `PropositionQuery` (1226-1419) | Move to `docs/concepts/context-and-schema.md`, `docs/how-to/query-propositions.md`. | +| Relations, projector architecture, graph and Prolog projection (1420-1647) | Move to `docs/concepts/storage-and-projections.md`, `docs/how-to/project-to-graph.md`, `docs/features/prolog-inference.md`. | +| Agent memory, memory projection, memory maintenance (1648-1979) | Move to `docs/how-to/agent-memory.md`, `docs/concepts/knowledge-hygiene.md`. | +| Proposition operations, Oracle (1980-2090) | Move to `docs/how-to/query-propositions.md`, `docs/how-to/oracle.md`. | +| Package structure (2091-2205) | Move to `docs/reference/package-structure.md`. | +| REST API and endpoints (2206-2334) | Move to `docs/features/web-api.md`. | +| Spring Boot integration, graph-backed storage, API-key security (2335-2572) | Move to `docs/how-to/choose-a-backend.md`, `docs/reference/configuration-properties.md`, `docs/features/web-api.md`. | +| Installation (2573-2585) | Keep as coordinates only. The working version lives in the quickstart. | +| Technology stack, references, license (2586-2621) | Keep. | + +Content moves: a section is deleted from the README as it lands under `docs/`, so one copy exists, +and it leaves a one-line link where it was so an existing bookmark still lands somewhere useful. +One PR per destination page. + +## Concept pages + +Read in order. Each page uses terms the page before it defines, and ends with a "Try it now" block +of five to ten lines that runs against the quickstart's setup, with the output to expect. + +1. `propositions.md`: claims as the system of record, with confidence, importance and decay. +2. `entity-resolution.md`: mentions matched to entities or minted as new, and the resolver chain. +3. `storage-and-projections.md`: the `PropositionStore` SPI, its repositories, and the views. +4. `context-and-schema.md`: `ContextId` scoping, the `DataDictionary`, and `SchemaAdherence`. +5. `knowledge-hygiene.md`: admission gates, reclamation and consolidation. + +## Feature pages + +One page per opt-in surface, opening with the activation condition: the exact bean or property that +switches the feature on. Template, in order: + +- Availability: DICE version, modules, cost to add. +- Activation condition: the bean or property, with the value that turns it on. +- When to use it, and when to leave it off. +- Impact: latency, memory, extra infrastructure, extra LLM calls. +- How to enable: full working configuration. +- Example: real code with real output. + +| Surface | Activation condition | +|---|---| +| Graph-backed storage | `embabel.dice.store.type=graph` | +| Vector index on Neo4j | `embabel.dice.store.vector-index.enabled=true` | +| Web API | `@Import(DiceRestConfiguration.class)` plus the beans the controllers need | +| API-key security | `dice.security.api-key.enabled=true` | +| Decay and stale pruning | `embabel.dice.store.decay.enabled=true`, with `prune-stale` false by default | +| Multi-signal collector | `embabel.dice.collector.enabled`, on unless set to false | +| Prolog inference | A `PrologProjector` bean. Experimental. | +| Concurrent extraction | A parallel or batched `ExtractionExecutionStrategy` on the pipeline | + +## Quickstart + +DICE publishes `dice`, `dice-ingestion`, `dice-storage`, `dice-storage-autoconfigure` and +`dice-report`, and no aggregator starter. Decide the coordinates before writing the page: + +- Add a `dice-spring-boot-starter` depending on `dice-storage-autoconfigure` and `dice-report`, + publish it, and the quickstart uses one coordinate. Needs a release first. +- Write against `dice-storage-autoconfigure` and `dice-report`, both explicit, with a comment + saying what each buys. Two coordinates, available today. + +1. Dependencies: the coordinates chosen above, at the current DICE version. +2. Configuration: an LLM provider inherited from embabel-agent, and the default in-memory store. +3. What autoconfiguration provides: the beans that now exist and what each is for. +4. Extract: one paragraph of text through `PropositionPipeline`, with confidence printed. +5. Persist: `persist(propositionRepository, namedEntityDataRepository)` on the `PersistablePropositions` the pipeline hands back unsaved. +6. Query: retrieve what was stored, by entity and by `ContextId`. +7. Report: produce one human-readable artifact through `dice-report`. +8. Where to go next: links into `docs/concepts/` and `docs/how-to/`. + +Constraints: + +- No Neo4j, no Docker, no external resource to create. +- `InMemoryPropositionRepository` needs an `EmbeddingService` for vector search, so the minimal path queries by entity and `ContextId`. +- Every snippet compiles, and a test in `dice-integration-tests` runs the page's exact code. +- Anything needing a decision (backend, resolver chain, schema) takes the default and links out. + +## Compatibility matrix + +Its own page under `docs/support/`, linked from the README. Four axes, plus a feature-availability +column. Values get pinned at the 1.0 release. + +| Axis | What we state | +|---|---| +| embabel-agent | Supported version range per DICE release, with the tested point release named. The build currently tracks 1.5.0-SNAPSHOT. | +| Spring Boot | Minimum and maximum tested, per DICE release. | +| JDK | Baseline is 21, inherited from `embabel-build-parent` and used by CI. Newer JDKs listed as tested or untested. | +| Neo4j | Version floor for the graph backend, reached through Drivine (`drivine4j-spring-boot-starter` 0.0.79). The in-memory backend runs without Neo4j. | + +## Docs rule + +Every feature PR ships its developer-doc page and its design-doc delta. Consumer-visible behaviour +updates the page under `docs/`, new rationale updates the `docs/design/` note, an opt-in feature +gets a feature page carrying its activation condition, a new property updates +`docs/reference/configuration-properties.md`, and a version-support change updates the compatibility +matrix. Internal refactors with no consumer-visible change, test-only changes and build changes are +exempt. Reviewers ask one question: with only this PR's docs, could a consumer use the feature? A no +is a blocking finding. A promise of later docs does not clear it. + +## Open questions + +- Starter module or two coordinates? Recommendation: two coordinates, which needs no release. +- Rendered site or Markdown in the repo? Recommendation: Markdown, revisited once the tree fills. +- Does DICE ship a CLI (schema validation, drift check)? Recommendation: no CLI before 1.0, and its + own quickstart if one lands. +- Promote `docs/design/` notes into concept pages? Recommendation: they stay contributor-facing, + and concept pages lift explanations from them and link back. From 5314a8d3adf432d5205e27f3f6349b1ecb2e73eb Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:29:21 -0400 Subject: [PATCH 6/6] Answer the review on the two specs Budget policy sits above DICE because embabel-agent owns the budget, not because a host has to build one. Multimodal input already exists upstream, so the out-of-scope line says what is actually left. Docs follow the framework and use AsciiDoc. --- specs/competitive-positioning.md | 10 +++++++-- specs/readme-to-docs-tree.md | 38 ++++++++++++++++++-------------- 2 files changed, 29 insertions(+), 19 deletions(-) diff --git a/specs/competitive-positioning.md b/specs/competitive-positioning.md index fdc57d8b..0149d8d2 100644 --- a/specs/competitive-positioning.md +++ b/specs/competitive-positioning.md @@ -246,7 +246,10 @@ retrieval ranking all depend on. - **Which working-memory capabilities belong in DICE and which in the consuming application.** Recommendation: activation ranking and per-turn lifecycle events in DICE, prompt-region and budget - policy in the application, since only the application knows its token budget. + policy above it. `embabel-agent` already carries the budget: `Budget` on `ProcessOptions` sets + cost, action and token caps and turns them into an `EarlyTerminationPolicy`, and consumption is + visible through the tool loop callbacks and the invocation events. DICE reads that ceiling and + does not define its own. - **Whether to close the schema version-identity gap.** Recommendation: treat it as a candidate module. No surveyed system offers it and no surveyed vendor has published roadmap intent, so nothing external forces the timing. @@ -261,7 +264,10 @@ retrieval ranking all depend on. - **LangMem's prompt optimisation.** Orthogonal to memory quality. - **Mem0's separate graph pipeline.** DICE has entity mentions plus a Neo4j projection over the same propositions. -- **Google's multimodal extraction.** The proposition model is format-agnostic. Add on demand. +- **Google's multimodal extraction.** Multimodal input is already available upstream + (`MultimodalContent` over images and documents, embabel/embabel-agent#43) and the proposition + model is format-agnostic, so feeding non-text material to extraction is wiring. Output support is + tracked upstream in embabel/embabel-agent#42. Add on demand. - **AWS's episodic reflection.** The abstraction pipeline already synthesises across propositions. - **Neo4j's POLE+O ontology.** Domain-specific subtypes. The proposition model is domain-agnostic. - **Neo4j's spaCy to GLiNER to LLM cascade.** Cost-effective and operationally heavy: model diff --git a/specs/readme-to-docs-tree.md b/specs/readme-to-docs-tree.md index 531791fa..4b71c57b 100644 --- a/specs/readme-to-docs-tree.md +++ b/specs/readme-to-docs-tree.md @@ -7,7 +7,7 @@ and feature pages, and the rule that keeps pages current. Writing the pages is o | Directory | What goes there | |---|---| -| `docs/README.md` | What DICE is, when to use it, links into the tree. | +| `docs/README.adoc` | What DICE is, when to use it, links into the tree. | | `docs/quickstart/` | One page, one path: dependencies through first report in 15 minutes. | | `docs/concepts/` | Five pages, read in order, propositions through knowledge hygiene. | | `docs/how-to/` | Task-shaped pages titled by what the reader wants, prerequisites at the top. | @@ -32,15 +32,15 @@ and feature pages, and the rule that keeps pages current. Writing the pages is o | What is DICE, benefits table, architecture overview, design notes (24-116) | Keep, trimmed. This is the landing page's job. | | Real-world example: Impromptu (117-126) | Keep, cut to a paragraph plus a link. | | Pipeline setup, conversation analysis (127-174) | Move to `docs/quickstart/`. | -| Key features, proposition pipeline, content dedup, mention filtering (175-489) | Move to `docs/concepts/propositions.md`, `docs/how-to/extract-from-documents.md`, `docs/how-to/mention-filtering.md`. | -| Entity extraction, entity resolution, resolution service (490-1225) | Move to `docs/concepts/entity-resolution.md` and `docs/how-to/tune-entity-resolution.md`. Largest block, split it. | -| Source analysis context, `ContextId`, `PropositionQuery` (1226-1419) | Move to `docs/concepts/context-and-schema.md`, `docs/how-to/query-propositions.md`. | -| Relations, projector architecture, graph and Prolog projection (1420-1647) | Move to `docs/concepts/storage-and-projections.md`, `docs/how-to/project-to-graph.md`, `docs/features/prolog-inference.md`. | -| Agent memory, memory projection, memory maintenance (1648-1979) | Move to `docs/how-to/agent-memory.md`, `docs/concepts/knowledge-hygiene.md`. | -| Proposition operations, Oracle (1980-2090) | Move to `docs/how-to/query-propositions.md`, `docs/how-to/oracle.md`. | -| Package structure (2091-2205) | Move to `docs/reference/package-structure.md`. | -| REST API and endpoints (2206-2334) | Move to `docs/features/web-api.md`. | -| Spring Boot integration, graph-backed storage, API-key security (2335-2572) | Move to `docs/how-to/choose-a-backend.md`, `docs/reference/configuration-properties.md`, `docs/features/web-api.md`. | +| Key features, proposition pipeline, content dedup, mention filtering (175-489) | Move to `docs/concepts/propositions.adoc`, `docs/how-to/extract-from-documents.adoc`, `docs/how-to/mention-filtering.adoc`. | +| Entity extraction, entity resolution, resolution service (490-1225) | Move to `docs/concepts/entity-resolution.adoc` and `docs/how-to/tune-entity-resolution.adoc`. Largest block, split it. | +| Source analysis context, `ContextId`, `PropositionQuery` (1226-1419) | Move to `docs/concepts/context-and-schema.adoc`, `docs/how-to/query-propositions.adoc`. | +| Relations, projector architecture, graph and Prolog projection (1420-1647) | Move to `docs/concepts/storage-and-projections.adoc`, `docs/how-to/project-to-graph.adoc`, `docs/features/prolog-inference.adoc`. | +| Agent memory, memory projection, memory maintenance (1648-1979) | Move to `docs/how-to/agent-memory.adoc`, `docs/concepts/knowledge-hygiene.adoc`. | +| Proposition operations, Oracle (1980-2090) | Move to `docs/how-to/query-propositions.adoc`, `docs/how-to/oracle.adoc`. | +| Package structure (2091-2205) | Move to `docs/reference/package-structure.adoc`. | +| REST API and endpoints (2206-2334) | Move to `docs/features/web-api.adoc`. | +| Spring Boot integration, graph-backed storage, API-key security (2335-2572) | Move to `docs/how-to/choose-a-backend.adoc`, `docs/reference/configuration-properties.adoc`, `docs/features/web-api.adoc`. | | Installation (2573-2585) | Keep as coordinates only. The working version lives in the quickstart. | | Technology stack, references, license (2586-2621) | Keep. | @@ -53,11 +53,11 @@ One PR per destination page. Read in order. Each page uses terms the page before it defines, and ends with a "Try it now" block of five to ten lines that runs against the quickstart's setup, with the output to expect. -1. `propositions.md`: claims as the system of record, with confidence, importance and decay. -2. `entity-resolution.md`: mentions matched to entities or minted as new, and the resolver chain. -3. `storage-and-projections.md`: the `PropositionStore` SPI, its repositories, and the views. -4. `context-and-schema.md`: `ContextId` scoping, the `DataDictionary`, and `SchemaAdherence`. -5. `knowledge-hygiene.md`: admission gates, reclamation and consolidation. +1. `propositions.adoc`: claims as the system of record, with confidence, importance and decay. +2. `entity-resolution.adoc`: mentions matched to entities or minted as new, and the resolver chain. +3. `storage-and-projections.adoc`: the `PropositionStore` SPI, its repositories, and the views. +4. `context-and-schema.adoc`: `ContextId` scoping, the `DataDictionary`, and `SchemaAdherence`. +5. `knowledge-hygiene.adoc`: admission gates, reclamation and consolidation. ## Feature pages @@ -125,7 +125,7 @@ column. Values get pinned at the 1.0 release. Every feature PR ships its developer-doc page and its design-doc delta. Consumer-visible behaviour updates the page under `docs/`, new rationale updates the `docs/design/` note, an opt-in feature gets a feature page carrying its activation condition, a new property updates -`docs/reference/configuration-properties.md`, and a version-support change updates the compatibility +`docs/reference/configuration-properties.adoc`, and a version-support change updates the compatibility matrix. Internal refactors with no consumer-visible change, test-only changes and build changes are exempt. Reviewers ask one question: with only this PR's docs, could a consumer use the feature? A no is a blocking finding. A promise of later docs does not clear it. @@ -133,7 +133,11 @@ is a blocking finding. A promise of later docs does not clear it. ## Open questions - Starter module or two coordinates? Recommendation: two coordinates, which needs no release. -- Rendered site or Markdown in the repo? Recommendation: Markdown, revisited once the tree fills. +- Which format, and rendered where? Decided: AsciiDoc, matching the framework reference, which + builds from `embabel-agent-docs/src/main/asciidoc` through the asciidoctor Maven plugin. GitHub + renders `.adoc` in the repo, so the tree stays readable before any site exists, and the existing + Markdown under `docs/design/` converts as those notes get touched. Whether DICE renders its own + site or publishes into the framework's is still open. - Does DICE ship a CLI (schema validation, drift check)? Recommendation: no CLI before 1.0, and its own quickstart if one lands. - Promote `docs/design/` notes into concept pages? Recommendation: they stay contributor-facing,