diff --git a/specs/competitive-positioning.md b/specs/competitive-positioning.md index a71c78c3..0149d8d2 100644 --- a/specs/competitive-positioning.md +++ b/specs/competitive-positioning.md @@ -1,198 +1,277 @@ -# DICE Competitive Positioning +# DICE competitive positioning -Based on deep analysis of Zep/Graphiti, Mem0, LangChain/LangMem, Google Vertex AI Memory Bank, AWS Bedrock AgentCore Memory, Microsoft Foundry Agent Service, and Neo4j Agent Memory. +Survey date: mid-2026. Zep/Graphiti, Mem0, Letta (MemGPT), Cognee, Hindsight, LangMem, Neo4j agent +memory (labs SDK and NAMS), Google Vertex AI Memory Bank, AWS Bedrock AgentCore Memory, Microsoft +Foundry. Several of them publish roadmap intent and shipped behaviour in the same posts, so +re-verify a claim before reusing it. -## DICE Strengths +## What DICE is -| Capability | DICE Implementation | Competitors | -|---|---|---| -| Batch classification | N propositions in 1 LLM call via `classifyBatch()` | Zep: sequential per-edge. Mem0: sequential per-fact. LangMem: sequential tool calls. | -| Auto-merge fast path | Embedding score >= 0.95 skips LLM entirely | Zep: text-identical fast path only (no similarity threshold). Others: none. | -| Outcome-dependent decay | Contradicted +0.15, merged *0.7, reinforced *0.85 | Zep: binary temporal invalidation. Mem0: hard delete. LangMem: none. | -| Canonical text dedup | Cheap string match before vector search | Zep: similar for identical edge text but only after vector search. | -| 5-way classification | IDENTICAL/SIMILAR/CONTRADICTORY/UNRELATED/GENERALIZES with edge-case guidance + few-shot | Mem0: only ADD/UPDATE/DELETE/NONE. LangMem: insert/update/delete tool calls. | -| JVM native | Only JVM-based memory system | All competitors are Python (Zep also Go). | -| Confidence + decay model | Per-proposition confidence with exponential time decay from GUM paper | Zep: no decay. Mem0: no confidence model. LangMem: asks for p(x) in prompts but no decay math. | -| Extraction quality | SNR-maximizing, confidence-qualified with hedging detection, role-aware (USER/AGENT/ALL), schema-bound | Comparable to LangMem prompts; better than Zep/Mem0. | -| Reinforcement frequency | `reinforceCount` tracks merge/reinforce frequency, queryable via `PropositionQuery` | Mem0: `mentions` counter on graph nodes. Zep/LangMem: none. Google/AWS/Microsoft: none. | -| Entity resolution | Fuzzy name, vector, exact name, partial name, agentic candidate searchers; LLM disambiguation | Zep: Neo4j entity nodes. Mem0: graph nodes. Google/AWS/Microsoft: none — flat fact strings. | -| Abstraction hierarchy | Multi-level propositions (level 0 = raw, level 1+ = synthesized) with source tracking | None of the competitors support multi-level abstraction. | -| Contradiction retention | Both propositions retained with reduced confidence | Zep: temporal invalidation. Mem0/Google/AWS/Microsoft: hard delete. LangMem: none. | -| Portability | Embeddable JVM library, no cloud dependency | Google: GCP only. AWS: AWS only. Microsoft: Azure only. Zep: Neo4j required. Mem0: self-hosted or cloud. LangMem: Python/LangGraph. | - -## vs Zep/Graphiti - -| Dimension | DICE | Zep | Edge | -|---|---|---|---| -| Ingestion speed | Batch classify + auto-merge + canonical dedup | Sequential only ("must be awaited") | **DICE** | -| Classification nuance | 5-way with edge cases + few-shot | Duplicate vs contradicted (binary) | **DICE** | -| Confidence model | Exponential decay + outcome-dependent adjustment + reinforceCount | No decay, no confidence scoring | **DICE** | -| Extraction quality | SNR, confidence-qualified, role-aware, schema-bound | Custom ontology via Pydantic, entity validation | Tie | -| Temporal model | System timestamps only (created/revised) | Bi-temporal (valid_at/invalid_at/expired_at) | **Zep** | -| Graph structure | Propositions + entity mentions, no graph DB required | Full knowledge graph in Neo4j with community detection | **Zep** | -| Retrieval | Vector similarity + canonical match | Cosine + BM25 + BFS + 5 rerankers | **Zep** | -| Infrastructure weight | Embeddable, JVM-native, no external deps | Requires Neo4j + embedding service + LLM | **DICE** | - -**Their moat**: Bi-temporal fact model, custom ontology via Pydantic, 5 reranking strategies, community subgraph summaries, Neo4j-backed graph traversal. +DICE is Domain-Integrated Context Engineering. It uses a domain model to structure the context an +LLM reads, and applies the same structure to what the LLM produces (README.md:24-36). -**Their weakness**: Sequential-only ingestion ("episodes must be added sequentially and awaited"), Python/Go only, heavy infrastructure (Neo4j required), no batch classification. +Natural language propositions are the system of record. Everything else derives from them: a Neo4j +graph, a Prolog fact base, vector embeddings, agent working memory, and reports (README.md:48-92, +`docs/design/architecture.md`). -**Attack angle**: DICE is embeddable — no Neo4j dependency. Batch pipeline is faster for high-throughput ingestion. Position as "memory for JVM agents" vs their "memory infrastructure platform." Bi-temporal model (GAP-4B) closes their biggest technical advantage. - -## vs Mem0 - -| Dimension | DICE | Mem0 | Edge | -|---|---|---|---| -| Classification | 5-way taxonomy with edge-case guidance | 4-op (ADD/UPDATE/DELETE/NONE) | **DICE** | -| Dedup pipeline | Canonical + auto-merge + batch LLM | Sequential per-fact, top-5 candidates | **DICE** | -| Frequency signal | `reinforceCount` on propositions, queryable | `mentions` counter on graph nodes/edges | Tie | -| Confidence model | Decay + outcome adjustment + qualification at extraction | None — no confidence scores | **DICE** | -| ID safety | Integer re-indexing prevents hallucination | Integer re-indexing prevents hallucination | Tie | -| Role-aware extraction | `ExtractionPerspective` enum (ALL/USER/AGENT) | Separate user vs agent prompts with penalty framing | Tie | -| Graph memory | No graph DB | Neo4j/Memgraph/Neptune/Kuzu | **Mem0** | -| Audit trail | Grounding chain + reinforceCount | Full SQLite history (old/new/event/actor) | **Mem0** | +A proposition carries: -**Their moat**: Graph memory with Neo4j/Memgraph/Neptune/Kuzu, vision support, procedural memory for agent traces, mentions counting. +- `confidence` and `importance`, each a `ZeroToOne` + (`dice/src/main/kotlin/com/embabel/dice/proposition/Proposition.kt:101-103`). +- `decay`. Effective confidence falls exponentially with age, anchored at the later of + `contentRevised` and `lastAccessed`, so using a claim refreshes it (`Proposition.kt:358-403`). +- A status: ACTIVE, SUPERSEDED, CONTRADICTED, PROMOTED, STALE. Reinforcement lifts a STALE + proposition back to ACTIVE (`Proposition.kt:31-54`). +- `grounding`. References to the source chunks the claim was extracted from. +- `provenanceEntries`. Typed source locator, chunk id, character offsets, content hash + (`dice/src/main/kotlin/com/embabel/dice/provenance/ProvenanceEntry.kt:18-52`). +- `reinforceCount`. How often the claim has been re-observed (`Proposition.kt:118`). -**Their weakness**: Coarse 4-operation model (ADD/UPDATE/DELETE/NONE) — no SIMILAR/GENERALIZES distinction. Sequential per-fact processing. Graph memory is a separate bolted-on pipeline. +The confidence-weighted proposition and the exponential decay formula come from "Creating General +User Models from Computer Use" (Shaikh, Sapkota, Rizvi, Horvitz, Park, Yang, Bernstein; +arXiv:2505.10831, UIST 2025, ACM DOI 10.1145/3746059.3747722), cited in the README beside the decay +maths. GUM is user-modelling research. -**Attack angle**: DICE's 5-way classification preserves nuance that Mem0's 4-op model loses. Unified revision pipeline vs their split vector+graph paths. ID hallucination prevention (GAP-6) adopts their best defensive technique. +## Unit of knowledge -## vs LangChain/LangMem - -| Dimension | DICE | LangMem | Edge | +| System | Unit of knowledge | What the unit carries | +|---|---|---| +| **Zep / Graphiti** | Episodes ingested into a temporal knowledge graph of entity nodes and edges | Bi-temporal edges: `valid_from`/`valid_until` for world time plus ingestion time; entity and edge attributes typed by Pydantic models; cross-session entity dedup | +| **Mem0** | Extracted facts, short natural language strings | Full SQLite change history (old value, new value, event, actor), mention counts. No confidence | +| **Letta (MemGPT)** | Memory blocks in core context (label, value, limit, description) and archival passages in Postgres/pgvector | Free text. The agent decides promotion and conflict resolution through tool calls | +| **Cognee** | LLM-extracted nodes and RDF triples | An `ontology_valid` flag from matching against a declared OWL/RDF ontology. No confidence or decay | +| **Neo4j agent-memory (labs SDK, NAMS)** | Entity and fact nodes, plus `(Message)` nodes chained by `[:NEXT]` per session | POLE+O typing, `valid_from`/`valid_until`, geospatial attributes | +| **LangMem** | Semantic, episodic and procedural memory items | Optional typed Pydantic profiles; consolidation state | +| **Hindsight** | Structured facts in a knowledge graph | Entity resolution that links "Alice" to "my coworker Alice" | +| **Google / AWS / Microsoft** | Extracted memories as flat fact strings or items | A strategy or topic label and an IAM scope | +| **DICE** | Proposition | Confidence, importance, decay, status, grounding, provenance entries, reinforce count | + +Three of those units carry a field DICE's proposition lacks. + +- **Bi-temporal edges (Zep / Graphiti).** World time is stored separately from ingestion time. + DICE's `TemporalMetadata` carries `observedAt`, `validFrom`, `validTo` and `invalidatedAt`, with + no transaction-time axis, so a query for what the store believed at a past instant has no basis. +- **Ontology grounding (Cognee).** Extracted entity and type names resolve against a declared + OWL/RDF ontology, with fuzzy matching, before any graph node is built + (https://docs.cognee.ai/core-concepts/ontologies). DICE declares types in a `DataDictionary` and + applies them at extraction, with no resolution against a stored ontology. +- **Change history (Mem0).** Old value, new value, event and actor per change. DICE records why a + collapse or merge decision was made + (`dice/src/main/kotlin/com/embabel/dice/spi/CollectorSignals.kt:115`) and where a claim came from + (`ProvenanceEntry`). Neither carries an extraction-run identifier, so a stored claim has no link + back to the run that produced it. + +## Projections + +Every DICE projection derives from the one proposition set and can be rebuilt from it, so adding a +representation means adding a projector: `dice-storage` for the Neo4j graph, the Prolog fact base +and lineage, `dice/src/main/kotlin/com/embabel/dice/projection/memory/MemoryProjector.kt:46` for +agent working memory, `dice-report` for rationale and structured reports. + +| System | Where knowledge lives | Retrieval | +|---|---|---| +| **Zep / Graphiti** | Neo4j property graph, with Leiden community clustering and summaries over it | Cosine plus BM25 plus BFS, with five reranking strategies | +| **Mem0** | A vector store, plus a separate graph pipeline over Neo4j, Memgraph, Neptune or Kuzu | Vector search; the graph pipeline runs its own extraction path | +| **Letta** | pgvector passages | Agent tool calls | +| **Cognee** | RDF-grounded graph plus embeddings | Search returns nodes | +| **Neo4j agent memory** | Native property graph | Hybrid vector plus up to 3-hop traversal | +| **LangMem** | Embedding space | Dilated-window retrieval | +| **Google / AWS / Microsoft** | Stored memories, with no exposed intermediate model | Similarity search | +| **DICE** | Propositions, projected into Neo4j, Prolog, vectors, memory and reports | `RetrievalRouter` over vector, entity, graph walk, temporal and hybrid modes | + +In every surveyed system the store is the knowledge model, so a second representation costs a second +extraction path or a migration. Mem0's graph pipeline is such a second representation, fed by its +own extraction over the source material. + +## Knowledge hygiene + +DICE splits hygiene into three interventions at three moments (`docs/design/knowledge-hygiene.md`). + +- **Admission.** Gates run at extraction: confidence qualification, deduplication against canonical + and stored propositions, conflict classification, trust scoring. Gating costs less than removing + junk later, and a low-confidence fact is easiest to judge while its extraction context is close. +- **Reclamation.** `DecaySweepPass` retires softly to STALE and never hard-deletes, with + dual-threshold hysteresis: stale below 0.1 effective confidence, recovery edge at 0.25 + (`dice/src/main/kotlin/com/embabel/dice/operations/consolidation/DecaySweepPass.kt:63-64`). + Revival happens only on reinforcement. +- **Consolidation.** Between-session passes. `ContradictionResolutionPass` retires the weaker of a + contradictory pair to CONTRADICTED, auto-merge collapses duplicates, abstraction synthesises + higher-level propositions from groups. + +The four managed services also form memory off the request path. + +| Service | Background formation | +|---|---| +| Bedrock AgentCore | Extraction then consolidation as background processes, with start and completion marks and success counts in the logs ([metrics](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html)) | +| Vertex AI Memory Bank | Asynchronous extraction, ingested via `add_memory()` or an end-of-conversation callback ([announcement](https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview), 2025-07-08) | +| LangMem | Hot-path tools or background managers ([writeup](https://rywalker.com/research/langmem)) | +| Neo4j | Background enrichment and a multi-stage extraction cascade ([labs](https://neo4j.com/labs/agent-memory/)) | + +No surveyed system documents a hard TTL delete. Decay, supersession and consolidation are the common +answer. + +Two DICE hygiene mechanisms are absent from every other system surveyed: a lifecycle status on the +stored unit, and a retirement path that keeps the retired unit. + +- Vertex AI Memory Bank deletes the superseded memory on contradiction. +- Microsoft Foundry discards the old value. +- Bedrock AgentCore writes a new entry, with no contradiction detection. +- Mem0 v3 (April 2026) is ADD-only, so supersession and contradiction are inexpressible in the model. +- Cognee keeps its non-conforming nodes. They are "stored, embedded, and returned by search exactly + like grounded ones" (https://docs.cognee.ai/core-concepts/ontologies), so the flag has no effect + on reads. + +## Schema governance + +Declaring a schema for extraction is common. What happens to the declaration varies. + +| System | Declaration | Enforcement | +|---|---|---| +| **Graphiti** | Pydantic entity and edge models; `set_ontology` with a `strict_ontology` flag | "Each entity is validated against the appropriate Pydantic model" before graph construction ([docs](https://help.getzep.com/graphiti/core-concepts/custom-entity-and-edge-types)) | +| **Cognee** | An OWL/RDF ontology file | Checked before any graph node is built. Content that fails to match is kept, tagged `ontology_valid=False`: "Nothing is rejected or discarded" ([docs](https://docs.cognee.ai/core-concepts/ontologies)) | +| **Neo4j labs SDK / NAMS** | `DomainSchema` of entity types and descriptions | Steers GLiNER extraction. No validation, versioning or rejection documented ([faq](https://neo4j.com/labs/agent-memory/faq/)) | +| **Bedrock AgentCore** | Output schema, self-managed strategy only | Built-in and overridden strategies "do not let you change the final output schema" ([docs](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory-custom-strategy.html)) | +| **Vertex Memory Bank / Mem0** | "Custom topics" and "custom categories": a label plus free-text prompt instructions, few-shot examples recommended | Prompt text, no type enforcement ([google](https://cloud.google.com/agent-builder/agent-engine/memory-bank/generate-memories); [mem0](https://docs.mem0.ai/open-source/features/custom-fact-extraction-prompt)) | +| **Letta** | None. Memory blocks are free-text segments | None | +| **Neo4j GRAPH TYPE** (general database feature, Cypher 25 preview in 2026.02, Enterprise, Infinigraph and all Aura tiers) | Nodes, labels and relationship connections via `SET`/`ADD`/`ALTER`/`DROP` | Hard-rejects a non-conforming write at write time ([blog](https://neo4j.com/blog/developer/graph-type-schema-enforcement-made-easy-preview/)) | +| **DICE** | A `DataDictionary` from embabel-agent (`com.embabel.agent.core.DataDictionary`), held by name in `SchemaRegistry` (`dice/src/main/kotlin/com/embabel/dice/common/SchemaRegistry.kt`) | `SchemaAdherence` over `entities` and `predicates` flags, with STRICT, DEFAULT and RELAXED presets (`dice/src/main/kotlin/com/embabel/dice/common/SchemaAdherence.kt:26-49`) | + +GRAPH TYPE is the strongest enforcement found, and it sits outside Neo4j's agent-memory product, +where `DomainSchema` remains unenforced extraction guidance. `SHOW CURRENT GRAPH TYPE` returns the +declaration, with no comparison against what the graph holds. + +Two capabilities are absent from every system surveyed, DICE included. No system gives an extraction +schema a version identity or history: Graphiti evolves a schema by adding attributes with no version +numbers, GRAPH TYPE's `ADD`/`ALTER`/`DROP` leave no discrete version record, and neither a DICE +declaration nor a stored proposition carries a version stamp. No system compares a declaration +against what a live store holds. No surveyed vendor has published roadmap intent for either. + +## Shared mechanisms + +Places where DICE and a surveyed system reached the same solution independently, beyond the decay +and consolidation overlap above. + +- **LLM extraction into a structured store.** All four surveyed memory services take a conversation + turn, run an LLM extraction, and persist the result into a vector, graph or relational store + (cited under knowledge hygiene above, plus + https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html). DICE's pipeline has the + same shape, with the proposition as the stored unit. +- **Integer re-indexing of proposition IDs across LLM calls.** DICE and Mem0 both renumber IDs per + call so the model cannot invent one. +- **Confidence-qualified, SNR-shaped extraction prompts.** DICE and LangMem arrived at the same + prompt controls independently (https://rywalker.com/research/langmem). + +## Remaining gaps + +Capabilities DICE lacks, against the four managed memory services: LangMem, Vertex AI Memory Bank, +Bedrock AgentCore Memory, Neo4j Agent Memory. The count column says how many of the four ship it. + +| Gap | Count | Evidence | DICE today | |---|---|---|---| -| Extraction prompts | SNR, confidence-qualified, role-aware, few-shot | Confidence-qualified, surprise-prioritized, SNR | Tie | -| Dedup/classification | Structured 5-way pipeline with fast paths | LLM tool calls (insert/update/delete), no structured classification | **DICE** | -| Batch processing | N propositions in 1 LLM call | Sequential tool calls | **DICE** | -| Prompt optimization | Not applicable | Gradient-based prompt evolution | **LangMem** | -| Retrieval | Vector similarity | Dilated windows + LLM-generated queries | **LangMem** | -| Graph memory | Entity mentions on propositions | Commented-out prototype | **DICE** | -| Background processing | Synchronous pipeline | Debounced async reflection | **LangMem** | -| Ecosystem | JVM/Spring native | Python/LangGraph locked | Depends on stack | - -**Their moat**: Excellent extraction prompts, prompt optimization via gradient analogy, debounced background reflection, dilated windows retrieval. +| Operational tooling and memory inspection | 4 | Bedrock emits CloudWatch metrics for latency, invocations, errors and memory creation count, with spans over CreateEvent, GetEvent, ListEvents, DeleteEvent and RetrieveMemoryRecords, plus extraction and consolidation logs ([metrics](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-memory-metrics.html)). Vertex lists stored memories in the Cloud Console (Memory Bank UI announcement, Google Developer Forums, no stable URL recorded). The NAMS dashboard shows health, entity count and queue lag with graph visualisation and Cypher queryability ([tour](https://medium.com/neo4j/a-tour-of-the-neo4j-agent-memory-service-nams-0f2d535a4fdb), 2026-06, GA unconfirmed). LangSmith traces execution paths and state transitions with cost, latency and error dashboards (langchain.com/resources/llm-observability-tools) | "Why was this proposition formed" needs custom logging over `CollectorTraceStore` | +| Background memory formation pipelines | 4 | Formation runs off the request path, cited under knowledge hygiene above | Async `@EventListener` on conversation analysis, a non-blocking `PropositionIncrementalAnalyzer` (`dice/src/main/kotlin/com/embabel/dice/incremental/proposition/PropositionIncrementalAnalyzer.kt`), and consolidation passes an application invokes. The scheduler that runs them is missing | +| TTL and eviction controls | 0 publish a TTL API | The AWS TTL setting and Google's retention policy are unverified | Decay rate multiplier defaults to 2.0 (`Proposition.kt:365`); sweep thresholds default to 0.1 and 0.25 (`DecaySweepPass.kt:63-64`). All are constructor parameters, so an operator has no retention policy to set | +| Procedural memory | 2 | LangMem's `procedural` type lets agents update their own prompt rules from feedback ([writeup](https://rywalker.com/research/langmem)). Neo4j stores tool usage and reasoning traces with similarity search over trace lineage ([labs](https://neo4j.com/labs/agent-memory/)). AWS and Google extract facts only | Prolog projection over propositions as a view layer, with no rule-formation path | +| Namespacing and scoping APIs | 2 | AWS uses hierarchical namespaces for fine-grained access control plus session headers `Mcp-Session-Id` and `X-Amzn-Bedrock-AgentCore-Runtime-Session-Id` ([memory](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html)). Neo4j chains `(Message)` nodes by `[:NEXT]` per session, with entity and fact nodes shared across sessions ([labs](https://neo4j.com/labs/agent-memory/)). Google scopes by identity and LangMem by embedding space, neither with a namespace API | `ContextId` is a query filter over shared storage (README.md:1238-1256). It scopes reads. The storage layer stays shared | +| Cross-session profile modelling | 4 aggregate, 0 publish a profile schema | AWS extracts user preferences, facts and session summaries across sessions. Google personalises long-term memories per user ([announcement](https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-memory-bank-in-public-preview), 2025-07-08). Neo4j persists preference and fact entities in a shared graph ([labs](https://neo4j.com/labs/agent-memory/)). LangMem keeps preferences in semantic memory with optional typed Pydantic profiles ([writeup](https://rywalker.com/research/langmem)) | `ContextId` scopes knowledge and defines no profile structure or cross-session aggregation | -**Their weakness**: Graph memory is literally commented-out code. No structured dedup pipeline — relies on LLM tool calls for consolidation. Tightly coupled to LangGraph ecosystem. +Absent from DICE and from every surveyed system: an extraction-run identifier linking a stored claim +to the run and schema that produced it; temporal anchoring, so relative dates are stored as literal +text; surprise-prioritised retention, so novel facts get no durability preference; versioning or an +audit trail on conflict policy. -**Attack angle**: LangMem has great prompts but weak infrastructure. DICE's explicit classification taxonomy + fast paths + batch processing is more reliable at scale. We've adopted their best prompt ideas and combined them with our superior pipeline mechanics. +Absent from DICE and present in Zep/Graphiti: a transaction-time axis, so DICE has no point-in-time +query over past belief and no temporal contradiction resolution. -## vs Google Vertex AI Memory Bank +## Candidate modules -| Dimension | DICE | Google Memory Bank | Edge | -|---|---|---|---| -| Data model | Rich `Proposition` with entity mentions, confidence, decay, grounding, reinforceCount | Flat `fact` string with no structured sub-components | **DICE** | -| Confidence/decay | Exponential decay from GUM paper, outcome-dependent adjustment | None — facts are binary (exist or deleted) | **DICE** | -| Entity resolution | Multi-strategy entity resolution with LLM disambiguation | None — no entity model | **DICE** | -| Classification nuance | 5-way (IDENTICAL/SIMILAR/CONTRADICTORY/UNRELATED/GENERALIZES) | 3-outcome (CREATED/UPDATED/DELETED), opaque LLM consolidation | **DICE** | -| Abstraction | Multi-level hierarchy with source tracking | None — all facts at same level | **DICE** | -| Contradiction handling | Both retained with reduced confidence | Contradicting memory deleted — history lost | **DICE** | -| Dedup transparency | Configurable thresholds + LLM classification | Opaque LLM-based consolidation, no control over merge logic | **DICE** | -| Graph projection | Entity mentions map to Neo4j relationships, Prolog facts | None | **DICE** | -| Managed service | No — self-hosted | Fully managed, zero infrastructure | **Google** | -| Multimodal | Text only | Images, video, audio extraction | **Google** | -| Topic-based extraction | Schema hints + extraction guidance | Managed + custom topics with per-topic filtering | **Google** | -| TTL | Threshold-based retirement via decay | Granular TTL (create, generate-created, generate-updated) | **Google** | -| Retrieval | Vector similarity + canonical match + entity-based queries | Euclidean distance similarity + regex/metadata filtering | **DICE** | - -**Their moat**: Fully managed GCP service with zero infrastructure overhead, multimodal extraction (images/video/audio), topic-based extraction with managed topics, built-in TTL, IAM-scoped access control, ADK integration with PreloadMemoryTool. - -**Their weakness**: Flat fact model with no entity resolution, no confidence or decay, no abstraction hierarchy, no reinforcement counting, opaque consolidation logic, `CreateMemory` bypasses dedup entirely, top_k defaults to 3, GCP vendor lock-in. +Each gap above consumes propositions and adds no field to the proposition model, so each can ship as +a separate module over the DICE state the table records. -**Attack angle**: Memory Bank solves "I don't want to build memory infrastructure" but trades away all the expressiveness that makes memory useful at scale. DICE's proposition model carries structured metadata (confidence, decay, entity mentions, grounding, reinforceCount) enabling richer consolidation, retrieval, and reasoning. For teams that need more than flat fact storage — entity-centric queries, confidence-weighted retrieval, graph projection — DICE is fundamentally more capable. +**Working memory.** A bounded active set held in a protected prompt region, ranked by an activation +score blending recency, reinforcement and decay, evicted into durable storage so evicted material +stays queryable. DICE supplies storage, extraction, decay, revision and pinning +(`dice/src/main/kotlin/com/embabel/dice/proposition/PropositionStore.kt:234-241`) as primitives. The +module adds four SPIs: -## vs AWS Bedrock AgentCore Memory +- activation ranking as a pluggable policy; +- authority and trust levels with a promotion gate; +- per-turn lifecycle events for reinforce, evict and reactivate; +- a token and unit budget over the resident set. -| Dimension | DICE | AWS AgentCore Memory | Edge | -|---|---|---|---| -| Data model | Rich `Proposition` with entity mentions, confidence, decay, grounding, reinforceCount | Flat `fact` string (`{"fact": "..."}`) with rigid output schema | **DICE** | -| Confidence/decay | Exponential decay + outcome-dependent adjustment | None — no confidence field on memory records | **DICE** | -| Entity resolution | Multi-strategy resolution with LLM disambiguation | None — facts are flat text strings, no entity linking | **DICE** | -| Classification nuance | 5-way taxonomy with edge-case guidance | 3-op consolidation (AddMemory/UpdateMemory/SkipMemory) | **DICE** | -| Contradiction handling | Both retained with reduced confidence | AddMemory creates new entry (contradictions not explicitly detected) | **DICE** | -| Provenance | Grounding chain links propositions to source chunks | None — no link back to source text | **DICE** | -| Dedup transparency | Configurable thresholds + deterministic fast paths + LLM classification | LLM-based only — non-deterministic, expensive per consolidation | **DICE** | -| Knowledge types | Explicit `KnowledgeType` enum (SEMANTIC/EPISODIC/PROCEDURAL/WORKING) with classifier | Implicit via strategy choice (semantic/preference/summary/episodic) | Tie | -| Managed service | No — self-hosted | Fully managed with control plane + data plane | **AWS** | -| Strategy tiers | N/A | Built-in, built-in with overrides, self-managed — three customization levels | **AWS** | -| Episodic memory | Propositions with EPISODIC knowledge type | Automatic episode detection with cross-episode reflection | **AWS** | -| Session management | Application-managed | Built-in session/actor model with branching | **AWS** | -| Framework integrations | JVM/Spring native | LangChain, LangGraph, AutoGen, Strands out of the box | **AWS** | -| Retrieval | Vector similarity + canonical match + entity-based + composable PropositionQuery | Cosine similarity only on flat text with metadata filters | **DICE** | -| Sync extraction | Synchronous pipeline | Async-only with `time.sleep(60)` in examples | **DICE** | - -**Their moat**: Fully managed AWS service with three strategy tiers (built-in → override → self-managed), episodic strategy with automatic episode detection and cross-episode reflection, session/actor model with branching, CDK/IaC support, framework integrations (LangChain/AutoGen/Strands). - -**Their weakness**: Flat `{"fact": "..."}` model with no entity resolution. No confidence/decay — all memories equally weighted. No provenance tracking. LLM-dependent deduplication is expensive and non-deterministic. Built-in schema is rigid and non-editable. Async-only extraction means stored information isn't immediately available. Deep AWS vendor lock-in (IAM, S3, SNS, KMS, Lambda). - -**Attack angle**: AgentCore Memory is a reasonable managed service for simple preference/fact storage in chatbots. But for systems that need structured knowledge with entity resolution, confidence-weighted decaying memory, provenance tracking, deterministic deduplication, and multi-strategy retrieval, DICE's proposition model is fundamentally more expressive. Both Google and AWS validate that agent memory is a critical capability — DICE provides it without vendor lock-in and with far richer knowledge representation. - -## vs Microsoft Foundry Agent Service - -| Dimension | DICE | Microsoft Foundry | Edge | -|---|---|---|---| -| Data model | Rich `Proposition` with entity mentions, confidence, decay, grounding, reinforceCount | Flat memory "items" — no structured sub-components | **DICE** | -| Memory types | Unified proposition model with `KnowledgeType` classifier (SEMANTIC/EPISODIC/PROCEDURAL/WORKING) | Two types only: user profile (static preferences) and chat summary (distilled conversation) | **DICE** | -| Confidence/decay | Exponential decay + outcome-dependent adjustment | None — memories are binary (exist or removed) | **DICE** | -| Entity resolution | Multi-strategy resolution with LLM disambiguation | None — no entity model | **DICE** | -| Classification nuance | 5-way (IDENTICAL/SIMILAR/CONTRADICTORY/UNRELATED/GENERALIZES) | Opaque LLM-based consolidation, no user-visible classification taxonomy | **DICE** | -| Contradiction handling | Both retained with reduced confidence | "Conflicting facts are resolved" — old value discarded | **DICE** | -| Abstraction | Multi-level hierarchy with source tracking | None — all memories at same level | **DICE** | -| Graph projection | Entity mentions map to Neo4j relationships, Prolog facts | None | **DICE** | -| Dedup transparency | Configurable thresholds + deterministic fast paths + LLM classification | Opaque LLM-based consolidation, "behavior can vary by memory type and may change during preview" | **DICE** | -| Retrieval | Vector similarity + canonical match + entity-based + composable PropositionQuery | Memory search (details opaque), scope-based filtering | **DICE** | -| Managed service | No — self-hosted | Fully managed Azure service, zero infrastructure | **Microsoft** | -| Integration model | JVM/Spring native, embeddable library | Memory search tool auto-attached to prompt agents, or low-level Memory Store APIs | **Microsoft** | -| Scale limits | Application-determined | 100 scopes/store, 10K memories/scope, 1K req/min | **DICE** | +**Session memory management.** Session-scoped active-context assembly and summarisation over +`MemoryProjector`, which classifies propositions by knowledge type for prompt injection +(`MemoryProjector.kt:46`), and the `Memory` tool, which runs hybrid vector plus keyword retrieval +over context-scoped propositions (`dice/src/main/kotlin/com/embabel/dice/agent/Memory.kt:112`). -**Their moat**: Fully managed Azure service with zero infrastructure overhead, two-method access (agent tool for simple use, APIs for advanced), integration with Azure AI Content Safety for prompt injection detection, built-in scope-based multi-tenancy. +**Background consolidation.** A scheduler over the existing passes, the +extraction-then-consolidation staging AWS and Google run, and lifecycle logging that makes a run +inspectable. -**Their weakness**: Simplest memory model of any competitor — only two memory types (user profile + chat summary), no entity resolution, no confidence or decay, no abstraction hierarchy, no reinforcement counting, opaque consolidation logic that "may change during preview," hard 10K memory limit per scope, Azure vendor lock-in. +**Procedural memory.** A formation path over the existing Prolog projection, turning agent feedback +into stored rules that later runs read back. -**Attack angle**: Foundry's memory is the thinnest of the managed offerings — even simpler than Google Memory Bank or AWS AgentCore. It's adequate for remembering user preferences across chatbot sessions, but lacks the structured knowledge representation needed for serious agent memory. DICE's proposition model, entity resolution, confidence decay, and composable queries operate in a fundamentally different capability tier. Microsoft validates the market need but their implementation is a minimal viable feature, not a memory system. +## Orthogonal research -## vs Neo4j Agent Memory +Mechanisms from non-LLM fields that map onto problems DICE has. -| Dimension | DICE | Neo4j Agent Memory | Edge | +| Field | Mechanism | DICE analog | Citation | |---|---|---|---| -| Classification nuance | 5-way with edge cases + few-shot | No explicit classification taxonomy — entity resolution handles dedup | **DICE** | -| Batch processing | N propositions in 1 LLM call | Sequential extraction through cascade stages | **DICE** | -| Confidence/decay | Exponential decay + outcome-dependent adjustment + reinforceCount | No decay model — entities are binary (exist or merged) | **DICE** | -| Contradiction handling | Both retained with reduced confidence | Entities merged or left distinct — no contradiction retention | **DICE** | -| Abstraction hierarchy | Multi-level propositions with source tracking | Flat — all entities/facts at same level | **DICE** | -| Extraction pipeline | Single LLM call, SNR-maximizing | spaCy → GLiNER → LLM cascade with 5 merge strategies | **Neo4j** | -| Graph structure | Propositions + entity mentions, Neo4j as projection | Full native Neo4j knowledge graph with POLE+O ontology | **Neo4j** | -| Temporal model | System timestamps only | Facts with valid_from/valid_until + geospatial queries | **Neo4j** | -| Reasoning traces | Not applicable | Trace → Step → ToolCall hierarchy with aggregated tool stats | **Neo4j** | -| Retrieval | Vector similarity + canonical match + entity-based + composable query | Hybrid vector + graph traversal (up to 3 hops) | **Neo4j** | -| Framework integrations | JVM/Spring native | 9 frameworks (LangChain, LlamaIndex, CrewAI, etc.) + MCP server | **Neo4j** | -| Infrastructure weight | Embeddable JVM library, no external deps | Requires Neo4j 5.11+ plus spaCy/GLiNER models | **DICE** | -| Observability | Application-managed | OpenTelemetry + Opik built-in | **Neo4j** | - -**Their moat**: Native Neo4j graph with multi-hop traversal and POLE+O ontology, multi-stage extraction cascade (spaCy → GLiNER → LLM) for cost/quality tradeoffs, reasoning memory with tool call statistics, 9 framework adapters + MCP server, geospatial and temporal fact queries, entity enrichment via Wikipedia/Diffbot. - -**Their weakness**: Neo4j 5.11+ hard dependency (significant infrastructure commitment). No classification taxonomy — entities merge or stay distinct with no SIMILAR/CONTRADICTORY/GENERALIZES nuance. No confidence decay — everything equally weighted forever. No abstraction hierarchy. No batch classification. No contradiction retention. Python only. Experimental Labs project with no SLAs. - -**Attack angle**: Neo4j Agent Memory is the most architecturally sophisticated competitor — both systems take knowledge representation seriously. But the competition is proposition-centric (DICE) vs entity-centric (Neo4j). DICE manages the **lifecycle** of knowledge claims — how they evolve, conflict, reinforce, and decay. Neo4j builds a **static graph** of entities and relationships. DICE's 5-way classification, confidence decay, and abstraction hierarchy address the harder problem of knowledge evolution. And DICE is embeddable with no infrastructure requirements — it already projects to Neo4j when graph structure is needed, without requiring it. - -## Key Remaining Gaps - -| Gap | Blocks us against | Impact | -|---|---|---| -| ~~ID hallucination prevention (GAP-6)~~ | ~~Mem0~~ | ~~DONE — integer re-indexing~~ | -| Surprise-prioritized retention (GAP-2) | LangMem | Novel facts don't get durable treatment | -| Temporal anchoring (GAP-4A) | Zep | Relative dates stored as literal text | -| Bi-temporal model (GAP-4B) | Zep | No point-in-time queries or temporal contradiction resolution | - -## Not Worth Chasing - -- **Zep's 5-reranker retrieval**: Deep feature tied to Neo4j graph traversal. Better to invest in bi-temporal model. -- **LangMem's prompt optimization**: Gradient-based prompt evolution is interesting but orthogonal to memory quality. DICE's pipeline mechanics matter more. -- **Mem0's graph memory**: DICE already has entity mentions + Neo4j projection via the graph projector. Adding a separate graph memory pipeline would duplicate effort. -- **Google's multimodal extraction**: Interesting for image/video-heavy use cases but orthogonal to memory quality. Can be added later if needed — the proposition model is format-agnostic. -- **AWS's episodic reflection**: Cross-episode insight generation is valuable but DICE's abstraction pipeline already synthesizes higher-level insights from proposition groups. Different mechanism, similar outcome. -- **Microsoft's user profile/chat summary model**: The simplest memory system of any competitor — just two flat memory types. Even less capable than Google or AWS. -- **Neo4j Agent Memory's POLE+O ontology**: Domain-specific entity subtypes (Person → Suspect/Witness/Victim) are useful for law enforcement/intelligence domains but add complexity for general-purpose memory. DICE's proposition model is domain-agnostic by design. -- **Neo4j Agent Memory's reasoning traces**: Capturing tool call statistics and decision workflows is interesting but orthogonal to memory quality. Could be added as a projection type if needed. -- **Neo4j Agent Memory's multi-stage extraction cascade**: Their spaCy → GLiNER → LLM pipeline is cost-effective but adds operational complexity (model downloads, dependency management). DICE's single LLM call is simpler; the cost savings don't justify the complexity for proposition extraction. -- **Managed service hosting**: Google, AWS, Microsoft, and Neo4j all validate that agent memory is a product category. DICE's value is in the richness of its knowledge model, not in being a managed service. The embeddable library model is a strength, not a gap. +| Truth maintenance (TMS/ATMS) | Justifications record why a belief holds; retracting a premise un-derives its dependents; ATMS labels a node with the assumption sets that support it | `ContradictionResolutionPass` retires the weaker of a contradictory pair to CONTRADICTED by comparing `effectiveConfidence()`, with no dependency record (`dice/src/main/kotlin/com/embabel/dice/operations/consolidation/ContradictionResolutionPass.kt:84-87`) | Doyle, "A Truth Maintenance System," *Artificial Intelligence* 12(3), 1979; de Kleer, "An Assumption-Based TMS," *Artificial Intelligence* 28, 1986 | +| AGM belief revision | Revision and contraction obey minimal-change postulates over a selection function | Decay adjustments (contradiction +0.15, merge ×0.7, reinforcement ×0.85) are hand-tuned constants (`dice/src/main/kotlin/com/embabel/dice/proposition/revision/LlmPropositionReviser.kt:528,670,694`) | Alchourrón, Gärdenfors, Makinson, "On the Logic of Theory Change," *J. Symbolic Logic* 50, 1985 | +| Provenance semirings | Provenance of a derived fact is a semiring expression over source tokens, composed under the query's + and × operators | `ProvenanceEntry` is a flat per-proposition list with no algebra for merge or abstraction | Green, Karvounarakis, Tannen, "Provenance Semirings," PODS 2007, DOI 10.1145/1265530.1265535 | +| Bitemporal databases | Valid time and transaction time as orthogonal axes, with defined as-of and point-in-time queries | `TemporalMetadata` carries `observedAt`/`validFrom`/`validTo`/`invalidatedAt` and no transaction-time axis | Snodgrass, *Developing Time-Oriented Database Applications in SQL*, Morgan Kaufmann, 1999 | +| Record linkage | Fellegi-Sunter match decisions from per-field m/u agreement probabilities against a likelihood-ratio threshold | Entity resolution uses fuzzy, vector, exact, partial and agentic searchers with LLM disambiguation | Fellegi, Sunter, "A Theory for Record Linkage," *JASA* 64, 1969, DOI 10.1080/01621459.1969.10501049 | +| ACT-R declarative memory | Base-level activation `B_i = ln(Σ_j t_j^-d)` folds recency and frequency into one retrieval score | `effectiveConfidence()` decays on recency alone. `reinforceCount` sits outside the decay maths, and a working-memory module needs both signals to rank on | Anderson & Schooler, "Reflections of the Environment in Memory," *Psychological Science* 2, 1991 | +| Argumentation frameworks | Arguments plus an attack relation; admissible, preferred and grounded semantics decide which sets survive collectively | `ContradictionResolutionPass` is pairwise strongest-wins, with pinned propositions branched out into a review event | Dung, "On the Acceptability of Arguments...," *Artificial Intelligence* 77(2), 1995 | + +Three borrowing opportunities: + +- **ATMS justification tracking, to make CONTRADICTED reversible.** The status flip follows a + confidence comparison at classification time and records no reason for the loss. A justification + set per proposition lets retracting the evidence un-derive the dependent status. +- **ACT-R base-level activation, to unify `reinforceCount` and decay.** Both signals exist and never + combine. The activation equation is a closed form for the ranking score a working-memory module + needs, and for decay that counts frequency of use. +- **Semiring-formalised provenance composition.** Treat auto-merge as the union-like operator and + abstraction synthesis (which requires all its sources) as the product-like one. That gives one + queryable answer to "what composed this fact" across both passes. + +AGM's postulates cover logical theories and DICE's beliefs are graded, so AGM serves as a checklist. +LLM disambiguation already covers what Fellegi-Sunter scoring would add. The first two opportunities +touch `Proposition.kt` and `ContradictionResolutionPass.kt`, which consolidation, projection and +retrieval ranking all depend on. + +## Open questions + +- **Which working-memory capabilities belong in DICE and which in the consuming application.** + Recommendation: activation ranking and per-turn lifecycle events in DICE, prompt-region and budget + policy above it. `embabel-agent` already carries the budget: `Budget` on `ProcessOptions` sets + cost, action and token caps and turns them into an `EarlyTerminationPolicy`, and consumption is + visible through the tool loop callbacks and the invocation events. DICE reads that ceiling and + does not define its own. +- **Whether to close the schema version-identity gap.** Recommendation: treat it as a candidate + module. No surveyed system offers it and no surveyed vendor has published roadmap intent, so + nothing external forces the timing. +- **Whether entity resolution can run concurrently in a working-memory module.** Recommendation: keep + it serial where shared identity is involved, matching the pipeline's serial resolution stage + (`docs/design/architecture.md`). + +## Out of scope + +- **Zep's five-reranker retrieval.** Deep feature tied to Neo4j traversal. The bi-temporal model is + the better investment from that system. +- **LangMem's prompt optimisation.** Orthogonal to memory quality. +- **Mem0's separate graph pipeline.** DICE has entity mentions plus a Neo4j projection over the same + propositions. +- **Google's multimodal extraction.** Multimodal input is already available upstream + (`MultimodalContent` over images and documents, embabel/embabel-agent#43) and the proposition + model is format-agnostic, so feeding non-text material to extraction is wiring. Output support is + tracked upstream in embabel/embabel-agent#42. Add on demand. +- **AWS's episodic reflection.** The abstraction pipeline already synthesises across propositions. +- **Neo4j's POLE+O ontology.** Domain-specific subtypes. The proposition model is domain-agnostic. +- **Neo4j's spaCy to GLiNER to LLM cascade.** Cost-effective and operationally heavy: model + downloads and dependency management. +- **Contract YAML as the authoring surface.** DICE's declared schema is a JVM type an application + owns. A YAML dialect would be a second source of truth. +- **Managed hosting.** The embeddable library is the distribution model. diff --git a/specs/readme-to-docs-tree.md b/specs/readme-to-docs-tree.md new file mode 100644 index 00000000..4b71c57b --- /dev/null +++ b/specs/readme-to-docs-tree.md @@ -0,0 +1,144 @@ +# Split the README into a docs tree + +Decides the `docs/` layout, which README section lands on which page, the shape of the quickstart +and feature pages, and the rule that keeps pages current. Writing the pages is out of scope. + +## Target tree + +| Directory | What goes there | +|---|---| +| `docs/README.adoc` | What DICE is, when to use it, links into the tree. | +| `docs/quickstart/` | One page, one path: dependencies through first report in 15 minutes. | +| `docs/concepts/` | Five pages, read in order, propositions through knowledge hygiene. | +| `docs/how-to/` | Task-shaped pages titled by what the reader wants, prerequisites at the top. | +| `docs/features/` | One page per opt-in surface, activation condition first. | +| `docs/reference/` | Configuration properties, package structure, REST endpoints. | +| `docs/production/` | Backend migration, decay settings, concurrency, LLM cost, observability, backup. | +| `docs/support/` | Compatibility matrix and FAQ. | + +| Location | Audience | Question it answers | +|---|---|---| +| `docs/design/` (19 notes and an index, unchanged) | DICE contributors | Why is it built this way? | +| `docs/` | DICE consumers | How do I use it? | +| `specs/` | Us | What are we building, and why does it matter commercially? | +| `README.md` | Anyone landing on the repo | What is this, and where do I start? | + +## README migration + +`README.md` is 2621 lines, target around 250. Line numbers move, so re-check a row before acting. + +| README section (lines) | Fate | +|---|---| +| What is DICE, benefits table, architecture overview, design notes (24-116) | Keep, trimmed. This is the landing page's job. | +| Real-world example: Impromptu (117-126) | Keep, cut to a paragraph plus a link. | +| Pipeline setup, conversation analysis (127-174) | Move to `docs/quickstart/`. | +| Key features, proposition pipeline, content dedup, mention filtering (175-489) | Move to `docs/concepts/propositions.adoc`, `docs/how-to/extract-from-documents.adoc`, `docs/how-to/mention-filtering.adoc`. | +| Entity extraction, entity resolution, resolution service (490-1225) | Move to `docs/concepts/entity-resolution.adoc` and `docs/how-to/tune-entity-resolution.adoc`. Largest block, split it. | +| Source analysis context, `ContextId`, `PropositionQuery` (1226-1419) | Move to `docs/concepts/context-and-schema.adoc`, `docs/how-to/query-propositions.adoc`. | +| Relations, projector architecture, graph and Prolog projection (1420-1647) | Move to `docs/concepts/storage-and-projections.adoc`, `docs/how-to/project-to-graph.adoc`, `docs/features/prolog-inference.adoc`. | +| Agent memory, memory projection, memory maintenance (1648-1979) | Move to `docs/how-to/agent-memory.adoc`, `docs/concepts/knowledge-hygiene.adoc`. | +| Proposition operations, Oracle (1980-2090) | Move to `docs/how-to/query-propositions.adoc`, `docs/how-to/oracle.adoc`. | +| Package structure (2091-2205) | Move to `docs/reference/package-structure.adoc`. | +| REST API and endpoints (2206-2334) | Move to `docs/features/web-api.adoc`. | +| Spring Boot integration, graph-backed storage, API-key security (2335-2572) | Move to `docs/how-to/choose-a-backend.adoc`, `docs/reference/configuration-properties.adoc`, `docs/features/web-api.adoc`. | +| Installation (2573-2585) | Keep as coordinates only. The working version lives in the quickstart. | +| Technology stack, references, license (2586-2621) | Keep. | + +Content moves: a section is deleted from the README as it lands under `docs/`, so one copy exists, +and it leaves a one-line link where it was so an existing bookmark still lands somewhere useful. +One PR per destination page. + +## Concept pages + +Read in order. Each page uses terms the page before it defines, and ends with a "Try it now" block +of five to ten lines that runs against the quickstart's setup, with the output to expect. + +1. `propositions.adoc`: claims as the system of record, with confidence, importance and decay. +2. `entity-resolution.adoc`: mentions matched to entities or minted as new, and the resolver chain. +3. `storage-and-projections.adoc`: the `PropositionStore` SPI, its repositories, and the views. +4. `context-and-schema.adoc`: `ContextId` scoping, the `DataDictionary`, and `SchemaAdherence`. +5. `knowledge-hygiene.adoc`: admission gates, reclamation and consolidation. + +## Feature pages + +One page per opt-in surface, opening with the activation condition: the exact bean or property that +switches the feature on. Template, in order: + +- Availability: DICE version, modules, cost to add. +- Activation condition: the bean or property, with the value that turns it on. +- When to use it, and when to leave it off. +- Impact: latency, memory, extra infrastructure, extra LLM calls. +- How to enable: full working configuration. +- Example: real code with real output. + +| Surface | Activation condition | +|---|---| +| Graph-backed storage | `embabel.dice.store.type=graph` | +| Vector index on Neo4j | `embabel.dice.store.vector-index.enabled=true` | +| Web API | `@Import(DiceRestConfiguration.class)` plus the beans the controllers need | +| API-key security | `dice.security.api-key.enabled=true` | +| Decay and stale pruning | `embabel.dice.store.decay.enabled=true`, with `prune-stale` false by default | +| Multi-signal collector | `embabel.dice.collector.enabled`, on unless set to false | +| Prolog inference | A `PrologProjector` bean. Experimental. | +| Concurrent extraction | A parallel or batched `ExtractionExecutionStrategy` on the pipeline | + +## Quickstart + +DICE publishes `dice`, `dice-ingestion`, `dice-storage`, `dice-storage-autoconfigure` and +`dice-report`, and no aggregator starter. Decide the coordinates before writing the page: + +- Add a `dice-spring-boot-starter` depending on `dice-storage-autoconfigure` and `dice-report`, + publish it, and the quickstart uses one coordinate. Needs a release first. +- Write against `dice-storage-autoconfigure` and `dice-report`, both explicit, with a comment + saying what each buys. Two coordinates, available today. + +1. Dependencies: the coordinates chosen above, at the current DICE version. +2. Configuration: an LLM provider inherited from embabel-agent, and the default in-memory store. +3. What autoconfiguration provides: the beans that now exist and what each is for. +4. Extract: one paragraph of text through `PropositionPipeline`, with confidence printed. +5. Persist: `persist(propositionRepository, namedEntityDataRepository)` on the `PersistablePropositions` the pipeline hands back unsaved. +6. Query: retrieve what was stored, by entity and by `ContextId`. +7. Report: produce one human-readable artifact through `dice-report`. +8. Where to go next: links into `docs/concepts/` and `docs/how-to/`. + +Constraints: + +- No Neo4j, no Docker, no external resource to create. +- `InMemoryPropositionRepository` needs an `EmbeddingService` for vector search, so the minimal path queries by entity and `ContextId`. +- Every snippet compiles, and a test in `dice-integration-tests` runs the page's exact code. +- Anything needing a decision (backend, resolver chain, schema) takes the default and links out. + +## Compatibility matrix + +Its own page under `docs/support/`, linked from the README. Four axes, plus a feature-availability +column. Values get pinned at the 1.0 release. + +| Axis | What we state | +|---|---| +| embabel-agent | Supported version range per DICE release, with the tested point release named. The build currently tracks 1.5.0-SNAPSHOT. | +| Spring Boot | Minimum and maximum tested, per DICE release. | +| JDK | Baseline is 21, inherited from `embabel-build-parent` and used by CI. Newer JDKs listed as tested or untested. | +| Neo4j | Version floor for the graph backend, reached through Drivine (`drivine4j-spring-boot-starter` 0.0.79). The in-memory backend runs without Neo4j. | + +## Docs rule + +Every feature PR ships its developer-doc page and its design-doc delta. Consumer-visible behaviour +updates the page under `docs/`, new rationale updates the `docs/design/` note, an opt-in feature +gets a feature page carrying its activation condition, a new property updates +`docs/reference/configuration-properties.adoc`, and a version-support change updates the compatibility +matrix. Internal refactors with no consumer-visible change, test-only changes and build changes are +exempt. Reviewers ask one question: with only this PR's docs, could a consumer use the feature? A no +is a blocking finding. A promise of later docs does not clear it. + +## Open questions + +- Starter module or two coordinates? Recommendation: two coordinates, which needs no release. +- Which format, and rendered where? Decided: AsciiDoc, matching the framework reference, which + builds from `embabel-agent-docs/src/main/asciidoc` through the asciidoctor Maven plugin. GitHub + renders `.adoc` in the repo, so the tree stays readable before any site exists, and the existing + Markdown under `docs/design/` converts as those notes get touched. Whether DICE renders its own + site or publishes into the framework's is still open. +- Does DICE ship a CLI (schema validation, drift check)? Recommendation: no CLI before 1.0, and its + own quickstart if one lands. +- Promote `docs/design/` notes into concept pages? Recommendation: they stay contributor-facing, + and concept pages lift explanations from them and link back.