From 4fecea7b18a57218c48aecec3eb0447bb1798656 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Sun, 30 Aug 2026 16:09:39 -0400 Subject: [PATCH 01/11] feat(metamodel): schema versioning core with per-type governance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New dice-metamodel module, first slice of the metamodel train: MetamodelVersion content-hash stamping over governed types only (GovernedTypeSelector — you version what you declare), property signatures (name, kind, type, cardinality) in the fingerprint, JVM-immutable value types, DeclaredSchema/DeclaredSchemaSource as the opt-in seam, and the MetamodelVersionStore contract. Pure JVM — no Spring, no database, no dice-core dependency. Refs #45. --- AGENTS.md | 1 + CHANGELOG.md | 16 + README.md | 2 + dice-metamodel/pom.xml | 50 ++ .../dice/metamodel/DeclaredSchemaSource.kt | 98 +++ .../dice/metamodel/GovernedTypeSelector.kt | 58 ++ .../dice/metamodel/MetamodelVersion.kt | 335 +++++++ .../dice/metamodel/MetamodelVersionStore.kt | 82 ++ .../dice/metamodel/DeclaredSchemaTest.kt | 64 ++ .../metamodel/MetamodelVersionStoreTest.kt | 105 +++ .../dice/metamodel/MetamodelVersionTest.kt | 814 ++++++++++++++++++ docs/design/INDEX.md | 6 +- docs/design/architecture.md | 11 +- docs/design/metamodel-versioning.md | 189 ++++ pom.xml | 6 + 15 files changed, 1833 insertions(+), 4 deletions(-) create mode 100644 CHANGELOG.md create mode 100644 dice-metamodel/pom.xml create mode 100644 dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt create mode 100644 dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt create mode 100644 dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt create mode 100644 dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt create mode 100644 dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt create mode 100644 dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt create mode 100644 dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt create mode 100644 docs/design/metamodel-versioning.md diff --git a/AGENTS.md b/AGENTS.md index e3816cc1..8094e7f3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,6 +11,7 @@ DICE (Domain-Integrated Context Engineering) is a proposition-first knowledge su | `dice-storage-autoconfigure` | Spring Boot auto-configuration that wires the right backend based on `embabel.dice.store.type`, schedules the decay tick, and provides auto-configuration for the multi-signal duplicate collector (properties prefix `embabel.dice.collector`) | | `dice-report` | Output projectors over propositions: rationale (why a fact is believed, with evidence), structured report, and surprising-link discovery | | `dice-ingestion` | Ingestion SPI (artifacts → chunks) with a content-hash dedup ledger so the same source isn't extracted twice | +| `dice-metamodel` | Schema versioning: `MetamodelVersion` content-hash stamps over the governed types of a `DataDictionary`, `GovernedTypeSelector`, the `DeclaredSchemaSource` opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM — no dependency on `dice` | | `dice-integration-tests` | Test-only: the cross-feature end-to-end canonical-flow harness | ## Build & test diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..6b71556a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,16 @@ +# Changelog + +Notable changes to DICE. Each entry states its compatibility impact on consumers +(assistant/me and anything else tracking `0.2.0-SNAPSHOT`): **additive** (safe to +pick up), **behavioral** (same API, different runtime behavior — read the note), +or **breaking** (consumer change required; the entry links the migration notes +and the consumer PRs that deliver it). + +## Unreleased + +### Added + +- `dice-metamodel` module, first slice: schema versioning — `MetamodelVersion` + content-hash stamping with per-type governance selection, declared-schema + opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM. + **Compatibility: additive.** New module; no existing API touched. diff --git a/README.md b/README.md index d4a92ef2..611a9f5e 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,8 @@ recover by reading a single class — see the design notes in [`docs/design/`](d - [Durable storage](docs/design/durable-storage.md) — backend selection, defense-in-depth dedup, two-phase save, materialised effective confidence, schema-as-beans, and the decay tick. - [Events](docs/design/events.md) — the domain-event model the store and pipeline emit. +- [Metamodel versioning](docs/design/metamodel-versioning.md) — content-hash schema stamps, per-type + governance (you version what you declare), the declared-schema opt-in seam, and version history. ## Real-World Example: Impromptu diff --git a/dice-metamodel/pom.xml b/dice-metamodel/pom.xml new file mode 100644 index 00000000..7df86561 --- /dev/null +++ b/dice-metamodel/pom.xml @@ -0,0 +1,50 @@ + + + 4.0.0 + + com.embabel.dice + dice-parent + 0.2.0-SNAPSHOT + + dice-metamodel + jar + Dice Metamodel + Schema versioning for DICE knowledge graphs: content-hash stamping, the declared-schema seam, and the version store contract + + + + + com.embabel.agent + embabel-agent-api + provided + + + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + + org.jetbrains.kotlin + kotlin-maven-plugin + + + -Xjvm-default=all + + + + + + + diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt new file mode 100644 index 00000000..c6c317f0 --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt @@ -0,0 +1,98 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DataDictionary + +/** + * The schema as declared: the stamped [version] plus the bare relationship type names it allows. + * + * The bare names travel alongside the stamp rather than being recovered from it, because + * [MetamodelVersion.relationshipNames] holds rendered `From-[name]->To` descriptors and + * reverse-parsing one is ambiguous — these names come from free-text and LLM extraction, and can + * themselves contain a `-[...]->`-shaped substring. Whoever builds a declaration holds the + * un-rendered names before anything was stamped, so it passes them straight through. Comparing + * declared relationships against what a graph actually holds needs those bare names, and that + * comparison is the next slice. + * + * @property version The stamped declared schema. + * @property relationshipTypeNames The bare relationship type names [version] allows. + */ +class DeclaredSchema( + val version: MetamodelVersion, + relationshipTypeNames: Set, +) { + + /** Copied to a JVM-immutable set so the declaration can't drift from its stamped [version]. */ + val relationshipTypeNames: Set = java.util.Set.copyOf(relationshipTypeNames) + + override fun equals(other: Any?): Boolean = + other is DeclaredSchema && + version == other.version && + relationshipTypeNames == other.relationshipTypeNames + + override fun hashCode(): Int = 31 * version.hashCode() + relationshipTypeNames.hashCode() + + override fun toString(): String = + "DeclaredSchema(version=$version, relationshipTypeNames=$relationshipTypeNames)" + + companion object { + + /** + * Declare the governed part of [dataDictionary]: stamp it and carry through the bare + * relationship names the same governed types declare. + * + * Use this rather than building the two halves separately. Picking a governed subset for + * the stamp but taking relationship names from the whole dictionary would declare + * relationships the stamp never covered, and the mismatch would only surface much later, + * as phantom disagreement in a drift check. + * + * @param dataDictionary The schema to declare. + * @param selector Which types are under governance. Defaults to all of them. + * @return The declaration. + */ + @JvmStatic + @JvmOverloads + fun from( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector = GovernedTypeSelector.ALL, + ): DeclaredSchema = DeclaredSchema( + version = MetamodelVersion.from(dataDictionary, selector), + relationshipTypeNames = MetamodelVersion.governedRelationshipTypeNames(dataDictionary, selector), + ) + } +} + +/** + * Supplies the schema an application has declared. + * + * This is the opt-in seam for the whole versioning story: no declared schema, no versioning. + * Nothing here stamps anything on its own, and the Spring wiring that arrives in a later slice + * activates only when a `DeclaredSchemaSource` bean is present. An application that hasn't decided + * what it governs is left alone. + * + * The module has no opinion on where a declared schema comes from — a consuming app implements this + * over whatever it already uses to define its types (a `DataDictionary`, a config file, a + * registry...) and wires it as a bean. There is no default implementation because there is no + * default declared schema. + */ +fun interface DeclaredSchemaSource { + + /** + * @return the current [DeclaredSchema]. + */ + fun declare(): DeclaredSchema +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt new file mode 100644 index 00000000..6fe7aa7c --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt @@ -0,0 +1,58 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DomainType + +/** + * Decides which domain types a [MetamodelVersion] stamp covers. + * + * A DICE domain is rarely all one thing. Part of it is closed-world — the types you have committed + * to, whose shape you want to notice changing. The rest is open-world: exploratory types that + * extraction proposes, that come and go, and that you would rather not have breaking a version + * comparison every time an LLM invents one. Governance is therefore per type and opt-in, in the + * spirit of Hibernate's `@Version`: you version what you declare, and nothing else. + * + * A selector is just a predicate over types, so the usual form is a lambda over names you already + * hold: + * + * ```kotlin + * val governed = setOf("Person", "Company") + * val version = MetamodelVersion.from(dataDictionary, GovernedTypeSelector { it.name in governed }) + * ``` + * + * Selecting a subset changes what the stamp is *about*, not how it is computed: adding an + * ungoverned type to the dictionary leaves the content hash alone, while touching a governed one + * changes it. + */ +fun interface GovernedTypeSelector { + + /** + * @param type A domain type from the dictionary being stamped. + * @return `true` when this type is under version governance and belongs in the stamp. + */ + fun governs(type: DomainType): Boolean + + companion object { + + /** + * Governs every type in the dictionary — the whole-schema stamp, and what + * [MetamodelVersion.from] uses when no selector is given. + */ + @JvmField + val ALL: GovernedTypeSelector = GovernedTypeSelector { true } + } +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt new file mode 100644 index 00000000..4951018f --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -0,0 +1,335 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.Cardinality +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DomainTypePropertyDefinition +import com.embabel.agent.core.NamedPropertyDefinition +import com.embabel.agent.core.PropertyDefinition +import java.security.MessageDigest +import java.util.Objects + +/** + * What a property looks like structurally: its name, whether it holds a plain value or points at + * another type, what that value or target is, and how many of them there can be. + * + * Names alone are not enough to version a schema. Turning `age` from a string into an integer, or a + * single `worksAt` into a list of them, is a real change to what the graph can hold — and with only + * names in the stamp both would leave [MetamodelVersion.contentHash] untouched, which is exactly the + * kind of silent drift versioning exists to catch. + * + * Descriptions and semantic metadata are deliberately left out. They steer extraction and read + * nicely in a prompt, but they don't change the shape of what gets stored, and re-wording a + * description shouldn't look like a schema change. + * + * @property name The property name. + * @property kind Whether the property holds a value or points at another domain type. + * @property type The value type (`string`, `integer`, ...), or the target type's name for a + * reference. Empty when [kind] is [Kind.UNKNOWN]. + * @property cardinality How many values the property holds — optional, one, a list, or a set. + */ +data class PropertySignature( + val name: String, + val kind: Kind, + val type: String, + val cardinality: Cardinality, +) : Comparable { + + /** Whether a property holds a value of its own or points at another type in the schema. */ + enum class Kind { + /** A plain value: string, integer, and so on. */ + VALUE, + + /** A reference to another domain type — what shows up as a relationship in the graph. */ + REFERENCE, + + /** + * A property kind the agent platform grew that we don't recognise yet. The name and + * cardinality still count towards the hash; the type is recorded as empty. + */ + UNKNOWN, + } + + override fun compareTo(other: PropertySignature): Int = ORDER.compare(this, other) + + companion object { + + /** Total order used to canonicalise a property set before hashing. */ + private val ORDER = compareBy({ it.name }, { it.kind }, { it.type }, { it.cardinality }) + + /** + * Read the signature off a property as the agent platform describes it. + * + * @param property The property definition to summarise. + * @return Its structural signature. + */ + @JvmStatic + fun of(property: PropertyDefinition): PropertySignature = when (property) { + is DomainTypePropertyDefinition -> + PropertySignature(property.name, Kind.REFERENCE, property.type.name, property.cardinality) + + is NamedPropertyDefinition -> + PropertySignature(property.name, Kind.VALUE, property.type, property.cardinality) + + else -> + PropertySignature(property.name, Kind.UNKNOWN, "", property.cardinality) + } + } +} + +/** + * An immutable stamp that captures the identity and structural content of the governed part of a + * [DataDictionary] at a point in time. + * + * Two [MetamodelVersion] instances with the same [contentHash] represent semantically equivalent + * schemas. A proposition can record the version it was created under via the + * `dice.metamodel.version` metadata key. + * + * The hash is computed here, from the structural fields, and is not something a caller can + * supply. That matters because the hash is load-bearing: it is the store's natural key and it is + * what [hasSameContentAs] compares. If it were an ordinary constructor argument, two structurally + * different schemas could claim the same hash, compare as equal, and overwrite each other in + * storage. + * + * Everything handed to the constructor is copied into genuinely immutable collections and + * canonicalised — type and relationship lists come out sorted and free of duplicates. A stamp whose + * collections could still be changed from the outside, or that listed the same relationship twice + * because two same-named types declared it, would disagree with its own precomputed hash. This is + * a plain class rather than a `data class` for the same reason: a generated `copy()` would hand its + * arguments straight to the fields and skip all of that. + * + * @property schemaName The [DataDictionary.name] at the time the stamp was taken. + * @property entityTypeNames Governed entity type names, sorted and deduplicated. + * @property entityTypeLabels Full label set per type (including inherited labels), keyed by type + * name. Captured so label-only changes are detectable and reflected in [contentHash]. + * @property entityTypeProperties Full property signature set per type (including inherited + * properties), keyed by type name. Signatures rather than bare names, so a property changing type + * or cardinality moves [contentHash]. + * @property relationshipNames Rendered `From-[name]->To` descriptors for the relationships the + * governed types declare, sorted and deduplicated. + * @property contentHash SHA-256 hex digest of the schema's entity types, label sets, property + * signatures, and allowed relationships — structural content only, derived from the four fields + * above. The schema name is intentionally excluded so that two structurally identical schemas + * are equal regardless of how they are named. Stable across JVM restarts; any change — even a + * property's type on a type whose name is preserved — produces a different hash. + */ +class MetamodelVersion( + schemaName: String, + entityTypeNames: List, + entityTypeLabels: Map>, + entityTypeProperties: Map>, + relationshipNames: List, +) { + + val schemaName: String = schemaName + + val entityTypeNames: List = immutableCopy(entityTypeNames.distinct().sorted()) + + val entityTypeLabels: Map> = immutableCopy(entityTypeLabels) + + val entityTypeProperties: Map> = immutableCopy(entityTypeProperties) + + val relationshipNames: List = immutableCopy(relationshipNames.distinct().sorted()) + + init { + // Only types named in entityTypeNames are walked when hashing, so a map entry keyed by + // anything else would never reach contentHash — two stamps holding different labels or + // properties could then share a hash and, with it, the store's natural key. Refuse the + // input rather than quietly ignoring half of it. `this.` is deliberate: the constructor + // parameters are still in scope here and would shadow the copied fields. + val known = this.entityTypeNames.toSet() + val strayLabelKeys = this.entityTypeLabels.keys - known + require(strayLabelKeys.isEmpty()) { + "entityTypeLabels is keyed by types missing from entityTypeNames: ${strayLabelKeys.sorted()}. " + + "Labels for a type that isn't listed never reach contentHash." + } + val strayPropertyKeys = this.entityTypeProperties.keys - known + require(strayPropertyKeys.isEmpty()) { + "entityTypeProperties is keyed by types missing from entityTypeNames: ${strayPropertyKeys.sorted()}. " + + "Properties for a type that isn't listed never reach contentHash." + } + } + + val contentHash: String = fingerprint() + + /** + * Returns `true` when this version and [other] have the same structural content — + * identical entity types, label sets, property signatures, and relationships — regardless + * of schema name. Uses [contentHash] for the comparison. + */ + fun hasSameContentAs(other: MetamodelVersion): Boolean = contentHash == other.contentHash + + /** + * The content hash itself: a deterministic encoding of the structural fields, hashed with + * SHA-256 and rendered as lowercase hex. + * + * This is a **persisted** format. The digest is the store's natural key, so changing the + * encoding orphans everything already saved against it. `MetamodelVersionTest` pins the + * digest of a fixed schema with a literal assertion for exactly that reason — if you + * change the encoding, change that literal deliberately and plan a migration. + */ + private fun fingerprint(): String { + // Build a collision-free fingerprint by length-prefixing every name, label, and property + // component (and counting each set) before hashing. A plain delimiter-joined encoding isn't + // safe here: these names come from free-text / LLM extraction and routinely contain ';', + // '[', '=' and spaces, so joining with those characters could make ["a;b"] and ["a", "b"] + // hash identically and hide a real, lossy schema change. Length-prefixing makes the encoding + // unambiguous, so distinct content always yields a distinct hash. + // Schema name is deliberately excluded: two structurally identical schemas must produce + // the same hash even when named differently (e.g. dev vs prod environments). + val hashInput = buildString { + append("types:").append(entityTypeNames.size).append('|') + entityTypeNames.forEach { name -> + appendSized(name) + val labels = entityTypeLabels[name].orEmpty().sorted() + append("labels:").append(labels.size).append('|') + labels.forEach { appendSized(it) } + val properties = entityTypeProperties[name].orEmpty().sorted() + append("props:").append(properties.size).append('|') + properties.forEach { property -> + appendSized(property.name) + appendSized(property.kind.name) + appendSized(property.type) + appendSized(property.cardinality.name) + } + } + append("rels:").append(relationshipNames.size).append('|') + relationshipNames.forEach { appendSized(it) } + } + + val digest = MessageDigest.getInstance("SHA-256") + val hashBytes = digest.digest(hashInput.toByteArray(Charsets.UTF_8)) + return hashBytes.joinToString("") { "%02x".format(it) } + } + + override fun equals(other: Any?): Boolean { + if (this === other) return true + if (other !is MetamodelVersion) return false + return schemaName == other.schemaName && + entityTypeNames == other.entityTypeNames && + entityTypeLabels == other.entityTypeLabels && + entityTypeProperties == other.entityTypeProperties && + relationshipNames == other.relationshipNames + } + + override fun hashCode(): Int = + Objects.hash(schemaName, entityTypeNames, entityTypeLabels, entityTypeProperties, relationshipNames) + + override fun toString(): String = + "MetamodelVersion(schemaName=$schemaName, contentHash=$contentHash, " + + "entityTypeNames=$entityTypeNames, entityTypeLabels=$entityTypeLabels, " + + "entityTypeProperties=$entityTypeProperties, relationshipNames=$relationshipNames)" + + companion object { + + /** Append [token] length-prefixed (`:`) so concatenation can't be ambiguous. */ + private fun StringBuilder.appendSized(token: String) { + append(token.length).append(':').append(token) + } + + /** + * Copy [values] into a list nothing can change afterwards — not even Java code holding the + * original, or calling `set` on what a getter hands back. + */ + private fun immutableCopy(values: List): List = java.util.List.copyOf(values) + + /** Copy a map of sets so both the map and every set inside it are genuinely immutable. */ + private fun immutableCopy(values: Map>): Map> = + java.util.Map.copyOf(values.mapValues { (_, set) -> java.util.Set.copyOf(set) }) + + /** + * Create a [MetamodelVersion] stamp covering every type in the given [DataDictionary] — + * the closed-world reading, where the whole dictionary is governed. + * + * @param dataDictionary The schema to stamp. + * @return An immutable version stamp. + */ + @JvmStatic + fun from(dataDictionary: DataDictionary): MetamodelVersion = + from(dataDictionary, GovernedTypeSelector.ALL) + + /** + * Create a [MetamodelVersion] stamp covering only the types [selector] governs. + * + * DICE domains are usually part closed-world and part open-world. You commit to some types + * and want to know the moment their shape moves; the rest are exploratory, proposed by + * extraction, and churn by design. Stamping the whole dictionary makes that churn look like + * a schema change — every new exploratory type produces a new content hash, and the version + * history fills with noise nobody chose. So governance is per type and opt-in, the way + * Hibernate's `@Version` is per entity: you version what you declare. + * + * Concretely, adding, removing or reshaping an ungoverned type leaves [contentHash] + * untouched; doing the same to a governed one changes it. Relationships follow the type + * that declares them — a governed type's outgoing relationship is part of that type's + * declared shape and stays in the stamp even when it points at an ungoverned type, while a + * relationship declared *by* an ungoverned type is left out entirely. + * + * @param dataDictionary The schema to stamp. + * @param selector Which of its types are under governance. [GovernedTypeSelector.ALL] + * reproduces [from] exactly. + * @return An immutable version stamp covering the governed subset. + */ + @JvmStatic + fun from(dataDictionary: DataDictionary, selector: GovernedTypeSelector): MetamodelVersion { + val governedTypes = dataDictionary.domainTypes.filter { selector.governs(it) } + + // A DataDictionary can legally hold two domain types that share a name but differ in + // shape (DynamicType is a data class, so same-named instances with different labels are + // not equal and both survive a set). Merge their labels and properties by union per name + // rather than letting associate() keep only the last — otherwise a label or property + // present under that name would vanish from the fingerprint, and later removing it + // wouldn't change the hash, hiding a real change. + val entityTypeLabels = governedTypes + .groupBy { it.name } + .mapValues { (_, types) -> types.flatMap { it.labels }.toSet() } + + val entityTypeProperties = governedTypes + .groupBy { it.name } + .mapValues { (_, types) -> types.flatMap { type -> type.properties.map(PropertySignature::of) }.toSet() } + + // Splitting one type into two same-named declarations, or merging two back into one, + // can render the same relationship descriptor twice. That's the same schema either way, + // so it has to be the same hash — the constructor sorts and deduplicates both lists. + val relationshipNames = dataDictionary.allowedRelationships() + .filter { selector.governs(it.from) } + .map { rel -> "${rel.from.name}-[${rel.name}]->${rel.to.name}" } + + return MetamodelVersion( + schemaName = dataDictionary.name, + entityTypeNames = governedTypes.map { it.name }, + entityTypeLabels = entityTypeLabels, + entityTypeProperties = entityTypeProperties, + relationshipNames = relationshipNames, + ) + } + + /** + * The bare relationship type names the governed types declare — the same relationships + * [from] renders into [relationshipNames], but un-rendered. + * + * Kept here so the governance rule ("a relationship belongs to the type that declares it") + * lives in one place, and a [DeclaredSchema] can't drift from the stamp beside it. + */ + internal fun governedRelationshipTypeNames( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector, + ): Set = dataDictionary.allowedRelationships() + .filter { selector.governs(it.from) } + .map { it.name } + .toSet() + } +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt new file mode 100644 index 00000000..e13afada --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt @@ -0,0 +1,82 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +/** + * Durable store for metamodel version stamps. History is the point: keeping every stamp a schema + * has ever had is what later lets you say when a shape changed and what knowledge was extracted + * under which version. Implementations must preserve saved stamps exactly as recorded, including + * set contents and ordering guarantees below. + * + * **What "save" means here.** [saveVersion] is an upsert on the natural key `(schemaName, + * contentHash)`, not a blind append. Saving a stamp whose key already exists doesn't create a + * second record: the existing one is matched and its non-key content is overwritten with what you + * just passed. In practice that's idempotence rather than mutation, because the key carries the + * content — the hash is derived from exactly the fields a re-save would overwrite, so anything + * landing on an existing key has identical content by construction. Nothing is ever deleted, and + * records with different keys always coexist, so history accumulates. Implementations are not + * expected to reject a re-save. + * + * There is no delete, and no drift or diffing here. This contract covers stamping and recall only; + * comparing a declaration against a live graph is a separate concern with its own store contract. + */ +interface MetamodelVersionStore { + + /** + * Save a version stamp, keyed on `(schemaName, contentHash)`. Saving the same version twice + * leaves one stored version, not two. + * + * @param version The version to save. + */ + fun saveVersion(version: MetamodelVersion) + + /** + * Return the most recently saved version for the given schema name, or `null` if no versions + * have been recorded. + * + * "Most recent" is logical write order, not stamp content — re-saving an existing version does + * not make it the latest again, since the write matched a record that was already there. + * + * @param schemaName The schema to look up. + * @return The latest [MetamodelVersion], or `null` if unknown. + */ + fun latestVersion(schemaName: String): MetamodelVersion? + + /** + * Return all saved versions for the given schema, newest first. + * + * @param schemaName The schema to look up. + * @return All [MetamodelVersion]s for the schema, newest first. Empty if there are none. + */ + fun versionHistory(schemaName: String): List + + /** + * Look up one exact stamp by its natural key. + * + * This is how you resolve a recorded hash — the one a proposition carries as the version it was + * extracted under — back into the schema shape it stood for. + * + * The default scans [versionHistory], which is correct for any implementation but reads the + * whole history to answer a keyed question. A backend that can push the lookup down (a database + * `MATCH` on the key rather than an in-memory `filter`) should override it. + * + * @param schemaName The schema the version belongs to. + * @param contentHash The [MetamodelVersion.contentHash] to find. + * @return The matching version, or `null` if this schema has no version with that hash. + */ + fun findVersion(schemaName: String, contentHash: String): MetamodelVersion? = + versionHistory(schemaName).firstOrNull { it.contentHash == contentHash } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt new file mode 100644 index 00000000..f217d17a --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt @@ -0,0 +1,64 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DomainTypePropertyDefinition +import com.embabel.agent.core.DynamicType +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test + +class DeclaredSchemaTest { + + private val company = DynamicType("Company") + private val person = DynamicType( + name = "Person", + ownProperties = listOf(DomainTypePropertyDefinition("worksAt", company)), + ) + private val sighting = DynamicType( + name = "Sighting", + ownProperties = listOf(DomainTypePropertyDefinition("about", person)), + ) + + private fun dictionary() = DataDictionary.fromDomainTypes("app", listOf(person, company, sighting)) + + @Test + fun `declaring everything carries every relationship name`() { + val declared = DeclaredSchema.from(dictionary()) + + assertEquals(setOf("worksAt", "about"), declared.relationshipTypeNames) + assertEquals(MetamodelVersion.from(dictionary()), declared.version) + } + + @Test + fun `bare relationship names follow governance, so they can't disagree with the stamp`() { + // `about` is declared by the ungoverned Sighting, so it is out of both halves. Taking the + // stamp from a subset but the names from the whole dictionary would declare a relationship + // the stamp never covered. + val declared = DeclaredSchema.from(dictionary(), GovernedTypeSelector { it.name in setOf("Person", "Company") }) + + assertEquals(setOf("worksAt"), declared.relationshipTypeNames) + assertEquals(listOf("Person-[worksAt]->Company"), declared.version.relationshipNames) + assertEquals(listOf("Company", "Person"), declared.version.entityTypeNames) + } + + @Test + fun `a source is just a supplier of the declaration`() { + val source = DeclaredSchemaSource { DeclaredSchema.from(dictionary()) } + + assertEquals(DeclaredSchema.from(dictionary()), source.declare()) + } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt new file mode 100644 index 00000000..874c9fbf --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt @@ -0,0 +1,105 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DynamicType +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test + +/** + * Covers the one piece of behaviour the contract itself ships: the default [findVersion], which a + * backend is free to override with a keyed lookup. A store implementation gets its own tests + * wherever it lives. + */ +class MetamodelVersionStoreTest { + + /** Minimal store honouring the contract: upsert on (schemaName, contentHash), newest first. */ + private class InMemoryVersionStore : MetamodelVersionStore { + + private val saved = mutableListOf() + + override fun saveVersion(version: MetamodelVersion) { + // Idempotent: re-saving an existing version keeps its original position in write order. + if (saved.none { it.schemaName == version.schemaName && it.contentHash == version.contentHash }) { + saved.add(version) + } + } + + override fun latestVersion(schemaName: String): MetamodelVersion? = + versionHistory(schemaName).firstOrNull() + + override fun versionHistory(schemaName: String): List = + saved.filter { it.schemaName == schemaName }.reversed() + } + + private fun version(schemaName: String, vararg typeNames: String): MetamodelVersion = + MetamodelVersion.from( + DataDictionary.fromDomainTypes(schemaName, typeNames.map { DynamicType(name = it) }), + ) + + @Test + fun `findVersion returns the stamp with that hash`() { + val store = InMemoryVersionStore() + val first = version("app", "Person") + val second = version("app", "Person", "Company") + store.saveVersion(first) + store.saveVersion(second) + + assertEquals(first, store.findVersion("app", first.contentHash)) + assertEquals(second, store.findVersion("app", second.contentHash)) + } + + @Test + fun `findVersion is scoped to the schema name`() { + // Two schemas can hold structurally identical versions — the hash excludes the name — so + // the lookup has to match on both halves of the key, not just the hash. + val store = InMemoryVersionStore() + val mine = version("mine", "Person") + store.saveVersion(mine) + + assertEquals(mine, store.findVersion("mine", mine.contentHash)) + assertNull(store.findVersion("yours", mine.contentHash)) + } + + @Test + fun `findVersion returns null for an unknown hash`() { + val store = InMemoryVersionStore() + store.saveVersion(version("app", "Person")) + + assertNull(store.findVersion("app", "not-a-hash")) + } + + @Test + fun `re-saving a version leaves one record, not two`() { + val store = InMemoryVersionStore() + val v = version("app", "Person") + store.saveVersion(v) + store.saveVersion(v) + + assertEquals(listOf(v), store.versionHistory("app")) + assertEquals(v, store.latestVersion("app")) + } + + @Test + fun `an empty store has no latest version and an empty history`() { + val store = InMemoryVersionStore() + + assertNull(store.latestVersion("app")) + assertEquals(emptyList(), store.versionHistory("app")) + assertNull(store.findVersion("app", "anything")) + } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt new file mode 100644 index 00000000..4b8259be --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt @@ -0,0 +1,814 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.Cardinality +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DomainType +import com.embabel.agent.core.DomainTypePropertyDefinition +import com.embabel.agent.core.DynamicType +import com.embabel.agent.core.PropertyDefinition +import com.embabel.agent.core.ValuePropertyDefinition +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Nested +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows + +class MetamodelVersionTest { + + private fun schemaWith(vararg typeNames: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + typeNames.map { DynamicType(name = it) }, + ) + + @Nested + inner class ContentHash { + + @Test + fun `identical schemas produce the same hash`() { + val a = MetamodelVersion.from(schemaWith("Person", "Company")) + val b = MetamodelVersion.from(schemaWith("Person", "Company")) + assertEquals(a.contentHash, b.contentHash) + assertTrue(a.hasSameContentAs(b)) + } + + @Test + fun `different type sets produce different hashes`() { + val a = MetamodelVersion.from(schemaWith("Person", "Company")) + val b = MetamodelVersion.from(schemaWith("Person", "Technology")) + assertNotEquals(a.contentHash, b.contentHash) + assertFalse(a.hasSameContentAs(b)) + } + + @Test + fun `type order does not affect hash`() { + val a = MetamodelVersion.from(schemaWith("Company", "Person")) + val b = MetamodelVersion.from(schemaWith("Person", "Company")) + assertEquals(a.contentHash, b.contentHash) + } + + @Test + fun `adding a type changes the hash`() { + val base = MetamodelVersion.from(schemaWith("Person")) + val extended = MetamodelVersion.from(schemaWith("Person", "Company")) + assertNotEquals(base.contentHash, extended.contentHash) + } + + @Test + fun `empty schema has a stable hash`() { + val a = MetamodelVersion.from(schemaWith()) + val b = MetamodelVersion.from(schemaWith()) + assertEquals(a.contentHash, b.contentHash) + } + + @Test + fun `a caller cannot supply the hash — it is always derived from the content`() { + // The whole point of deriving it: two versions that differ structurally can never claim + // the same hash, because there is no way to hand one in. + val one = MetamodelVersion( + schemaName = "s", + entityTypeNames = listOf("A"), + entityTypeLabels = mapOf("A" to setOf("A")), + entityTypeProperties = mapOf("A" to emptySet()), + relationshipNames = emptyList(), + ) + val two = MetamodelVersion( + schemaName = "s", + entityTypeNames = listOf("A", "B"), + entityTypeLabels = mapOf("A" to setOf("A"), "B" to setOf("B")), + entityTypeProperties = mapOf("A" to emptySet(), "B" to emptySet()), + relationshipNames = emptyList(), + ) + assertNotEquals(one.contentHash, two.contentHash) + assertFalse(one.hasSameContentAs(two)) + } + + @Test + fun `same types under different schema names produce the same hash`() { + // contentHash covers structural content only; the schema name is excluded so that + // dev/prod variants of the same schema compare as equal. + val a = MetamodelVersion.from( + DataDictionary.fromDomainTypes("schema-dev", listOf(DynamicType("Person"))) + ) + val b = MetamodelVersion.from( + DataDictionary.fromDomainTypes("schema-prod", listOf(DynamicType("Person"))) + ) + assertEquals(a.contentHash, b.contentHash) + assertTrue(a.hasSameContentAs(b)) + } + } + + @Nested + inner class GoldenHash { + + /** + * A fixed schema whose every hashed ingredient is spelled out: two type names, one label + * apiece, three properties on one of them, and one relationship. + */ + private fun goldenSchema(): DataDictionary { + val company = DynamicType(name = "Company") + val person = DynamicType( + name = "Person", + ownProperties = listOf( + ValuePropertyDefinition("age"), + ValuePropertyDefinition("email"), + DomainTypePropertyDefinition("worksAt", company), + ), + ) + return DataDictionary.fromDomainTypes("golden-schema", listOf(person, company)) + } + + @Test + fun `the fixture hashes exactly these ingredients`() { + // Guards the golden vector below: if this fails, the fixture changed, not the format. + val version = MetamodelVersion.from(goldenSchema()) + assertEquals(listOf("Company", "Person"), version.entityTypeNames) + assertEquals(setOf("Company"), version.entityTypeLabels["Company"]) + assertEquals(setOf("Person"), version.entityTypeLabels["Person"]) + assertEquals(emptySet(), version.entityTypeProperties["Company"]) + assertEquals( + setOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE), + PropertySignature("email", PropertySignature.Kind.VALUE, "string", Cardinality.ONE), + PropertySignature("worksAt", PropertySignature.Kind.REFERENCE, "Company", Cardinality.ONE), + ), + version.entityTypeProperties["Person"], + ) + assertEquals(listOf("Person-[worksAt]->Company"), version.relationshipNames) + } + + @Test + fun `golden vector — the digest of a fixed schema is pinned to a literal`() { + // This literal is the persisted hash format, deliberately frozen. contentHash is the + // store's natural key and what extracted data records as the version it was created + // under, so changing how the fingerprint is encoded orphans everything already saved + // against it. If you mean to change the format, change this literal in the same commit + // and plan the migration — do not "fix" the test by pasting in whatever the new code + // produces. This vector was last regenerated when property signatures (type and + // cardinality, not just the name) went into the encoding, before anything had been + // persisted against the old form. + assertEquals( + "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00", + MetamodelVersion.from(goldenSchema()).contentHash, + ) + } + + @Test + fun `the golden digest does not depend on the schema name`() { + val renamed = DataDictionary.fromDomainTypes("some-other-name", goldenSchema().domainTypes) + assertEquals( + "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00", + MetamodelVersion.from(renamed).contentHash, + ) + } + } + + @Nested + inner class Relationships { + + /** Same types and properties throughout; only the relationship list varies. */ + private fun versionWithRelationships(vararg relationshipNames: String): MetamodelVersion = + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Company", "Person"), + entityTypeLabels = mapOf("Company" to setOf("Company"), "Person" to setOf("Person")), + entityTypeProperties = mapOf("Company" to emptySet(), "Person" to emptySet()), + relationshipNames = relationshipNames.toList(), + ) + + @Test + fun `adding a relationship changes the hash`() { + val before = versionWithRelationships() + val after = versionWithRelationships("Person-[WORKS_AT]->Company") + assertNotEquals(before.contentHash, after.contentHash) + assertFalse(before.hasSameContentAs(after)) + } + + @Test + fun `removing one relationship of two changes the hash`() { + val both = versionWithRelationships("Person-[WORKS_AT]->Company", "Person-[FOUNDED]->Company") + val one = versionWithRelationships("Person-[WORKS_AT]->Company") + assertNotEquals(both.contentHash, one.contentHash) + } + + @Test + fun `relationship order does not affect the hash`() { + val a = versionWithRelationships("Person-[WORKS_AT]->Company", "Person-[FOUNDED]->Company") + val b = versionWithRelationships("Person-[FOUNDED]->Company", "Person-[WORKS_AT]->Company") + assertEquals(a.contentHash, b.contentHash) + assertTrue(a.hasSameContentAs(b)) + } + + @Test + fun `dropping a relationship property from a dictionary changes the hash end to end`() { + val company = DynamicType(name = "Company") + val withRelationship = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf(DomainTypePropertyDefinition("worksAt", company)), + ), + company, + ), + ) + val withoutRelationship = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person"), company), + ) + + val before = MetamodelVersion.from(withRelationship) + val after = MetamodelVersion.from(withoutRelationship) + + assertEquals(listOf("Person-[worksAt]->Company"), before.relationshipNames) + assertEquals(emptyList(), after.relationshipNames) + assertNotEquals(before.contentHash, after.contentHash) + } + + @Test + fun `a repeated descriptor is deduplicated, not counted twice`() { + val once = versionWithRelationships("Person-[WORKS_AT]->Company") + val twice = versionWithRelationships("Person-[WORKS_AT]->Company", "Person-[WORKS_AT]->Company") + assertEquals(listOf("Person-[WORKS_AT]->Company"), twice.relationshipNames) + assertEquals(once.contentHash, twice.contentHash) + } + + @Test + fun `splitting a type into two same-named declarations hashes like the merged one`() { + // A DataDictionary can hold two "Person" types that both declare worksAt. That renders + // the same descriptor twice, but it is the same schema as one Person declaring it once, + // so it has to be the same hash. + val company = DynamicType(name = "Company") + val worksAt = DomainTypePropertyDefinition("worksAt", company) + val merged = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age"), worksAt), + ), + company, + ), + ) + val split = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age"), worksAt)), + DynamicType(name = "Person", ownProperties = listOf(worksAt)), + company, + ), + ) + + val mergedVersion = MetamodelVersion.from(merged) + val splitVersion = MetamodelVersion.from(split) + + assertEquals(listOf("Person-[worksAt]->Company"), splitVersion.relationshipNames) + assertEquals(mergedVersion.contentHash, splitVersion.contentHash) + assertTrue(mergedVersion.hasSameContentAs(splitVersion)) + } + } + + @Nested + inner class PropertySignatures { + + /** One Person with a single property, described however the test needs it. */ + private fun personWith(property: PropertyDefinition): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = listOf(property))), + ) + + @Test + fun `changing a property's type produces a different hash`() { + val asString = MetamodelVersion.from(personWith(ValuePropertyDefinition("age", type = "string"))) + val asInteger = MetamodelVersion.from(personWith(ValuePropertyDefinition("age", type = "integer"))) + + assertEquals(asString.entityTypeNames, asInteger.entityTypeNames) + assertEquals(asString.entityTypeLabels, asInteger.entityTypeLabels) + assertNotEquals(asString.contentHash, asInteger.contentHash) + assertFalse(asString.hasSameContentAs(asInteger)) + } + + @Test + fun `changing a property's cardinality produces a different hash`() { + val one = MetamodelVersion.from( + personWith(ValuePropertyDefinition("nickname", cardinality = Cardinality.ONE)) + ) + val list = MetamodelVersion.from( + personWith(ValuePropertyDefinition("nickname", cardinality = Cardinality.LIST)) + ) + + assertNotEquals(one.contentHash, list.contentHash) + } + + @Test + fun `changing a relationship's cardinality produces a different hash`() { + val company = DynamicType(name = "Company") + val one = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", company, Cardinality.ONE)) + ) + val many = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", company, Cardinality.LIST)) + ) + + // The rendered descriptor is identical; only the cardinality behind it moved. + assertEquals(one.relationshipNames, many.relationshipNames) + assertNotEquals(one.contentHash, many.contentHash) + } + + @Test + fun `retargeting a relationship produces a different hash`() { + val toCompany = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", DynamicType("Company"))) + ) + val toCharity = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", DynamicType("Charity"))) + ) + + assertNotEquals(toCompany.contentHash, toCharity.contentHash) + } + + @Test + fun `a value property and a reference of the same name hash differently`() { + // "worksAt: string" and "worksAt -> Company" are different schemas even though the + // property name, the declared type name, and the cardinality all read the same. + val asValue = MetamodelVersion.from(personWith(ValuePropertyDefinition("worksAt", type = "Company"))) + val asReference = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", DynamicType("Company"))) + ) + + assertNotEquals(asValue.contentHash, asReference.contentHash) + } + + @Test + fun `descriptions and metadata are not part of the signature`() { + // They steer extraction, but they don't change what the graph can hold. + val plain = MetamodelVersion.from(personWith(ValuePropertyDefinition("age"))) + val documented = MetamodelVersion.from( + personWith( + ValuePropertyDefinition( + name = "age", + description = "how many years the person has been alive", + metadata = mapOf("predicate" to "is aged"), + ) + ) + ) + + assertEquals(plain.contentHash, documented.contentHash) + } + + @Test + fun `the signature records name, kind, type and cardinality`() { + val version = MetamodelVersion.from( + personWith(ValuePropertyDefinition("nicknames", type = "string", cardinality = Cardinality.SET)) + ) + + assertEquals( + setOf(PropertySignature("nicknames", PropertySignature.Kind.VALUE, "string", Cardinality.SET)), + version.entityTypeProperties["Person"], + ) + } + } + + @Nested + inner class Immutability { + + private fun version(): MetamodelVersion = MetamodelVersion.from( + DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age"))), + DynamicType("Company"), + ), + ), + ) + + @Test + fun `the collections a stamp hands back cannot be mutated`() { + // Kotlin's read-only types are a compile-time promise only — a Java caller sees plain + // java.util collections through the getters. These have to refuse at runtime, or a + // caller could reshape a stamp out from under its own precomputed hash. + val version = version() + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeNames as MutableList).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.relationshipNames as MutableList).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeLabels as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeLabels["Person"] as MutableSet).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties["Person"] as MutableSet).clear() + } + } + + @Test + fun `mutating what the caller passed in does not change the stamp`() { + val names = mutableListOf("Person") + val labels = mutableMapOf("Person" to mutableSetOf("Person")) + val properties = mutableMapOf( + "Person" to mutableSetOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ) + val relationships = mutableListOf("Person-[worksAt]->Company") + + val version = MetamodelVersion("test", names, labels, properties, relationships) + val hashAtConstruction = version.contentHash + + names.add("Company") + labels["Person"]!!.add("Agent") + properties["Person"]!!.clear() + relationships.clear() + + assertEquals(listOf("Person"), version.entityTypeNames) + assertEquals(setOf("Person"), version.entityTypeLabels["Person"]) + assertEquals(1, version.entityTypeProperties["Person"]!!.size) + assertEquals(listOf("Person-[worksAt]->Company"), version.relationshipNames) + assertEquals(hashAtConstruction, version.contentHash) + } + } + + @Nested + inner class StructuralConsistency { + + @Test + fun `labels keyed by a type that is not listed are rejected`() { + // Only listed types are walked when hashing, so accepting this would let two stamps + // with different labels share a content hash — and the store's natural key with it. + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = mapOf("Person" to setOf("Person"), "Ghost" to setOf("Ghost")), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + ) + } + assertTrue(thrown.message!!.contains("entityTypeLabels"), thrown.message) + assertTrue(thrown.message!!.contains("Ghost"), thrown.message) + } + + @Test + fun `properties keyed by a type that is not listed are rejected`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Ghost" to setOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ), + relationshipNames = emptyList(), + ) + } + assertTrue(thrown.message!!.contains("entityTypeProperties"), thrown.message) + assertTrue(thrown.message!!.contains("Ghost"), thrown.message) + } + + @Test + fun `entity type names are sorted and deduplicated on the way in`() { + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person", "Company", "Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + ) + assertEquals(listOf("Company", "Person"), version.entityTypeNames) + } + } + + @Nested + inner class VersionMetadata { + + @Test + fun `entity type names are sorted`() { + val version = MetamodelVersion.from(schemaWith("Zebra", "Apple", "Mango")) + assertEquals(listOf("Apple", "Mango", "Zebra"), version.entityTypeNames) + } + + @Test + fun `schema name is captured`() { + val dict = DataDictionary.fromDomainTypes("my-schema", listOf(DynamicType("Person"))) + val version = MetamodelVersion.from(dict) + assertEquals("my-schema", version.schemaName) + } + + @Test + fun `per-type label sets are captured including inherited labels`() { + val dict = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", parents = listOf(DynamicType(name = "Agent")))), + ) + val version = MetamodelVersion.from(dict) + assertEquals(setOf("Person", "Agent"), version.entityTypeLabels["Person"]) + } + + @Test + fun `per-type property sets are captured`() { + val dict = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age"), ValuePropertyDefinition("email")), + ), + ), + ) + val version = MetamodelVersion.from(dict) + assertEquals( + setOf("age", "email"), + version.entityTypeProperties["Person"]!!.map { it.name }.toSet(), + ) + } + + @Test + fun `same-named types with different shapes are merged, not dropped`() { + // A DataDictionary can hold two "Person" types with different shapes. The fingerprint + // must union both, never silently keep only the last — otherwise a label or property + // would disappear from the hash and its later removal would go undetected. + val dict = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + parents = listOf(DynamicType(name = "Agent")), + ownProperties = listOf(ValuePropertyDefinition("age")), + ), + DynamicType( + name = "Person", + parents = listOf(DynamicType(name = "Robot")), + ownProperties = listOf(ValuePropertyDefinition("email")), + ), + ), + ) + val version = MetamodelVersion.from(dict) + assertEquals(setOf("Person", "Agent", "Robot"), version.entityTypeLabels["Person"]) + assertEquals( + setOf("age", "email"), + version.entityTypeProperties["Person"]!!.map { it.name }.toSet(), + ) + // The name is deduped in the sorted name list, not repeated. + assertEquals(listOf("Person"), version.entityTypeNames) + } + } + + @Nested + inner class Labels { + + /** Same type name, but a different parent — so the label set differs while the name does not. */ + private fun personWithParent(parent: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", parents = listOf(DynamicType(name = parent)))), + ) + + @Test + fun `a label-only change produces a different hash`() { + val a = MetamodelVersion.from(personWithParent("Agent")) + val b = MetamodelVersion.from(personWithParent("Actor")) + // The type name set is identical; only the label sets differ. + assertEquals(a.entityTypeNames, b.entityTypeNames) + assertNotEquals(a.contentHash, b.contentHash) + assertFalse(a.hasSameContentAs(b)) + } + + @Test + fun `identical label sets produce the same hash`() { + val a = MetamodelVersion.from(personWithParent("Agent")) + val b = MetamodelVersion.from(personWithParent("Agent")) + assertEquals(a.contentHash, b.contentHash) + } + } + + @Nested + inner class Properties { + + /** Same type name, but a different property set. */ + private fun personWithProperties(vararg props: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = props.map { ValuePropertyDefinition(it) })), + ) + + @Test + fun `a property-only change produces a different hash`() { + val a = MetamodelVersion.from(personWithProperties("age")) + val b = MetamodelVersion.from(personWithProperties("age", "email")) + assertEquals(a.entityTypeNames, b.entityTypeNames) + assertEquals(a.entityTypeLabels, b.entityTypeLabels) + assertNotEquals(a.contentHash, b.contentHash) + assertFalse(a.hasSameContentAs(b)) + } + + @Test + fun `identical property sets produce the same hash`() { + val a = MetamodelVersion.from(personWithProperties("age", "email")) + val b = MetamodelVersion.from(personWithProperties("email", "age")) + assertEquals(a.contentHash, b.contentHash) + } + + @Test + fun `a property name containing the set delimiter does not collide with a split set`() { + // ["a;b"] and ["a", "b"] are genuinely different property sets. A delimiter-joined + // encoding would serialise both as "a;b;" and hash them identically, hiding a real + // (lossy) schema change. Length-prefixed encoding keeps them distinct. + val joined = MetamodelVersion.from(personWithProperties("a;b")) + val split = MetamodelVersion.from(personWithProperties("a", "b")) + assertNotEquals(joined.contentHash, split.contentHash) + assertFalse(joined.hasSameContentAs(split)) + } + } + + @Nested + inner class GovernedSubset { + + private val governed = GovernedTypeSelector { it.name in setOf("Person", "Company") } + + private fun dictionaryOf(vararg types: DomainType): DataDictionary = + DataDictionary.fromDomainTypes("test", types.toList()) + + @Test + fun `the default selector governs everything`() { + val dict = dictionaryOf(DynamicType("Person"), DynamicType("Company")) + assertEquals( + MetamodelVersion.from(dict).contentHash, + MetamodelVersion.from(dict, GovernedTypeSelector.ALL).contentHash, + ) + assertEquals(listOf("Company", "Person"), MetamodelVersion.from(dict, GovernedTypeSelector.ALL).entityTypeNames) + } + + @Test + fun `adding an ungoverned type leaves the hash alone`() { + // The point of per-type governance: extraction proposing a new exploratory type is not + // a schema change, and must not fill the version history with stamps nobody chose. + val before = dictionaryOf(DynamicType("Person"), DynamicType("Company")) + val after = dictionaryOf(DynamicType("Person"), DynamicType("Company"), DynamicType("Sighting")) + + assertEquals( + MetamodelVersion.from(before, governed).contentHash, + MetamodelVersion.from(after, governed).contentHash, + ) + } + + @Test + fun `adding a governed type changes the hash`() { + val before = dictionaryOf(DynamicType("Person")) + val after = dictionaryOf(DynamicType("Person"), DynamicType("Company")) + + assertNotEquals( + MetamodelVersion.from(before, governed).contentHash, + MetamodelVersion.from(after, governed).contentHash, + ) + } + + @Test + fun `reshaping an ungoverned type leaves the hash alone`() { + // Not just the type set: labels and properties on an ungoverned type are out too. + val plain = dictionaryOf(DynamicType("Person"), DynamicType("Sighting")) + val reshaped = dictionaryOf( + DynamicType("Person"), + DynamicType( + name = "Sighting", + parents = listOf(DynamicType("Observation")), + ownProperties = listOf(ValuePropertyDefinition("seenAt")), + ), + ) + + assertEquals( + MetamodelVersion.from(plain, governed).contentHash, + MetamodelVersion.from(reshaped, governed).contentHash, + ) + } + + @Test + fun `reshaping a governed type changes the hash`() { + val plain = dictionaryOf(DynamicType("Person"), DynamicType("Sighting")) + val reshaped = dictionaryOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age"))), + DynamicType("Sighting"), + ) + + assertNotEquals( + MetamodelVersion.from(plain, governed).contentHash, + MetamodelVersion.from(reshaped, governed).contentHash, + ) + } + + @Test + fun `only governed types are stamped`() { + val version = MetamodelVersion.from( + dictionaryOf(DynamicType("Person"), DynamicType("Company"), DynamicType("Sighting")), + governed, + ) + assertEquals(listOf("Company", "Person"), version.entityTypeNames) + assertNull(version.entityTypeLabels["Sighting"]) + assertNull(version.entityTypeProperties["Sighting"]) + } + + @Test + fun `a relationship declared by an ungoverned type is left out`() { + val person = DynamicType("Person") + val withSighting = dictionaryOf( + person, + DynamicType(name = "Sighting", ownProperties = listOf(DomainTypePropertyDefinition("about", person))), + ) + val withoutSighting = dictionaryOf(person) + + val version = MetamodelVersion.from(withSighting, governed) + assertEquals(emptyList(), version.relationshipNames) + assertEquals(MetamodelVersion.from(withoutSighting, governed).contentHash, version.contentHash) + } + + @Test + fun `a governed type's relationship to an ungoverned type stays in the stamp`() { + // The relationship belongs to the type that declares it. Person saying it has a + // `spotted` Sighting is part of Person's declared shape, whether or not Sighting is + // itself governed — and whether or not Sighting is even in the dictionary. + val sighting = DynamicType("Sighting") + val person = DynamicType( + name = "Person", + ownProperties = listOf(DomainTypePropertyDefinition("spotted", sighting)), + ) + + val version = MetamodelVersion.from(dictionaryOf(person, sighting), governed) + assertEquals(listOf("Person-[spotted]->Sighting"), version.relationshipNames) + // Which means listing the ungoverned type in the dictionary changes nothing. + assertEquals(MetamodelVersion.from(dictionaryOf(person), governed).contentHash, version.contentHash) + } + + @Test + fun `dropping a governed type's relationship changes the hash`() { + val sighting = DynamicType("Sighting") + val with = dictionaryOf( + DynamicType(name = "Person", ownProperties = listOf(DomainTypePropertyDefinition("spotted", sighting))), + sighting, + ) + val without = dictionaryOf(DynamicType("Person"), sighting) + + assertNotEquals( + MetamodelVersion.from(with, governed).contentHash, + MetamodelVersion.from(without, governed).contentHash, + ) + } + + @Test + fun `governing nothing stamps an empty schema`() { + val version = MetamodelVersion.from( + dictionaryOf(DynamicType("Person"), DynamicType("Company")), + GovernedTypeSelector { false }, + ) + assertEquals(emptyList(), version.entityTypeNames) + assertEquals(emptyList(), version.relationshipNames) + assertEquals( + MetamodelVersion.from(DataDictionary.fromDomainTypes("test", emptyList())).contentHash, + version.contentHash, + ) + } + + @Test + fun `the schema name is still captured when only a subset is governed`() { + val version = MetamodelVersion.from( + DataDictionary.fromDomainTypes("my-schema", listOf(DynamicType("Person"), DynamicType("Sighting"))), + governed, + ) + assertEquals("my-schema", version.schemaName) + } + } +} diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index f691f3c0..e02f4ec7 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -70,10 +70,13 @@ you need. layers, gated by an API-key filter. - [report.md](report.md) — `dice-report`'s pure projectors that turn queried propositions into human-facing artifacts: structured breakdowns, discovered links, LLM-generated rationale. +- [metamodel-versioning.md](metamodel-versioning.md) — stamping a schema with a content hash so it + can be compared later: per-type governance (you version what you declare), the declared-schema + opt-in seam, and the version store's accumulating history. ## Modules -DICE ships as six Maven modules; [architecture.md](architecture.md#module-map) has the full +DICE ships as seven Maven modules; [architecture.md](architecture.md#module-map) has the full dependency map. Quick pointer to where each is documented: | Module | Documented in | @@ -83,5 +86,6 @@ dependency map. Quick pointer to where each is documented: | `dice-storage-autoconfigure` | [durable-storage.md](durable-storage.md) | | `dice-ingestion` | [ingestion.md](ingestion.md) | | `dice-report` | [report.md](report.md) | +| `dice-metamodel` | [metamodel-versioning.md](metamodel-versioning.md) | | `dice-integration-tests` | not separately documented — exercises the above end-to-end | diff --git a/docs/design/architecture.md b/docs/design/architecture.md index bfddee49..27678091 100644 --- a/docs/design/architecture.md +++ b/docs/design/architecture.md @@ -16,6 +16,7 @@ DICE is a multi-module Maven build. Each module's intent, and what it's allowed | `dice-storage-autoconfigure` | Spring Boot autoconfiguration that wires `dice-storage`'s beans (repository, projectors, trust scorer) into a host application. Depends on `dice-storage`. | | `dice-ingestion` | Content-hash dedup ledger and source adapters that sit in front of `PropositionPipeline`, so the same artifact is never extracted twice concurrently. Depends on `dice`. | | `dice-report` | Rationale and structured report generation over propositions and their lineage. Depends on `dice`. | +| `dice-metamodel` | Schema versioning: content-hash stamps over the governed part of a `DataDictionary`, the declared-schema seam, and the version store contract. Pure JVM — depends on no other DICE module. | | `dice-integration-tests` | End-to-end tests exercising the real Neo4j backend and full pipeline across module boundaries. Depends on `dice`, `dice-ingestion`, `dice-report` (and transitively `dice-storage`). Not shipped. | ```mermaid @@ -25,6 +26,7 @@ flowchart TB autoconf["dice-storage-autoconfigure
(Spring Boot wiring)"] ingestion["dice-ingestion
(dedup ledger)"] report["dice-report
(rationale/reports)"] + metamodel["dice-metamodel
(schema versioning)"] itest["dice-integration-tests"] storage --> dice @@ -37,9 +39,11 @@ flowchart TB ``` `dice` never depends on any other DICE module — it's the leaf of the graph, so every other module -can be added or removed without touching core logic. `dice-storage-autoconfigure` is the only -module that knows about Spring Boot autoconfiguration; plain `dice-storage` stays framework-neutral -so it can be wired by hand outside Spring Boot. +can be added or removed without touching core logic. `dice-metamodel` is a second leaf, with no +edge at all: it stamps a schema and never touches the proposition model, so it stands on Embabel's +agent core types alone. `dice-storage-autoconfigure` is the only module that knows about Spring +Boot autoconfiguration; plain `dice-storage` stays framework-neutral so it can be wired by hand +outside Spring Boot. ### Subsystem design docs @@ -62,6 +66,7 @@ Each subsystem below the module level has its own design note: - [durable-storage](durable-storage.md) — `dice-storage` backend, schema, indexes - [events](events.md) — `DiceEvent` model and emitters - [report](report.md) — `dice-report` rationale and structured reports +- [metamodel-versioning](metamodel-versioning.md) — `MetamodelVersion` stamping, per-type governance - [web-api](web-api.md) — REST surface (`DiscoveryController` and friends) ## System-level map diff --git a/docs/design/metamodel-versioning.md b/docs/design/metamodel-versioning.md new file mode 100644 index 00000000..7e544a6e --- /dev/null +++ b/docs/design/metamodel-versioning.md @@ -0,0 +1,189 @@ +# Metamodel versioning: stamping a schema you can compare later + +DICE extracts entities and relationships against a metamodel — the entity types, their labels and +properties, and the relationships allowed between them. That schema moves. It gets edited as a +domain is understood better, and it can quietly diverge from what a live graph actually holds, +because an integration that used to declare a type can be switched off while its data stays behind. + +Before any of that can be reasoned about, a schema needs an identity you can write down. That's what +this note covers: turning a mutable `DataDictionary` into an immutable stamp, deciding which types +the stamp is even about, and keeping the stamps around so history is answerable. Comparing two +stamps, or a stamp against a live graph, comes later — see [the tiers ahead](#the-tiers-ahead). + +The types live in `dice-metamodel`. It's a small, pure-JVM module: `MetamodelVersion`, +`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, and the `MetamodelVersionStore` +contract. It depends on Embabel's agent core types and nothing else — not `dice`, not a database +driver, not Spring. + +## Declare, stamp, store + +```mermaid +flowchart LR + dict["DataDictionary
(mutable, whole domain)"] + sel["GovernedTypeSelector
which types we version"] + dss["DeclaredSchemaSource
(opt-in seam)"] + stamp["MetamodelVersion
contentHash"] + store[("MetamodelVersionStore")] + history["latestVersion / versionHistory
findVersion(schema, hash)"] + + dict --> dss + sel --> dss + dss -->|declare| stamp + stamp -->|saveVersion| store + store --> history +``` + +Three moving parts, in order. An application says what it governs (`DeclaredSchemaSource`). Stamping +freezes that into a `MetamodelVersion` with a content hash. The store keeps every stamp, so the hash +a piece of extracted data carries can be resolved back into the schema shape it stood for. + +## Why a version is a content hash + +The live schema is mutable. What we store, compare, and stamp onto extracted data has to not be. +`MetamodelVersion.from(dataDictionary)` takes an immutable snapshot — sorted entity type names, the +full label set per type, the full property *signature* set per type, sorted relationship descriptors +— and fingerprints it as a SHA-256 `contentHash`. That stamp is the fixed point. A proposition can +record which one it was extracted under via the `dice.metamodel.version` metadata key, which is how +you later tell which stored knowledge a schema change actually touches. + +Four choices in the fingerprint carry weight. + +**The schema name is excluded.** Two structurally identical schemas hash identically no matter what +they're called, so a dev environment and a prod environment compare cleanly. The cost is that +`contentHash` alone can't tell two same-shaped schemas apart by name, which is why the store keys on +`(schemaName, contentHash)` together. + +**The hash is derived, not passed in.** `MetamodelVersion` computes it from its own structural +fields; there is no constructor parameter for it. That's not tidiness — the hash is the store's +natural key and what `hasSameContentAs` compares. If a caller could supply it, two schemas with +different types could claim the same hash, compare equal, and overwrite each other in storage. + +**Properties go in as signatures, not names.** A property contributes its name, whether it holds a +value or points at another type, that value type (`string`, `integer`, ...) or target type name, and +its cardinality. With names alone, `age` turning from a string into an integer, or a single `worksAt` +becoming a list of them, would leave the hash untouched — a real change to what the graph can hold, +invisible. Descriptions and semantic metadata are deliberately left out: they steer extraction and +matter in a prompt, but they don't change the shape of what gets stored, so re-wording one shouldn't +read as a schema change. + +**Every name, label, and signature component is length-prefixed** (`:`) before hashing, +and each set is preceded by its size. These names come from free-text and LLM extraction; they +routinely contain `;`, `[`, `=`, and spaces. A delimiter-joined encoding would let `["a;b"]` and +`["a", "b"]` produce the same digest — and a schema change that collapsed one into the other would be +invisible. Losing a property silently is exactly the failure versioning exists to catch, so the +encoding is unambiguous rather than merely tidy. + +Concretely, the hashed form is `types:|` followed, for each type name in sorted order, by the +length-prefixed name, then `labels:|` and its sorted labels, then `props:|` and its sorted +signatures — each signature contributing name, kind, type, and cardinality as four length-prefixed +tokens. Then `rels:|` and the sorted relationship descriptors. The schema name appears nowhere. + +Two subtleties about input. A `DataDictionary` can legally hold two domain types sharing a name but +differing in shape, so `from` unions their labels and properties per name rather than letting the +last one win — otherwise a label under that name never reaches the fingerprint, and removing it later +wouldn't change the hash. The same split can render one relationship descriptor twice, so the +constructor sorts *and* deduplicates the type and relationship lists: declaring a type once or +splitting it in two is the same schema, and has to be the same hash. + +The constructor is strict about the rest, too. It copies every collection into a genuinely immutable +one — the JVM kind that throws, not just Kotlin's read-only view, which a Java caller sees straight +through — so nothing can be reshaped out from under the precomputed hash. And it rejects a label or +property map keyed by a type missing from `entityTypeNames`: only listed types are walked when +hashing, so such an entry would never reach `contentHash`, and two structurally different stamps +could end up sharing the store's natural key. `MetamodelVersion` is a plain class rather than a +`data class` for the same reason — a generated `copy()` would hand its arguments straight to the +fields and skip all of it. + +The encoding is a persisted format. `MetamodelVersionTest` pins the digest of a fixed schema to a +literal for that reason — changing how the fingerprint is built orphans every hash already recorded +against it, so it's a migration, not a refactor. + +## Versioning is per type, and opt-in + +A DICE domain is rarely all one thing. Part of it is closed-world: the types you have committed to, +whose shape you want to notice changing. The rest is open-world — exploratory types that extraction +proposes, that come and go, and that nobody has decided about yet. + +Stamping the whole dictionary treats both the same, and that's wrong in a specific, annoying way: +every new exploratory type produces a new content hash, so the version history fills with entries +nobody chose and no comparison means anything. So governance is per type. `GovernedTypeSelector` is +a predicate over `DomainType`, and `MetamodelVersion.from(dictionary, selector)` stamps only what it +governs: + +```kotlin +val governed = setOf("Person", "Company") +val version = MetamodelVersion.from(dataDictionary, GovernedTypeSelector { it.name in governed }) +``` + +Add, remove or reshape an ungoverned type and `contentHash` doesn't move. Do the same to a governed +one and it does. `from(dictionary)` with no selector is the govern-everything case — the same stamp +as before, for a domain that is closed-world throughout. + +The model here is Hibernate's `@Version`: you version what you declare, per entity, and everything +else is left alone. Nothing is inferred, and nothing is versioned by default. + +Relationships follow the type that declares them. A governed type's outgoing relationship is part of +that type's declared shape, so it stays in the stamp even when it points at an ungoverned type; a +relationship declared *by* an ungoverned type is left out entirely. That rule is what makes the +guarantee hold end to end — otherwise an exploratory type wiring itself to `Person` would move +`Person`'s hash. + +`DeclaredSchema.from(dictionary, selector)` applies the same rule to both halves of a declaration, +which is why it exists. A declaration carries the bare relationship type names alongside the stamp, +because `relationshipNames` holds rendered `From-[name]->To` descriptors and reverse-parsing one is +ambiguous when the names themselves can contain a `-[...]->`-shaped substring. Building the stamp +from a governed subset while taking the names from the whole dictionary would declare relationships +the stamp never covered — a mismatch that only surfaces much later, as phantom disagreement. + +## The opt-in seam + +`DeclaredSchemaSource` is where an application says "here is what I govern". It's a single method +returning a `DeclaredSchema`, and it has no default implementation — there's no such thing as a +default declared schema. A consuming app maps whatever it already uses to define its types: + +```kotlin +class MyAppDeclaredSchemaSource( + private val dataDictionary: DataDictionary, +) : DeclaredSchemaSource { + + private val governed = GovernedTypeSelector { it.name in setOf("Person", "Company") } + + override fun declare(): DeclaredSchema = DeclaredSchema.from(dataDictionary, governed) +} +``` + +This is the on-switch for the whole story: no declared schema, no versioning. Nothing stamps on its +own, and the Spring wiring that lands in a later slice activates only when a `DeclaredSchemaSource` +bean is present. An app that hasn't decided what it governs is left entirely alone — which is the +right default for a substrate whose whole point is accepting knowledge it wasn't expecting. + +## History accumulates + +`MetamodelVersionStore` is a port: `saveVersion`, `latestVersion`, `versionHistory`, and +`findVersion(schemaName, contentHash)`. No delete. Versions accumulate, and that's the point — +correlating what a graph holds today against the stamp it was extracted under is only possible if +the old stamps are still there. + +`saveVersion` is an upsert on `(schemaName, contentHash)`, not a blind append. In practice that's +idempotence rather than mutation, because the key carries the content: the hash is derived from +exactly the fields a re-save would overwrite, so anything landing on an existing key has identical +content by construction. "Append-only" is the shape you observe; it isn't a promise the interface +makes, and an implementation isn't expected to reject a re-save. + +`findVersion` is the reverse lookup — a recorded hash back to the schema shape it named. It ships +with a default that scans `versionHistory`, correct everywhere and efficient nowhere; a backend that +can push a keyed lookup down to the database should override it. There's no implementation in this +module. Storage is a separate concern, and a stamp is useful in memory long before anything durable +exists. + +## The tiers ahead + +Versioning is the first of three escalating tiers, and they ship in that order deliberately. +**Stamp and observe** is this slice: identity, history, no opinions. **Detect and report** comes +next — comparing two declared stamps, and comparing a declaration against what a live graph +actually contains, then recording the result. **Quarantine** is last: acting on a lossy change by +marking affected propositions stale rather than deleting them. Each tier is only safe to build on +the one below it, and each is worth having on its own — you can stamp for a year without ever +detecting, and detect for a year without ever quarantining. Rejecting undeclared types at write +time is not on the list yet: extraction is LLM-driven and a type nobody declared is often a real +finding, so throwing it away is the one thing that can't be undone later. diff --git a/pom.xml b/pom.xml index 4bf74b0e..9157f565 100644 --- a/pom.xml +++ b/pom.xml @@ -28,6 +28,7 @@ dice-storage-autoconfigure dice-report dice-ingestion + dice-metamodel dice-integration-tests dice-user-guide @@ -81,6 +82,11 @@ dice-report ${project.version} + + com.embabel.dice + dice-metamodel + ${project.version} + From e087a80a18e696c9f85d6f45036c8fb7fafab00e Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Sun, 30 Aug 2026 20:23:54 -0400 Subject: [PATCH 02/11] docs(metamodel): voice pass on versioning docs and KDoc Rewrite the design doc, KDoc, and comments from this slice in the repo's documentation voice. Comment and doc text only; no code change. --- AGENTS.md | 2 +- CHANGELOG.md | 4 +- README.md | 2 +- .../dice/metamodel/DeclaredSchemaSource.kt | 26 +-- .../dice/metamodel/GovernedTypeSelector.kt | 19 +- .../dice/metamodel/MetamodelVersion.kt | 130 +++++------ .../dice/metamodel/MetamodelVersionStore.kt | 39 ++-- .../dice/metamodel/DeclaredSchemaTest.kt | 4 +- .../metamodel/MetamodelVersionStoreTest.kt | 4 +- .../dice/metamodel/MetamodelVersionTest.kt | 55 +++-- docs/design/INDEX.md | 4 +- docs/design/architecture.md | 8 +- docs/design/metamodel-versioning.md | 211 +++++++++--------- 13 files changed, 246 insertions(+), 262 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8094e7f3..e69d15d0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,7 +11,7 @@ DICE (Domain-Integrated Context Engineering) is a proposition-first knowledge su | `dice-storage-autoconfigure` | Spring Boot auto-configuration that wires the right backend based on `embabel.dice.store.type`, schedules the decay tick, and provides auto-configuration for the multi-signal duplicate collector (properties prefix `embabel.dice.collector`) | | `dice-report` | Output projectors over propositions: rationale (why a fact is believed, with evidence), structured report, and surprising-link discovery | | `dice-ingestion` | Ingestion SPI (artifacts → chunks) with a content-hash dedup ledger so the same source isn't extracted twice | -| `dice-metamodel` | Schema versioning: `MetamodelVersion` content-hash stamps over the governed types of a `DataDictionary`, `GovernedTypeSelector`, the `DeclaredSchemaSource` opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM — no dependency on `dice` | +| `dice-metamodel` | Schema versioning: `MetamodelVersion` content-hash stamps over the governed types of a `DataDictionary`, `GovernedTypeSelector`, the `DeclaredSchemaSource` opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM, with no dependency on `dice` | | `dice-integration-tests` | Test-only: the cross-feature end-to-end canonical-flow harness | ## Build & test diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b71556a..ed877519 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ and the consumer PRs that deliver it). ### Added -- `dice-metamodel` module, first slice: schema versioning — `MetamodelVersion` - content-hash stamping with per-type governance selection, declared-schema +- `dice-metamodel` module, first slice of schema versioning: `MetamodelVersion` + content-hash stamping with per-type governance selection, the declared-schema opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM. **Compatibility: additive.** New module; no existing API touched. diff --git a/README.md b/README.md index 611a9f5e..365e45c3 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,7 @@ recover by reading a single class — see the design notes in [`docs/design/`](d two-phase save, materialised effective confidence, schema-as-beans, and the decay tick. - [Events](docs/design/events.md) — the domain-event model the store and pipeline emit. - [Metamodel versioning](docs/design/metamodel-versioning.md) — content-hash schema stamps, per-type - governance (you version what you declare), the declared-schema opt-in seam, and version history. + governance, the declared-schema opt-in seam, and version history. ## Real-World Example: Impromptu diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt index c6c317f0..fae0628d 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt @@ -22,11 +22,11 @@ import com.embabel.agent.core.DataDictionary * * The bare names travel alongside the stamp rather than being recovered from it, because * [MetamodelVersion.relationshipNames] holds rendered `From-[name]->To` descriptors and - * reverse-parsing one is ambiguous — these names come from free-text and LLM extraction, and can + * reverse-parsing one is ambiguous: these names come from free-text and LLM extraction, and can * themselves contain a `-[...]->`-shaped substring. Whoever builds a declaration holds the - * un-rendered names before anything was stamped, so it passes them straight through. Comparing - * declared relationships against what a graph actually holds needs those bare names, and that - * comparison is the next slice. + * un-rendered names before anything is stamped, so it passes them straight through. Comparing + * declared relationships against what a graph holds needs those bare names, and that comparison is + * the next slice. * * @property version The stamped declared schema. * @property relationshipTypeNames The bare relationship type names [version] allows. @@ -36,7 +36,7 @@ class DeclaredSchema( relationshipTypeNames: Set, ) { - /** Copied to a JVM-immutable set so the declaration can't drift from its stamped [version]. */ + /** Copied into a JVM-immutable set so the declaration can't drift from its stamped [version]. */ val relationshipTypeNames: Set = java.util.Set.copyOf(relationshipTypeNames) override fun equals(other: Any?): Boolean = @@ -56,7 +56,7 @@ class DeclaredSchema( * relationship names the same governed types declare. * * Use this rather than building the two halves separately. Picking a governed subset for - * the stamp but taking relationship names from the whole dictionary would declare + * the stamp while taking relationship names from the whole dictionary would declare * relationships the stamp never covered, and the mismatch would only surface much later, * as phantom disagreement in a drift check. * @@ -79,15 +79,13 @@ class DeclaredSchema( /** * Supplies the schema an application has declared. * - * This is the opt-in seam for the whole versioning story: no declared schema, no versioning. - * Nothing here stamps anything on its own, and the Spring wiring that arrives in a later slice - * activates only when a `DeclaredSchemaSource` bean is present. An application that hasn't decided - * what it governs is left alone. + * This is the opt-in seam for versioning: with no declared schema, nothing is stamped. The Spring + * wiring that arrives in a later slice activates only when a `DeclaredSchemaSource` bean is + * present, so an application that hasn't decided what it governs is left alone. * - * The module has no opinion on where a declared schema comes from — a consuming app implements this - * over whatever it already uses to define its types (a `DataDictionary`, a config file, a - * registry...) and wires it as a bean. There is no default implementation because there is no - * default declared schema. + * A declared schema can come from anywhere. A consuming app implements this over whatever it + * already uses to define its types (a `DataDictionary`, a config file, a registry...) and wires it + * as a bean. There is no default implementation, because there is no default declared schema. */ fun interface DeclaredSchemaSource { diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt index 6fe7aa7c..7acfc216 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt @@ -20,23 +20,22 @@ import com.embabel.agent.core.DomainType /** * Decides which domain types a [MetamodelVersion] stamp covers. * - * A DICE domain is rarely all one thing. Part of it is closed-world — the types you have committed + * A DICE domain is rarely all one thing. Part of it is closed-world: the types you have committed * to, whose shape you want to notice changing. The rest is open-world: exploratory types that - * extraction proposes, that come and go, and that you would rather not have breaking a version - * comparison every time an LLM invents one. Governance is therefore per type and opt-in, in the - * spirit of Hibernate's `@Version`: you version what you declare, and nothing else. + * extraction proposes, that come and go, and that would otherwise break a version comparison every + * time an LLM invents one. Governance is therefore per type and opt-in, in the spirit of + * Hibernate's `@Version`. * - * A selector is just a predicate over types, so the usual form is a lambda over names you already - * hold: + * A selector is a predicate over types, so the usual form is a lambda over names you already hold: * * ```kotlin * val governed = setOf("Person", "Company") * val version = MetamodelVersion.from(dataDictionary, GovernedTypeSelector { it.name in governed }) * ``` * - * Selecting a subset changes what the stamp is *about*, not how it is computed: adding an - * ungoverned type to the dictionary leaves the content hash alone, while touching a governed one - * changes it. + * Selecting a subset changes which types the stamp covers, and leaves the encoding alone. Adding an + * ungoverned type to the dictionary leaves the content hash as it was, while touching a governed + * one changes it. */ fun interface GovernedTypeSelector { @@ -49,7 +48,7 @@ fun interface GovernedTypeSelector { companion object { /** - * Governs every type in the dictionary — the whole-schema stamp, and what + * Governs every type in the dictionary. This is the whole-schema stamp, and what * [MetamodelVersion.from] uses when no selector is given. */ @JvmField diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt index 4951018f..fdd15504 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -27,20 +27,18 @@ import java.util.Objects * What a property looks like structurally: its name, whether it holds a plain value or points at * another type, what that value or target is, and how many of them there can be. * - * Names alone are not enough to version a schema. Turning `age` from a string into an integer, or a - * single `worksAt` into a list of them, is a real change to what the graph can hold — and with only - * names in the stamp both would leave [MetamodelVersion.contentHash] untouched, which is exactly the - * kind of silent drift versioning exists to catch. + * The stamp holds signatures rather than bare names. Turning `age` from a string into an integer, + * or a single `worksAt` into a list of them, changes what the graph can hold; with only names in + * the stamp, both would leave [MetamodelVersion.contentHash] untouched. * - * Descriptions and semantic metadata are deliberately left out. They steer extraction and read - * nicely in a prompt, but they don't change the shape of what gets stored, and re-wording a - * description shouldn't look like a schema change. + * Descriptions and semantic metadata are left out. They steer extraction, but they don't change the + * shape of what gets stored, so re-wording a description doesn't count as a schema change. * * @property name The property name. * @property kind Whether the property holds a value or points at another domain type. * @property type The value type (`string`, `integer`, ...), or the target type's name for a * reference. Empty when [kind] is [Kind.UNKNOWN]. - * @property cardinality How many values the property holds — optional, one, a list, or a set. + * @property cardinality How many values the property holds: optional, one, a list, or a set. */ data class PropertySignature( val name: String, @@ -54,12 +52,12 @@ data class PropertySignature( /** A plain value: string, integer, and so on. */ VALUE, - /** A reference to another domain type — what shows up as a relationship in the graph. */ + /** A reference to another domain type, which shows up as a relationship in the graph. */ REFERENCE, /** - * A property kind the agent platform grew that we don't recognise yet. The name and - * cardinality still count towards the hash; the type is recorded as empty. + * A property kind the agent platform has added that this module doesn't recognise. The + * name and cardinality still count towards the hash; the type is recorded as empty. */ UNKNOWN, } @@ -72,7 +70,7 @@ data class PropertySignature( private val ORDER = compareBy({ it.name }, { it.kind }, { it.type }, { it.cardinality }) /** - * Read the signature off a property as the agent platform describes it. + * Read the structural signature off a property definition. * * @param property The property definition to summarise. * @return Its structural signature. @@ -99,14 +97,13 @@ data class PropertySignature( * schemas. A proposition can record the version it was created under via the * `dice.metamodel.version` metadata key. * - * The hash is computed here, from the structural fields, and is not something a caller can - * supply. That matters because the hash is load-bearing: it is the store's natural key and it is - * what [hasSameContentAs] compares. If it were an ordinary constructor argument, two structurally - * different schemas could claim the same hash, compare as equal, and overwrite each other in - * storage. + * The hash is computed here from the structural fields, and a caller cannot supply it. It is the + * store's natural key and what [hasSameContentAs] compares, so if it were an ordinary constructor + * argument, two structurally different schemas could claim the same hash, compare as equal, and + * overwrite each other in storage. * - * Everything handed to the constructor is copied into genuinely immutable collections and - * canonicalised — type and relationship lists come out sorted and free of duplicates. A stamp whose + * Everything handed to the constructor is copied into JVM-immutable collections and + * canonicalised: type and relationship lists come out sorted and free of duplicates. A stamp whose * collections could still be changed from the outside, or that listed the same relationship twice * because two same-named types declared it, would disagree with its own precomputed hash. This is * a plain class rather than a `data class` for the same reason: a generated `copy()` would hand its @@ -122,10 +119,10 @@ data class PropertySignature( * @property relationshipNames Rendered `From-[name]->To` descriptors for the relationships the * governed types declare, sorted and deduplicated. * @property contentHash SHA-256 hex digest of the schema's entity types, label sets, property - * signatures, and allowed relationships — structural content only, derived from the four fields - * above. The schema name is intentionally excluded so that two structurally identical schemas - * are equal regardless of how they are named. Stable across JVM restarts; any change — even a - * property's type on a type whose name is preserved — produces a different hash. + * signatures, and allowed relationships, derived from the four fields above. The schema name is + * excluded so that two structurally identical schemas are equal regardless of how they are named. + * Stable across JVM restarts. Any structural change produces a different hash, including a + * property's type changing on a type whose name is unchanged. */ class MetamodelVersion( schemaName: String, @@ -147,10 +144,9 @@ class MetamodelVersion( init { // Only types named in entityTypeNames are walked when hashing, so a map entry keyed by - // anything else would never reach contentHash — two stamps holding different labels or - // properties could then share a hash and, with it, the store's natural key. Refuse the - // input rather than quietly ignoring half of it. `this.` is deliberate: the constructor - // parameters are still in scope here and would shadow the copied fields. + // anything else never reaches contentHash, and two stamps holding different labels or + // properties could share a hash and the store's natural key with it. `this.` is required + // here: the constructor parameters are still in scope and would shadow the copied fields. val known = this.entityTypeNames.toSet() val strayLabelKeys = this.entityTypeLabels.keys - known require(strayLabelKeys.isEmpty()) { @@ -167,30 +163,28 @@ class MetamodelVersion( val contentHash: String = fingerprint() /** - * Returns `true` when this version and [other] have the same structural content — - * identical entity types, label sets, property signatures, and relationships — regardless - * of schema name. Uses [contentHash] for the comparison. + * Returns `true` when this version and [other] have the same structural content (entity types, + * label sets, property signatures, and relationships), regardless of schema name. Compares + * [contentHash]. */ fun hasSameContentAs(other: MetamodelVersion): Boolean = contentHash == other.contentHash /** - * The content hash itself: a deterministic encoding of the structural fields, hashed with - * SHA-256 and rendered as lowercase hex. + * A deterministic encoding of the structural fields, hashed with SHA-256 and rendered as + * lowercase hex. * - * This is a **persisted** format. The digest is the store's natural key, so changing the - * encoding orphans everything already saved against it. `MetamodelVersionTest` pins the - * digest of a fixed schema with a literal assertion for exactly that reason — if you - * change the encoding, change that literal deliberately and plan a migration. + * This is a persisted format. The digest is the store's natural key, so changing the encoding + * orphans everything already saved against it. `MetamodelVersionTest` pins the digest of a + * fixed schema with a literal assertion; changing the encoding means changing that literal + * deliberately and planning a migration. */ private fun fingerprint(): String { - // Build a collision-free fingerprint by length-prefixing every name, label, and property - // component (and counting each set) before hashing. A plain delimiter-joined encoding isn't - // safe here: these names come from free-text / LLM extraction and routinely contain ';', - // '[', '=' and spaces, so joining with those characters could make ["a;b"] and ["a", "b"] - // hash identically and hide a real, lossy schema change. Length-prefixing makes the encoding - // unambiguous, so distinct content always yields a distinct hash. - // Schema name is deliberately excluded: two structurally identical schemas must produce - // the same hash even when named differently (e.g. dev vs prod environments). + // Every name, label and property component is length-prefixed, and each set is preceded by + // its size. These names come from free-text and LLM extraction and routinely contain ';', + // '[', '=' and spaces, so a delimiter-joined encoding could make ["a;b"] and ["a", "b"] + // hash identically and hide a lossy schema change. + // The schema name is excluded, so two structurally identical schemas produce the same hash + // even when named differently (e.g. dev vs prod environments). val hashInput = buildString { append("types:").append(entityTypeNames.size).append('|') entityTypeNames.forEach { name -> @@ -241,19 +235,16 @@ class MetamodelVersion( append(token.length).append(':').append(token) } - /** - * Copy [values] into a list nothing can change afterwards — not even Java code holding the - * original, or calling `set` on what a getter hands back. - */ + /** Copy [values] into a JVM-immutable list, which a Java caller can't mutate via a getter. */ private fun immutableCopy(values: List): List = java.util.List.copyOf(values) - /** Copy a map of sets so both the map and every set inside it are genuinely immutable. */ + /** Copy a map of sets so both the map and every set inside it are JVM-immutable. */ private fun immutableCopy(values: Map>): Map> = java.util.Map.copyOf(values.mapValues { (_, set) -> java.util.Set.copyOf(set) }) /** - * Create a [MetamodelVersion] stamp covering every type in the given [DataDictionary] — - * the closed-world reading, where the whole dictionary is governed. + * Create a [MetamodelVersion] stamp covering every type in [dataDictionary], which is the + * right stamp for a domain that is closed-world throughout. * * @param dataDictionary The schema to stamp. * @return An immutable version stamp. @@ -265,18 +256,18 @@ class MetamodelVersion( /** * Create a [MetamodelVersion] stamp covering only the types [selector] governs. * - * DICE domains are usually part closed-world and part open-world. You commit to some types - * and want to know the moment their shape moves; the rest are exploratory, proposed by + * DICE domains are usually part closed-world and part open-world. Some types are committed + * to, and their shape changing is worth noticing; the rest are exploratory, proposed by * extraction, and churn by design. Stamping the whole dictionary makes that churn look like - * a schema change — every new exploratory type produces a new content hash, and the version - * history fills with noise nobody chose. So governance is per type and opt-in, the way - * Hibernate's `@Version` is per entity: you version what you declare. + * a schema change: every new exploratory type produces a new content hash, and the version + * history fills with entries nobody chose. Governance is therefore per type and opt-in, the + * way Hibernate's `@Version` is per entity. * - * Concretely, adding, removing or reshaping an ungoverned type leaves [contentHash] - * untouched; doing the same to a governed one changes it. Relationships follow the type - * that declares them — a governed type's outgoing relationship is part of that type's - * declared shape and stays in the stamp even when it points at an ungoverned type, while a - * relationship declared *by* an ungoverned type is left out entirely. + * Adding, removing or reshaping an ungoverned type leaves [contentHash] untouched; doing + * the same to a governed one changes it. Relationships follow the type that declares them. + * A governed type's outgoing relationship is part of that type's declared shape and stays + * in the stamp even when it points at an ungoverned type, while a relationship declared + * *by* an ungoverned type is left out entirely. * * @param dataDictionary The schema to stamp. * @param selector Which of its types are under governance. [GovernedTypeSelector.ALL] @@ -289,10 +280,9 @@ class MetamodelVersion( // A DataDictionary can legally hold two domain types that share a name but differ in // shape (DynamicType is a data class, so same-named instances with different labels are - // not equal and both survive a set). Merge their labels and properties by union per name - // rather than letting associate() keep only the last — otherwise a label or property - // present under that name would vanish from the fingerprint, and later removing it - // wouldn't change the hash, hiding a real change. + // not equal and both survive a set). Labels and properties are unioned per name. + // Keeping only the last would drop a label or property from the fingerprint, and + // removing it later wouldn't change the hash. val entityTypeLabels = governedTypes .groupBy { it.name } .mapValues { (_, types) -> types.flatMap { it.labels }.toSet() } @@ -302,8 +292,8 @@ class MetamodelVersion( .mapValues { (_, types) -> types.flatMap { type -> type.properties.map(PropertySignature::of) }.toSet() } // Splitting one type into two same-named declarations, or merging two back into one, - // can render the same relationship descriptor twice. That's the same schema either way, - // so it has to be the same hash — the constructor sorts and deduplicates both lists. + // can render the same relationship descriptor twice. It is the same schema either way, + // so it has to be the same hash; the constructor sorts and deduplicates both lists. val relationshipNames = dataDictionary.allowedRelationships() .filter { selector.governs(it.from) } .map { rel -> "${rel.from.name}-[${rel.name}]->${rel.to.name}" } @@ -318,10 +308,10 @@ class MetamodelVersion( } /** - * The bare relationship type names the governed types declare — the same relationships - * [from] renders into [relationshipNames], but un-rendered. + * The bare relationship type names the governed types declare: the same relationships + * [from] renders into [relationshipNames], un-rendered. * - * Kept here so the governance rule ("a relationship belongs to the type that declares it") + * Kept here so the governance rule (a relationship belongs to the type that declares it) * lives in one place, and a [DeclaredSchema] can't drift from the stamp beside it. */ internal fun governedRelationshipTypeNames( diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt index e13afada..33c8e0fd 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt @@ -16,28 +16,27 @@ package com.embabel.dice.metamodel /** - * Durable store for metamodel version stamps. History is the point: keeping every stamp a schema - * has ever had is what later lets you say when a shape changed and what knowledge was extracted - * under which version. Implementations must preserve saved stamps exactly as recorded, including - * set contents and ordering guarantees below. + * Durable store for metamodel version stamps. Keeping every stamp a schema has ever had is what + * later lets you say when a shape changed and what knowledge was extracted under which version. + * Implementations must preserve saved stamps exactly as recorded, including set contents and the + * ordering guarantees below. * * **What "save" means here.** [saveVersion] is an upsert on the natural key `(schemaName, - * contentHash)`, not a blind append. Saving a stamp whose key already exists doesn't create a - * second record: the existing one is matched and its non-key content is overwritten with what you - * just passed. In practice that's idempotence rather than mutation, because the key carries the - * content — the hash is derived from exactly the fields a re-save would overwrite, so anything - * landing on an existing key has identical content by construction. Nothing is ever deleted, and - * records with different keys always coexist, so history accumulates. Implementations are not - * expected to reject a re-save. + * contentHash)`. Saving a stamp whose key already exists matches the existing record and + * overwrites its non-key content, so no second record appears. That is idempotence in practice, + * because the key carries the content: the hash is derived from exactly the fields a re-save would + * overwrite, so anything landing on an existing key has identical content by construction. Nothing + * is deleted, and records with different keys always coexist, so history accumulates. + * Implementations are not expected to reject a re-save. * - * There is no delete, and no drift or diffing here. This contract covers stamping and recall only; - * comparing a declaration against a live graph is a separate concern with its own store contract. + * This contract covers stamping and recall. Comparing a declaration against a live graph is a + * separate concern with its own store contract. */ interface MetamodelVersionStore { /** * Save a version stamp, keyed on `(schemaName, contentHash)`. Saving the same version twice - * leaves one stored version, not two. + * leaves one stored version. * * @param version The version to save. */ @@ -47,8 +46,8 @@ interface MetamodelVersionStore { * Return the most recently saved version for the given schema name, or `null` if no versions * have been recorded. * - * "Most recent" is logical write order, not stamp content — re-saving an existing version does - * not make it the latest again, since the write matched a record that was already there. + * "Most recent" means logical write order. Re-saving an existing version does not make it the + * latest again, because the write matched a record that was already there. * * @param schemaName The schema to look up. * @return The latest [MetamodelVersion], or `null` if unknown. @@ -66,12 +65,12 @@ interface MetamodelVersionStore { /** * Look up one exact stamp by its natural key. * - * This is how you resolve a recorded hash — the one a proposition carries as the version it was - * extracted under — back into the schema shape it stood for. + * Resolves a recorded hash, such as the one a proposition carries as the version it was + * extracted under, back into the schema shape it stood for. * * The default scans [versionHistory], which is correct for any implementation but reads the - * whole history to answer a keyed question. A backend that can push the lookup down (a database - * `MATCH` on the key rather than an in-memory `filter`) should override it. + * whole history to answer a keyed question. A backend that can push the lookup down to the + * database (a keyed `MATCH` rather than an in-memory `filter`) should override it. * * @param schemaName The schema the version belongs to. * @param contentHash The [MetamodelVersion.contentHash] to find. diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt index f217d17a..b745d2e0 100644 --- a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt @@ -46,8 +46,8 @@ class DeclaredSchemaTest { @Test fun `bare relationship names follow governance, so they can't disagree with the stamp`() { // `about` is declared by the ungoverned Sighting, so it is out of both halves. Taking the - // stamp from a subset but the names from the whole dictionary would declare a relationship - // the stamp never covered. + // stamp from a governed subset while taking the names from the whole dictionary would + // declare a relationship the stamp never covered. val declared = DeclaredSchema.from(dictionary(), GovernedTypeSelector { it.name in setOf("Person", "Company") }) assertEquals(setOf("worksAt"), declared.relationshipTypeNames) diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt index 874c9fbf..e83a0f88 100644 --- a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt @@ -65,8 +65,8 @@ class MetamodelVersionStoreTest { @Test fun `findVersion is scoped to the schema name`() { - // Two schemas can hold structurally identical versions — the hash excludes the name — so - // the lookup has to match on both halves of the key, not just the hash. + // Two schemas can hold structurally identical versions, because the hash excludes the + // name, so the lookup has to match on both halves of the key. val store = InMemoryVersionStore() val mine = version("mine", "Person") store.saveVersion(mine) diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt index 4b8259be..195a042d 100644 --- a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt @@ -77,8 +77,6 @@ class MetamodelVersionTest { @Test fun `a caller cannot supply the hash — it is always derived from the content`() { - // The whole point of deriving it: two versions that differ structurally can never claim - // the same hash, because there is no way to hand one in. val one = MetamodelVersion( schemaName = "s", entityTypeNames = listOf("A"), @@ -99,8 +97,8 @@ class MetamodelVersionTest { @Test fun `same types under different schema names produce the same hash`() { - // contentHash covers structural content only; the schema name is excluded so that - // dev/prod variants of the same schema compare as equal. + // The schema name is excluded from contentHash, so dev and prod variants of one schema + // compare as equal. val a = MetamodelVersion.from( DataDictionary.fromDomainTypes("schema-dev", listOf(DynamicType("Person"))) ) @@ -153,14 +151,13 @@ class MetamodelVersionTest { @Test fun `golden vector — the digest of a fixed schema is pinned to a literal`() { - // This literal is the persisted hash format, deliberately frozen. contentHash is the - // store's natural key and what extracted data records as the version it was created - // under, so changing how the fingerprint is encoded orphans everything already saved - // against it. If you mean to change the format, change this literal in the same commit - // and plan the migration — do not "fix" the test by pasting in whatever the new code - // produces. This vector was last regenerated when property signatures (type and - // cardinality, not just the name) went into the encoding, before anything had been - // persisted against the old form. + // This literal pins the persisted hash format. contentHash is the store's natural key + // and what extracted data records as the version it was created under, so changing how + // the fingerprint is encoded orphans everything already saved against it. Changing the + // format means changing this literal in the same commit and planning the migration; + // don't "fix" the test by pasting in whatever the new code produces. This vector was + // last regenerated when property signatures (type and cardinality, not just the name) + // went into the encoding, before anything had been persisted against the old form. assertEquals( "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00", MetamodelVersion.from(goldenSchema()).contentHash, @@ -249,9 +246,9 @@ class MetamodelVersionTest { @Test fun `splitting a type into two same-named declarations hashes like the merged one`() { - // A DataDictionary can hold two "Person" types that both declare worksAt. That renders - // the same descriptor twice, but it is the same schema as one Person declaring it once, - // so it has to be the same hash. + // A DataDictionary can hold two "Person" types that both declare worksAt, which renders + // the same descriptor twice. It is the same schema as one Person declaring it once, so + // it has to be the same hash. val company = DynamicType(name = "Company") val worksAt = DomainTypePropertyDefinition("worksAt", company) val merged = DataDictionary.fromDomainTypes( @@ -399,7 +396,7 @@ class MetamodelVersionTest { @Test fun `the collections a stamp hands back cannot be mutated`() { - // Kotlin's read-only types are a compile-time promise only — a Java caller sees plain + // Kotlin's read-only types are a compile-time promise; a Java caller sees plain // java.util collections through the getters. These have to refuse at runtime, or a // caller could reshape a stamp out from under its own precomputed hash. val version = version() @@ -468,7 +465,7 @@ class MetamodelVersionTest { @Test fun `labels keyed by a type that is not listed are rejected`() { // Only listed types are walked when hashing, so accepting this would let two stamps - // with different labels share a content hash — and the store's natural key with it. + // with different labels share a content hash, and the store's natural key with it. val thrown = assertThrows { MetamodelVersion( schemaName = "test", @@ -561,8 +558,8 @@ class MetamodelVersionTest { @Test fun `same-named types with different shapes are merged, not dropped`() { // A DataDictionary can hold two "Person" types with different shapes. The fingerprint - // must union both, never silently keep only the last — otherwise a label or property - // would disappear from the hash and its later removal would go undetected. + // unions both. Keeping only the last would drop a label or property from the hash, and + // its later removal would go undetected. val dict = DataDictionary.fromDomainTypes( "test", listOf( @@ -584,7 +581,7 @@ class MetamodelVersionTest { setOf("age", "email"), version.entityTypeProperties["Person"]!!.map { it.name }.toSet(), ) - // The name is deduped in the sorted name list, not repeated. + // The name is deduped in the sorted name list. assertEquals(listOf("Person"), version.entityTypeNames) } } @@ -592,7 +589,7 @@ class MetamodelVersionTest { @Nested inner class Labels { - /** Same type name, but a different parent — so the label set differs while the name does not. */ + /** Same type name with a different parent, so the label set differs while the name doesn't. */ private fun personWithParent(parent: String): DataDictionary = DataDictionary.fromDomainTypes( "test", @@ -646,9 +643,9 @@ class MetamodelVersionTest { @Test fun `a property name containing the set delimiter does not collide with a split set`() { - // ["a;b"] and ["a", "b"] are genuinely different property sets. A delimiter-joined - // encoding would serialise both as "a;b;" and hash them identically, hiding a real - // (lossy) schema change. Length-prefixed encoding keeps them distinct. + // ["a;b"] and ["a", "b"] are different property sets. A delimiter-joined encoding would + // serialise both as "a;b;" and hash them identically, hiding a lossy schema change. + // Length-prefixed encoding keeps them distinct. val joined = MetamodelVersion.from(personWithProperties("a;b")) val split = MetamodelVersion.from(personWithProperties("a", "b")) assertNotEquals(joined.contentHash, split.contentHash) @@ -676,8 +673,8 @@ class MetamodelVersionTest { @Test fun `adding an ungoverned type leaves the hash alone`() { - // The point of per-type governance: extraction proposing a new exploratory type is not - // a schema change, and must not fill the version history with stamps nobody chose. + // Per-type governance exists so that extraction proposing a new exploratory type + // doesn't fill the version history with stamps nobody chose. val before = dictionaryOf(DynamicType("Person"), DynamicType("Company")) val after = dictionaryOf(DynamicType("Person"), DynamicType("Company"), DynamicType("Sighting")) @@ -700,7 +697,7 @@ class MetamodelVersionTest { @Test fun `reshaping an ungoverned type leaves the hash alone`() { - // Not just the type set: labels and properties on an ungoverned type are out too. + // Labels and properties on an ungoverned type are excluded along with its name. val plain = dictionaryOf(DynamicType("Person"), DynamicType("Sighting")) val reshaped = dictionaryOf( DynamicType("Person"), @@ -760,7 +757,7 @@ class MetamodelVersionTest { fun `a governed type's relationship to an ungoverned type stays in the stamp`() { // The relationship belongs to the type that declares it. Person saying it has a // `spotted` Sighting is part of Person's declared shape, whether or not Sighting is - // itself governed — and whether or not Sighting is even in the dictionary. + // itself governed, and whether or not Sighting is in the dictionary at all. val sighting = DynamicType("Sighting") val person = DynamicType( name = "Person", @@ -769,7 +766,7 @@ class MetamodelVersionTest { val version = MetamodelVersion.from(dictionaryOf(person, sighting), governed) assertEquals(listOf("Person-[spotted]->Sighting"), version.relationshipNames) - // Which means listing the ungoverned type in the dictionary changes nothing. + // Listing the ungoverned type in the dictionary therefore changes nothing. assertEquals(MetamodelVersion.from(dictionaryOf(person), governed).contentHash, version.contentHash) } diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index e02f4ec7..01c0a4d8 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -71,8 +71,8 @@ you need. - [report.md](report.md) — `dice-report`'s pure projectors that turn queried propositions into human-facing artifacts: structured breakdowns, discovered links, LLM-generated rationale. - [metamodel-versioning.md](metamodel-versioning.md) — stamping a schema with a content hash so it - can be compared later: per-type governance (you version what you declare), the declared-schema - opt-in seam, and the version store's accumulating history. + can be compared later: per-type governance, the declared-schema opt-in seam, and the version + store's accumulating history. ## Modules diff --git a/docs/design/architecture.md b/docs/design/architecture.md index 27678091..6e2570cb 100644 --- a/docs/design/architecture.md +++ b/docs/design/architecture.md @@ -16,7 +16,7 @@ DICE is a multi-module Maven build. Each module's intent, and what it's allowed | `dice-storage-autoconfigure` | Spring Boot autoconfiguration that wires `dice-storage`'s beans (repository, projectors, trust scorer) into a host application. Depends on `dice-storage`. | | `dice-ingestion` | Content-hash dedup ledger and source adapters that sit in front of `PropositionPipeline`, so the same artifact is never extracted twice concurrently. Depends on `dice`. | | `dice-report` | Rationale and structured report generation over propositions and their lineage. Depends on `dice`. | -| `dice-metamodel` | Schema versioning: content-hash stamps over the governed part of a `DataDictionary`, the declared-schema seam, and the version store contract. Pure JVM — depends on no other DICE module. | +| `dice-metamodel` | Schema versioning: content-hash stamps over the governed part of a `DataDictionary`, the declared-schema seam, and the version store contract. Pure JVM. Depends on no other DICE module. | | `dice-integration-tests` | End-to-end tests exercising the real Neo4j backend and full pipeline across module boundaries. Depends on `dice`, `dice-ingestion`, `dice-report` (and transitively `dice-storage`). Not shipped. | ```mermaid @@ -39,9 +39,9 @@ flowchart TB ``` `dice` never depends on any other DICE module — it's the leaf of the graph, so every other module -can be added or removed without touching core logic. `dice-metamodel` is a second leaf, with no -edge at all: it stamps a schema and never touches the proposition model, so it stands on Embabel's -agent core types alone. `dice-storage-autoconfigure` is the only module that knows about Spring +can be added or removed without touching core logic. `dice-metamodel` is a second leaf with no +edges: it stamps a schema, and depends only on Embabel's agent core types. +`dice-storage-autoconfigure` is the only module that knows about Spring Boot autoconfiguration; plain `dice-storage` stays framework-neutral so it can be wired by hand outside Spring Boot. diff --git a/docs/design/metamodel-versioning.md b/docs/design/metamodel-versioning.md index 7e544a6e..cdb8ead7 100644 --- a/docs/design/metamodel-versioning.md +++ b/docs/design/metamodel-versioning.md @@ -1,19 +1,20 @@ -# Metamodel versioning: stamping a schema you can compare later +# Metamodel versioning: stamping, governance, and history -DICE extracts entities and relationships against a metamodel — the entity types, their labels and +DICE stamps the governed part of a metamodel with a content hash, so a schema has an identity you +can record against extracted data and compare later. + +DICE extracts entities and relationships against a metamodel: the entity types, their labels and properties, and the relationships allowed between them. That schema moves. It gets edited as a -domain is understood better, and it can quietly diverge from what a live graph actually holds, -because an integration that used to declare a type can be switched off while its data stays behind. +domain is understood better, and it can diverge from what a live graph holds, because an +integration that used to declare a type can be switched off while its data stays behind. -Before any of that can be reasoned about, a schema needs an identity you can write down. That's what -this note covers: turning a mutable `DataDictionary` into an immutable stamp, deciding which types -the stamp is even about, and keeping the stamps around so history is answerable. Comparing two -stamps, or a stamp against a live graph, comes later — see [the tiers ahead](#the-tiers-ahead). +This note covers turning a mutable `DataDictionary` into an immutable stamp, deciding which types +the stamp is about, and keeping the stamps so history is answerable. Comparing two stamps, or a +stamp against a live graph, comes later — see [the tiers ahead](#the-tiers-ahead). -The types live in `dice-metamodel`. It's a small, pure-JVM module: `MetamodelVersion`, +The types live in `dice-metamodel`, a small pure-JVM module: `MetamodelVersion`, `GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, and the `MetamodelVersionStore` -contract. It depends on Embabel's agent core types and nothing else — not `dice`, not a database -driver, not Spring. +contract. It depends on Embabel's agent core types and nothing else. ## Declare, stamp, store @@ -33,82 +34,80 @@ flowchart LR store --> history ``` -Three moving parts, in order. An application says what it governs (`DeclaredSchemaSource`). Stamping -freezes that into a `MetamodelVersion` with a content hash. The store keeps every stamp, so the hash -a piece of extracted data carries can be resolved back into the schema shape it stood for. +Three moving parts, in order. An application says what it governs (`DeclaredSchemaSource`). +Stamping freezes that into a `MetamodelVersion` with a content hash. The store keeps every stamp, +so the hash a piece of extracted data carries can be resolved back into the schema shape it stood +for. ## Why a version is a content hash -The live schema is mutable. What we store, compare, and stamp onto extracted data has to not be. -`MetamodelVersion.from(dataDictionary)` takes an immutable snapshot — sorted entity type names, the -full label set per type, the full property *signature* set per type, sorted relationship descriptors -— and fingerprints it as a SHA-256 `contentHash`. That stamp is the fixed point. A proposition can -record which one it was extracted under via the `dice.metamodel.version` metadata key, which is how -you later tell which stored knowledge a schema change actually touches. +The live schema is mutable, so stamping means taking an immutable snapshot of it. +`MetamodelVersion.from(dataDictionary)` snapshots sorted entity type names, the full label set per +type, the full property *signature* set per type, and sorted relationship descriptors, then +fingerprints them as a SHA-256 `contentHash`. A proposition records the version it was extracted +under in the `dice.metamodel.version` metadata key, which is how you later tell which stored +knowledge a schema change touches. -Four choices in the fingerprint carry weight. +Four choices in the fingerprint matter. -**The schema name is excluded.** Two structurally identical schemas hash identically no matter what +**The schema name is excluded.** Two structurally identical schemas hash identically whatever they're called, so a dev environment and a prod environment compare cleanly. The cost is that -`contentHash` alone can't tell two same-shaped schemas apart by name, which is why the store keys on -`(schemaName, contentHash)` together. +`contentHash` alone can't tell two same-shaped schemas apart by name, which is why the store keys +on `(schemaName, contentHash)` together. -**The hash is derived, not passed in.** `MetamodelVersion` computes it from its own structural -fields; there is no constructor parameter for it. That's not tidiness — the hash is the store's -natural key and what `hasSameContentAs` compares. If a caller could supply it, two schemas with -different types could claim the same hash, compare equal, and overwrite each other in storage. +**The hash is derived inside the class.** `MetamodelVersion` computes it from its own structural +fields; there is no constructor parameter for it. The hash is the store's natural key and what +`hasSameContentAs` compares. If a caller could supply it, two schemas with different types could +claim the same hash, compare equal, and overwrite each other in storage. **Properties go in as signatures, not names.** A property contributes its name, whether it holds a -value or points at another type, that value type (`string`, `integer`, ...) or target type name, and -its cardinality. With names alone, `age` turning from a string into an integer, or a single `worksAt` -becoming a list of them, would leave the hash untouched — a real change to what the graph can hold, -invisible. Descriptions and semantic metadata are deliberately left out: they steer extraction and -matter in a prompt, but they don't change the shape of what gets stored, so re-wording one shouldn't -read as a schema change. +value or points at another type, that value type (`string`, `integer`, ...) or target type name, +and its cardinality. With names alone, `age` turning from a string into an integer, or a single +`worksAt` becoming a list of them, would leave the hash untouched, though both change what the +graph can hold. Descriptions and semantic metadata are left out: they steer extraction, but they +don't change the shape of what gets stored, so re-wording one doesn't count as a schema change. **Every name, label, and signature component is length-prefixed** (`:`) before hashing, -and each set is preceded by its size. These names come from free-text and LLM extraction; they +and each set is preceded by its size. These names come from free-text and LLM extraction, and routinely contain `;`, `[`, `=`, and spaces. A delimiter-joined encoding would let `["a;b"]` and -`["a", "b"]` produce the same digest — and a schema change that collapsed one into the other would be -invisible. Losing a property silently is exactly the failure versioning exists to catch, so the -encoding is unambiguous rather than merely tidy. - -Concretely, the hashed form is `types:|` followed, for each type name in sorted order, by the -length-prefixed name, then `labels:|` and its sorted labels, then `props:|` and its sorted -signatures — each signature contributing name, kind, type, and cardinality as four length-prefixed -tokens. Then `rels:|` and the sorted relationship descriptors. The schema name appears nowhere. - -Two subtleties about input. A `DataDictionary` can legally hold two domain types sharing a name but -differing in shape, so `from` unions their labels and properties per name rather than letting the -last one win — otherwise a label under that name never reaches the fingerprint, and removing it later -wouldn't change the hash. The same split can render one relationship descriptor twice, so the -constructor sorts *and* deduplicates the type and relationship lists: declaring a type once or -splitting it in two is the same schema, and has to be the same hash. - -The constructor is strict about the rest, too. It copies every collection into a genuinely immutable -one — the JVM kind that throws, not just Kotlin's read-only view, which a Java caller sees straight -through — so nothing can be reshaped out from under the precomputed hash. And it rejects a label or -property map keyed by a type missing from `entityTypeNames`: only listed types are walked when -hashing, so such an entry would never reach `contentHash`, and two structurally different stamps -could end up sharing the store's natural key. `MetamodelVersion` is a plain class rather than a -`data class` for the same reason — a generated `copy()` would hand its arguments straight to the -fields and skip all of it. +`["a", "b"]` produce the same digest, hiding a schema change that collapsed one into the other. +Length-prefixing makes the encoding unambiguous, so distinct content always yields a distinct hash. + +The hashed form is `types:|` followed, for each type name in sorted order, by the length-prefixed +name, then `labels:|` and its sorted labels, then `props:|` and its sorted signatures — each +signature contributing name, kind, type, and cardinality as four length-prefixed tokens. Then +`rels:|` and the sorted relationship descriptors. The schema name appears nowhere. + +Two things about the input. A `DataDictionary` can legally hold two domain types sharing a name but +differing in shape, so `from` unions their labels and properties per name. Keeping only the last +would drop a label from the fingerprint, and removing that label later wouldn't change the hash. +The same split can render one relationship descriptor twice, so the constructor sorts *and* +deduplicates the type and relationship lists: declaring a type once or splitting it in two is the +same schema, and has to be the same hash. + +The constructor is strict about the rest, too. It copies every collection into a JVM-immutable one. +Kotlin's read-only view is a compile-time promise that a Java caller sees straight through, so it +wouldn't stop anything being reshaped out from under the precomputed hash. The constructor also +rejects a label or property map keyed by a type missing from `entityTypeNames`: only listed types +are walked when hashing, so such an entry would never reach `contentHash`, and two structurally +different stamps could end up sharing the store's natural key. `MetamodelVersion` is a plain class +rather than a `data class` for the same reason: a generated `copy()` would hand its arguments +straight to the fields and skip all of that. The encoding is a persisted format. `MetamodelVersionTest` pins the digest of a fixed schema to a -literal for that reason — changing how the fingerprint is built orphans every hash already recorded -against it, so it's a migration, not a refactor. +literal, because changing how the fingerprint is built orphans every hash already recorded against +it, and needs a migration. ## Versioning is per type, and opt-in A DICE domain is rarely all one thing. Part of it is closed-world: the types you have committed to, -whose shape you want to notice changing. The rest is open-world — exploratory types that extraction +whose shape you want to notice changing. The rest is open-world: exploratory types that extraction proposes, that come and go, and that nobody has decided about yet. -Stamping the whole dictionary treats both the same, and that's wrong in a specific, annoying way: -every new exploratory type produces a new content hash, so the version history fills with entries -nobody chose and no comparison means anything. So governance is per type. `GovernedTypeSelector` is -a predicate over `DomainType`, and `MetamodelVersion.from(dictionary, selector)` stamps only what it -governs: +Stamping the whole dictionary treats both alike. Every new exploratory type then produces a new +content hash, the version history fills with entries nobody chose, and comparisons stop meaning +anything. Governance is therefore per type. `GovernedTypeSelector` is a predicate over +`DomainType`, and `MetamodelVersion.from(dictionary, selector)` stamps only what it governs: ```kotlin val governed = setOf("Person", "Company") @@ -116,30 +115,29 @@ val version = MetamodelVersion.from(dataDictionary, GovernedTypeSelector { it.na ``` Add, remove or reshape an ungoverned type and `contentHash` doesn't move. Do the same to a governed -one and it does. `from(dictionary)` with no selector is the govern-everything case — the same stamp -as before, for a domain that is closed-world throughout. +one and it does. `from(dictionary)` with no selector governs everything, which is the right stamp +for a domain that is closed-world throughout. -The model here is Hibernate's `@Version`: you version what you declare, per entity, and everything -else is left alone. Nothing is inferred, and nothing is versioned by default. +The model is Hibernate's `@Version`: governance is declared per entity, and nothing is versioned by +default. Relationships follow the type that declares them. A governed type's outgoing relationship is part of that type's declared shape, so it stays in the stamp even when it points at an ungoverned type; a -relationship declared *by* an ungoverned type is left out entirely. That rule is what makes the -guarantee hold end to end — otherwise an exploratory type wiring itself to `Person` would move -`Person`'s hash. +relationship declared *by* an ungoverned type is left out entirely. Without that rule, an +exploratory type wiring itself to `Person` would move `Person`'s hash. `DeclaredSchema.from(dictionary, selector)` applies the same rule to both halves of a declaration, which is why it exists. A declaration carries the bare relationship type names alongside the stamp, because `relationshipNames` holds rendered `From-[name]->To` descriptors and reverse-parsing one is ambiguous when the names themselves can contain a `-[...]->`-shaped substring. Building the stamp from a governed subset while taking the names from the whole dictionary would declare relationships -the stamp never covered — a mismatch that only surfaces much later, as phantom disagreement. +the stamp never covered, and the mismatch would only surface much later, as phantom disagreement. ## The opt-in seam -`DeclaredSchemaSource` is where an application says "here is what I govern". It's a single method -returning a `DeclaredSchema`, and it has no default implementation — there's no such thing as a -default declared schema. A consuming app maps whatever it already uses to define its types: +`DeclaredSchemaSource` is where an application states what it governs: a single method returning a +`DeclaredSchema`. It has no default implementation, because there is no default declared schema. A +consuming app maps whatever it already uses to define its types: ```kotlin class MyAppDeclaredSchemaSource( @@ -152,38 +150,41 @@ class MyAppDeclaredSchemaSource( } ``` -This is the on-switch for the whole story: no declared schema, no versioning. Nothing stamps on its -own, and the Spring wiring that lands in a later slice activates only when a `DeclaredSchemaSource` -bean is present. An app that hasn't decided what it governs is left entirely alone — which is the -right default for a substrate whose whole point is accepting knowledge it wasn't expecting. +Versioning starts here: with no declared schema, nothing is stamped. The Spring wiring that lands +in a later slice activates only when a `DeclaredSchemaSource` bean is present, so an application +that hasn't decided what it governs is left alone. ## History accumulates -`MetamodelVersionStore` is a port: `saveVersion`, `latestVersion`, `versionHistory`, and -`findVersion(schemaName, contentHash)`. No delete. Versions accumulate, and that's the point — -correlating what a graph holds today against the stamp it was extracted under is only possible if -the old stamps are still there. +`MetamodelVersionStore` is a port with four operations: `saveVersion`, `latestVersion`, +`versionHistory`, and `findVersion(schemaName, contentHash)`. There is no delete. Correlating what +a graph holds today against the stamp it was extracted under only works while the old stamps are +still there. -`saveVersion` is an upsert on `(schemaName, contentHash)`, not a blind append. In practice that's -idempotence rather than mutation, because the key carries the content: the hash is derived from -exactly the fields a re-save would overwrite, so anything landing on an existing key has identical -content by construction. "Append-only" is the shape you observe; it isn't a promise the interface -makes, and an implementation isn't expected to reject a re-save. +`saveVersion` is an upsert on `(schemaName, contentHash)`. Re-saving is idempotent, because the key +carries the content: the hash is derived from exactly the fields a re-save would overwrite, so +anything landing on an existing key has identical content by construction. The interface doesn't +promise append-only storage, and an implementation isn't expected to reject a re-save. -`findVersion` is the reverse lookup — a recorded hash back to the schema shape it named. It ships -with a default that scans `versionHistory`, correct everywhere and efficient nowhere; a backend that -can push a keyed lookup down to the database should override it. There's no implementation in this -module. Storage is a separate concern, and a stamp is useful in memory long before anything durable -exists. +`findVersion` resolves a recorded hash back to the schema shape it named. The default scans +`versionHistory`, which is correct for any implementation but reads the whole history to answer a +keyed question; a backend that can push the lookup down to the database should override it. This +module ships no implementation. Storage is a separate concern, and a stamp is useful in memory +before anything durable exists. ## The tiers ahead -Versioning is the first of three escalating tiers, and they ship in that order deliberately. -**Stamp and observe** is this slice: identity, history, no opinions. **Detect and report** comes -next — comparing two declared stamps, and comparing a declaration against what a live graph -actually contains, then recording the result. **Quarantine** is last: acting on a lossy change by -marking affected propositions stale rather than deleting them. Each tier is only safe to build on -the one below it, and each is worth having on its own — you can stamp for a year without ever -detecting, and detect for a year without ever quarantining. Rejecting undeclared types at write -time is not on the list yet: extraction is LLM-driven and a type nobody declared is often a real -finding, so throwing it away is the one thing that can't be undone later. +Versioning is the first of three escalating tiers, shipped in that order. + +**Stamp and observe** is this slice: identity and history. + +**Detect and report** comes next: comparing two declared stamps, comparing a declaration against +what a live graph contains, and recording the result. + +**Quarantine** is last: acting on a lossy change by marking affected propositions stale rather than +deleting them. + +Each tier builds on the one below it, and each is useful on its own. You can stamp for a year +without detecting, and detect for a year without quarantining. Rejecting undeclared types at write +time is not on the list yet: extraction is LLM-driven, a type nobody declared is often a real +finding, and discarding it is the one thing that can't be undone later. From a6d827bc191a60f87779867d134a1cb294b5ab06 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Mon, 31 Aug 2026 00:52:41 -0400 Subject: [PATCH 03/11] Add declared aliases and stamp provenance to metamodel versioning Declared renames enter as aliases on property signatures and entity types, hashed only when non-empty so every existing stamp keeps its digest, guarded against reuse collisions and duplicate-name ambiguity, and defensively copied down to the alias set inside each signature. Stamps also carry optional origin and last-stamped provenance, capped and never hashed. Prior art: Iceberg field identity adapted to declaration-time aliases, Snowflake's schema-evolution record adapted to two provenance pairs. --- CHANGELOG.md | 28 + dice-metamodel/pom.xml | 35 +- .../dice/metamodel/DeclaredSchemaSource.kt | 31 +- .../dice/metamodel/MetamodelVersion.kt | 275 ++++++- .../embabel/dice/metamodel/SchemaAliases.kt | 89 +++ .../metamodel/MetamodelJavaCompatTest.java | 201 +++++ .../dice/metamodel/DeclaredSchemaTest.kt | 63 ++ .../dice/metamodel/MetamodelVersionTest.kt | 707 ++++++++++++++++++ .../dice/metamodel/SchemaAliasesTest.kt | 132 ++++ docs/design/metamodel-versioning.md | 180 ++++- 10 files changed, 1718 insertions(+), 23 deletions(-) create mode 100644 dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt create mode 100644 dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java create mode 100644 dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/SchemaAliasesTest.kt diff --git a/CHANGELOG.md b/CHANGELOG.md index ed877519..0421adb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,3 +14,31 @@ and the consumer PRs that deliver it). content-hash stamping with per-type governance selection, the declared-schema opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM. **Compatibility: additive.** New module; no existing API touched. + +- Declared renames and stamp provenance in `dice-metamodel`. **EXPERIMENTAL** + (shape may change before 1.0): `SchemaAliases`, `StampProvenance`, + `PropertySignature.aliases`, `MetamodelVersion.entityTypeAliases`, and + `MetamodelVersion.origin`/`lastStamped`. A declaration states the names a type + or property used to go by, so a later comparison pairs a rename instead of + reading it as a removal and an addition; provenance records who caused the + first and the most recent stamp, and is never hashed. + **Compatibility: additive.** `contentHash` is unchanged for any schema that + declares no aliases — the new hash blocks serialize only when non-empty, and + the pinned golden digest is asserted unchanged, including for a stamp rebuilt + through the public constructor. The new constructor parameters carry + `@JvmOverloads`, so the existing `PropertySignature(String, Kind, String, + Cardinality)` and `MetamodelVersion(String, List, Map, Map, List)` descriptors + survive. `SchemaAliases` reaches `MetamodelVersion.from` and + `DeclaredSchema.from` through separate overloads that take it as a required + parameter, which leaves the shipped one- and two-argument forms and their + Kotlin `from$default` synthetics byte-identical; widening those functions with + a third defaulted parameter would have replaced + `DeclaredSchema.Companion.from$default(Companion, DataDictionary, + GovernedTypeSelector, int, Object)` with a wider descriptor and broken any + caller already compiled against it. `MetamodelJavaCompatTest` calls every + static form. The changed Kotlin synthetic constructor, `copy`, `copy$default` + and `componentN` signatures on `PropertySignature` are the accepted boundary: + Kotlin callers recompile, and no consumer holds a compiled reference to them. + `StampProvenance` caps `actor` and `trigger` at 256 **characters** + (`String.length`), so a storage backend sizing a column in bytes needs room + for the up-to-1024 UTF-8 bytes those characters can take. diff --git a/dice-metamodel/pom.xml b/dice-metamodel/pom.xml index 7df86561..d9d92f18 100644 --- a/dice-metamodel/pom.xml +++ b/dice-metamodel/pom.xml @@ -35,7 +35,12 @@ - + org.jetbrains.kotlin kotlin-maven-plugin @@ -43,6 +48,34 @@ -Xjvm-default=all + + + compile + compile + + compile + + + + src/main/java + src/main/kotlin + + + + + test-compile + test-compile + + test-compile + + + + src/test/java + src/test/kotlin + + + + diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt index fae0628d..959ee949 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt @@ -69,8 +69,37 @@ class DeclaredSchema( fun from( dataDictionary: DataDictionary, selector: GovernedTypeSelector = GovernedTypeSelector.ALL, + ): DeclaredSchema = from(dataDictionary, selector, SchemaAliases.NONE) + + /** + * Declare the governed part of [dataDictionary], carrying the former names [aliases] + * declares for its types and properties. + * + * This is how a rename gets recorded as one. The stamp carries the former names, and a + * later comparison pairs the old name with the new one rather than reading the change as a + * type or property disappearing. + * + * [aliases] has no default, and the shorter form above stays a separate function. Adding a + * third defaulted parameter to it would replace its synthetic `from$default` descriptor + * with a wider one, which is a link error for any caller already compiled against it. + * + * @param dataDictionary The schema to declare. + * @param selector Which types are under governance. [GovernedTypeSelector.ALL] governs all + * of them. + * @param aliases Former names for the schema's types and properties. [SchemaAliases.NONE] + * declares none. Experimental: shape may change before 1.0. + * @return The declaration. + * @throws IllegalArgumentException when a declared type name appears in another type's + * alias set, or when aliases are declared for a property name the governed types hold + * more than one signature for. + */ + @JvmStatic + fun from( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector, + aliases: SchemaAliases, ): DeclaredSchema = DeclaredSchema( - version = MetamodelVersion.from(dataDictionary, selector), + version = MetamodelVersion.from(dataDictionary, selector, aliases), relationshipTypeNames = MetamodelVersion.governedRelationshipTypeNames(dataDictionary, selector), ) } diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt index fdd15504..3d1d12aa 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -39,12 +39,21 @@ import java.util.Objects * @property type The value type (`string`, `integer`, ...), or the target type's name for a * reference. Empty when [kind] is [Kind.UNKNOWN]. * @property cardinality How many values the property holds: optional, one, a list, or a set. + * @property aliases Names this property used to go by, so a rename pairs up instead of reading as a + * removal and an addition. Declared through [SchemaAliases] and empty unless someone declared + * them. Part of the signature, so an alias-only edit moves [MetamodelVersion.contentHash]. + * Held as handed in, including a mutable set: a `data class` component can't be normalised on the + * way through, since a constructor `val` takes no initialiser and the generated `copy` calls that + * same constructor. [MetamodelVersion] therefore copies the set into an immutable one when it + * takes a signature, and a signature that never reaches a stamp keeps whatever set it was built + * with. Experimental: shape may change before 1.0. */ -data class PropertySignature( +data class PropertySignature @JvmOverloads constructor( val name: String, val kind: Kind, val type: String, val cardinality: Cardinality, + val aliases: Set = emptySet(), ) : Comparable { /** Whether a property holds a value of its own or points at another type in the schema. */ @@ -66,8 +75,24 @@ data class PropertySignature( companion object { - /** Total order used to canonicalise a property set before hashing. */ + /** + * Total order used to canonicalise a property set before hashing. Aliases come last, so a + * set holding two signatures that differ only in their aliases still sorts the same way + * every time. + */ private val ORDER = compareBy({ it.name }, { it.kind }, { it.type }, { it.cardinality }) + .thenComparator { left, right -> compareAliases(left.aliases, right.aliases) } + + /** Compare two alias sets as sorted lists: element by element, then by size. */ + private fun compareAliases(left: Set, right: Set): Int { + val sortedLeft = left.sorted() + val sortedRight = right.sorted() + for (i in 0 until minOf(sortedLeft.size, sortedRight.size)) { + val comparison = sortedLeft[i].compareTo(sortedRight[i]) + if (comparison != 0) return comparison + } + return sortedLeft.size.compareTo(sortedRight.size) + } /** * Read the structural signature off a property definition. @@ -89,6 +114,51 @@ data class PropertySignature( } } +/** + * Who or what caused a stamp to be taken. + * + * Both fields are opaque strings the host supplies and DICE never interprets. `actor` is whoever + * asked — a deploy pipeline, an operator, a scheduled job. `trigger` is what prompted it, and is + * where an extraction-run reference lands once that exists; keeping it a plain string means this + * module gains no dependency on the run model. + * + * Provenance never reaches [MetamodelVersion.contentHash] and never affects equality. It is + * informational: two stamps of the same schema taken for different reasons are the same schema. + * Hashing it would give one schema as many content hashes as it had causes, and the store keys on + * that hash. + * + * Both fields are capped at [MAX_LENGTH] **characters**, counted as `String.length` (UTF-16 code + * units). They land in a database column, and an unbounded host-supplied string is how a stamp + * write starts failing at the driver. A storage backend sizing a column has to size it in bytes: a + * 256-character value can reach 1024 bytes in UTF-8, and more once surrogate pairs are involved, so + * the column needs headroom rather than a matching 256. + * + * Experimental: shape may change before 1.0. + * + * @property actor Who asked for the stamp. Null when the host didn't say. + * @property trigger What prompted it. Null when the host didn't say. + */ +data class StampProvenance @JvmOverloads constructor( + val actor: String? = null, + val trigger: String? = null, +) { + + init { + require(actor == null || actor.length <= MAX_LENGTH) { + "actor is ${actor!!.length} characters; the cap is $MAX_LENGTH." + } + require(trigger == null || trigger.length <= MAX_LENGTH) { + "trigger is ${trigger!!.length} characters; the cap is $MAX_LENGTH." + } + } + + companion object { + + /** Longest an [actor] or [trigger] may be, in characters. */ + const val MAX_LENGTH: Int = 256 + } +} + /** * An immutable stamp that captures the identity and structural content of the governed part of a * [DataDictionary] at a point in time. @@ -118,18 +188,30 @@ data class PropertySignature( * or cardinality moves [contentHash]. * @property relationshipNames Rendered `From-[name]->To` descriptors for the relationships the * governed types declare, sorted and deduplicated. + * @property entityTypeAliases Names each entity type used to go by, keyed by its current name, so a + * type rename pairs up instead of reading as a type vanishing and another appearing. Declared + * through [SchemaAliases] and empty unless someone declared them. Hashed, so an alias-only edit + * moves [contentHash]. Experimental: shape may change before 1.0. + * @property origin Who or what caused this schema to be stamped for the first time. Informational, + * never hashed. Experimental: shape may change before 1.0. + * @property lastStamped Who or what caused the most recent stamp of this schema. Informational, + * never hashed. Experimental: shape may change before 1.0. * @property contentHash SHA-256 hex digest of the schema's entity types, label sets, property - * signatures, and allowed relationships, derived from the four fields above. The schema name is - * excluded so that two structurally identical schemas are equal regardless of how they are named. - * Stable across JVM restarts. Any structural change produces a different hash, including a - * property's type changing on a type whose name is unchanged. + * signatures, type aliases, and allowed relationships. The schema name is excluded so that two + * structurally identical schemas are equal regardless of how they are named, and provenance is + * excluded because the cause of a stamp doesn't change the schema it stamps. Stable across JVM + * restarts. Any structural change produces a different hash, including a property's type changing + * on a type whose name is unchanged. */ -class MetamodelVersion( +class MetamodelVersion @JvmOverloads constructor( schemaName: String, entityTypeNames: List, entityTypeLabels: Map>, entityTypeProperties: Map>, relationshipNames: List, + entityTypeAliases: Map> = emptyMap(), + val origin: StampProvenance? = null, + val lastStamped: StampProvenance? = null, ) { val schemaName: String = schemaName @@ -138,10 +220,13 @@ class MetamodelVersion( val entityTypeLabels: Map> = immutableCopy(entityTypeLabels) - val entityTypeProperties: Map> = immutableCopy(entityTypeProperties) + val entityTypeProperties: Map> = + immutableCopy(entityTypeProperties.mapValues { (_, signatures) -> signatures.map(::withImmutableAliases).toSet() }) val relationshipNames: List = immutableCopy(relationshipNames.distinct().sorted()) + val entityTypeAliases: Map> = immutableCopy(entityTypeAliases) + init { // Only types named in entityTypeNames are walked when hashing, so a map entry keyed by // anything else never reaches contentHash, and two stamps holding different labels or @@ -158,6 +243,22 @@ class MetamodelVersion( "entityTypeProperties is keyed by types missing from entityTypeNames: ${strayPropertyKeys.sorted()}. " + "Properties for a type that isn't listed never reach contentHash." } + val strayAliasKeys = this.entityTypeAliases.keys - known + require(strayAliasKeys.isEmpty()) { + "entityTypeAliases is keyed by types missing from entityTypeNames: ${strayAliasKeys.sorted()}. " + + "Aliases for a type that isn't listed never reach contentHash." + } + + // An entry mapping a type to no former names at all hashes differently from having no entry, + // while saying the same thing, so two stamps of one schema could land on different keys. + val emptyAliasKeys = this.entityTypeAliases.filterValues { it.isEmpty() }.keys + require(emptyAliasKeys.isEmpty()) { + "entityTypeAliases holds empty alias sets for: ${emptyAliasKeys.sorted()}. " + + "Drop the entry instead; it means the same thing and hashes the same as the types around it." + } + + requireNoTypeAliasReuse(known, this.entityTypeAliases) + requireNoAliasesOnDuplicateNames(this.entityTypeProperties) } val contentHash: String = fingerprint() @@ -185,10 +286,15 @@ class MetamodelVersion( // hash identically and hide a lossy schema change. // The schema name is excluded, so two structurally identical schemas produce the same hash // even when named differently (e.g. dev vs prod environments). + // Alias blocks are written only when there is something in them, so a schema that declares + // no former names renders exactly the bytes this encoding produced before aliases existed + // and keeps every hash already recorded against it. Each block is `:|` followed + // by length-prefixed entries in sorted order, which is what the rest of the encoding does. val hashInput = buildString { append("types:").append(entityTypeNames.size).append('|') entityTypeNames.forEach { name -> appendSized(name) + appendAliasBlock("typealiases", entityTypeAliases[name].orEmpty()) val labels = entityTypeLabels[name].orEmpty().sorted() append("labels:").append(labels.size).append('|') labels.forEach { appendSized(it) } @@ -199,6 +305,7 @@ class MetamodelVersion( appendSized(property.kind.name) appendSized(property.type) appendSized(property.cardinality.name) + appendAliasBlock("aliases", property.aliases) } } append("rels:").append(relationshipNames.size).append('|') @@ -210,6 +317,11 @@ class MetamodelVersion( return hashBytes.joinToString("") { "%02x".format(it) } } + /** + * Structural equality. Provenance is left out for the same reason it is left out of + * [contentHash]: it records why a stamp was taken, and two stamps of one schema taken for + * different reasons are still one schema. + */ override fun equals(other: Any?): Boolean { if (this === other) return true if (other !is MetamodelVersion) return false @@ -217,16 +329,24 @@ class MetamodelVersion( entityTypeNames == other.entityTypeNames && entityTypeLabels == other.entityTypeLabels && entityTypeProperties == other.entityTypeProperties && - relationshipNames == other.relationshipNames + relationshipNames == other.relationshipNames && + entityTypeAliases == other.entityTypeAliases } - override fun hashCode(): Int = - Objects.hash(schemaName, entityTypeNames, entityTypeLabels, entityTypeProperties, relationshipNames) + override fun hashCode(): Int = Objects.hash( + schemaName, + entityTypeNames, + entityTypeLabels, + entityTypeProperties, + relationshipNames, + entityTypeAliases, + ) override fun toString(): String = "MetamodelVersion(schemaName=$schemaName, contentHash=$contentHash, " + "entityTypeNames=$entityTypeNames, entityTypeLabels=$entityTypeLabels, " + - "entityTypeProperties=$entityTypeProperties, relationshipNames=$relationshipNames)" + "entityTypeProperties=$entityTypeProperties, relationshipNames=$relationshipNames, " + + "entityTypeAliases=$entityTypeAliases, origin=$origin, lastStamped=$lastStamped)" companion object { @@ -235,6 +355,16 @@ class MetamodelVersion( append(token.length).append(':').append(token) } + /** + * Append `:|` and the sorted, length-prefixed [aliases], writing nothing at all + * when there are none. + */ + private fun StringBuilder.appendAliasBlock(tag: String, aliases: Set) { + if (aliases.isEmpty()) return + append(tag).append(':').append(aliases.size).append('|') + aliases.sorted().forEach { appendSized(it) } + } + /** Copy [values] into a JVM-immutable list, which a Java caller can't mutate via a getter. */ private fun immutableCopy(values: List): List = java.util.List.copyOf(values) @@ -242,6 +372,67 @@ class MetamodelVersion( private fun immutableCopy(values: Map>): Map> = java.util.Map.copyOf(values.mapValues { (_, set) -> java.util.Set.copyOf(set) }) + /** + * Re-wrap a signature's alias set as a JVM-immutable one. The signature is a data class, so + * its alias set arrives however the caller built it; a caller who kept a mutable set and + * added to it afterwards would leave the stamp disagreeing with its own precomputed hash. + * + * The copy is unconditional. An empty set is the dangerous case, not the safe one: a caller + * holding an empty `mutableSetOf` can add to it after the stamp is built, which moves the + * signature's own `hashCode` while it sits in a hash-based set and leaves it unfindable in + * the collection that contains it. `Set.copyOf` of an empty set is a shared singleton, so + * skipping it buys nothing. + */ + private fun withImmutableAliases(signature: PropertySignature): PropertySignature = + signature.copy(aliases = java.util.Set.copyOf(signature.aliases)) + + /** + * Reject a declared type name showing up in another type's alias set. + * + * One name would then belong to two types at once: the live type that carries it, and the + * renamed type that still claims it as a former name. Nothing downstream can tell which of + * the two a piece of data under that name belongs to. Reusing a retired name means first + * deleting the alias that still claims it. + */ + private fun requireNoTypeAliasReuse( + declaredTypeNames: Set, + entityTypeAliases: Map>, + ) { + entityTypeAliases.forEach { (typeName, aliases) -> + val reused = (aliases - typeName).filter { it in declaredTypeNames }.sorted() + require(reused.isEmpty()) { + "Entity type '$typeName' declares alias(es) $reused, and those name types the " + + "schema still declares. Retire the alias before reusing the name." + } + } + } + + /** + * Reject aliases on a property name a type holds more than one signature for. + * + * Two same-named domain types can each declare `age` with a different shape, and the union + * keeps both signatures. A comparison has no way to say which of the two an old name refers + * to, so the declaration is refused and the name compares as a removal plus an addition. + */ + private fun requireNoAliasesOnDuplicateNames( + entityTypeProperties: Map>, + ) { + entityTypeProperties.forEach { (typeName, signatures) -> + signatures + .groupBy { it.name } + .filterValues { withName -> withName.size > 1 } + .forEach { (propertyName, withName) -> + val declared = withName.flatMap { it.aliases }.distinct().sorted() + require(declared.isEmpty()) { + "Property '$propertyName' on entity type '$typeName' has " + + "${withName.size} signatures and declares alias(es) $declared. " + + "Retire the alias: a name with more than one signature can't say " + + "which one an old name meant." + } + } + } + } + /** * Create a [MetamodelVersion] stamp covering every type in [dataDictionary], which is the * right stamp for a domain that is closed-world throughout. @@ -251,7 +442,7 @@ class MetamodelVersion( */ @JvmStatic fun from(dataDictionary: DataDictionary): MetamodelVersion = - from(dataDictionary, GovernedTypeSelector.ALL) + from(dataDictionary, GovernedTypeSelector.ALL, SchemaAliases.NONE) /** * Create a [MetamodelVersion] stamp covering only the types [selector] governs. @@ -275,7 +466,40 @@ class MetamodelVersion( * @return An immutable version stamp covering the governed subset. */ @JvmStatic - fun from(dataDictionary: DataDictionary, selector: GovernedTypeSelector): MetamodelVersion { + fun from(dataDictionary: DataDictionary, selector: GovernedTypeSelector): MetamodelVersion = + from(dataDictionary, selector, SchemaAliases.NONE) + + /** + * Create a [MetamodelVersion] stamp covering the types [selector] governs, carrying the + * former names [aliases] declares for them. + * + * The upstream `PropertyDefinition` has nowhere to hold a former name, so aliases are + * declared alongside the dictionary and applied while the stamp is built: each signature + * picks up the former names declared for its property, and the type-level map is carried + * through for the governed types. Aliases declared for a type the selector doesn't govern + * are dropped, the same way everything else about an ungoverned type is. + * + * [aliases] has no default, and the two shorter overloads are separate functions rather + * than defaulted parameters on this one. A default here would fold all three into one + * function whose synthetic `from$default` descriptor replaced the shipped one, which is a + * link error for any caller already compiled against it. + * + * @param dataDictionary The schema to stamp. + * @param selector Which of its types are under governance. [GovernedTypeSelector.ALL] + * governs all of them. + * @param aliases Former names for the schema's types and properties. [SchemaAliases.NONE] + * declares none. Experimental: shape may change before 1.0. + * @return An immutable version stamp covering the governed subset. + * @throws IllegalArgumentException when a declared type name appears in another type's + * alias set, or when aliases are declared for a property name the governed types hold + * more than one signature for. + */ + @JvmStatic + fun from( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector, + aliases: SchemaAliases, + ): MetamodelVersion { val governedTypes = dataDictionary.domainTypes.filter { selector.governs(it) } // A DataDictionary can legally hold two domain types that share a name but differ in @@ -287,9 +511,21 @@ class MetamodelVersion( .groupBy { it.name } .mapValues { (_, types) -> types.flatMap { it.labels }.toSet() } + // Decorating inside this loop is the only place it can happen: the stamp is immutable + // and hashes at construction. Every signature sharing a property name picks up the same + // declared alias set, so decoration can neither create nor collapse a duplicate, and + // the constructor's duplicate-name guard sees exactly the duplicates the union holds. val entityTypeProperties = governedTypes .groupBy { it.name } - .mapValues { (_, types) -> types.flatMap { type -> type.properties.map(PropertySignature::of) }.toSet() } + .mapValues { (typeName, types) -> + types.flatMap { type -> + type.properties.map { property -> + val signature = PropertySignature.of(property) + val declared = aliases.propertyAliasesFor(typeName, signature.name) + if (declared.isEmpty()) signature else signature.copy(aliases = declared) + } + }.toSet() + } // Splitting one type into two same-named declarations, or merging two back into one, // can render the same relationship descriptor twice. It is the same schema either way, @@ -298,12 +534,21 @@ class MetamodelVersion( .filter { selector.governs(it.from) } .map { rel -> "${rel.from.name}-[${rel.name}]->${rel.to.name}" } + val governedNames = governedTypes.map { it.name }.toSet() + val entityTypeAliases = aliases.typeAliases.filterKeys { it in governedNames } + + // The two refusals run here as well as in the constructor so a declaration that can't + // be stamped fails at the seam that wrote it, naming the alias to retire. + requireNoTypeAliasReuse(governedNames, entityTypeAliases) + requireNoAliasesOnDuplicateNames(entityTypeProperties) + return MetamodelVersion( schemaName = dataDictionary.name, entityTypeNames = governedTypes.map { it.name }, entityTypeLabels = entityTypeLabels, entityTypeProperties = entityTypeProperties, relationshipNames = relationshipNames, + entityTypeAliases = entityTypeAliases, ) } diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt new file mode 100644 index 00000000..f51ca047 --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt @@ -0,0 +1,89 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +/** + * The former names a schema's types and properties have gone by, declared alongside the schema + * itself. + * + * A rename with no alias behind it looks like a removal and an addition, which reads as loss. An + * alias tells the comparison that the two names are the same thing, so the rename pairs up. + * Iceberg and Delta get this from stable field ids assigned when a column is created; DICE + * extracts its schema from LLM output and has no id to assign, so the declaration carries the old + * name instead. + * + * Names are exact and case-sensitive. LLM extraction drifts on case, and treating `worksAt` and + * `worksat` as the same name here would quietly pair two properties an operator never said were + * the same one. + * + * Aliases accumulate. A type renamed `A` to `B` to `C` declares `{A, B}`, so a comparison across + * non-adjacent stamps still pairs. Retiring a name means deleting it from the declaration. + * + * Empty alias sets are dropped on the way in: declaring no former names for a type means the same + * thing as saying nothing about it, and an empty entry would otherwise change the content hash + * while meaning nothing. + * + * Experimental: shape may change before 1.0. + * + * @property typeAliases Current entity type name to the names it used to have. + * @property propertyAliases Entity type name, then current property name, to the names that + * property used to have. + */ +class SchemaAliases @JvmOverloads constructor( + typeAliases: Map> = emptyMap(), + propertyAliases: Map>> = emptyMap(), +) { + + val typeAliases: Map> = copyDroppingEmpty(typeAliases) + + val propertyAliases: Map>> = java.util.Map.copyOf( + propertyAliases + .mapValues { (_, byProperty) -> copyDroppingEmpty(byProperty) } + .filterValues { it.isNotEmpty() }, + ) + + /** The former names declared for [propertyName] on [typeName], empty when none are. */ + fun propertyAliasesFor(typeName: String, propertyName: String): Set = + propertyAliases[typeName]?.get(propertyName).orEmpty() + + override fun equals(other: Any?): Boolean = + other is SchemaAliases && + typeAliases == other.typeAliases && + propertyAliases == other.propertyAliases + + override fun hashCode(): Int = 31 * typeAliases.hashCode() + propertyAliases.hashCode() + + override fun toString(): String = + "SchemaAliases(typeAliases=$typeAliases, propertyAliases=$propertyAliases)" + + companion object { + + /** No former names declared anywhere. What the stamping entry points use by default. */ + @JvmField + val NONE: SchemaAliases = SchemaAliases() + + /** + * Copy into a JVM-immutable map of JVM-immutable sets, leaving out keys whose alias set is + * empty. + */ + private fun copyDroppingEmpty(values: Map>): Map> = + java.util.Map.copyOf( + values + .filterValues { it.isNotEmpty() } + .mapValues { (_, aliases) -> java.util.Set.copyOf(aliases) }, + ) + } +} diff --git a/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java new file mode 100644 index 00000000..91c8e359 --- /dev/null +++ b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java @@ -0,0 +1,201 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel; + +import com.embabel.agent.core.Cardinality; +import com.embabel.agent.core.DataDictionary; +import com.embabel.agent.core.DomainType; +import com.embabel.agent.core.DynamicType; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Calls every entry point the way a Java consumer compiled against the previous release does. + *

+ * Aliases and provenance were added as trailing parameters with defaults, so a Kotlin caller sees + * no change. A Java caller sees whatever descriptors the compiler emitted, which is why + * {@code @JvmOverloads} is on those constructors and factories: without it, adding a parameter + * would delete the descriptor an already-compiled consumer is linked against, and the failure + * would be a {@code NoSuchMethodError} at runtime rather than a compile error here. + */ +class MetamodelJavaCompatTest { + + /** DynamicType has no Java-friendly overloads upstream, so every argument is spelled out. */ + private static DomainType type(String name) { + return new DynamicType(name, "", List.of(), List.of(), true); + } + + private static DataDictionary goldenSchema() { + return DataDictionary.fromDomainTypes("golden-schema", List.of(type("Person"), type("Company"))); + } + + @Test + void theFourArgumentPropertySignatureConstructorStillExists() { + PropertySignature signature = new PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE); + + assertEquals("age", signature.getName()); + assertEquals(Set.of(), signature.getAliases()); + } + + @Test + void thePropertySignatureConstructorAlsoTakesAliases() { + PropertySignature signature = new PropertySignature( + "emailAddress", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, Set.of("email")); + + assertEquals(Set.of("email"), signature.getAliases()); + } + + @Test + void theFiveArgumentMetamodelVersionConstructorStillExists() { + MetamodelVersion version = new MetamodelVersion( + "test", + List.of("Person"), + Map.of("Person", Set.of("Person")), + Map.of("Person", Set.of()), + List.of()); + + assertEquals(List.of("Person"), version.getEntityTypeNames()); + assertEquals(Map.of(), version.getEntityTypeAliases()); + assertNull(version.getOrigin()); + assertNull(version.getLastStamped()); + } + + @Test + void theMetamodelVersionConstructorAlsoTakesAliasesAndProvenance() { + MetamodelVersion version = new MetamodelVersion( + "test", + List.of("Person"), + Map.of("Person", Set.of("Person")), + Map.of("Person", Set.of()), + List.of(), + Map.of("Person", Set.of("Human")), + new StampProvenance("deploy-pipeline", "release-1"), + new StampProvenance("operator", null)); + + assertEquals(Set.of("Human"), version.getEntityTypeAliases().get("Person")); + assertEquals("deploy-pipeline", version.getOrigin().getActor()); + assertEquals("operator", version.getLastStamped().getActor()); + assertNull(version.getLastStamped().getTrigger()); + } + + @Test + void theOneAndTwoArgumentStampingFactoriesStillExist() { + MetamodelVersion whole = MetamodelVersion.from(goldenSchema()); + MetamodelVersion governed = MetamodelVersion.from(goldenSchema(), GovernedTypeSelector.ALL); + + assertEquals(whole.getContentHash(), governed.getContentHash()); + assertEquals(List.of("Company", "Person"), whole.getEntityTypeNames()); + } + + @Test + void theStampingFactoryAlsoTakesAliases() { + MetamodelVersion version = MetamodelVersion.from( + goldenSchema(), + GovernedTypeSelector.ALL, + new SchemaAliases(Map.of("Person", Set.of("Human")), Map.of())); + + assertEquals(Set.of("Human"), version.getEntityTypeAliases().get("Person")); + } + + @Test + void theOneAndTwoArgumentDeclarationFactoriesStillExist() { + DeclaredSchema whole = DeclaredSchema.from(goldenSchema()); + DeclaredSchema governed = DeclaredSchema.from(goldenSchema(), GovernedTypeSelector.ALL); + + assertEquals(whole, governed); + assertEquals(Set.of(), whole.getRelationshipTypeNames()); + } + + @Test + void theDeclarationFactoryAlsoTakesAliases() { + DeclaredSchema declared = DeclaredSchema.from( + goldenSchema(), + GovernedTypeSelector.ALL, + new SchemaAliases(Map.of("Person", Set.of("Human")), Map.of())); + + assertEquals(Set.of("Human"), declared.getVersion().getEntityTypeAliases().get("Person")); + } + + @Test + void theNoArgumentAliasAndProvenanceConstructorsExist() { + assertEquals(Map.of(), new SchemaAliases().getTypeAliases()); + assertEquals(Map.of(), SchemaAliases.NONE.getPropertyAliases()); + assertNull(new StampProvenance().getActor()); + assertEquals("ci", new StampProvenance("ci").getActor()); + } + + @Test + void theCollectionsAStampHandsBackRefuseMutationFromJava() { + MetamodelVersion version = MetamodelVersion.from( + goldenSchema(), + GovernedTypeSelector.ALL, + new SchemaAliases(Map.of("Person", Set.of("Human")), Map.of())); + + assertTrue(throwsOnMutation(() -> version.getEntityTypeAliases().remove("Person"))); + assertTrue(throwsOnMutation(() -> version.getEntityTypeAliases().get("Person").add("Sneaky"))); + } + + @Test + void theShippedKotlinDefaultSyntheticKeepsItsDescriptor() throws Exception { + // A Kotlin caller that omits a defaulted argument links against the $default synthetic + // rather than the function itself. DeclaredSchema.from shipped with one defaulted + // parameter, so that synthetic is part of the module's binary surface. Adding a third + // defaulted parameter would have rewritten its descriptor and turned every already + // compiled `DeclaredSchema.from(dictionary)` into a NoSuchMethodError, which is why + // SchemaAliases arrives on a separate overload that requires it. + Class companion = Class.forName("com.embabel.dice.metamodel.DeclaredSchema$Companion"); + + assertNotNull(companion.getDeclaredMethod( + "from$default", + companion, + DataDictionary.class, + GovernedTypeSelector.class, + int.class, + Object.class)); + } + + @Test + void theStampingFactoriesTakeNoDefaultedParameters() throws Exception { + // MetamodelVersion.from shipped as two overloads with no defaults, so it has no $default + // synthetic to preserve. Keeping it that way means the alias overload can never widen one. + Class companion = Class.forName("com.embabel.dice.metamodel.MetamodelVersion$Companion"); + + long defaultSynthetics = Arrays.stream(companion.getDeclaredMethods()) + .filter(method -> method.getName().equals("from$default")) + .count(); + + assertEquals(0, defaultSynthetics); + } + + private static boolean throwsOnMutation(Runnable mutation) { + try { + mutation.run(); + return false; + } catch (UnsupportedOperationException expected) { + return true; + } + } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt index b745d2e0..0adf3ecc 100644 --- a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt @@ -18,8 +18,10 @@ package com.embabel.dice.metamodel import com.embabel.agent.core.DataDictionary import com.embabel.agent.core.DomainTypePropertyDefinition import com.embabel.agent.core.DynamicType +import com.embabel.agent.core.ValuePropertyDefinition import org.junit.jupiter.api.Assertions.* import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows class DeclaredSchemaTest { @@ -61,4 +63,65 @@ class DeclaredSchemaTest { assertEquals(DeclaredSchema.from(dictionary()), source.declare()) } + + @Test + fun `declaring no aliases matches declaring none explicitly`() { + assertEquals( + DeclaredSchema.from(dictionary()).version.contentHash, + DeclaredSchema.from(dictionary(), GovernedTypeSelector.ALL, SchemaAliases.NONE).version.contentHash, + ) + } + + @Test + fun `declared aliases reach the stamp`() { + val declared = DeclaredSchema.from( + dictionary(), + GovernedTypeSelector.ALL, + SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("worksAt" to setOf("employer"))), + ), + ) + + assertEquals(setOf("Human"), declared.version.entityTypeAliases["Person"]) + assertEquals( + setOf("employer"), + declared.version.entityTypeProperties["Person"]!!.single { it.name == "worksAt" }.aliases, + ) + assertNotEquals(DeclaredSchema.from(dictionary()).version.contentHash, declared.version.contentHash) + } + + @Test + fun `a type alias naming another declared type is refused`() { + val thrown = assertThrows { + DeclaredSchema.from( + dictionary(), + GovernedTypeSelector.ALL, + SchemaAliases(typeAliases = mapOf("Person" to setOf("Company"))), + ) + } + assertTrue(thrown.message!!.contains("Company"), thrown.message) + } + + @Test + fun `aliases on a property name the merge holds two signatures for are refused`() { + // Two same-named Person declarations each carry their own `age`, so the union holds two + // signatures and an old name can't say which of them it meant. + val duplicated = DataDictionary.fromDomainTypes( + "app", + listOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age", type = "string"))), + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age", type = "integer"))), + ), + ) + + val thrown = assertThrows { + DeclaredSchema.from( + duplicated, + GovernedTypeSelector.ALL, + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("years")))), + ) + } + assertTrue(thrown.message!!.contains("years"), thrown.message) + } } diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt index 195a042d..bc68a68e 100644 --- a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt @@ -172,6 +172,51 @@ class MetamodelVersionTest { MetamodelVersion.from(renamed).contentHash, ) } + + @Test + fun `an alias-free declaration still hashes to the digest pinned before aliases existed`() { + // The literal below was produced by the encoding as it stood before PropertySignature + // carried aliases and MetamodelVersion carried entityTypeAliases. Alias blocks are + // written only when they hold something, so a schema declaring no former names has to + // render the same bytes and keep every hash already recorded against it. All four ways + // of saying "no aliases" have to land on it. + val pinned = "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00" + + assertEquals(pinned, MetamodelVersion.from(goldenSchema()).contentHash) + assertEquals(pinned, MetamodelVersion.from(goldenSchema(), GovernedTypeSelector.ALL).contentHash) + assertEquals( + pinned, + MetamodelVersion.from(goldenSchema(), GovernedTypeSelector.ALL, SchemaAliases.NONE).contentHash, + ) + assertEquals( + pinned, + MetamodelVersion.from( + goldenSchema(), + GovernedTypeSelector.ALL, + SchemaAliases(typeAliases = emptyMap(), propertyAliases = emptyMap()), + ).contentHash, + ) + } + + @Test + fun `rebuilding the golden stamp through the public constructor hashes to the same literal`() { + // The storage mapper reconstructs a stamp field by field rather than from a dictionary. + // Passing an explicitly empty alias map and no provenance has to reproduce the pinned + // digest, or a row written before aliases existed could never be read back. + val fromDictionary = MetamodelVersion.from(goldenSchema()) + val rebuilt = MetamodelVersion( + schemaName = fromDictionary.schemaName, + entityTypeNames = fromDictionary.entityTypeNames, + entityTypeLabels = fromDictionary.entityTypeLabels, + entityTypeProperties = fromDictionary.entityTypeProperties, + relationshipNames = fromDictionary.relationshipNames, + entityTypeAliases = emptyMap(), + ) + assertEquals( + "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00", + rebuilt.contentHash, + ) + } } @Nested @@ -808,4 +853,666 @@ class MetamodelVersionTest { assertEquals("my-schema", version.schemaName) } } + + @Nested + inner class Aliases { + + private fun personWith(vararg properties: PropertyDefinition): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = properties.toList())), + ) + + private fun stamp(dictionary: DataDictionary, aliases: SchemaAliases): MetamodelVersion = + MetamodelVersion.from(dictionary, GovernedTypeSelector.ALL, aliases) + + @Test + fun `a declared property alias lands on the signature`() { + val version = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + + assertEquals( + setOf( + PropertySignature( + "emailAddress", + PropertySignature.Kind.VALUE, + "string", + Cardinality.ONE, + setOf("email"), + ), + ), + version.entityTypeProperties["Person"], + ) + } + + @Test + fun `declaring a property alias changes the hash`() { + val plain = stamp(personWith(ValuePropertyDefinition("emailAddress")), SchemaAliases.NONE) + val aliased = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + + assertNotEquals(plain.contentHash, aliased.contentHash) + assertFalse(plain.hasSameContentAs(aliased)) + } + + @Test + fun `the order aliases are declared in does not affect the hash`() { + val forwards = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases( + propertyAliases = mapOf("Person" to mapOf("emailAddress" to linkedSetOf("email", "contact"))), + ), + ) + val backwards = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases( + propertyAliases = mapOf("Person" to mapOf("emailAddress" to linkedSetOf("contact", "email"))), + ), + ) + + assertEquals(forwards.contentHash, backwards.contentHash) + } + + @Test + fun `different alias sets on one property hash differently`() { + val one = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + val two = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email", "contact")))), + ) + + assertNotEquals(one.contentHash, two.contentHash) + } + + @Test + fun `an alias containing the block delimiter does not collide with a split set`() { + // Same reasoning as the property-name case: alias entries are length-prefixed, so + // ["a;b"] and ["a", "b"] can't serialise to the same bytes. + val joined = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("a;b")))), + ) + val split = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("a", "b")))), + ) + + assertNotEquals(joined.contentHash, split.contentHash) + } + + @Test + fun `an alias for a property the type doesn't have changes nothing`() { + val plain = stamp(personWith(ValuePropertyDefinition("age")), SchemaAliases.NONE) + val stale = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("retired" to setOf("gone")))), + ) + + assertEquals(plain.contentHash, stale.contentHash) + } + + @Test + fun `an alias equal to the property's own name is kept and hashed`() { + // It matches nothing at diff time — nothing looks data up by property name — so it is + // inert there. It is still part of the signature, so it moves the hash. + val plain = stamp(personWith(ValuePropertyDefinition("age")), SchemaAliases.NONE) + val selfAliased = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("age")))), + ) + + assertEquals(setOf("age"), selfAliased.entityTypeProperties["Person"]!!.single().aliases) + assertNotEquals(plain.contentHash, selfAliased.contentHash) + } + + @Test + fun `a declared type alias is carried and changes the hash`() { + val plain = stamp(personWith(), SchemaAliases.NONE) + val aliased = stamp(personWith(), SchemaAliases(typeAliases = mapOf("Person" to setOf("Human")))) + + assertEquals(mapOf("Person" to setOf("Human")), aliased.entityTypeAliases) + assertEquals(emptyMap>(), plain.entityTypeAliases) + assertNotEquals(plain.contentHash, aliased.contentHash) + } + + @Test + fun `type alias order does not affect the hash`() { + val forwards = stamp(personWith(), SchemaAliases(typeAliases = mapOf("Person" to linkedSetOf("Human", "Actor")))) + val backwards = stamp(personWith(), SchemaAliases(typeAliases = mapOf("Person" to linkedSetOf("Actor", "Human")))) + + assertEquals(forwards.contentHash, backwards.contentHash) + } + + @Test + fun `a type alias and a property alias of the same name hash differently`() { + // The two blocks carry different tags and sit in different places, so declaring "old" + // as a former type name is a different schema from declaring it as a former property + // name. + val asTypeAlias = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(typeAliases = mapOf("Person" to setOf("old"))), + ) + val asPropertyAlias = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("old")))), + ) + + assertNotEquals(asTypeAlias.contentHash, asPropertyAlias.contentHash) + } + + @Test + fun `aliases accumulate across successive renames`() { + val version = stamp( + DataDictionary.fromDomainTypes("test", listOf(DynamicType("C"))), + SchemaAliases(typeAliases = mapOf("C" to setOf("A", "B"))), + ) + + assertEquals(setOf("A", "B"), version.entityTypeAliases["C"]) + } + + @Test + fun `a type may list its own name, which is what a rename and back leaves behind`() { + // A renamed to B and back to A accumulates {A, B}, so the alias set holds the current + // name. The reuse guard is about other types' names. + val version = stamp( + DataDictionary.fromDomainTypes("test", listOf(DynamicType("A"))), + SchemaAliases(typeAliases = mapOf("A" to setOf("A", "B"))), + ) + + assertEquals(setOf("A", "B"), version.entityTypeAliases["A"]) + } + + @Test + fun `aliases for an ungoverned type are dropped`() { + // Everything else about an ungoverned type is invisible to the stamp; aliases follow. + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType("Person"), + DynamicType(name = "Sighting", ownProperties = listOf(ValuePropertyDefinition("seenAt"))), + ), + ) + val governed = GovernedTypeSelector { it.name == "Person" } + + val plain = MetamodelVersion.from(dictionary, governed) + val aliased = MetamodelVersion.from( + dictionary, + governed, + SchemaAliases( + typeAliases = mapOf("Sighting" to setOf("Observation")), + propertyAliases = mapOf("Sighting" to mapOf("seenAt" to setOf("spottedAt"))), + ), + ) + + assertEquals(emptyMap>(), aliased.entityTypeAliases) + assertEquals(plain.contentHash, aliased.contentHash) + } + + @Test + fun `an explicitly empty alias set is dropped rather than hashed`() { + val plain = stamp(personWith(ValuePropertyDefinition("age")), SchemaAliases.NONE) + val declaredEmpty = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases( + typeAliases = mapOf("Person" to emptySet()), + propertyAliases = mapOf("Person" to mapOf("age" to emptySet())), + ), + ) + + assertEquals(emptyMap>(), declaredEmpty.entityTypeAliases) + assertEquals(plain.contentHash, declaredEmpty.contentHash) + } + } + + @Nested + inner class SignatureOrdering { + + private fun signature(name: String, aliases: Set = emptySet()): PropertySignature = + PropertySignature(name, PropertySignature.Kind.VALUE, "string", Cardinality.ONE, aliases) + + @Test + fun `aliases break ties only after name, kind, type and cardinality`() { + val aliasedAge = signature("age", setOf("zzz")) + val plainEmail = signature("email") + + // The name still decides, whatever the aliases say. + assertTrue(aliasedAge < plainEmail) + + val aliasedString = PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, setOf("zzz"), + ) + val plainInteger = PropertySignature( + "age", PropertySignature.Kind.VALUE, "integer", Cardinality.ONE, + ) + assertTrue(plainInteger < aliasedString) + } + + @Test + fun `signatures differing only in aliases sort deterministically`() { + val none = signature("age") + val one = signature("age", setOf("b")) + val two = signature("age", setOf("a1", "b")) + + assertTrue(none < two) + assertTrue(two < one) + assertEquals(listOf(none, two, one), listOf(one, none, two).sorted()) + assertEquals(listOf(none, two, one), listOf(two, one, none).sorted()) + } + + @Test + fun `alias order inside the set does not change the ordering`() { + val forwards = signature("age", linkedSetOf("a", "b")) + val backwards = signature("age", linkedSetOf("b", "a")) + + assertEquals(0, forwards.compareTo(backwards)) + } + } + + @Nested + inner class AliasGuards { + + private fun personTwice(vararg propertyTypes: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + propertyTypes.map { propertyType -> + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age", type = propertyType)), + ) + }, + ) + + @Test + fun `type aliases keyed by a type that is not listed are rejected`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Ghost" to setOf("Spectre")), + ) + } + assertTrue(thrown.message!!.contains("entityTypeAliases"), thrown.message) + assertTrue(thrown.message!!.contains("Ghost"), thrown.message) + } + + @Test + fun `an empty type alias set is rejected`() { + // An entry with no former names in it hashes differently from having no entry at all, + // while meaning the same thing, so two stamps of one schema could land on two keys. + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Person" to emptySet()), + ) + } + assertTrue(thrown.message!!.contains("empty alias sets"), thrown.message) + assertTrue(thrown.message!!.contains("Person"), thrown.message) + } + + @Test + fun `a declared type name in another type's alias set is rejected by the constructor`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Human", "Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Human" to setOf("Person")), + ) + } + assertTrue(thrown.message!!.contains("Human"), thrown.message) + assertTrue(thrown.message!!.contains("Person"), thrown.message) + assertTrue(thrown.message!!.contains("Retire the alias"), thrown.message) + } + + @Test + fun `a declared type name in another type's alias set is rejected at the stamping seam`() { + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType("Human"), DynamicType("Person")), + ) + + val thrown = assertThrows { + MetamodelVersion.from( + dictionary, + GovernedTypeSelector.ALL, + SchemaAliases(typeAliases = mapOf("Human" to setOf("Person"))), + ) + } + assertTrue(thrown.message!!.contains("Person"), thrown.message) + } + + @Test + fun `reusing the name of an ungoverned type is allowed, because the stamp never sees it`() { + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType("Human"), DynamicType("Person")), + ) + + val version = MetamodelVersion.from( + dictionary, + GovernedTypeSelector { it.name == "Human" }, + SchemaAliases(typeAliases = mapOf("Human" to setOf("Person"))), + ) + assertEquals(setOf("Person"), version.entityTypeAliases["Human"]) + } + + @Test + fun `aliases on a property name with more than one signature are rejected by the constructor`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, setOf("years"), + ), + PropertySignature( + "age", PropertySignature.Kind.VALUE, "integer", Cardinality.ONE, setOf("years"), + ), + ), + ), + relationshipNames = emptyList(), + ) + } + assertTrue(thrown.message!!.contains("age"), thrown.message) + assertTrue(thrown.message!!.contains("years"), thrown.message) + assertTrue(thrown.message!!.contains("Retire the alias"), thrown.message) + } + + @Test + fun `aliases on a property name with more than one signature are rejected at the stamping seam`() { + // Two same-named Person declarations each carry their own `age`, so the union holds two + // signatures for one name and an old name can't say which it meant. + val thrown = assertThrows { + MetamodelVersion.from( + personTwice("string", "integer"), + GovernedTypeSelector.ALL, + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("years")))), + ) + } + assertTrue(thrown.message!!.contains("age"), thrown.message) + assertTrue(thrown.message!!.contains("years"), thrown.message) + } + + @Test + fun `a duplicated property name with no aliases declared is fine`() { + val version = MetamodelVersion.from(personTwice("string", "integer")) + assertEquals(2, version.entityTypeProperties["Person"]!!.size) + } + + @Test + fun `aliases on a single-signature property survive a duplicate elsewhere on the type`() { + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf( + ValuePropertyDefinition("age", type = "string"), + ValuePropertyDefinition("emailAddress"), + ), + ), + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age", type = "integer")), + ), + ), + ) + + val version = MetamodelVersion.from( + dictionary, + GovernedTypeSelector.ALL, + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + + assertEquals( + setOf("email"), + version.entityTypeProperties["Person"]!!.single { it.name == "emailAddress" }.aliases, + ) + } + } + + @Nested + inner class Provenance { + + /** One fixed schema, stamped with whatever provenance a test wants on it. */ + private fun personStamp( + origin: StampProvenance? = null, + lastStamped: StampProvenance? = null, + ): MetamodelVersion = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = mapOf("Person" to setOf("Person")), + entityTypeProperties = mapOf("Person" to emptySet()), + relationshipNames = emptyList(), + entityTypeAliases = emptyMap(), + origin = origin, + lastStamped = lastStamped, + ) + + @Test + fun `provenance defaults to absent`() { + val version = MetamodelVersion.from( + DataDictionary.fromDomainTypes("test", listOf(DynamicType("Person"))), + ) + assertNull(version.origin) + assertNull(version.lastStamped) + } + + @Test + fun `provenance is carried`() { + val version = personStamp( + origin = StampProvenance("deploy-pipeline", "release-1"), + lastStamped = StampProvenance("operator", "manual-recheck"), + ) + + assertEquals(StampProvenance("deploy-pipeline", "release-1"), version.origin) + assertEquals(StampProvenance("operator", "manual-recheck"), version.lastStamped) + } + + @Test + fun `provenance never reaches the content hash`() { + // Two stamps of one schema taken for different reasons are the same schema, and the + // hash is the store's natural key. + val bare = personStamp() + val attributed = personStamp( + origin = StampProvenance("deploy-pipeline", "release-1"), + lastStamped = StampProvenance("operator", "manual-recheck"), + ) + + assertEquals(bare.contentHash, attributed.contentHash) + assertTrue(bare.hasSameContentAs(attributed)) + } + + @Test + fun `provenance is not part of equality`() { + val bare = personStamp() + val attributed = personStamp(origin = StampProvenance("deploy-pipeline", "release-1")) + + assertEquals(bare, attributed) + assertEquals(bare.hashCode(), attributed.hashCode()) + } + + @Test + fun `both fields are optional`() { + assertNull(StampProvenance().actor) + assertNull(StampProvenance().trigger) + assertEquals("ci", StampProvenance("ci").actor) + assertNull(StampProvenance("ci").trigger) + } + + @Test + fun `an actor at the cap is accepted and one over it is rejected`() { + assertEquals( + StampProvenance.MAX_LENGTH, + StampProvenance(actor = "a".repeat(StampProvenance.MAX_LENGTH)).actor!!.length, + ) + + val thrown = assertThrows { + StampProvenance(actor = "a".repeat(StampProvenance.MAX_LENGTH + 1)) + } + assertTrue(thrown.message!!.contains("actor"), thrown.message) + } + + @Test + fun `a trigger at the cap is accepted and one over it is rejected`() { + assertEquals( + StampProvenance.MAX_LENGTH, + StampProvenance(trigger = "t".repeat(StampProvenance.MAX_LENGTH)).trigger!!.length, + ) + + val thrown = assertThrows { + StampProvenance(trigger = "t".repeat(StampProvenance.MAX_LENGTH + 1)) + } + assertTrue(thrown.message!!.contains("trigger"), thrown.message) + } + } + + @Nested + inner class AliasImmutability { + + @Test + fun `the alias collections a stamp hands back cannot be mutated`() { + val version = MetamodelVersion.from( + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age")))), + ), + GovernedTypeSelector.ALL, + SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("age" to setOf("years"))), + ), + ) + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeAliases as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeAliases["Person"] as MutableSet).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties["Person"]!!.single().aliases as MutableSet).add("Sneaky") + } + } + + @Test + fun `mutating the alias sets the caller passed in does not change the stamp`() { + val typeAliases = mutableSetOf("Human") + val signatureAliases = mutableSetOf("years") + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, signatureAliases, + ), + ), + ), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Person" to typeAliases), + ) + val hashAtConstruction = version.contentHash + + typeAliases.add("Actor") + signatureAliases.add("yearsOld") + + assertEquals(setOf("Human"), version.entityTypeAliases["Person"]) + assertEquals(setOf("years"), version.entityTypeProperties["Person"]!!.single().aliases) + assertEquals(hashAtConstruction, version.contentHash) + } + + @Test + fun `filling in an alias set that was empty at construction does not change the stamp`() { + // The empty case is the dangerous one. A signature built with an empty mutable set that + // the stamp stored by reference would change its own hashCode when the caller added an + // alias, leaving it unfindable in the hash-based set holding it and disagreeing with a + // contentHash computed while it looked alias-free. + val aliases = mutableSetOf() + val signature = PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, aliases, + ) + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf("Person" to setOf(signature)), + relationshipNames = emptyList(), + ) + val hashAtConstruction = version.contentHash + + aliases.add("years") + + val stored = version.entityTypeProperties["Person"]!! + assertEquals(emptySet(), stored.single().aliases) + assertEquals(hashAtConstruction, version.contentHash) + + // The signature is still findable under the identity it was hashed with, so nothing has + // shifted position in the set that holds it. + assertTrue( + stored.contains( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ) + + // And the stamp still hashes as the alias-free schema it was built from. + val neverAliased = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ), + relationshipNames = emptyList(), + ) + assertEquals(neverAliased.contentHash, version.contentHash) + } + + @Test + fun `an initially empty alias set is replaced by an immutable one`() { + val signature = PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, mutableSetOf(), + ) + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf("Person" to setOf(signature)), + relationshipNames = emptyList(), + ) + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties["Person"]!!.single().aliases as MutableSet) + .add("Sneaky") + } + } + } } diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/SchemaAliasesTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/SchemaAliasesTest.kt new file mode 100644 index 00000000..23450d8d --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/SchemaAliasesTest.kt @@ -0,0 +1,132 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows + +class SchemaAliasesTest { + + @Test + fun `NONE declares nothing`() { + assertEquals(emptyMap>(), SchemaAliases.NONE.typeAliases) + assertEquals(emptyMap>>(), SchemaAliases.NONE.propertyAliases) + assertEquals(SchemaAliases(), SchemaAliases.NONE) + } + + @Test + fun `an empty type alias set is dropped`() { + // Saying a type has no former names is the same as saying nothing about it, and an empty + // entry would otherwise be refused by the stamp's guard. + val aliases = SchemaAliases(typeAliases = mapOf("Person" to emptySet(), "Company" to setOf("Corp"))) + + assertEquals(mapOf("Company" to setOf("Corp")), aliases.typeAliases) + } + + @Test + fun `an empty property alias set is dropped, and a type left with none goes with it`() { + val aliases = SchemaAliases( + propertyAliases = mapOf( + "Person" to mapOf("age" to emptySet(), "emailAddress" to setOf("email")), + "Company" to mapOf("name" to emptySet()), + ), + ) + + assertEquals(mapOf("Person" to mapOf("emailAddress" to setOf("email"))), aliases.propertyAliases) + } + + @Test + fun `propertyAliasesFor answers empty for anything undeclared`() { + val aliases = SchemaAliases( + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + + assertEquals(setOf("email"), aliases.propertyAliasesFor("Person", "emailAddress")) + assertEquals(emptySet(), aliases.propertyAliasesFor("Person", "age")) + assertEquals(emptySet(), aliases.propertyAliasesFor("Company", "emailAddress")) + } + + @Test + fun `alias names are case-sensitive`() { + // LLM extraction drifts on case. Folding it here would pair two names nobody said were the + // same one. + val aliases = SchemaAliases(typeAliases = mapOf("Person" to setOf("Human"))) + + assertNotEquals(aliases, SchemaAliases(typeAliases = mapOf("Person" to setOf("human")))) + assertNotEquals(aliases, SchemaAliases(typeAliases = mapOf("person" to setOf("Human")))) + } + + @Test + fun `mutating what the caller passed in does not change the declaration`() { + val types = mutableMapOf("Person" to mutableSetOf("Human")) + val properties = mutableMapOf("Person" to mutableMapOf("emailAddress" to mutableSetOf("email"))) + + val aliases = SchemaAliases(types, properties) + + types["Ghost"] = mutableSetOf("Spectre") + types["Person"]!!.add("Actor") + properties["Person"]!!["age"] = mutableSetOf("years") + properties["Person"]!!["emailAddress"]!!.add("contact") + + assertEquals(mapOf("Person" to setOf("Human")), aliases.typeAliases) + assertEquals(mapOf("Person" to mapOf("emailAddress" to setOf("email"))), aliases.propertyAliases) + } + + @Test + fun `the collections a declaration hands back cannot be mutated`() { + val aliases = SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.typeAliases as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.typeAliases["Person"] as MutableSet).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.propertyAliases as MutableMap>>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.propertyAliases["Person"]!!["emailAddress"] as MutableSet).add("Sneaky") + } + } + + @Test + fun `equality is by content`() { + val one = SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + val two = SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + + assertEquals(one, two) + assertEquals(one.hashCode(), two.hashCode()) + assertTrue(one.toString().contains("Human")) + } +} diff --git a/docs/design/metamodel-versioning.md b/docs/design/metamodel-versioning.md index cdb8ead7..9643ffc8 100644 --- a/docs/design/metamodel-versioning.md +++ b/docs/design/metamodel-versioning.md @@ -13,8 +13,10 @@ the stamp is about, and keeping the stamps so history is answerable. Comparing t stamp against a live graph, comes later — see [the tiers ahead](#the-tiers-ahead). The types live in `dice-metamodel`, a small pure-JVM module: `MetamodelVersion`, -`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, and the `MetamodelVersionStore` -contract. It depends on Embabel's agent core types and nothing else. +`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `SchemaAliases`, `StampProvenance`, +and the `MetamodelVersionStore` contract. It depends on Embabel's agent core types and nothing else. +`SchemaAliases`, `StampProvenance`, and the alias fields on `PropertySignature` and +`MetamodelVersion` are experimental; their shape may change before 1.0. ## Declare, stamp, store @@ -74,9 +76,26 @@ routinely contain `;`, `[`, `=`, and spaces. A delimiter-joined encoding would l Length-prefixing makes the encoding unambiguous, so distinct content always yields a distinct hash. The hashed form is `types:|` followed, for each type name in sorted order, by the length-prefixed -name, then `labels:|` and its sorted labels, then `props:|` and its sorted signatures — each -signature contributing name, kind, type, and cardinality as four length-prefixed tokens. Then -`rels:|` and the sorted relationship descriptors. The schema name appears nowhere. +name, an optional `typealiases:|` block, then `labels:|` and its sorted labels, then +`props:|` and its sorted signatures — each signature contributing name, kind, type, and +cardinality as four length-prefixed tokens, followed by an optional `aliases:|` block. Then +`rels:|` and the sorted relationship descriptors. The schema name appears nowhere, and neither +does stamp provenance. + +The two alias blocks are written only when they hold something, which is what lets aliases be added +to a shipped encoding at all. A schema that declares no former names renders exactly the bytes this +encoding produced before aliases existed, so every hash already recorded against it still resolves. +`MetamodelVersionTest` pins that. One test asserts the golden digest for the same dictionary stamped +four ways: `from(dictionary)`, `from(dictionary, GovernedTypeSelector.ALL)`, the same with an +explicit `SchemaAliases.NONE`, and the same with an explicitly empty +`SchemaAliases(emptyMap(), emptyMap())`. A second asserts it for a stamp rebuilt field by field +through the public constructor with an empty alias map, which is the path a storage mapper takes. + +The block shape is the one the rest of the encoding already uses: `:|` and then +length-prefixed entries in sorted order. Position keeps the two tags apart — the type block sits +between a type's name and its labels, the property block after a signature's fourth token — so +declaring `old` as a former type name and declaring it as a former property name are different +digests. Two things about the input. A `DataDictionary` can legally hold two domain types sharing a name but differing in shape, so `from` unions their labels and properties per name. Keeping only the last @@ -85,7 +104,8 @@ The same split can render one relationship descriptor twice, so the constructor deduplicates the type and relationship lists: declaring a type once or splitting it in two is the same schema, and has to be the same hash. -The constructor is strict about the rest, too. It copies every collection into a JVM-immutable one. +The constructor is strict about the rest, too. It copies every collection into a JVM-immutable one, +down to the alias set inside each property signature, which arrives however the caller built it. Kotlin's read-only view is a compile-time promise that a Java caller sees straight through, so it wouldn't stop anything being reshaped out from under the precomputed hash. The constructor also rejects a label or property map keyed by a type missing from `entityTypeNames`: only listed types @@ -154,6 +174,146 @@ Versioning starts here: with no declared schema, nothing is stamped. The Spring in a later slice activates only when a `DeclaredSchemaSource` bean is present, so an application that hasn't decided what it governs is left alone. +## Declared renames + +Renaming a type or a property is the change a content hash reads worst. `email` becoming +`emailAddress` is one property with a new spelling, and a stamp comparison sees a removal and an +addition, which is the same shape as deleting a property and inventing an unrelated one. A declared +alias says what the name used to be, so the comparison can pair the two. + +Iceberg and Delta hold identity in stable field ids assigned when a column is created, leaving the +name as a label over an identity the format already carries. DICE has no id to assign. Its types +come from LLM extraction against a `DataDictionary` an application edits, and the upstream +`PropertyDefinition` has nowhere to put an id even if DICE minted one. The adapted form is the +former name itself, declared at the moment the rename is made: + +```kotlin +DeclaredSchema.from( + dataDictionary, + governed, + SchemaAliases( + typeAliases = mapOf("Organisation" to setOf("Company")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ), +) +``` + +Types get the mechanism as well as properties because a type rename is the more destructive of the +two: every proposition labelled with the old name and every referrer pointing at it move at once. + +Alias names are exact and case-sensitive. LLM extraction drifts on case, and folding `worksAt` into +`worksat` here would pair two names nobody declared as the same one. + +Aliases accumulate. A type renamed `A` to `B` to `C` declares `{A, B}`, so a comparison across +non-adjacent stamps still pairs. Retiring a name means deleting it from the declaration. + +`SchemaAliases` is a declaration-time input; the stamp carries the result. `MetamodelVersion.from` +decorates each signature with the former names declared for its property, inside the per-type loop +right after the signature is read, because the stamp is immutable and hashes at construction. Type +aliases land in `entityTypeAliases`. Aliases declared for a type the selector doesn't govern are +dropped, along with everything else about an ungoverned type. + +A property alias equal to the property's own name, or naming a property that still exists on both +sides of a comparison, matches nothing: nothing looks data up by property name, so a stale property +alias can mislead nothing. It stays part of the signature and moves the hash. A type alias naming +another declared type is a different case, and the third guard below rejects it. + +Comparing two stamps is the next slice. Until it lands, an alias is a recorded intention that moves +the hash and nothing more. + +### The four guards + +```mermaid +flowchart TB + aliases["SchemaAliases
typeAliases, propertyAliases"] + seam["MetamodelVersion.from / DeclaredSchema.from"] + ctor["MetamodelVersion constructor
(also the storage reconstruction path)"] + g1{"alias keys all in
entityTypeNames?"} + g2{"every alias set
non-empty?"} + g3{"alias names another
declared type?"} + g4{"aliases on a property name
with two signatures?"} + stamp["Stamp: entityTypeAliases and
PropertySignature.aliases, both hashed"] + reject["IllegalArgumentException
naming the alias to retire"] + + aliases --> seam + seam -->|"ungoverned keys dropped,
signatures decorated"| ctor + ctor --> g1 + g1 -->|no| reject + g1 -->|yes| g2 + g2 -->|no| reject + g2 -->|yes| g3 + g3 -->|yes| reject + g3 -->|no| g4 + g4 -->|yes| reject + g4 -->|no| stamp +``` + +**Alias-map keys are a subset of `entityTypeNames`.** Only listed types are walked when hashing, so +an entry keyed by anything else never reaches `contentHash`. This is the rule the label and property +maps already follow. + +**Every alias set is non-empty.** An entry mapping a type to no former names hashes differently from +having no entry at all while saying the same thing, so two stamps of one schema could land on two +different natural keys. `SchemaAliases` drops empty sets on the way in, which keeps a harmless +declaration from becoming an error at the seam. + +**No declared type name appears in another type's alias set.** This is the reuse collision. A schema +renames `Company` to `Organisation`, keeps `{Company}` as the alias, and later declares a fresh +`Company` for something unrelated. A live type now shares a retired name, and the comparison has two +bad options: sweep the new `Company`'s data into the renamed type's quarantine matching, or treat +the old label as declared forever and never report drift on it. Reusing a retired name therefore +requires deleting the alias that still claims it first. A type listing its own name is fine — a +rename to `B` and back to `A` accumulates `{A, B}` — because the guard is about other types' names. + +**No aliases on a property name a type holds more than one signature for.** Two same-named domain +types can each declare `age` with a different shape, and the union keeps both signatures. An old +name has no way to say which of the two it meant, so the declaration is refused. + +All four run in the `MetamodelVersion` constructor, so the public constructors and the storage +mapper's reconstruction path are covered. The last two also run at `MetamodelVersion.from` and +`DeclaredSchema.from`, which is where duplicates and reused names actually become visible, and where +the message can name the alias to retire. + +### What the duplicate-name refusal costs + +A dictionary can legally evolve into duplicate-hood. Someone adds a second `Person` declaration +carrying its own `age`, and a name that had one signature now has two. If an alias was standing on +that name, every stamp and every drift check from that moment throws, with a message naming the +alias to retire. That is a loud, conservative hard stop on a schema that was fine the day before. + +Retiring the alias clears the throw and costs the pairing. The name then diffs through the duplicate +fallback as a removal plus an addition, the drift policy reads that as lossy, and an additive +evolution earns a quarantine sweep. Both outcomes cost more than an ordinary signature change would, +and both beat the only other option, which is guessing which of the two signatures the old name +referred to. + +## Stamp provenance + +A stamp records who caused it, in two pairs: `origin` for the first stamp of a schema, `lastStamped` +for the most recent. Each is a `StampProvenance(actor, trigger)` of two opaque host-supplied +strings, capped at 256 characters because they land in a database column and an unbounded +host-supplied string is how a stamp write starts failing at the driver. The cap counts characters, +which is what `String.length` gives; a storage backend sizes its column in bytes, so 256 characters +needs room for the 1024 bytes UTF-8 can take to encode them. + +A single pair would answer only the second question. The drift check that arrives in a later slice +re-stamps on every boot and on a schedule, so one last-writer-wins field converges on whichever +instance booted last, and the cause of the original stamp is gone within a deploy cycle. Keeping +`origin` separate means the first cause survives every routine re-stamp. + +Snowflake's `SCHEMA_EVOLUTION_RECORD` is the same shape: the evolution event is recorded alongside +the schema rather than folded into the schema's identity. Provenance is informational here for the +same reason. Two stamps of one schema taken for different reasons are the same schema, and the hash +is the store's natural key, so hashing the cause would give one schema as many identities as it had +causes. It is excluded from `contentHash` and from equality. + +`trigger` is where an extraction-run reference lands once that exists. It stays a plain string, so +this module gains no dependency on the run model. + +The persistence rules that keep those two pairs alive through routine re-saves belong to the storage +slice: `origin` is first-write-wins, and `lastStamped` moves only when the incoming value is +non-null, so a routine re-stamp carrying no provenance can never erase cause. + ## History accumulates `MetamodelVersionStore` is a port with four operations: `saveVersion`, `latestVersion`, @@ -166,6 +326,14 @@ carries the content: the hash is derived from exactly the fields a re-save would anything landing on an existing key has identical content by construction. The interface doesn't promise append-only storage, and an implementation isn't expected to reject a re-save. +This is where schema registries have already landed. AWS Glue Schema Registry and Confluent Schema +Registry both identify a schema version by a fingerprint of its content and answer a re-registration +of an identical schema with the existing version rather than a new one. Deriving identity from the +content is what makes registration idempotent, and it is why a client that re-registers on every +boot doesn't inflate the history. DICE keys on `(schemaName, contentHash)` for that reason, and an +application stamping its declared schema on every start is exactly the client those registries are +built for. + `findVersion` resolves a recorded hash back to the schema shape it named. The default scans `versionHistory`, which is correct for any implementation but reads the whole history to answer a keyed question; a backend that can push the lookup down to the database should override it. This From d8bd3188162b49ed4060a203ce2e7ef4519721cd Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Mon, 31 Aug 2026 14:25:09 -0400 Subject: [PATCH 04/11] Remove stamp provenance until a caller records it --- CHANGELOG.md | 15 +-- .../dice/metamodel/MetamodelVersion.kt | 62 +---------- .../metamodel/MetamodelJavaCompatTest.java | 26 ++--- .../dice/metamodel/MetamodelVersionTest.kt | 100 +----------------- docs/design/metamodel-versioning.md | 48 ++++----- 5 files changed, 37 insertions(+), 214 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0421adb0..fdd37fcd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,13 +15,11 @@ and the consumer PRs that deliver it). opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM. **Compatibility: additive.** New module; no existing API touched. -- Declared renames and stamp provenance in `dice-metamodel`. **EXPERIMENTAL** - (shape may change before 1.0): `SchemaAliases`, `StampProvenance`, - `PropertySignature.aliases`, `MetamodelVersion.entityTypeAliases`, and - `MetamodelVersion.origin`/`lastStamped`. A declaration states the names a type - or property used to go by, so a later comparison pairs a rename instead of - reading it as a removal and an addition; provenance records who caused the - first and the most recent stamp, and is never hashed. +- Declared renames in `dice-metamodel`. **EXPERIMENTAL** (shape may change + before 1.0): `SchemaAliases`, `PropertySignature.aliases`, and + `MetamodelVersion.entityTypeAliases`. A declaration states the names a type or + property used to go by, so a later comparison pairs a rename instead of + reading it as a removal and an addition. **Compatibility: additive.** `contentHash` is unchanged for any schema that declares no aliases — the new hash blocks serialize only when non-empty, and the pinned golden digest is asserted unchanged, including for a stamp rebuilt @@ -39,6 +37,3 @@ and the consumer PRs that deliver it). static form. The changed Kotlin synthetic constructor, `copy`, `copy$default` and `componentN` signatures on `PropertySignature` are the accepted boundary: Kotlin callers recompile, and no consumer holds a compiled reference to them. - `StampProvenance` caps `actor` and `trigger` at 256 **characters** - (`String.length`), so a storage backend sizing a column in bytes needs room - for the up-to-1024 UTF-8 bytes those characters can take. diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt index 3d1d12aa..1d476181 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -114,51 +114,6 @@ data class PropertySignature @JvmOverloads constructor( } } -/** - * Who or what caused a stamp to be taken. - * - * Both fields are opaque strings the host supplies and DICE never interprets. `actor` is whoever - * asked — a deploy pipeline, an operator, a scheduled job. `trigger` is what prompted it, and is - * where an extraction-run reference lands once that exists; keeping it a plain string means this - * module gains no dependency on the run model. - * - * Provenance never reaches [MetamodelVersion.contentHash] and never affects equality. It is - * informational: two stamps of the same schema taken for different reasons are the same schema. - * Hashing it would give one schema as many content hashes as it had causes, and the store keys on - * that hash. - * - * Both fields are capped at [MAX_LENGTH] **characters**, counted as `String.length` (UTF-16 code - * units). They land in a database column, and an unbounded host-supplied string is how a stamp - * write starts failing at the driver. A storage backend sizing a column has to size it in bytes: a - * 256-character value can reach 1024 bytes in UTF-8, and more once surrogate pairs are involved, so - * the column needs headroom rather than a matching 256. - * - * Experimental: shape may change before 1.0. - * - * @property actor Who asked for the stamp. Null when the host didn't say. - * @property trigger What prompted it. Null when the host didn't say. - */ -data class StampProvenance @JvmOverloads constructor( - val actor: String? = null, - val trigger: String? = null, -) { - - init { - require(actor == null || actor.length <= MAX_LENGTH) { - "actor is ${actor!!.length} characters; the cap is $MAX_LENGTH." - } - require(trigger == null || trigger.length <= MAX_LENGTH) { - "trigger is ${trigger!!.length} characters; the cap is $MAX_LENGTH." - } - } - - companion object { - - /** Longest an [actor] or [trigger] may be, in characters. */ - const val MAX_LENGTH: Int = 256 - } -} - /** * An immutable stamp that captures the identity and structural content of the governed part of a * [DataDictionary] at a point in time. @@ -192,14 +147,9 @@ data class StampProvenance @JvmOverloads constructor( * type rename pairs up instead of reading as a type vanishing and another appearing. Declared * through [SchemaAliases] and empty unless someone declared them. Hashed, so an alias-only edit * moves [contentHash]. Experimental: shape may change before 1.0. - * @property origin Who or what caused this schema to be stamped for the first time. Informational, - * never hashed. Experimental: shape may change before 1.0. - * @property lastStamped Who or what caused the most recent stamp of this schema. Informational, - * never hashed. Experimental: shape may change before 1.0. * @property contentHash SHA-256 hex digest of the schema's entity types, label sets, property * signatures, type aliases, and allowed relationships. The schema name is excluded so that two - * structurally identical schemas are equal regardless of how they are named, and provenance is - * excluded because the cause of a stamp doesn't change the schema it stamps. Stable across JVM + * structurally identical schemas are equal regardless of how they are named. Stable across JVM * restarts. Any structural change produces a different hash, including a property's type changing * on a type whose name is unchanged. */ @@ -210,8 +160,6 @@ class MetamodelVersion @JvmOverloads constructor( entityTypeProperties: Map>, relationshipNames: List, entityTypeAliases: Map> = emptyMap(), - val origin: StampProvenance? = null, - val lastStamped: StampProvenance? = null, ) { val schemaName: String = schemaName @@ -317,11 +265,7 @@ class MetamodelVersion @JvmOverloads constructor( return hashBytes.joinToString("") { "%02x".format(it) } } - /** - * Structural equality. Provenance is left out for the same reason it is left out of - * [contentHash]: it records why a stamp was taken, and two stamps of one schema taken for - * different reasons are still one schema. - */ + /** Structural equality: same schema name, types, labels, property signatures, relationships, and aliases. */ override fun equals(other: Any?): Boolean { if (this === other) return true if (other !is MetamodelVersion) return false @@ -346,7 +290,7 @@ class MetamodelVersion @JvmOverloads constructor( "MetamodelVersion(schemaName=$schemaName, contentHash=$contentHash, " + "entityTypeNames=$entityTypeNames, entityTypeLabels=$entityTypeLabels, " + "entityTypeProperties=$entityTypeProperties, relationshipNames=$relationshipNames, " + - "entityTypeAliases=$entityTypeAliases, origin=$origin, lastStamped=$lastStamped)" + "entityTypeAliases=$entityTypeAliases)" companion object { diff --git a/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java index 91c8e359..8140f4f4 100644 --- a/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java +++ b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java @@ -28,17 +28,16 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; /** * Calls every entry point the way a Java consumer compiled against the previous release does. *

- * Aliases and provenance were added as trailing parameters with defaults, so a Kotlin caller sees - * no change. A Java caller sees whatever descriptors the compiler emitted, which is why - * {@code @JvmOverloads} is on those constructors and factories: without it, adding a parameter - * would delete the descriptor an already-compiled consumer is linked against, and the failure - * would be a {@code NoSuchMethodError} at runtime rather than a compile error here. + * Aliases were added as trailing parameters with defaults, so a Kotlin caller sees no change. A + * Java caller sees whatever descriptors the compiler emitted, which is why {@code @JvmOverloads} is + * on those constructors and factories: without it, adding a parameter would delete the descriptor + * an already-compiled consumer is linked against, and the failure would be a + * {@code NoSuchMethodError} at runtime rather than a compile error here. */ class MetamodelJavaCompatTest { @@ -79,26 +78,19 @@ void theFiveArgumentMetamodelVersionConstructorStillExists() { assertEquals(List.of("Person"), version.getEntityTypeNames()); assertEquals(Map.of(), version.getEntityTypeAliases()); - assertNull(version.getOrigin()); - assertNull(version.getLastStamped()); } @Test - void theMetamodelVersionConstructorAlsoTakesAliasesAndProvenance() { + void theMetamodelVersionConstructorAlsoTakesAliases() { MetamodelVersion version = new MetamodelVersion( "test", List.of("Person"), Map.of("Person", Set.of("Person")), Map.of("Person", Set.of()), List.of(), - Map.of("Person", Set.of("Human")), - new StampProvenance("deploy-pipeline", "release-1"), - new StampProvenance("operator", null)); + Map.of("Person", Set.of("Human"))); assertEquals(Set.of("Human"), version.getEntityTypeAliases().get("Person")); - assertEquals("deploy-pipeline", version.getOrigin().getActor()); - assertEquals("operator", version.getLastStamped().getActor()); - assertNull(version.getLastStamped().getTrigger()); } @Test @@ -140,11 +132,9 @@ void theDeclarationFactoryAlsoTakesAliases() { } @Test - void theNoArgumentAliasAndProvenanceConstructorsExist() { + void theNoArgumentAliasConstructorExists() { assertEquals(Map.of(), new SchemaAliases().getTypeAliases()); assertEquals(Map.of(), SchemaAliases.NONE.getPropertyAliases()); - assertNull(new StampProvenance().getActor()); - assertEquals("ci", new StampProvenance("ci").getActor()); } @Test diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt index bc68a68e..7b5aa2f7 100644 --- a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt @@ -201,8 +201,8 @@ class MetamodelVersionTest { @Test fun `rebuilding the golden stamp through the public constructor hashes to the same literal`() { // The storage mapper reconstructs a stamp field by field rather than from a dictionary. - // Passing an explicitly empty alias map and no provenance has to reproduce the pinned - // digest, or a row written before aliases existed could never be read back. + // Passing an explicitly empty alias map has to reproduce the pinned digest, or a row + // written before aliases existed could never be read back. val fromDictionary = MetamodelVersion.from(goldenSchema()) val rebuilt = MetamodelVersion( schemaName = fromDictionary.schemaName, @@ -1290,102 +1290,6 @@ class MetamodelVersionTest { } } - @Nested - inner class Provenance { - - /** One fixed schema, stamped with whatever provenance a test wants on it. */ - private fun personStamp( - origin: StampProvenance? = null, - lastStamped: StampProvenance? = null, - ): MetamodelVersion = MetamodelVersion( - schemaName = "test", - entityTypeNames = listOf("Person"), - entityTypeLabels = mapOf("Person" to setOf("Person")), - entityTypeProperties = mapOf("Person" to emptySet()), - relationshipNames = emptyList(), - entityTypeAliases = emptyMap(), - origin = origin, - lastStamped = lastStamped, - ) - - @Test - fun `provenance defaults to absent`() { - val version = MetamodelVersion.from( - DataDictionary.fromDomainTypes("test", listOf(DynamicType("Person"))), - ) - assertNull(version.origin) - assertNull(version.lastStamped) - } - - @Test - fun `provenance is carried`() { - val version = personStamp( - origin = StampProvenance("deploy-pipeline", "release-1"), - lastStamped = StampProvenance("operator", "manual-recheck"), - ) - - assertEquals(StampProvenance("deploy-pipeline", "release-1"), version.origin) - assertEquals(StampProvenance("operator", "manual-recheck"), version.lastStamped) - } - - @Test - fun `provenance never reaches the content hash`() { - // Two stamps of one schema taken for different reasons are the same schema, and the - // hash is the store's natural key. - val bare = personStamp() - val attributed = personStamp( - origin = StampProvenance("deploy-pipeline", "release-1"), - lastStamped = StampProvenance("operator", "manual-recheck"), - ) - - assertEquals(bare.contentHash, attributed.contentHash) - assertTrue(bare.hasSameContentAs(attributed)) - } - - @Test - fun `provenance is not part of equality`() { - val bare = personStamp() - val attributed = personStamp(origin = StampProvenance("deploy-pipeline", "release-1")) - - assertEquals(bare, attributed) - assertEquals(bare.hashCode(), attributed.hashCode()) - } - - @Test - fun `both fields are optional`() { - assertNull(StampProvenance().actor) - assertNull(StampProvenance().trigger) - assertEquals("ci", StampProvenance("ci").actor) - assertNull(StampProvenance("ci").trigger) - } - - @Test - fun `an actor at the cap is accepted and one over it is rejected`() { - assertEquals( - StampProvenance.MAX_LENGTH, - StampProvenance(actor = "a".repeat(StampProvenance.MAX_LENGTH)).actor!!.length, - ) - - val thrown = assertThrows { - StampProvenance(actor = "a".repeat(StampProvenance.MAX_LENGTH + 1)) - } - assertTrue(thrown.message!!.contains("actor"), thrown.message) - } - - @Test - fun `a trigger at the cap is accepted and one over it is rejected`() { - assertEquals( - StampProvenance.MAX_LENGTH, - StampProvenance(trigger = "t".repeat(StampProvenance.MAX_LENGTH)).trigger!!.length, - ) - - val thrown = assertThrows { - StampProvenance(trigger = "t".repeat(StampProvenance.MAX_LENGTH + 1)) - } - assertTrue(thrown.message!!.contains("trigger"), thrown.message) - } - } - @Nested inner class AliasImmutability { diff --git a/docs/design/metamodel-versioning.md b/docs/design/metamodel-versioning.md index 9643ffc8..12b0fdbe 100644 --- a/docs/design/metamodel-versioning.md +++ b/docs/design/metamodel-versioning.md @@ -13,10 +13,10 @@ the stamp is about, and keeping the stamps so history is answerable. Comparing t stamp against a live graph, comes later — see [the tiers ahead](#the-tiers-ahead). The types live in `dice-metamodel`, a small pure-JVM module: `MetamodelVersion`, -`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `SchemaAliases`, `StampProvenance`, -and the `MetamodelVersionStore` contract. It depends on Embabel's agent core types and nothing else. -`SchemaAliases`, `StampProvenance`, and the alias fields on `PropertySignature` and -`MetamodelVersion` are experimental; their shape may change before 1.0. +`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `SchemaAliases`, and the +`MetamodelVersionStore` contract. It depends on Embabel's agent core types and nothing else. +`SchemaAliases` and the alias fields on `PropertySignature` and `MetamodelVersion` are +experimental; their shape may change before 1.0. ## Declare, stamp, store @@ -79,8 +79,7 @@ The hashed form is `types:|` followed, for each type name in sorted order, by name, an optional `typealiases:|` block, then `labels:|` and its sorted labels, then `props:|` and its sorted signatures — each signature contributing name, kind, type, and cardinality as four length-prefixed tokens, followed by an optional `aliases:|` block. Then -`rels:|` and the sorted relationship descriptors. The schema name appears nowhere, and neither -does stamp provenance. +`rels:|` and the sorted relationship descriptors. The schema name appears nowhere. The two alias blocks are written only when they hold something, which is what lets aliases be added to a shipped encoding at all. A schema that declares no former names renders exactly the bytes this @@ -287,32 +286,23 @@ evolution earns a quarantine sweep. Both outcomes cost more than an ordinary sig and both beat the only other option, which is guessing which of the two signatures the old name referred to. -## Stamp provenance +## Stamp provenance waits for a caller -A stamp records who caused it, in two pairs: `origin` for the first stamp of a schema, `lastStamped` -for the most recent. Each is a `StampProvenance(actor, trigger)` of two opaque host-supplied -strings, capped at 256 characters because they land in a database column and an unbounded -host-supplied string is how a stamp write starts failing at the driver. The cap counts characters, -which is what `String.length` gives; a storage backend sizes its column in bytes, so 256 characters -needs room for the 1024 bytes UTF-8 can take to encode them. +A stamp says nothing about who or what caused it. That is deliberate for now. Recording the cause +means fixing a type for it, deciding how long its strings may be, finding it somewhere to live in +every backend, and settling what a re-save does to a value that is already there — a pile of +commitments made on behalf of a caller that doesn't exist yet. Nothing in DICE stamps with a cause +today: `MetamodelVersion.from` builds a stamp from a dictionary, and the drift check that arrives +in a later slice re-stamps the +declared schema without anything to attribute it to. -A single pair would answer only the second question. The drift check that arrives in a later slice -re-stamps on every boot and on a schedule, so one last-writer-wins field converges on whichever -instance booted last, and the cause of the original stamp is gone within a deploy cycle. Keeping -`origin` separate means the first cause survives every routine re-stamp. +Provenance returns with the first stamping caller that records it. Whoever that caller is will +settle the shape, which is a better basis for the decision than a guess made here. -Snowflake's `SCHEMA_EVOLUTION_RECORD` is the same shape: the evolution event is recorded alongside -the schema rather than folded into the schema's identity. Provenance is informational here for the -same reason. Two stamps of one schema taken for different reasons are the same schema, and the hash -is the store's natural key, so hashing the cause would give one schema as many identities as it had -causes. It is excluded from `contentHash` and from equality. - -`trigger` is where an extraction-run reference lands once that exists. It stays a plain string, so -this module gains no dependency on the run model. - -The persistence rules that keep those two pairs alive through routine re-saves belong to the storage -slice: `origin` is first-write-wins, and `lastStamped` moves only when the incoming value is -non-null, so a routine re-stamp carrying no provenance can never erase cause. +Snowflake's `SCHEMA_EVOLUTION_RECORD` shows where it lands when it does arrive: the evolution event +sits alongside the schema rather than inside the schema's identity. Two stamps of one schema taken +for different reasons are the same schema, and the hash is the store's natural key, so hashing the +cause would give one schema as many identities as it had causes. ## History accumulates From 0a011f5783ca8216380c81c2a24717d34e91a5f4 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Tue, 1 Sep 2026 13:00:14 -0400 Subject: [PATCH 05/11] Define the stamping contract for dice.metamodel.version The key existed with nothing specifying who writes it or what the value means. The KDoc now states the contract: the extraction persistence path stamps the declared schema's content hash onto canonical proposition metadata, a missing key marks pre-governance extraction, and the value is opaque. A test proves the stamp round-trips through the in-memory store and propositions are selectable by it. Production stamping lands in a follow-up slice once the extraction-run stack merges. --- CHANGELOG.md | 8 ++ .../embabel/dice/common/DiceMetadataKeys.kt | 15 ++- .../MetamodelVersionStampingTest.kt | 104 ++++++++++++++++++ 3 files changed, 125 insertions(+), 2 deletions(-) create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/MetamodelVersionStampingTest.kt diff --git a/CHANGELOG.md b/CHANGELOG.md index fdd37fcd..174f4124 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,3 +37,11 @@ and the consumer PRs that deliver it). static form. The changed Kotlin synthetic constructor, `copy`, `copy$default` and `componentN` signatures on `PropertySignature` are the accepted boundary: Kotlin callers recompile, and no consumer holds a compiled reference to them. + +- `DiceMetadataKeys.METAMODEL_VERSION` metadata key and stamping contract. + Propositions can carry the declared schema version hash under this key to + record which schema governed their extraction. The key is defined here with + its contract; production wiring that stamps propositions at persistence time + lands in a follow-up slice after the extraction-run stack merges. + **Compatibility: additive.** New metadata key only; no existing API or code + touched. diff --git a/dice/src/main/kotlin/com/embabel/dice/common/DiceMetadataKeys.kt b/dice/src/main/kotlin/com/embabel/dice/common/DiceMetadataKeys.kt index f2f48bf7..902dac75 100644 --- a/dice/src/main/kotlin/com/embabel/dice/common/DiceMetadataKeys.kt +++ b/dice/src/main/kotlin/com/embabel/dice/common/DiceMetadataKeys.kt @@ -36,8 +36,19 @@ object DiceMetadataKeys { /** * Content hash of the schema active at extraction time. * - * Stamping a proposition with this key lets drift detection later identify which - * propositions were extracted under an older schema version. + * The extraction persistence path writes this key onto a canonical proposition's metadata map + * at persistence time to record which declared schema version governed its extraction. The value + * is the content hash computed by the metamodel versioning module for the schema snapshot in + * effect at extraction. The value is opaque: consumers should compare it by equality to detect + * schema changes across propositions, and should avoid parsing or inspecting its structure. + * + * A missing key means the proposition was extracted before schema governance was adopted and + * has no declared version. Downstream drift detection and version-aware operations must treat + * unversioned propositions explicitly (e.g., assume they came from a known prior schema, or + * exclude them from compatibility checks). + * + * Production wiring that stamps propositions at persistence time lands in a follow-up slice + * after the extraction-run stack merges. */ const val METAMODEL_VERSION = "dice.metamodel.version" diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/MetamodelVersionStampingTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/MetamodelVersionStampingTest.kt new file mode 100644 index 00000000..f80eae7e --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/MetamodelVersionStampingTest.kt @@ -0,0 +1,104 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition + +import com.embabel.agent.core.ContextId +import com.embabel.dice.common.DiceMetadataKeys +import com.embabel.dice.proposition.store.InMemoryPropositionRepository +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test + +class MetamodelVersionStampingTest { + + private val testContextId = ContextId("test-context") + private val testContentHash = "a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2w3x4y5z6a7b8c9d0" + + @Test + fun `proposition metadata with METAMODEL_VERSION survives save and retrieve`() { + val repo = InMemoryPropositionRepository() + + val original = Proposition( + contextId = testContextId, + text = "Alice works at Acme", + mentions = listOf(EntityMention(span = "alice", type = "Person", resolvedId = "alice")), + confidence = 0.95, + ).withMetadataValue(DiceMetadataKeys.METAMODEL_VERSION, testContentHash) + + val saved = repo.save(original) + val retrieved = repo.findById(saved.id) + + assertNotNull(retrieved) + assertEquals(testContentHash, retrieved!!.metadata[DiceMetadataKeys.METAMODEL_VERSION]) + } + + @Test + fun `propositions can be selected by METAMODEL_VERSION metadata value`() { + val repo = InMemoryPropositionRepository() + val hash1 = "hash1" + val hash2 = "hash2" + + val prop1 = Proposition( + contextId = testContextId, + text = "Alice works at Acme", + mentions = emptyList(), + confidence = 0.95, + ).withMetadataValue(DiceMetadataKeys.METAMODEL_VERSION, hash1) + + val prop2 = Proposition( + contextId = testContextId, + text = "Bob works at Globex", + mentions = emptyList(), + confidence = 0.95, + ).withMetadataValue(DiceMetadataKeys.METAMODEL_VERSION, hash2) + + val prop3 = Proposition( + contextId = testContextId, + text = "Carol works at Initech", + mentions = emptyList(), + confidence = 0.95, + ).withMetadataValue(DiceMetadataKeys.METAMODEL_VERSION, hash1) + + repo.save(prop1) + repo.save(prop2) + repo.save(prop3) + + val hash1Props = repo.findAll().filter { + it.metadata[DiceMetadataKeys.METAMODEL_VERSION] == hash1 + } + + assertEquals(2, hash1Props.size) + assertTrue(hash1Props.any { it.text.contains("Alice") }) + assertTrue(hash1Props.any { it.text.contains("Carol") }) + } + + @Test + fun `proposition without METAMODEL_VERSION has no value for the key`() { + val repo = InMemoryPropositionRepository() + + val prop = Proposition( + contextId = testContextId, + text = "Alice works at Acme", + mentions = emptyList(), + confidence = 0.95, + ) + + val saved = repo.save(prop) + val retrieved = repo.findById(saved.id) + + assertNotNull(retrieved) + assertNull(retrieved!!.metadata[DiceMetadataKeys.METAMODEL_VERSION]) + } +} From 1226c14d9cf9068046b5689563f4cc3fbce1854f Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Tue, 1 Sep 2026 17:16:26 -0400 Subject: [PATCH 06/11] Retire the word seam from versioning docs and test names --- AGENTS.md | 2 +- README.md | 2 +- dice-metamodel/pom.xml | 2 +- .../kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt | 2 +- .../kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt | 2 +- .../kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt | 4 ++-- 6 files changed, 7 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e69d15d0..a789a8b9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,7 +11,7 @@ DICE (Domain-Integrated Context Engineering) is a proposition-first knowledge su | `dice-storage-autoconfigure` | Spring Boot auto-configuration that wires the right backend based on `embabel.dice.store.type`, schedules the decay tick, and provides auto-configuration for the multi-signal duplicate collector (properties prefix `embabel.dice.collector`) | | `dice-report` | Output projectors over propositions: rationale (why a fact is believed, with evidence), structured report, and surprising-link discovery | | `dice-ingestion` | Ingestion SPI (artifacts → chunks) with a content-hash dedup ledger so the same source isn't extracted twice | -| `dice-metamodel` | Schema versioning: `MetamodelVersion` content-hash stamps over the governed types of a `DataDictionary`, `GovernedTypeSelector`, the `DeclaredSchemaSource` opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM, with no dependency on `dice` | +| `dice-metamodel` | Schema versioning: `MetamodelVersion` content-hash stamps over the governed types of a `DataDictionary`, `GovernedTypeSelector`, the `DeclaredSchemaSource` opt-in, and the `MetamodelVersionStore` contract. Pure JVM, with no dependency on `dice` | | `dice-integration-tests` | Test-only: the cross-feature end-to-end canonical-flow harness | ## Build & test diff --git a/README.md b/README.md index 365e45c3..fd9e296c 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,7 @@ recover by reading a single class — see the design notes in [`docs/design/`](d two-phase save, materialised effective confidence, schema-as-beans, and the decay tick. - [Events](docs/design/events.md) — the domain-event model the store and pipeline emit. - [Metamodel versioning](docs/design/metamodel-versioning.md) — content-hash schema stamps, per-type - governance, the declared-schema opt-in seam, and version history. + governance, the declared-schema opt-in, and version history. ## Real-World Example: Impromptu diff --git a/dice-metamodel/pom.xml b/dice-metamodel/pom.xml index d9d92f18..4fa5ec2a 100644 --- a/dice-metamodel/pom.xml +++ b/dice-metamodel/pom.xml @@ -10,7 +10,7 @@ dice-metamodel jar Dice Metamodel - Schema versioning for DICE knowledge graphs: content-hash stamping, the declared-schema seam, and the version store contract + Schema versioning for DICE knowledge graphs: content-hash stamping, the declared-schema contract, and the version store contract + + org.jetbrains + annotations + 26.0.2 + provided + + org.springframework.boot diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt index 64afc9bd..469a6a46 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt @@ -16,6 +16,7 @@ package com.embabel.dice.metamodel import com.embabel.agent.core.DataDictionary +import org.jetbrains.annotations.ApiStatus /** * The schema as declared: the stamped [version] plus the bare relationship type names it allows. @@ -31,6 +32,7 @@ import com.embabel.agent.core.DataDictionary * @property version The stamped declared schema. * @property relationshipTypeNames The bare relationship type names [version] allows. */ +@ApiStatus.Experimental class DeclaredSchema( val version: MetamodelVersion, relationshipTypeNames: Set, @@ -116,6 +118,7 @@ class DeclaredSchema( * already uses to define its types (a `DataDictionary`, a config file, a registry...) and wires it * as a bean. There is no default implementation, because there is no default declared schema. */ +@ApiStatus.Experimental fun interface DeclaredSchemaSource { /** diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt index 7acfc216..1c77ee48 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt @@ -16,6 +16,7 @@ package com.embabel.dice.metamodel import com.embabel.agent.core.DomainType +import org.jetbrains.annotations.ApiStatus /** * Decides which domain types a [MetamodelVersion] stamp covers. @@ -37,6 +38,7 @@ import com.embabel.agent.core.DomainType * ungoverned type to the dictionary leaves the content hash as it was, while touching a governed * one changes it. */ +@ApiStatus.Experimental fun interface GovernedTypeSelector { /** diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt index 2a7b87d2..adb85fd9 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -20,6 +20,7 @@ import com.embabel.agent.core.DataDictionary import com.embabel.agent.core.DomainTypePropertyDefinition import com.embabel.agent.core.NamedPropertyDefinition import com.embabel.agent.core.PropertyDefinition +import org.jetbrains.annotations.ApiStatus import java.security.MessageDigest import java.util.Objects @@ -48,6 +49,7 @@ import java.util.Objects * takes a signature, and a signature that never reaches a stamp keeps whatever set it was built * with. Experimental: shape may change before 1.0. */ +@ApiStatus.Experimental data class PropertySignature @JvmOverloads constructor( val name: String, val kind: Kind, @@ -153,6 +155,7 @@ data class PropertySignature @JvmOverloads constructor( * restarts. Any structural change produces a different hash, including a property's type changing * on a type whose name is unchanged. */ +@ApiStatus.Experimental class MetamodelVersion @JvmOverloads constructor( schemaName: String, entityTypeNames: List, diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt index 33c8e0fd..ab8d9a0a 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt @@ -15,6 +15,8 @@ */ package com.embabel.dice.metamodel +import org.jetbrains.annotations.ApiStatus + /** * Durable store for metamodel version stamps. Keeping every stamp a schema has ever had is what * later lets you say when a shape changed and what knowledge was extracted under which version. @@ -32,6 +34,7 @@ package com.embabel.dice.metamodel * This contract covers stamping and recall. Comparing a declaration against a live graph is a * separate concern with its own store contract. */ +@ApiStatus.Experimental interface MetamodelVersionStore { /** diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt index f51ca047..28aea42c 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt @@ -15,6 +15,8 @@ */ package com.embabel.dice.metamodel +import org.jetbrains.annotations.ApiStatus + /** * The former names a schema's types and properties have gone by, declared alongside the schema * itself. @@ -42,6 +44,7 @@ package com.embabel.dice.metamodel * @property propertyAliases Entity type name, then current property name, to the names that * property used to have. */ +@ApiStatus.Experimental class SchemaAliases @JvmOverloads constructor( typeAliases: Map> = emptyMap(), propertyAliases: Map>> = emptyMap(), From 726a51d80057f3e1f221f4d5a7fdf14f4c9f54cd Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Wed, 2 Sep 2026 16:50:04 -0400 Subject: [PATCH 09/11] Name no consumer in the changelog header --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf419a7a..cd5fa989 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,7 @@ # Changelog Notable changes to DICE. Each entry states its compatibility impact on consumers -(assistant/me and anything else tracking `0.2.0-SNAPSHOT`): **additive** (safe to +(anything tracking `0.2.0-SNAPSHOT`): **additive** (safe to pick up), **behavioral** (same API, different runtime behavior — read the note), or **breaking** (consumer change required; the entry links the migration notes and the consumer PRs that deliver it). From 3c2fcda81f43244a86bef7eeb480925bc3da716e Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Thu, 3 Sep 2026 16:58:40 -0400 Subject: [PATCH 10/11] Answer the code review on the metamodel stamp Manage org.jetbrains:annotations in dice-parent. Neither embabel BOM manages it, so each module pinned its own version and the two had already drifted apart. Simplify the alias comparator. Name the Java compatibility tests in sentences, which is what @DisplayName is for. --- dice-metamodel/pom.xml | 1 - .../com/embabel/dice/metamodel/MetamodelVersion.kt | 8 +++----- .../dice/metamodel/MetamodelJavaCompatTest.java | 13 +++++++++++++ dice/pom.xml | 1 - pom.xml | 11 +++++++++++ 5 files changed, 27 insertions(+), 7 deletions(-) diff --git a/dice-metamodel/pom.xml b/dice-metamodel/pom.xml index b05e338f..67a95652 100644 --- a/dice-metamodel/pom.xml +++ b/dice-metamodel/pom.xml @@ -31,7 +31,6 @@ org.jetbrains annotations - 26.0.2 provided diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt index adb85fd9..954762c8 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -87,13 +87,11 @@ data class PropertySignature @JvmOverloads constructor( /** Compare two alias sets as sorted lists: element by element, then by size. */ private fun compareAliases(left: Set, right: Set): Int { - val sortedLeft = left.sorted() - val sortedRight = right.sorted() - for (i in 0 until minOf(sortedLeft.size, sortedRight.size)) { - val comparison = sortedLeft[i].compareTo(sortedRight[i]) + left.sorted().zip(right.sorted()).forEach { (leftAlias, rightAlias) -> + val comparison = leftAlias.compareTo(rightAlias) if (comparison != 0) return comparison } - return sortedLeft.size.compareTo(sortedRight.size) + return left.size.compareTo(right.size) } /** diff --git a/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java index 8140f4f4..a2ac7a2d 100644 --- a/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java +++ b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java @@ -19,6 +19,7 @@ import com.embabel.agent.core.DataDictionary; import com.embabel.agent.core.DomainType; import com.embabel.agent.core.DynamicType; +import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import java.util.Arrays; @@ -51,6 +52,7 @@ private static DataDictionary goldenSchema() { } @Test + @DisplayName("the four-argument property signature constructor still exists") void theFourArgumentPropertySignatureConstructorStillExists() { PropertySignature signature = new PropertySignature( "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE); @@ -60,6 +62,7 @@ void theFourArgumentPropertySignatureConstructorStillExists() { } @Test + @DisplayName("the property signature constructor also takes aliases") void thePropertySignatureConstructorAlsoTakesAliases() { PropertySignature signature = new PropertySignature( "emailAddress", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, Set.of("email")); @@ -68,6 +71,7 @@ void thePropertySignatureConstructorAlsoTakesAliases() { } @Test + @DisplayName("the five-argument metamodel version constructor still exists") void theFiveArgumentMetamodelVersionConstructorStillExists() { MetamodelVersion version = new MetamodelVersion( "test", @@ -81,6 +85,7 @@ void theFiveArgumentMetamodelVersionConstructorStillExists() { } @Test + @DisplayName("the metamodel version constructor also takes aliases") void theMetamodelVersionConstructorAlsoTakesAliases() { MetamodelVersion version = new MetamodelVersion( "test", @@ -94,6 +99,7 @@ void theMetamodelVersionConstructorAlsoTakesAliases() { } @Test + @DisplayName("the one and two-argument stamping factories still exist") void theOneAndTwoArgumentStampingFactoriesStillExist() { MetamodelVersion whole = MetamodelVersion.from(goldenSchema()); MetamodelVersion governed = MetamodelVersion.from(goldenSchema(), GovernedTypeSelector.ALL); @@ -103,6 +109,7 @@ void theOneAndTwoArgumentStampingFactoriesStillExist() { } @Test + @DisplayName("the stamping factory also takes aliases") void theStampingFactoryAlsoTakesAliases() { MetamodelVersion version = MetamodelVersion.from( goldenSchema(), @@ -113,6 +120,7 @@ void theStampingFactoryAlsoTakesAliases() { } @Test + @DisplayName("the one and two-argument declaration factories still exist") void theOneAndTwoArgumentDeclarationFactoriesStillExist() { DeclaredSchema whole = DeclaredSchema.from(goldenSchema()); DeclaredSchema governed = DeclaredSchema.from(goldenSchema(), GovernedTypeSelector.ALL); @@ -122,6 +130,7 @@ void theOneAndTwoArgumentDeclarationFactoriesStillExist() { } @Test + @DisplayName("the declaration factory also takes aliases") void theDeclarationFactoryAlsoTakesAliases() { DeclaredSchema declared = DeclaredSchema.from( goldenSchema(), @@ -132,12 +141,14 @@ void theDeclarationFactoryAlsoTakesAliases() { } @Test + @DisplayName("the no-argument alias constructor exists") void theNoArgumentAliasConstructorExists() { assertEquals(Map.of(), new SchemaAliases().getTypeAliases()); assertEquals(Map.of(), SchemaAliases.NONE.getPropertyAliases()); } @Test + @DisplayName("the collections a stamp hands back refuse mutation from Java") void theCollectionsAStampHandsBackRefuseMutationFromJava() { MetamodelVersion version = MetamodelVersion.from( goldenSchema(), @@ -149,6 +160,7 @@ void theCollectionsAStampHandsBackRefuseMutationFromJava() { } @Test + @DisplayName("the shipped Kotlin default synthetic keeps its descriptor") void theShippedKotlinDefaultSyntheticKeepsItsDescriptor() throws Exception { // A Kotlin caller that omits a defaulted argument links against the $default synthetic // rather than the function itself. DeclaredSchema.from shipped with one defaulted @@ -168,6 +180,7 @@ void theShippedKotlinDefaultSyntheticKeepsItsDescriptor() throws Exception { } @Test + @DisplayName("the stamping factories take no defaulted parameters") void theStampingFactoriesTakeNoDefaultedParameters() throws Exception { // MetamodelVersion.from shipped as two overloads with no defaults, so it has no $default // synthetic to preserve. Keeping it that way means the alias overload can never widen one. diff --git a/dice/pom.xml b/dice/pom.xml index 09616eca..ba1c7432 100644 --- a/dice/pom.xml +++ b/dice/pom.xml @@ -61,7 +61,6 @@ org.jetbrains annotations - 26.0.1 1.41 + + 26.0.2 @@ -94,6 +99,12 @@ drivine4j-spring-boot-starter ${drivine.version} + + + org.jetbrains + annotations + ${jetbrains.annotations.version} + From e423a0ab2b7dc05d6c70d63ac9ebcb6c1258633d Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Thu, 3 Sep 2026 23:10:22 -0400 Subject: [PATCH 11/11] Answer the agent findings on the metamodel stamp Cross-reference equals and hasSameContentAs, since one compares the schema name and the other does not. Run the two declaration refusals through one requireDeclarable so the constructor and from cannot drift. Group governed types by name once. Say why the store key carries the schema name: two schemas with the same shape share a hash, and history is per schema. Say when findVersion's default is enough and when to override it. A design note explains why three classes write their own equals. --- .../dice/metamodel/MetamodelVersion.kt | 73 ++++++++++++------- .../dice/metamodel/MetamodelVersionStore.kt | 16 +++- docs/design/metamodel-versioning.md | 21 ++++++ 3 files changed, 81 insertions(+), 29 deletions(-) diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt index 954762c8..aa9170eb 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -206,8 +206,7 @@ class MetamodelVersion @JvmOverloads constructor( "Drop the entry instead; it means the same thing and hashes the same as the types around it." } - requireNoTypeAliasReuse(known, this.entityTypeAliases) - requireNoAliasesOnDuplicateNames(this.entityTypeProperties) + requireDeclarable(known, this.entityTypeAliases, this.entityTypeProperties) } val contentHash: String = fingerprint() @@ -216,6 +215,9 @@ class MetamodelVersion @JvmOverloads constructor( * Returns `true` when this version and [other] have the same structural content (entity types, * label sets, property signatures, and relationships), regardless of schema name. Compares * [contentHash]. + * + * [equals] also compares [schemaName]; this does not. Use this to ask whether two schemas have + * the same shape, and [equals] to ask whether two stamps are the same stamp. */ fun hasSameContentAs(other: MetamodelVersion): Boolean = contentHash == other.contentHash @@ -266,7 +268,14 @@ class MetamodelVersion @JvmOverloads constructor( return hashBytes.joinToString("") { "%02x".format(it) } } - /** Structural equality: same schema name, types, labels, property signatures, relationships, and aliases. */ + /** + * Structural equality: same schema name, types, labels, property signatures, relationships, and + * aliases. + * + * This compares [schemaName]; [hasSameContentAs] and [contentHash] leave it out. Two stamps of + * identically shaped schemas under different names are therefore unequal here and equal there, + * and a `Set` keys the way the store does, on schema and content both. + */ override fun equals(other: Any?): Boolean { if (this === other) return true if (other !is MetamodelVersion) return false @@ -331,6 +340,21 @@ class MetamodelVersion @JvmOverloads constructor( private fun withImmutableAliases(signature: PropertySignature): PropertySignature = signature.copy(aliases = java.util.Set.copyOf(signature.aliases)) + /** + * The refusals a declaration has to pass, run as one so the constructor and [from] can't + * drift on which checks apply. [from] runs it first, so a declaration that can't be stamped + * fails at the call that stamped it; the constructor runs it again, so a stamp built by + * hand meets the same guard. + */ + private fun requireDeclarable( + declaredTypeNames: Set, + entityTypeAliases: Map>, + entityTypeProperties: Map>, + ) { + requireNoTypeAliasReuse(declaredTypeNames, entityTypeAliases) + requireNoAliasesOnDuplicateNames(entityTypeProperties) + } + /** * Reject a declared type name showing up in another type's alias set. * @@ -449,28 +473,26 @@ class MetamodelVersion @JvmOverloads constructor( // A DataDictionary can legally hold two domain types that share a name but differ in // shape (DynamicType is a data class, so same-named instances with different labels are - // not equal and both survive a set). Labels and properties are unioned per name. - // Keeping only the last would drop a label or property from the fingerprint, and - // removing it later wouldn't change the hash. - val entityTypeLabels = governedTypes - .groupBy { it.name } - .mapValues { (_, types) -> types.flatMap { it.labels }.toSet() } + // not equal and both survive a set). Labels and properties are unioned per name, off + // the one grouping. Keeping only the last would drop a label or property from the + // fingerprint, and removing it later wouldn't change the hash. + val typesByName = governedTypes.groupBy { it.name } + + val entityTypeLabels = typesByName.mapValues { (_, types) -> types.flatMap { it.labels }.toSet() } // Decorating inside this loop is the only place it can happen: the stamp is immutable // and hashes at construction. Every signature sharing a property name picks up the same // declared alias set, so decoration can neither create nor collapse a duplicate, and // the constructor's duplicate-name guard sees exactly the duplicates the union holds. - val entityTypeProperties = governedTypes - .groupBy { it.name } - .mapValues { (typeName, types) -> - types.flatMap { type -> - type.properties.map { property -> - val signature = PropertySignature.of(property) - val declared = aliases.propertyAliasesFor(typeName, signature.name) - if (declared.isEmpty()) signature else signature.copy(aliases = declared) - } - }.toSet() - } + val entityTypeProperties = typesByName.mapValues { (typeName, types) -> + types.flatMap { type -> + type.properties.map { property -> + val signature = PropertySignature.of(property) + val declared = aliases.propertyAliasesFor(typeName, signature.name) + if (declared.isEmpty()) signature else signature.copy(aliases = declared) + } + }.toSet() + } // Splitting one type into two same-named declarations, or merging two back into one, // can render the same relationship descriptor twice. It is the same schema either way, @@ -479,17 +501,16 @@ class MetamodelVersion @JvmOverloads constructor( .filter { selector.governs(it.from) } .map { rel -> "${rel.from.name}-[${rel.name}]->${rel.to.name}" } - val governedNames = governedTypes.map { it.name }.toSet() + val governedNames = typesByName.keys val entityTypeAliases = aliases.typeAliases.filterKeys { it in governedNames } - // The two refusals run here as well as in the constructor so a declaration that can't - // be stamped fails at the call that stamped it, naming the alias to retire. - requireNoTypeAliasReuse(governedNames, entityTypeAliases) - requireNoAliasesOnDuplicateNames(entityTypeProperties) + // Runs here as well as in the constructor so a declaration that can't be stamped fails + // at the call that stamped it, naming the alias to retire. + requireDeclarable(governedNames, entityTypeAliases, entityTypeProperties) return MetamodelVersion( schemaName = dataDictionary.name, - entityTypeNames = governedTypes.map { it.name }, + entityTypeNames = governedNames.toList(), entityTypeLabels = entityTypeLabels, entityTypeProperties = entityTypeProperties, relationshipNames = relationshipNames, diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt index ab8d9a0a..62e06587 100644 --- a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt @@ -31,6 +31,14 @@ import org.jetbrains.annotations.ApiStatus * is deleted, and records with different keys always coexist, so history accumulates. * Implementations are not expected to reject a re-save. * + * **Why the schema name is in the key.** [MetamodelVersion.contentHash] excludes the schema name + * on purpose, so two schemas with the same shape share a hash: a schema and its staging copy, or a + * schema forked under a new name. History is per schema, which is what [latestVersion] and + * [versionHistory] answer, so the same content has to be a separate record under each name. + * Keyed on the hash alone, one schema adopting a shape another had earlier would land on the other + * schema's record and pull that schema's history into its own. The name in the key is what keeps + * two schemas' histories from bleeding into each other. + * * This contract covers stamping and recall. Comparing a declaration against a live graph is a * separate concern with its own store contract. */ @@ -71,9 +79,11 @@ interface MetamodelVersionStore { * Resolves a recorded hash, such as the one a proposition carries as the version it was * extracted under, back into the schema shape it stood for. * - * The default scans [versionHistory], which is correct for any implementation but reads the - * whole history to answer a keyed question. A backend that can push the lookup down to the - * database (a keyed `MATCH` rather than an in-memory `filter`) should override it. + * The default scans [versionHistory], which is correct for any implementation and reads the + * whole history to answer a keyed question. That is fine for the in-memory reference and for + * a test double. A durable backend should override it with a keyed lookup, since a long-lived + * schema's history only grows and this default grows with it; the Drivine store does, with a + * `MATCH` on the natural key. * * @param schemaName The schema the version belongs to. * @param contentHash The [MetamodelVersion.contentHash] to find. diff --git a/docs/design/metamodel-versioning.md b/docs/design/metamodel-versioning.md index f99e9d6f..b5ff7b86 100644 --- a/docs/design/metamodel-versioning.md +++ b/docs/design/metamodel-versioning.md @@ -336,6 +336,27 @@ keyed question; a backend that can push the lookup down to the database should o module ships no implementation. Storage is a separate concern, and a stamp is useful in memory before anything durable exists. +## Plain classes, not data classes + +`MetamodelVersion`, `DeclaredSchema` and `SchemaAliases` each write their own `equals`, `hashCode` +and `toString`. That is one deliberate pattern, for one reason: each of them copies what the +constructor is handed into a JVM-immutable collection in its body, and a `data class` cannot do +that. A constructor `val` takes no initialiser, so the generated `equals` and `copy` would read the +raw arguments and skip the copy, and a stamp whose collections could still be changed from the +outside would disagree with its own precomputed hash. + +`PropertySignature` is the exception that proves it. It is a `data class`, and its `aliases` set is +therefore held as handed in; `MetamodelVersion` copies that set into an immutable one when it takes +a signature. The KDoc on each class says as much, and this section is here so the pattern is read +as a module decision and not raised class by class. + +Two notions of equality live on `MetamodelVersion`, and both are meant. `equals` compares the schema +name along with the content, so a `Set` keys the way the store does. `contentHash` +and `hasSameContentAs` leave the name out, so two schemas with the same shape under different names +compare equal there. That is also why the store's natural key is `(schemaName, contentHash)` and not +the hash alone: history is per schema, and one schema adopting a shape another had earlier must not +land on the other schema's record. + ## The tiers ahead Versioning is the first of three escalating tiers, shipped in that order.